Copernicus Data Infrastructure at IT.NRW

Efficient processing of Copernicus satellite and in situ data

One of the core tasks of the State Office for Information Technology and Statistics (IT.NRW) is to develop and operate the IT infrastructure for the North Rhine-Westphalian (NRW) state administration. The state office also advises and supports the state authorities and institutions in the use of information technology. As a provider of infrastructure, geospatial data and services, IT.NRW plays an important role in the spatial data infrastructure in NRW. One of IT.NRW’s goals is to further develop its own infrastructure to better support on-demand processing of very large amounts of data for creating information products in the context of Copernicus. This development takes place in joint pilot projects with the state administration stakeholders, such as the competence centers for remote sensing of the LANUV NRW, Geobasis NRW or Wald und Holz NRW. IT.NRW applies an agile process to develop structures and answer questions that contribute to the sustainable development of an efficient Copernicus data infrastructure for the NRW state administration.

After the successful realization of a first pilot project supporting the use case of detection and qualification of orchards in NRW, IT.NRW continued to improve the stability and scalability of the Kubernetes-based infrastructure. The consortium, in close collaboration with IT.NRW, replaced the central workflow execution component with a workflow engine based on the Argo framework. This pilot project was driven by Geobasis NRW requirements. Fully integrated with Kubernetes, Argo enables the scalable and reliable execution of complex workflows using process steps capsulated in Docker containers. A dedicated focus was on the management of metadata on Copernicus raw data as well as on process results in a central metadata index.

In close collaboration with con terra and IT.NRW, 52°North designed the system architecture. We focused on the transition of existing workflows to Argo and the integration of the newly introduced metadata index. In addition, our team supported the deployment and setup of the new software solution on IT.NRW premises.

In order to reflect and sustainably integrate the requirements of IT.NRW and the related competence centers, our team held a dedicated workshop to identify the most important aspects of the platform. We concurrently improved the architecture design based on insights gained and developed first system components. Due to the short project life, we established an agile feedback process with bi-weekly Jour Fixe meetings. This ensured the efficient and focused realization of the workflow system that is now capable of processing large amounts of Sentinel data using modern elastic scalability approaches.

con terra GmbH (main contractor), Germany

EFTAS, Germany