Sep 7 – 11, 2015
Europe/Berlin timezone

Big data in critical infrastructure: Production and failover infrastructure in DWD's central data management

Sep 10, 2015, 10:40 AM
40m
Gaede HS (30.22)

Gaede HS

30.22

Plenary Talks Plenary talks

Speaker

Mr Daniel Lee (DWD)

Description

The German Weather Service (DWD) provides a wide variety of services for the protection of life and property in the form of weather and climate information. One core task is safeguarding aviation, marine safety and terrestrial traffic. Another is warning before meteorological events that could endanger public safety and order. Additionally, we monitor the climate and are active in multiple research fields, from ensemble numerical weather forecasting to applications of weather data in new areas. Data is recorded, processed and transformed into time-critical products and securely archived 24 hours per day, 365 days per year.

The DWD maintains a high productivity, redundant infrastructure in order to provide these services reliably and on demand. We ensure deliverability with multiple tiers of failover strategies, enabling us to manage and monitor production even when faced with major hardware or software failures.

Specialized systems allow rapid access to large, cross-sectional binary files in file system caches for near-real-time applications, while an automated tape archive provides short-term access to long-term archival data. Simultaneously, observational data is processed and stored in relational databases in order to allow comfortable processing of long time series data. Various application layers are used to post-process products in order to refine them for domain-specific queries.

Demands for weather and climate based data and services, as well as the associated needs for processing power, network transfer capabilities and storage capacity are constantly increasing. It is the DWD's goal not only to maintain a production infrastructure with high quality and availability, but also to continue to evolve to meet these demands. Doing so while maintaining our tradition of quality, speed and reliability is one of the major challenges facing the DWD. Some current projects designed to meet these goals are introduced in the outlook.

Primary author

Mr Daniel Lee (DWD)

Presentation materials