Building a data management platform with Azure Synapse Analytics

About the customer:

Our customer is a wholly owned subsidiary of a large German group. With a holistic, customer-specific offer, it drives the digitization of all companies in the group with around 5,100 employees successfully, integratively and value-adding.

In order to find and implement the right solutions for the companies, the customer relies on a comprehensive, market-oriented product portfolio for topics such as the cloud, big data, Internet of Things and artificial intelligence, as well as a high level of consulting and methodological expertise.

Especially with its own ventures, the customer is in the dynamic development of open data platforms. These enable the comprehensive change from classic work and organizational structures to self-organization and company-wide, agile networks.

The challenge:

Since mid-2020, the customer has been faced with the task of developing a uniform, central and open data platform in accordance with the group’s operator specifications.

A large number of group-wide projects with requirements for high data throughput should be able to benefit from the advantages of a modern data platform within a very short time thanks to the automated provision. Thanks to the configurations specified in-house, all corporate requirements regarding security and compatibility are met in a future-proof manner and, in addition to quick and easy integration into existing systems, they offer cost savings, reliable performance and high reliability.

The implementation:

The aim of the project is to offer a group-wide and uniform data management platform for IT projects within the group. The entire infrastructure is set up according to group-wide operating specifications. The connections run mainly via the Microsoft backbone and not publicly via the Internet. The data management platform can be ordered for all IT teams via a corporate portal and is automatically made available on it.

An order process triggers a Gitlab pipeline, which uses Terraform to set up the infrastructure within 30 minutes and set the necessary authorizations and configure it.

The Microsoft Azure Cloud was chosen as the provider for building a data management platform. This option offers intelligent solutions for storing, managing and analyzing growing volumes of complex customer data.

A Data Lake Gen2 architecture is deployed for data storage, which in combination with Synapse Analytics offers a simple interface for data scientists. The processing is carried out on the underlying Apache Spark pools and SQL pools.

In the future, Synapse Analytics will ensure integration with the data catalog.

The result:

The introduction of the data management platform enables fast, barrier-free and cost-efficient access to the data collected by the group. This not only improves access to and management of the existing data catalogue, but also makes it easier to use the data profitably.

Thanks to the technical support of PROTOS Technologie GmbH, it is possible to receive the platform completely set up and preconfigured within approx. 30 minutes after ordering. The user-friendly operation reduces the inhibition threshold to work with data.

For data scientists, using Synapse Analytics brings elementary operational and organizational advantages, as it supports the integration of different systems. While the internal data integration takes over the management and administration of the data, the data connection prevents data from disappearing into the data swamp. The innovative Data Lake House paradigm is also supported by Synapse Analytics. Big Data Warehouse (Big Data SQL Pools) queries and Spark Jobs (Apache Spark) can also be run in Synapse Analytics.

As a certified HashiCorp & Microsoft Partner (Silver – Data Analytics), PROTOS Technologie GmbH has been supporting the customer since the beginning of 2020 in setting up the Azure reference architecture for the data management platform, which can be self-provisioned via a service portal. Thanks to the cooperation, the customer was able to develop a uniform, central and open data platform according to the operator specifications of the group. In addition to the consulting service, PROTOS implements the highly automated provision of the platform based on HashiCorp Terraform.

Graphics: Prosymbols, Freepik, Becris, Ralf Schmitzer from Flaticon.com