Learn how to use your programming skill to build modern data processing pipeline

Name: Learn how to use your programming skill to build modern data processing pipeline
Start: 2023-04-20T18:30:00+02:00
End: 2023-04-20T21:00:00+02:00
Location: bluehands GmbH & Co.mmunication KG

bluehands GmbH & Co.mmunication KG, Waldstrasse 63, Karlsruhe, 76133

GDG Karlsruhe

Learn how to use your programming skill to build modern data processing pipelines. - Data Engineering for dummies (aka Software Devs 😉)

Apr 20, 2023, 4:30 – 7:00 PM

31 RSVP'd

Key Themes

NetworkingWeb

About this event

This time we will have Sascha Dittmann with a talk about building a modern data processing pipeline.

Thanks to bluehands GmbH for hosting us 

When

Thursday, April 20, 2023
4:30 PM – 7:00 PM UTC

Where

bluehands GmbH & Co.mmunication KG
Waldstrasse 63
Karlsruhe, 76133

Agenda

4:30 PM	Doors open
4:45 PM	Orga Intro
5:00 PM	Learn how to use your programming skill to build modern data processing pipeline - By Sascha Dittmann	Data quality has become a much discussed topic in the fields of data engineering and data science. It’s become clear that data validation is crucial to ensuring the reliability of data products and insights produced by an organisation’s data pipelines. But can you do it completely with open source software? Apache Airflow and dbt (data build tool) are among the prominent open source tools in the data engineering ecosystem, and while dbt offers some data testing capabilities, another open source data tool, Great Expectations, enhances the pipeline with data validation and can add layers of robustness. Join expert Sascha Dittmann to explore the "dAG stack" and learn how to combine the functions of these three open source tools to build, test, validate, document, and orchestrate an entire pipeline, end to end, from scratch.
5:45 PM	Networking