Practical Datastream: CloudSQL to BigQuery Change Data Capture on Google Cloud

GDG San Diego
Wed, Aug 18, 2021, 6:00 PM (PDT)

23 RSVP'ed

Datastream from Google is a serverless change data capture and replication service. This allows organizations to replicate data across multiple databases, storage systems and is especially useful for replicating OLTP data in MySQL into an OLAP database such as BigQuery. This talk walks through setting up connection profiles, streams and touch on some useful debugging if things don't go as planned

About this event

Datastream from Google is a serverless change data capture and replication service. This allows organizations to replicate data across multiple databases, storage systems and is especially useful for replicating OLTP data in MySQL into an OLAP database such as BigQuery. This talk will walk through setting up connection profiles, streams and touch on some useful debugging for when things don't go as planned.

 
What happens when data in one database, such as an Online Transaction Processing (OLTP) database running on MySQL has a change that needs to be reflected in an Online Analytics Processing (OLAP) database? 

How do you capture that change of data? 

With a Change Data Capture process of course! 

The Datastream product from Google does just this, and has support for Oracle and MySQL sources, with more to come as the product matures. Currently the only supported target is Cloud Storage, but by using BigQuery's External table support we can view this changed data in BigQuery. But things are prone to breaking, so we'll also review some troubleshooting methods for sorting out what went wrong where.

We will review each service and outline the common use cases and strengths. A passing familiarity with SQL is encouraged but not required.

Featured Presentation

Speaker

Facilitators

When

Wednesday, Aug 18
6:00 PM - 8:00 PM (PDT)

Host

  • Svetlin Stanchev

    Svetlin Stanchev

    Objectist Software Inc

    Owner

Organizers

  • Svetlin Stanchev

    Svetlin Stanchev

    Objectist Software Inc

    GDG Organizer

    View Profile
  • Kyle Murley

    Kyle Murley

    Google Cloud Platform

    GCP Customer Engineer, San Diego Enterprise

    View Profile