Analytics Infrastructure Powered by Logical Decoding and Kafka
Presented by:
Jeff Klukas is a data scientist and systems engineer who loves code and automation. He helped discover the Higgs particle at the Large Hadron Collider and now works on data systems for Simple, a consumer technology company that's building banking for the way people think rather than the way banks work.
Logical decoding provides a comprehensive, real-time stream of the changes happening within a Postgres instance, which has tremendous potential for analytics use cases. This talk gives an overview of some common patterns for building pipelines to feed a data warehouse, then dives into the details of Simple's Postgres -> Kafka pipeline, how we use it to warehouse Postgres change history to Amazon Redshift, and how it feeds user-facing streaming applications. We'll finish by comparing Simple's approach to Confluent's bottledwater-pg and where Postgres' integration with the Kafka ecosystem could go in the future.
- Date:
- Duration:
- 50 min
- Room:
- Conference:
- PGConf US 2017 [PgConf.US]
- Language:
- Track:
- Use Cases
- Difficulty:
- Medium