Will partitioning and sharding improve performance?
Pilar de Teodoro is physicist and works at the European Space Agency (ESA)/ESAC Science Data Centre (ESDC) in Madrid, Spain. After many years working in Oracle Corp., she moved to the European Space Agency working first for the Gaia mission being in charge of the databases for the largest survey of objects in the universe dealing with hundreds of Terabytes of data. After Gaia, she joined the ESDC team where all archives from the different ESA missions are kept. PostgreSQL is the database system most used in the science archives helping the the singularity of every space mission and it is the pillar of the ESA Sky project.
No video of the event yet, sorry!
What is the best solution to join 2 very big tables (>1TB) with billion rows and hundreds of columns with the help of distribution and partitioning? We have analysed different distribution and partition methods to achieve the best performance for a crossmatch, that is, identifying different rows from 2 tables (star catalogues) referring to the same star. Crossmatching helps scientists around the world to understand better our universe identifying objects observed by different space missions.
- 20 min
- Postgres Conference 2020
- Case Studies