now has an SQL context, so do we need DuckDB
makes an interesting guess on how the adoption of these tools will break down:
🎱Those who value non-SQL-centric data pipelines will choose Polars, and use its SQL context when needed.
🎱Those who already love SQL will choose DuckDB hands down.
🎱If you like to write re-usable, functional, unit-tested code, you will use Polars.
🎱If you have tabular data that uses lots of analytics and groupings with aggregates, you will pick DuckDB.
He also argues that simplicity is underrated, and in the near future, the value of a data engineer won’t come from writing code but from the ability to be wise and build good, scalable, clean, and robust systems.