State-of-the-art data platform design
19 hours ago
On this story, I’ll attempt to shed some light on the advantages of contemporary data warehouse solutions (DWH) in comparison with other data platform architecture types. I’d dare to say that DWH is the most well-liked platform amongst data engineers in the intervening time. It offers invaluable advantages in comparison with other solution types but in addition has some well-known limitations. Need to learn data engineering? This story is a superb place to begin since it explains data engineering at its core — the DWH solution on the centre of the architecture diagram. We are going to see how data will be ingested and transformed in several DWHs available available in the market.
I’d prefer to open the discussion with experienced users too. It will be great to know your opinion and see what you could have to say on this topic.
Key characteristics of a knowledge warehouse
A serverless, distributed SQL engine (BigQuery, Snowflake, Redshift, Microsoft Azure Synapse, Teradata.) is what we call a contemporary data warehouse (DWH). It’s a SQL-first data architecture [1] where data is stored in a knowledge warehouse, and we are able to use all some great benefits of using denormalized star schema [2] datasets because most of the trendy data warehouses are distributed and scale well, which suggests there isn’t a have to worry about table keys and indices. It suits well for ad-hoc analytical queries on Big Data.
Most of the trendy data warehouse solutions can process structured and unstructured data and are very convenient for data analysts with good SQL skills.
Modern data warehouses integrate easily with business intelligence solutions like Looker, Tableau, Sisense, and Mode, which use ANSI-SQL to process data. Within the diagram below I attempted to map a standard data transformation journey and tools used (not a whole list in fact). We are able to see that…