Building a Flight Data Quality Pipeline with Snowflake and Azure Blob
I recently had the chance to build something in Snowflake, and I figured it was worth documenting the process since there were a few things that weren't immediately obvious coming from a Microsoft Fabric / Azure background. The project: a data quality pipeline that ingests BTS On-Time Performance flight data into Snowflake via Azure Blob Storage, validates every record against a set of quality rules, and routes clean vs. rejected records into separate tables. I filtered the data to Dallas/Fort Worth International Airport (DFW), partly because the domain is interesting and partly because it made the project feel less like a toy example. The full source is on GitHub: snowflake-dfw-flight-pipeline The Setup Snowflake's external stage + Snowpipe combination is the equivalent of what I'd normally do with Azure Data Factory and a Blob trigger — files land in storage, something detects them, and they get loaded into a table. The implementation is a bit more explicit in Snowf...