A metadata-driven ETL framework using Azure Data Factory boosts scalability, flexibility, and security in integrating diverse data sources with minimal rework.

In a recent blog post, Microsoft announced the general availability (GA) of their serverless, code-free Extract-Transform-Load (ETL) capability inside of Azure Data Factory called Mapping Data Flows.

In “ Designing a metadata-driven ETL framework with Azure ADF: An architectural perspective,” I laid the groundwork for a scalable, metadata-driven ETL framework using Azure Data Factory (ADF).

Kensu gathers metadata from Azure Data Factory pipeline runs and datasets used in the pipelines, feeding valuable insights into data lineage, schema changes, and performance metrics.