Data Flows: How to capture changed data
Why Stack Overflow recommends it
Stack Overflow users link it when answering questions about incremental loads and change detection in Azure Data Factory, especially where hashing or data flows can replace timestamp or key columns.
Recommended by 3 people across 3 different months, 2021 to 2023.
What it is
A short walkthrough by Mark from the Azure Data Factory team showing how to build a mapping data flow in Azure Data Factory and Synapse Analytics that captures changed data. It covers incremental change detection and de-duplication using hashing.
What people said
Recommends Mark Kromer MSFT · 2021-05 · Stack Overflow · on “In Azure Data Factory, is it possible to filter the Source query of a Copy Activity, using a column from another Source?”
You can join them and filter and lookup in ADF. But you're trying to do it all in the Copy activity. This is exactly the kind of use case that the data flow activity was created for: [this video]
Recommends Ziya Mert Karakas · 2023-03 · Stack Overflow · on “Beat methods to validate/test the output data with original data”
You should be able to do verify the whole dataset through validate activity if thats what you want (that the dataset meets a certain criteria etc.): [LINK] Dataset validation directly on Alteryx output is not supported, in which case you need to store the dataset somewhere that you just transformed in Alteryx like a blob store, then you can validate there. Validation activity usually supports file storage systems like datalake, (S)FTP, or a relational database. You can also do this through dataflows but you need to store it somewhere, as ADF cannot store any files or data by itself. Once you have both datasets stored, you can for example do this through dataflows by self-joining the datasets
Recommends Kranthi Pakala · 2023-09 · Stack Overflow · on “Incremental copy data from AWS RDS hosted MariaDB to blob storage without write access”
As you don't have much control over your source AWS Maria DB and also Dataflow isn't an option, you may consider switching your solution to Azure Databricks and write a logic using pyspark similar to that of the implementation explained in this tutorial - Data Flows: How to capture changed data [this video] You may explore writing similar custom logic in Python and execute that using Azure Function activity or a Custom activity in ADF.
Cites Mark Kromer MSFT · 2021-04 · Stack Overflow · on “Load data incrementally in azure data factory”
There is a source property in both Copy & Data flow for "last modified date/time". You can also use the pipeline incremental pattern with waterlines or hashing techniques. [this video] [LINK]
Cites Kranthi Pakala · 2022-12 · Stack Overflow · on “Incremental Load without Last Modified Date and Primary Key field in Azure Data Factory”
If you source database support native Change Data Capture feature, then you can use ADF Mapping data flow with no timestamp or ID columns are required to identify the changes since it uses the native change data capture technology in the databases. For complete demonstration, please refer to this public documentation: Change data capture in Azure Data Factory and Azure Synapse Analytics [LINK] Another possible approach is if you can access both old data (Previously loaded data to your sink) and latest data (source) with changes then you can use mapping data flow in ADF and implement hashing to compare the both the datasets and pick the changed data as per your requirement. You can refer to t