Azure Data Factory Tutorial | Introduction to ETL in Azure
Why Stack Overflow recommends it
Stack Overflow users link it in Azure data movement answers as a quick, visual intro to Data Factory for copying files such as CSVs from blob storage into SQL or other formats.
Recommended by 2 people across 4 different months, 2019 to 2023.
What it is
A 25-minute introduction to Azure Data Factory by Azure trainer Adam Marczak. It explains the basic building blocks of the service and walks through a common scenario of loading data from Blob Storage into an Azure SQL database, with sample code on GitHub.
What people said
Recommends Adam Marczak · 2019-07 · Stack Overflow · on “Load CSV files to Azure SQL Warehouse via curl”
You can maybe do something like this. Upload to blob storage uisng this script. This is pure HTTP call to blob using PowerShell but you can produce the same result using bash/batch and curl. # Storage Account $accountname="" $container="" # Access Key $key = "" # Blob details $fileContent = "content" $blobName="file.txt" $bloburl = " [LINK] " $BlobOperation = "PUT" $filelen = $fileContent.Length $RESTAPI_URL = " [LINK] "; $date=(Get-Date).ToUniversalTime() $datestr=$date.ToString("R"); $strtosign = "$BlobOperation`n`n`n$filelen`n`n`n`n`n`n`n`n`nx-ms-blob-type:BlockBlob`nx-ms-date:$datestr`nx-ms-version:2015-04-05`n/" $strtosign = $strtosign + $accountname + "/" + $container + "/" +$blobName
Recommends Adam Marczak · 2019-08 · Stack Overflow · on “Azure SQL DB - data file export (.csv) from azure sql”
Honestly the best in terms of professional approach is to use Azure Data Factory and installation of Integration Runtime on the on premises. You of course can use BCP but it will be cumbersome in the long run. A lot of scripts, tables, maintenance. No logging, no metrics, no alerts... Don't do it honestly. SSIS is another option butin my opinion it takes more effort than ADF solution. Azure Data Factory will allow you to do this in professional way using user interface with no coding. It also can be parametrized so you just change name of table name parameter and suddenly you are exporting 20, 50 or 100 tables at ease. Here is video example and intro into data factory if you want to see quic
Recommends Adam Marczak · 2019-08 · Stack Overflow · on “Azure function to convert encoded json IOT Hub data to csv on azure data lake store”
I would really not save them into blob but just leave them in IoT hub and make functions trigger of that. But if you really it need from blob then. Make function trigger like this [FunctionName("BlobTriggerCSharp")] public static void Run( [BlobTrigger("input/{name}")] Stream myBlob, [Blob("output/{name}"), FileAccess.Write] Stream outputBlob string name, ILogger log) { // parse JSON with JsonConvert // parse using base64 decode from SystemProperties.contentType field // write parsed output to outputBlob stream } Then in ADF Placed this in demo container as file called demo.json on blob storage. I created source dataset of type JSON in ADF [LINK] which looks like this [LINK] You can test if
Recommends Adam Marczak · 2019-08 · Stack Overflow · on “What is the easiest way to pull data from a blob and load it into a table in SQL Server?”
I would create Azure Logic App that would HTTP trigger (this will be manually ran) Iterate through all blobs present on storage account For each element use blob connector which has action called Extract archive to extract zipped files into blob Use Azure SQL BCP to pull this, if this is different format than CSV then use Azure Data Factory Later if you expect new blobs do similar flow except use New or modified blob trigger. For Azure SQL BCP from BLOB example with Logic Apps check this article [LINK] And if you want general intro into Azure Logic Apps check this video [LINK] In here there is also new blob trigger demo. In case you need data factory feel free to check this video. It has blo
Recommends Adam Marczak · 2019-09 · Stack Overflow · on “CSV to AVRO conversion in Azure”
That's actually pretty straightforward to do with Azure Data Factory and Blob Storage . This should be also very cheap because you pay per second when executing in ADF so you only pay for conversion time. No infra required. If your CSV looks like this ID,Name,Surname 1,Adam,Marczak 2,Tom,Kowalski 3,John,Johnson Upload it to blob storage into input container [LINK] Add linked service for blob storage in ADF [LINK] [LINK] [LINK] Select your storage [LINK] Add dataset [LINK] Of blob type [LINK] And set it to CSV format [LINK] With values as such [LINK] Add another dataset [LINK] Of blob type [LINK] And select Avro type [LINK] With value likes [LINK] Add pipeline [LINK] Drag-n-drop Copy Data act
Recommends Kahfi Nadhika · 2023-09 · Stack Overflow · on “Blob File Not Appearing in Query Editor”
[this video] Here's the link for the tutorial that I followed, I copied all the steps from the tutorial. I hope this helps
Cites IpsitaDash-MT · 2021-06 · Stack Overflow · on “Migrate azure databricks delta table to Azure Synapse SQL pool”
Below are the steps to copy data from Azure data bricks delta tables to Azure Synapse using azure data factory Step1: Create a data set with "Azure data bricks delta lake" connector to point to your delta table [LINK] Step2: Create a data set with "Azure Synapse Analytics" connector to point to your Synapse Analytics [LINK] Step3: Use copy activity. In copy activity use step1 data set as source and step2 data set as sink Where you have the Delta files in the Azure Databricks are the source data set, just you must create the pipeline to perform the copy activity on the Datasets to reflect on the sink Datasets that will be your Azure Synapse SQL pool. [LINK] Refer this link: Basic scenario how