Create an Azure Data Factory package¶

An Azure Data Factory package triggers a pipeline in one of your data factories and waits for it to finish. Use it to put an existing ADF pipeline into an orchestration flow, so that what happens before and after it is governed in one place.

Before you start¶

You need an Azure App Registration data provider in the workspace whose service principal can see the data factory. Everything else — subscription, resource group, factory, pipeline — is discovered from that provider.

Selecting a data provider loads its factories; selecting a factory loads that factory's pipelines. If either list comes back empty, the service principal cannot see the resource; see Troubleshooting.

Steps¶

1. Create the package¶

Go to Orchestration → Packages, select Create, and choose Azure Data Factory from the Azure area.

The editor opens with three fields, the lower two disabled until the one above them is set.

2. Name the package and pick the data provider¶

Give it a name that says what the pipeline achieves, not what it is called in Azure — "Trigger the daily warehouse load" rather than "Run adf-pipeline-3".

The Azure App Registration data provider whose service principal connects to Azure Data Factory. Select a data provider to load its factories and pipelines.

3. Pick the data factory¶

The list loads from Azure for the selected provider. Refresh re-reads it — needed when a factory was created in Azure after this page loaded.

4. Pick the pipeline¶

Pipelines are listed for the selected factory, again with a refresh button.

That is the whole configuration: provider, factory, pipeline.

5. Save¶

The package is a Draft until you publish it.

Choosing what belongs in ADF and what belongs here¶

The two tools overlap, and the sensible division is not obvious.

Keep inside one ADF pipeline the work that shares a failure boundary — copy activities that must all succeed or all be retried together, and anything that depends on ADF-specific features like data flows or integration runtimes.

Split into separate packages the steps that different people care about, or that you want to retry independently. A failed pipeline is one red package in the orchestration log; three packages tell you which third of the night's work failed.

A useful pattern is one package per pipeline, wired together in a process map, so the dependencies between pipelines are visible in Timextender rather than encoded in ADF triggers that nobody remembers.

Best practices¶

Name the package for the outcome. The pipeline name is already recorded on the Definition tab; the package name is what appears in schedules, logs and process maps.

Set a timeout longer than the pipeline's normal runtime, since the package waits for completion. Too short and you will fail runs that were going to succeed.

Give the service principal least privilege. It needs to read the factory and start the pipeline, not contribute to the resource group.

Fill in Properties → Action. "Check the ADF run history in the Azure portal for pipeline X" saves the next person a lookup.

Troubleshooting¶

The factory list is empty. The service principal cannot see any data factory in its subscriptions. Check the App Registration has a role assignment on the factory or its resource group, then use Refresh.

The pipeline list is empty but the factory is listed. The principal can see the factory but not enumerate its pipelines. That is usually a narrower role than intended — Data Factory Contributor or a custom role including pipeline read.

A pipeline I just created is not listed. Use the Refresh button next to the field. The list is fetched when the page loads.

The package fails but the pipeline succeeded in Azure. Check which run the log names. The package reports the run it started; a pipeline that also runs on its own ADF trigger can succeed separately while the triggered run failed.

Authentication failures after the package has been working. The client secret on the data provider has expired. Azure App Registration secrets have a finite lifetime, and nothing warns you before the day it stops.

Reference¶

Fields¶

Validation, if something is missing: Please select a data provider, Please select a data factory, Please select a pipeline. Hover Save to see which apply.

What is stored, and what the engine expects¶

The saved package records the data provider, the factory and the pipeline name. The execution engine additionally resolves and records the subscription id and resource group from the factory, and reads the tenant, application id and secret from the data provider at run time rather than storing them on the package.

Because the engine resolves credentials at run time, rotating the secret on the data provider takes effect without editing the packages that use it.

Execution semantics¶

The engine starts the pipeline run, then polls its status, sleeping between checks until the run reaches a final state. Exit code 0 is returned only for Succeeded; every other final state exits non-zero and fails the package.

Related¶

• Packages overview
• Create a Power BI Refresh package — the other trigger-and-wait Azure type
• Trigger Fabric data pipelines with Timextender Orchestration