Create an Azure Databricks package¶

An Azure Databricks package runs an existing job on a Databricks workspace and can pass it parameters. Use it when the notebook or pipeline logic belongs in Databricks but its place in the nightly sequence belongs in orchestration.

Before you start¶

• An Azure App Registration data provider in the workspace
• The principal able to enumerate resource groups — the chain cannot get past that field otherwise, and it is selection-only
• The principal granted access to the job in Databricks
• A job that already exists. This package type triggers jobs; it does not define them.

The selection chain¶

Data provider → subscription → resource group → Databricks workspace → job, then optional job parameters.

The Azure Databricks workspace that hosts the job. Pick a resource group first to load its workspaces.

Job parameters¶

Once a job is selected, its parameters are read from the job definition and listed for you to fill in:

Read from the job definition. Leave a value empty to use the job's own default.

That last part is the useful bit. You do not have to restate every parameter — override only what differs for this package, and the job's own defaults cover the rest. It also means a parameter added to the job later does not silently break the package.

Steps¶

1. Create the package¶

Orchestration → Packages → Create, then Azure Databricks from the Azure area.

2. Choose the data provider¶

The Azure App Registration data provider whose service principal authenticates against Azure and Databricks.

3. Choose the subscription¶

4. Resource group, workspace, job¶

Each loads from the one above. Then fill in only the job parameters you want to override.

5. Save¶

Saves as a Draft.

Best practices¶

Override the minimum. Leaving a parameter empty uses the job's default, which keeps the package working when the job evolves.

Do not encode secrets in job parameters. Use a Databricks secret scope, or a Timextender global parameter if the value must come from orchestration.

Give the job a stable name. The package stores a reference; renaming the job in Databricks means re-picking it here.

Watch cluster start-up in your timeout. A job on a cold job-cluster can spend several minutes starting before any work happens.

Troubleshooting¶

The resource group list is empty. Grant the principal subscription-level Reader, or Microsoft.Resources/subscriptions/resourceGroups/read. Until then the package cannot be completed, because the field only accepts a selection.

The workspace lists but the job list is empty. Permission on the job is missing in Databricks. Azure RBAC alone is not enough.

"The selected … no longer exists in Azure. Please select another." Something referenced was deleted or renamed. Re-pick from the refreshed list — this message names which level of the chain broke.

A parameter I set is ignored. Check the name matches the job's parameter exactly. An unmatched name is not an error; it is simply not used.

Reference¶

Fields¶

Validation: Please select a data provider, Please select a subscription, Please select a resource group, Please select a Databricks workspace, Please select a job, and The selected {level} no longer exists in Azure. Please select another.

Related¶

• Packages overview
• Azure Databricks package in Timextender Orchestration — the earlier desktop-era article