Add option to include upstream tasks for `bundle run --only`

[guizsantos]

Describe the issue

When developing a Databricks Job, it is good practice to only execute the tasks that got updated. Oftentimes upstream or downstream tasks have to be executed as well to reproduce the pipeline reliably.

Currently, Databricks CLI supports executing specific tasks in a Job by means of the --only flag but for some reason there is no native way to automatically include upstream or downstream tasks of the selected tasks. However, via the UI, it is possible to do this:

It'd be really great to have a flag --include-downstream/--include-upstream to do just that. Ideally, it would be possible to include upstream/downstream tasks on a per-task basis such as dbt does with its + operator (dbt run --select +model_name+), but adding a general flag that would do this for all of the tasks selected using the --only flag would be a great addition as it is.

Some examples with expected behaviors:

databricks bundle run job_name --only task1,task8 --include-upstream

Would run task1 and its upstream tasks plus task8 and its upstream tasks

databricks bundle run job_name --only task1,task8 --include-downstream

Would run task1 and its downstream tasks plus task8 and its downstream tasks

databricks bundle run job_name --only task1,task8 --include-upstream --include-downstream

Would run task1 and its upstream tasks and its downstream tasks plus task8 and its upstream tasks and its downstream tasks

Otherwise, mimicking dbt + operator would also work but I guess it would require far more effort to make happen.

OS and CLI version

Databricks CLI v0.282.0

Is this a regression?

No

Debug Logs

N/A

[vitorweiss]

Having a native way to select specific tasks and modularly include their upstream and downstream dependencies is necessary when developing and iterating on Databricks Jobs.

The ability to run a subset of tasks while optionally expanding execution to their upstream and/or downstream dependencies would enable more reliable local and iterative development, without requiring manual task selection or UI-based execution. A flag-based or task-level mechanism to include upstream and downstream tasks would make the CLI workflow significantly more flexible and consistent with real pipeline dependency structures.

This kind of modular task selection is an important gap in the current CLI behavior and would meaningfully improve job development and testing workflows.

[PauloBaptista77]

That'd be a great adition to the platform, having a way of running only the required models would be extremely helpful for developing taks!!

[Indi-RafaelAndrade]

Would be a great addition, very helpful!!

[Schadenlord]

Fully support this idea! The ability to include upstream/downstream dependencies, similar to how dbt does it, would make iterative development much smoother. It’s a bit tedious to select tasks manually right now, so having a native way to handle this in the CLI would be awesome

[afberg]

Hi, we are tracking this internally but haven't been able to prioritise the effort yet. We are working on other improvements in the area and will hopefully be able to tackle it soon.

In the meantime, thank you for your patience.