New hidden `install-pipelines-cli` command that creates `pipelines` symlink to `databricks` #3009

alyssa-db · 2025-06-06T22:48:51Z

Changes

New file: pipelines.go (package pipelines)
New file: install_pipelines_cli.go (package pipelines)
- To databricks, implements a hidden install-pipelines-cli command that creates a symlink named pipelines pointing to the real databricks binary.
- Handles symlink creation, permission checks, and error handling for existing files/symlinks.
New folder: pipelines/install-pipelines-cli acceptance test
main.go returns a root depending on if the binary is invoked as pipelines/pipelines.exe or databricks

Why

Symlink installation: This enables users to invoke pipelines as a standalone CLI, which internally points to the main databricks binary. This is a step towards a dedicated Pipelines CLI experience, while minimizing code duplication.
Stub CLI: Lays groundwork for future pipelines-specific commands and workflows
Testing: Acceptance test.

Tests

Added acceptance tests for install-pipelines-cli in pipelines to check if success message pipelines successfully installed is printed
Test descriptions:
1. Fresh installation:
  Creates temporary directory and verifies successful symlink creation.
  Validates CLI output and stub command functionality.
2. Already installed scenario:
  Confirms graceful handling when symlink already exists.
3. File collision test:
  Ensures installation fails when regular file exists at target path.
4. Aliased binary execution:
  Verifies installation works when databricks binary is renamed/called via alias.

…atabricks CLI with tests

cmd/cmd.go

cmd/dlt/dlt.go

cmd/dlt/dlt_test.go

kanterov

Left few comments

acceptance/dlt/install-dlt/script

cmd/dlt/dlt.go

Makefile

denik · 2025-06-11T08:53:03Z

acceptance/acceptance_test.go

@@ -150,7 +150,8 @@ func testAccept(t *testing.T, inprocessMode bool, singleTest string) int {
 	repls.SetPath(execPath, "[CLI]")

 	// Make helper scripts available
-	t.Setenv("PATH", fmt.Sprintf("%s%c%s", filepath.Join(cwd, "bin"), os.PathListSeparator, os.Getenv("PATH")))
+	pathDirs := []string{filepath.Join(cwd, "bin"), buildDir}
+	t.Setenv("PATH", fmt.Sprintf("%s%c%s", strings.Join(pathDirs, string(os.PathListSeparator)), os.PathListSeparator, os.Getenv("PATH")))


This adds buildDir to PATH? What for?

If the goal is to expose dlt command to test then we should

modify BuildCLI to link databricks to dlt

export $DLT env var pointing to built dlt (+ replacement)

The intention here was probably so dlt can be found on the build path when it's built, but we should probably export the $DLT env var the same way we do it for CLI and other envs.

qq: maybe I misunderstood, wdym by "+ replacement" here?

acceptance/dlt/install-dlt/output.txt

jefferycheng1 · 2025-06-11T09:07:59Z

I think it's better to write it as an acceptance test, for example as here
https://github.com/databricks/cli/tree/main/acceptance/cmd/unknown-subcommand

Here you can see a short guide how to do that https://github.com/databricks/cli/tree/main/acceptance

@denik / @kanterov high level question here: we should be definitely adding acceptance tests, but in general does it hurt to have unit tests as well? A quick look through the repo and it feels like some files have more unit tests while others don't.

For this command specifically, I think it's helpful to test the edge cases for which PATH might not contain databricks, target local isn't writable etc... WDYT?

denik · 2025-06-11T09:16:30Z

@denik / @kanterov high level question here: we should be definitely adding acceptance tests, but in general does it hurt to have unit tests as well? A quick look through the repo and it feels like some files have more unit tests while others don't.

For this command specifically, I think it's helpful to test the edge cases for which PATH might not contain databricks, target local isn't writable etc... WDYT?

Depends on code being test; if it's a pure function / class, makes sense to cover edge cases with unit tests. If it involves reading env vars, command line arguments then acceptance tests typically provide more realistic setup.

cmd/dlt/dlt.go

cmd/cmd.go

cmd/dlt/install_dlt.go

acceptance/dlt/install-dlt/script

cmd/dlt/install_dlt.go

acceptance/dlt/install-dlt/script

shreyas-goenka · 2025-06-18T09:18:09Z

acceptance/dlt/install-dlt/script

+
+title "Executable not called as databricks"
+cp $CLI notdatabricks
+dlt=$(trace errcode ./notdatabricks install-dlt -o json | jq -r .symlink_path)


It's also not clear here what's being asserted.

In the case that the user has databricks aliased, want to make sure the install-dlt command still works.

cmd/dlt/install_dlt.go

andrewnester · 2025-06-18T15:13:17Z

main.go

+// binary to be used for both DLT and databricks CLI commands.
+func getCommand(ctx context.Context) *cobra.Command {
+	invokedAs := filepath.Base(os.Args[0])
+	if invokedAs == "dlt" || (runtime.GOOS == "windows" && invokedAs == "dlt.exe") {


nit: you can replace it with strings.StartsWith for invokedAs as nit

andrewnester

Approved to unblock but first symlink needs to be renamed from dlt to pipelines

pietern

Please update the PR title and summary to reflect the new state of the PR.

acceptance/pipelines/install-pipelines/script

cmd/pipelines/install_pipelines.go

main.go

github-actions · 2025-06-19T19:06:11Z

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

PR number: 3009
Commit SHA: 9156ad52393b0c5260b7573fe18c969c2ae82723

Checks will be approved automatically on success.

Already addressed

New hidden install-dlt command that creates symlink in same PATH as d…

60260b7

…atabricks CLI with tests

alyssa-db requested review from pietern, andrewnester, shreyas-goenka, denik and anton-107 as code owners June 6, 2025 22:48

andrewnester reviewed Jun 10, 2025

View reviewed changes

alyssa-db added 4 commits June 10, 2025 11:39

addressed code comments prior to testing

6750e38

added acceptance tests

359a77d

modified acceptance test PATH to add databricks and helper tools

d12a848

Merge branch 'databricks:main' into install-dlt-feat

e3c41fc

alyssa-db requested a review from kanterov June 10, 2025 23:38

kanterov reviewed Jun 11, 2025

View reviewed changes

acceptance/dlt/install-dlt/script Outdated Show resolved Hide resolved

cmd/dlt/dlt.go Outdated Show resolved Hide resolved

cmd/dlt/dlt.go Outdated Show resolved Hide resolved

denik reviewed Jun 11, 2025

View reviewed changes

jefferycheng1 reviewed Jun 11, 2025

View reviewed changes

cmd/dlt/dlt.go Outdated Show resolved Hide resolved

jefferycheng1 reviewed Jun 11, 2025

View reviewed changes

cmd/dlt/dlt.go Outdated Show resolved Hide resolved

jefferycheng1 reviewed Jun 11, 2025

View reviewed changes

cmd/cmd.go Outdated Show resolved Hide resolved

jefferycheng1 reviewed Jun 11, 2025

View reviewed changes

cmd/cmd.go Outdated Show resolved Hide resolved

made dlt path be based on executable

ca3d09b

alyssa-db force-pushed the install-dlt-feat branch from 8711e41 to ca3d09b Compare June 11, 2025 20:15

added acceptance test python script, refactored install-dlt

d168e96

alyssa-db force-pushed the install-dlt-feat branch from d1f1df0 to d168e96 Compare June 12, 2025 01:07

alyssa-db added 3 commits June 12, 2025 12:03

draft acceptance test and acceptance test DLT setup

c282769

acceptance test logic

e771366

Merge remote-tracking branch 'upstream/main' into install-dlt-feat

9fac6c7

alyssa-db force-pushed the install-dlt-feat branch 2 times, most recently from 085a428 to 9d53ff8 Compare June 12, 2025 23:39

windows path

14b7d31

alyssa-db requested review from lennartkats-db, shreyas-goenka and jefferycheng1 June 18, 2025 02:43

alyssa-db force-pushed the install-dlt-feat branch from ac4799e to 45c7df1 Compare June 18, 2025 05:50

not clean paths windows

12b1e64

alyssa-db force-pushed the install-dlt-feat branch from 45c7df1 to 12b1e64 Compare June 18, 2025 06:23

error order

2d8da98

shreyas-goenka reviewed Jun 18, 2025

View reviewed changes

alyssa-db force-pushed the install-dlt-feat branch from 554264e to 515956f Compare June 18, 2025 10:42

removed json, added relative path to tests

1da3ff0

alyssa-db force-pushed the install-dlt-feat branch from 515956f to 1da3ff0 Compare June 18, 2025 11:02

alyssa-db requested a review from shreyas-goenka June 18, 2025 11:12

andrewnester approved these changes Jun 18, 2025

View reviewed changes

alyssa-db enabled auto-merge June 18, 2025 18:43

alyssa-db disabled auto-merge June 18, 2025 20:08

andrewnester approved these changes Jun 18, 2025

View reviewed changes

alyssa-db added 3 commits June 18, 2025 15:49

modified invoke statement

be778d6

Merge branch 'main' into install-dlt-feat

f9dc119

renamed dlt to pipelines

fb62458

alyssa-db requested a review from andrewnester June 19, 2025 01:38

pietern previously requested changes Jun 19, 2025

View reviewed changes

acceptance/pipelines/install-pipelines/script Outdated Show resolved Hide resolved

cmd/pipelines/install_pipelines.go Outdated Show resolved Hide resolved

cmd/pipelines/install_pipelines.go Outdated Show resolved Hide resolved

main.go Show resolved Hide resolved

alyssa-db changed the title ~~New hidden install-dlt command that creates dlt symlink to databricks~~ New hidden install-pipelines-cli command that creates pipelines symlink to databricks Jun 19, 2025

alyssa-db changed the title ~~New hidden install-pipelines-cli command that creates pipelines symlink to databricks~~ New hidden install-pipelines-cli command that creates pipelines symlink to databricks Jun 19, 2025

renamed to install-pipelines-cli

9156ad5

alyssa-db force-pushed the install-dlt-feat branch from 0fd2c95 to 9156ad5 Compare June 19, 2025 19:06

alyssa-db requested a review from pietern June 19, 2025 19:15

andrewnester approved these changes Jun 20, 2025

View reviewed changes

andrewnester merged commit d5108bf into databricks:main Jun 20, 2025
9 checks passed

New hidden install-pipelines-cli command that creates pipelines symlink to databricks #3009

New hidden install-pipelines-cli command that creates pipelines symlink to databricks #3009

Uh oh!

Conversation

alyssa-db commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kanterov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

denik Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

denik Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jefferycheng1 Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jefferycheng1 commented Jun 11, 2025

Uh oh!

denik commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shreyas-goenka Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

alyssa-db Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andrewnester Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

andrewnester left a comment

Choose a reason for hiding this comment

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 19, 2025

Uh oh!

Uh oh!

Uh oh!

New hidden `install-pipelines-cli` command that creates `pipelines` symlink to `databricks` #3009

New hidden `install-pipelines-cli` command that creates `pipelines` symlink to `databricks` #3009

alyssa-db commented Jun 6, 2025 •

edited

Loading

denik Jun 11, 2025 •

edited

Loading