dify/cli/test/e2e

Dify CLI — E2E Test Suite

End-to-end tests that exercise the real difyctl binary against a live Dify server. Every test uses an isolated temporary config directory so no state leaks between test files.

Directory layout

test/e2e/
├── setup/
│   ├── env.ts              — Load & validate DIFY_E2E_* env vars (CE + EE)
│   ├── global-setup.ts     — CE/EE-aware bootstrap: account creation, token
│   │                         minting, workspace provisioning, DSL import
│   └── global-teardown.ts  — Delete conversations created during the run
│
├── helpers/
│   ├── cli.ts              — run(), withAuthFixture(), mintFreshToken(),
│   │                         injectAuth(), spawn_background()
│   ├── assert.ts           — assertExitCode, assertJson, assertErrorEnvelope,
│   │                         assertNoAnsi, assertPipeFriendlyJson, ...
│   ├── cleanup-registry.ts — registerConversation() / cleanupRegisteredConversations()
│   ├── retry.ts            — withRetry(fn, { attempts, delayMs })
│   └── skip.ts             — optionalIt(), optionalDescribe(),
│                             enterpriseOnlyIt(), enterpriseOnlyDescribe(), isEE()
│
└── suites/
    ├── auth/
    │   ├── status.e2e.ts   — auth status (text + JSON + SSO)
    │   ├── use.e2e.ts      — workspace switching ([EE] cases require 2 workspaces)
    │   ├── whoami.e2e.ts   — whoami + external SSO session checks
    │   ├── devices.e2e.ts  — devices list + revoke (runs near-last)
    │   └── logout.e2e.ts   — logout + local credential cleanup (runs last)
    ├── config/
    │   └── config.e2e.ts   — config path/get/set/unset/view, env override
    ├── discovery/
    │   ├── get-app-list.e2e.ts           — basic get app list
    │   ├── get-app-single.e2e.ts         — get single app by ID
    │   ├── describe-app.e2e.ts           — describe app
    │   └── get-app-all-workspaces.e2e.ts — get app -A ([EE] multi-workspace cases)
    └── run/
        ├── run-app-basic.e2e.ts     — basic run, -o json, --inputs, streaming,
        │                              conversation, CI mode
        ├── run-app-streaming.e2e.ts — Ctrl+C / error-event / chunk timing
        ├── run-app-file.e2e.ts      — --file upload (local + remote URL)
        └── run-app-hitl.e2e.ts      — HITL pause + resume

Edition support

difyctl supports two Dify editions. The test suite adapts automatically:

Edition	DIFY_E2E_EDITION	Workspaces	EE-only cases
Community Edition (CE)	ce (default)	1	Skipped
Enterprise Edition (EE)	ee	2 (auto-created)	Active

EE-only test cases

Tests that require Enterprise Edition features (workspace switching between independent workspaces, cross-workspace app query, etc.) are tagged [EE] in their names and wrapped with enterpriseOnlyIt() / enterpriseOnlyDescribe() from helpers/skip.ts. In CE mode these tests are automatically skipped.

// helpers/skip.ts usage
const eeIt = enterpriseOnlyIt(caps)
eeIt('[EE][P0] cross-workspace query returns apps from all workspaces', async () => {
  // test body
})

Setup

Copy the credential template and fill in your values:

cp cli/test/e2e/.env.e2e.example cli/.env.e2e
# edit cli/.env.e2e with real credentials

Community Edition (CE) — minimum 3 vars

Variable	Description
DIFY_E2E_HOST	Server base URL (http://localhost)
DIFY_E2E_EMAIL	Account email — created automatically by global-setup
DIFY_E2E_PASSWORD	Account password

global-setup will:

1. Register the account (idempotent — safe to rerun)
2. Login and mint a bearer token via the device flow
3. Import all DSL fixtures into the single workspace
4. Publish apps and set access_mode → public

Enterprise Edition (EE) — 5 required vars

Variable	Description
DIFY_E2E_EDITION	Must be ee
DIFY_E2E_HOST	Console/API base URL
DIFY_E2E_EMAIL	Member account email — created via enterprise API
DIFY_E2E_PASSWORD	Member account password
DIFY_E2E_ENTERPRISE_API_URL	Enterprise admin API base URL (https://.../inner/api)
DIFY_E2E_ENTERPRISE_API_SECRET_KEY	Enterprise admin API secret key

Optional:

Variable	Description
DIFY_E2E_CONSOLE_URL	Console URL if different from DIFY_E2E_HOST

global-setup will:

1. Create the member account via the enterprise admin API (idempotent)
2. Login and obtain a session cookie
3. Create two workspaces (e2e-primary-auto, e2e-secondary-auto) via the enterprise API
4. Import DSL fixtures into both workspaces
5. Publish apps and set access_mode → public via the enterprise API

Optional overrides (both editions)

Variable	Description
DIFY_E2E_TOKEN	Pre-minted bearer token — skips device-flow mint
DIFY_E2E_SSO_TOKEN	External SSO bearer token (dfoe_...)
DIFY_E2E_WORKSPACE_ID	Override primary workspace ID
DIFY_E2E_WORKSPACE_NAME	Override primary workspace name
DIFY_E2E_WS2_ID	Override secondary workspace ID (EE)
DIFY_E2E_CHAT_APP_ID	Override echo-chat app ID
DIFY_E2E_WORKFLOW_APP_ID	Override echo-workflow app ID
DIFY_E2E_FILE_APP_ID	Override file-upload app ID
DIFY_E2E_FILE_CHAT_APP_ID	Override file-chat app ID
DIFY_E2E_HITL_APP_ID	Override HITL main app ID
DIFY_E2E_HITL_EXTERNAL_APP_ID
DIFY_E2E_HITL_SINGLE_ACTION_APP_ID
DIFY_E2E_HITL_MULTI_NODE_APP_ID
DIFY_E2E_WS2_APP_ID	Override secondary workspace app ID (EE)

Running tests

cd cli

# Community Edition (default)
bun run test:e2e

# Enterprise Edition
DIFY_E2E_EDITION=ee bun run test:e2e

# Run only [P0] smoke cases
bun run test:e2e:smoke

# Run only EE-tagged cases (P0 smoke)
DIFY_E2E_EDITION=ee bun run test:e2e:smoke --testNamePattern "\[EE\]"

# Run offline-safe config tests only (no network required)
bun run test:e2e:local

# Run a single file
bun vitest --config vitest.e2e.config.ts test/e2e/suites/auth/status.e2e.ts

Test execution order

Files run sequentially (fileParallelism: false) in this order:

login → status → use → whoami → help → config → output → error-handling
  → framework → discovery → run (basic / streaming / file / HITL)
  → devices → logout

devices and logout run last because they revoke real server sessions.

Design decisions

Decision	Rationale
CE/EE edition flag	DIFY_E2E_EDITION=ce/ee controls global-setup bootstrap path and activates/skips [EE]-tagged tests.
[EE] tag convention	Test names include [EE] to make skipped cases visible in the report and to allow --testNamePattern "\[EE\]" filtering.
enterpriseOnlyIt(caps)	Returns it in EE mode, it.skip in CE mode — no runtime assertions needed, skip is declarative.
No mocking	All HTTP traffic goes to the real server — this catches real integration regressions.
Isolated config dirs	Each test creates a fresh withTempConfig() dir; session state never leaks between tests.
withAuthFixture()	Combines withTempConfig + injectAuth into a single fixture; reduces beforeEach boilerplate.
injectAuth() bypasses Device Flow	Non-auth tests skip the browser step; only auth/ suites exercise the real flow.
mintFreshToken()	logout and devices-revoke tests mint a disposable dfoa_ token via the device flow API.
Global retry: 0	Flaky network calls use withRetry() locally; global retry masks non-idempotent failures.
Conversation cleanup	registerConversation() + global-teardown delete staging conversations after the run.