workflow_dispatch boolean inputs in GitHub Actions do not apply default
values when triggered via gh CLI without explicit --field flags.
They arrive as empty string ('') which does not equal true, causing all
if: inputs.suite_XXX == true conditions to evaluate as false.
Fix: change all suite if conditions from == true to != 'false' so that
both explicit true and the default empty string are treated as enabled.
Previously, passing positional file args to pnpm test:e2e had no effect
because vitest.e2e.config.ts include list took precedence, causing every
parallel job to run the full suite and hit the 20-minute timeout.
vitest.e2e.config.ts:
- Add DIFY_E2E_INCLUDE env var (comma-separated globs, replaces SINGLE_FILE)
- DIFY_E2E_SINGLE_FILE kept as deprecated alias for back-compat
- When set, include list is built from DIFY_E2E_INCLUDE instead of the
hardcoded full-suite list
cli-e2e.yml:
- Every suite job now sets DIFY_E2E_INCLUDE to target its own files
- Remove positional file args (they were silently ignored by vitest config)
- Standardise smoke filter: FILTER_ARGS array -> inline -t arg
- timeout-minutes: discovery 20->15, run/* 20->35 (CI ~5x slower)
Workflow (cli-e2e.yml):
- Plan B: split into parallel jobs after a shared provision step
- provision job: mints token + provisions DSL apps, outputs IDs to downstream jobs
- suite-run: matrix of 5 parallel jobs (basic/streaming/conversation/file/hitl)
- suite-last: serial, waits for all parallel jobs; runs use/devices/logout/agent
- Add DIFY_E2E_NO_KEYRING=1 globally (Linux CI has no keychain)
- timeout-minutes: 30 → 60; smoke filter: --testNamePattern → -t
New script (cli/scripts/e2e-provision.ts):
- Standalone Bun script: login + mint token + discover workspaces + provision apps
- Outputs to GITHUB_OUTPUT + .provision-output.json
Test fixes:
- run-app-streaming: DIFY_E2E_NO_KEYRING=1 in spawn-based tests
- run-app-basic: fix cache test (DIFY_CACHE_DIR + app-info.yml path)
- run-app-conversation/file: SSO injectAuth uses new hosts.yml format
- get-app-list: tag filter auto-provisions e2e-test tag
- vitest.e2e.config.ts: DIFY_E2E_SINGLE_FILE override + agent suite
vitest.e2e.config.ts:
Read VITEST_RETRY env var (default 0) to set the global retry count.
Local runs keep retry=0; per-test withRetry() stays the precise tool
for known flaky paths. CI sets retry=2 to handle transient server 500s.
cli-e2e.yml:
Pass VITEST_RETRY=2 to the test step so each failing test gets up to
2 automatic retries before being reported as a failure.
Bash array elements containing ** globs are not expanded when passed
as arguments to pnpm, causing 'No test files found'. Replace with
process substitution + find to reliably collect .e2e.ts paths.
Replace fixed 'smoke/full' choice with individual boolean inputs:
suite_discovery (default: true)
suite_run (default: true)
suite_auth (default: false)
suite_config (default: false)
Combined with a 'test_scope' choice (smoke=[P0] / full=all cases).
This allows any combination of suites to be run in a single dispatch
without breaking up into multiple workflows.
Default behaviour (no inputs): discovery + run, all cases.
Drop the middleware/API/provision steps that spun up a full Dify
Docker stack. All DIFY_E2E_* vars are now injected from repository
Secrets so the workflow targets an existing staging server instead
of building one on every run.
Also removed: dify_version input (no longer needed), Dump Dify logs
step (no Docker stack to inspect). Timeout reduced from 45→30 min.
- Override pnpm to v11 (packageManager field) before CLI install step to
resolve ERR_PNPM_BAD_PM_VERSION conflict with setup-web's pnpm@9
- Add per-suite dedicated tokens (logoutToken / devicesToken) in global-setup
via device flow to prevent token cross-contamination between suites
- Translate all Chinese comments and test names to English across e2e suites
- Fix injectAuth: add tokenId field so devices revoke correctly detects selfHit
- Fix HITL test: read action id dynamically from pause response instead of
hardcoding 'submit'
Triggers on pull_request when cli/** files change.
Spins up a full Dify stack via docker compose, provisions
an admin account + test apps, then runs the E2E smoke suite
(P0 cases only for CI speed).
gigglewang0417