refactor(e2e): unify seeded runtime orchestration (#39700)
This commit is contained in:
yyh
@@ -139,21 +139,6 @@ jobs:
|
||||
exit 1
|
||||
fi
|
||||
|
||||
teardown_external_runtime() {
|
||||
local run_status=$?
|
||||
trap - EXIT
|
||||
if ! vp run e2e:middleware:down; then
|
||||
echo "::error title=E2E teardown failed::External runtime middleware did not shut down cleanly."
|
||||
if [[ "$run_status" -eq 0 ]]; then
|
||||
run_status=1
|
||||
fi
|
||||
fi
|
||||
exit "$run_status"
|
||||
}
|
||||
|
||||
trap teardown_external_runtime EXIT
|
||||
vp run e2e:middleware:up
|
||||
vp run e2e:post-merge:prepare
|
||||
vp run e2e:post-merge
|
||||
|
||||
- name: Upload Cucumber report
|
||||
|
||||
@@ -22,6 +22,8 @@ E2E_MODEL_PROVIDER_CREDENTIALS_JSON='{"openai_api_key":"replace-with-real-key"}'
|
||||
# from @external-model because other external-model scenarios may not use dify-agent.
|
||||
# When enabled, the E2E runner also starts the shellctl local sandbox required by
|
||||
# dify-agent's dify.shell/config runtime layer.
|
||||
# Use E2E_AGENT_BACKEND_URL only for an already-running backend; do not set it
|
||||
# together with E2E_START_AGENT_BACKEND.
|
||||
# E2E_START_AGENT_BACKEND=1
|
||||
# E2E_AGENT_BACKEND_PORT=5050
|
||||
# E2E_AGENT_BACKEND_URL=http://127.0.0.1:5050
|
||||
|
||||
+8
-3
@@ -8,9 +8,11 @@ Run commands from the repository root. Install dependencies and browsers once wi
|
||||
|
||||
- Existing initialized instance: `pnpm -C e2e e2e`
|
||||
- Reset, initialize, and run deterministic scenarios: `pnpm -C e2e e2e:full`
|
||||
- Prepare and run scenarios backed by shared fixtures: `E2E_START_AGENT_BACKEND=1 pnpm -C e2e e2e:prepared`
|
||||
- Tagged subset: `pnpm -C e2e e2e -- --tags @smoke`
|
||||
- Headed debugging: `pnpm -C e2e e2e:headed -- --tags @smoke`
|
||||
- External runtime preparation and run: `pnpm -C e2e e2e:external:prepare`, then `pnpm -C e2e e2e:external`
|
||||
- Prepare and run external runtime scenarios: `E2E_START_AGENT_BACKEND=1 pnpm -C e2e e2e:external`
|
||||
- Seed against existing middleware without running Cucumber: `pnpm -C e2e seed -- --profile <prepared|external-runtime|post-merge>`
|
||||
- Reset persisted E2E state: `pnpm -C e2e e2e:reset`
|
||||
- Middleware lifecycle: `pnpm -C e2e e2e:middleware:up` and `pnpm -C e2e e2e:middleware:down`
|
||||
- Scoped static checks: `vp check e2e`
|
||||
@@ -20,7 +22,8 @@ The runner reuses `web/.next/BUILD_ID` when present. Set `E2E_FORCE_WEB_BUILD=1`
|
||||
## Runtime Ownership
|
||||
|
||||
- `scripts/setup.ts` owns reset, middleware, backend, and frontend startup.
|
||||
- `scripts/run-cucumber.ts` owns E2E orchestration and Cucumber invocation.
|
||||
- `scripts/run-cucumber.ts` is the only E2E runtime orchestrator. It owns service lifetime, optional seed execution, Cucumber invocation, and teardown.
|
||||
- `scripts/seed-runner.ts` owns fixture creation and verification against an already-running runtime; it never starts services.
|
||||
- `support/web-server.ts` owns frontend reuse, readiness, and shutdown.
|
||||
- `features/support/hooks.ts` owns shared auth bootstrap, scenario lifecycle, and diagnostics.
|
||||
- `features/support/world.ts` owns `DifyWorld`, the per-scenario behavior `BrowserContext`, and its authenticated setup and cleanup client. Browser and API identities remain separate so unauthenticated and logout journeys cannot invalidate fixture ownership.
|
||||
@@ -33,12 +36,14 @@ An uninitialized instance is installed and authenticated lazily; an initialized
|
||||
## Tags And External Runtime
|
||||
|
||||
- Default scenarios use shared authenticated storage state. `@unauthenticated` creates a clean context; `@authenticated` is an intent and selection tag only.
|
||||
- `@prepared` requires the strict post-merge seed profile.
|
||||
- `@prepared` requires the prepared fixtures; the post-merge seed profile includes them.
|
||||
- `@external-model` and `@external-tool` identify scenarios that call real external runtimes. Deterministic commands exclude these tags; external commands are opt-in.
|
||||
- `@microphone` uses the checked-in fake audio fixture and an isolated Chromium context.
|
||||
- `@browser-smoke` runs focused keyboard and navigation coverage in Chromium and WebKit CI lanes.
|
||||
- Feature-owned services use their own tags. Agent v2 runtime scenarios use `@agent-backend-runtime` and require the explicit runtime-availability step. Set `E2E_START_AGENT_BACKEND=1` to start it locally, or provide `E2E_AGENT_BACKEND_URL` / `AGENT_BACKEND_BASE_URL`.
|
||||
|
||||
Seed and Cucumber must share one runtime lifecycle. Combined commands own reset, middleware, services, seed, Cucumber, and teardown; CI must not reproduce that lifecycle in workflow YAML. `E2E_START_AGENT_BACKEND=1` starts a managed local backend before the API; it is mutually exclusive with an explicit Agent backend URL.
|
||||
|
||||
Do not overload runtime tags to imply unrelated services or silently skip behavior when a required fixture is missing.
|
||||
|
||||
## Browser, API, And Contract Boundaries
|
||||
|
||||
@@ -48,16 +48,15 @@ Use `the Agent v2 configuration should be saved automatically` for Configure aut
|
||||
|
||||
## Seed and fixture contract
|
||||
|
||||
Seed scripts create or update environment-owned models, plugins, datasets, Agents, and workflows. `fixtures.steps.ts` resolves and validates those resources before a dependent behavior runs. Missing, inactive, unindexed, or drifted fixtures must throw and fail the scenario; never return `skipped`.
|
||||
Seed tasks create or update environment-owned models, plugins, datasets, Agents, and workflows. `fixtures.steps.ts` resolves and validates those resources before a dependent behavior runs. Missing, inactive, unindexed, or drifted fixtures must throw and fail the scenario; never return `skipped`.
|
||||
|
||||
`@prepared` scenarios are excluded from deterministic PR core. Post-merge runs:
|
||||
|
||||
```bash
|
||||
pnpm -C e2e e2e:post-merge:prepare
|
||||
pnpm -C e2e e2e:post-merge
|
||||
E2E_START_AGENT_BACKEND=1 pnpm -C e2e e2e:post-merge
|
||||
```
|
||||
|
||||
The strict seed must finish without blocked tasks. The concrete resource inventory and defaults belong to the seed profile and environment configuration rather than this guidance.
|
||||
The command owns runtime setup, strict seed, Cucumber, and teardown. The strict seed must finish without blocked tasks. The concrete resource inventory and defaults belong to the seed profile and environment configuration rather than this guidance.
|
||||
|
||||
Organize fixture helpers by the product resource or infrastructure capability they own, not by the feature file that happens to consume them. Keep runtime readiness adapters separate from Console resource fixtures, and keep all fixture state in the current `SeedContext` or scenario `DifyWorld` rather than module globals.
|
||||
|
||||
|
||||
+5
-3
@@ -5,7 +5,7 @@
|
||||
"scripts": {
|
||||
"e2e": "tsx ./scripts/run-cucumber.ts",
|
||||
"e2e:external": "tsx ./scripts/run-external-runtime.ts",
|
||||
"e2e:external:prepare": "tsx ./scripts/prepare-external-runtime.ts",
|
||||
"e2e:external:prepare": "tsx ./scripts/run-cucumber.ts --seed-only --profile external-runtime",
|
||||
"e2e:full": "tsx ./scripts/run-cucumber.ts --full",
|
||||
"e2e:full:headed": "tsx ./scripts/run-cucumber.ts --full --headed",
|
||||
"e2e:headed": "tsx ./scripts/run-cucumber.ts --headed",
|
||||
@@ -13,9 +13,11 @@
|
||||
"e2e:middleware:down": "tsx ./scripts/setup.ts middleware-down",
|
||||
"e2e:middleware:up": "tsx ./scripts/setup.ts middleware-up",
|
||||
"e2e:post-merge": "tsx ./scripts/run-post-merge.ts",
|
||||
"e2e:post-merge:prepare": "tsx ./scripts/seed.ts --pack agent-v2 --profile post-merge",
|
||||
"e2e:post-merge:prepare": "tsx ./scripts/run-cucumber.ts --seed-only --profile post-merge",
|
||||
"e2e:prepared": "tsx ./scripts/run-prepared.ts",
|
||||
"e2e:prepared:prepare": "tsx ./scripts/run-cucumber.ts --seed-only --profile prepared",
|
||||
"e2e:reset": "tsx ./scripts/setup.ts reset",
|
||||
"seed": "tsx ./scripts/seed.ts",
|
||||
"seed": "tsx ./scripts/run-cucumber.ts --seed-only",
|
||||
"test:unit": "vitest run",
|
||||
"type-check": "tsc"
|
||||
},
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
import { e2eDir, isMainModule, runCommandOrThrow } from './common'
|
||||
import './env-register'
|
||||
|
||||
const main = async () => {
|
||||
await runCommandOrThrow({
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/seed.ts', '--pack', 'agent-v2', '--profile', 'external-runtime'],
|
||||
cwd: e2eDir,
|
||||
})
|
||||
}
|
||||
|
||||
if (isMainModule(import.meta.url)) {
|
||||
void main().catch((error) => {
|
||||
console.error(error instanceof Error ? error.message : String(error))
|
||||
process.exit(1)
|
||||
})
|
||||
}
|
||||
+102
-150
@@ -7,56 +7,16 @@ import { startLoggedProcess, stopManagedProcess, waitForUrl } from '../support/p
|
||||
import { startWebServer, stopWebServer } from '../support/web-server'
|
||||
import { apiURL, baseURL, reuseExistingWebServer } from '../test-env'
|
||||
import { e2eDir, isMainModule, runCommand } from './common'
|
||||
import { parseRunOptions, shouldStartManagedAgentBackend } from './run-options'
|
||||
import { runSeed } from './seed-runner'
|
||||
import { resetState, startMiddleware, stopMiddleware } from './setup'
|
||||
import './env-register'
|
||||
|
||||
type RunOptions = {
|
||||
forwardArgs: string[]
|
||||
full: boolean
|
||||
headed: boolean
|
||||
}
|
||||
|
||||
const parseArgs = (argv: string[]): RunOptions => {
|
||||
let full = false
|
||||
let headed = false
|
||||
const forwardArgs: string[] = []
|
||||
|
||||
for (const [index, arg] of argv.entries()) {
|
||||
if (arg === '--') {
|
||||
forwardArgs.push(...argv.slice(index + 1))
|
||||
return { forwardArgs, full, headed }
|
||||
}
|
||||
|
||||
if (arg === '--full') {
|
||||
full = true
|
||||
continue
|
||||
}
|
||||
|
||||
if (arg === '--headed') {
|
||||
headed = true
|
||||
continue
|
||||
}
|
||||
|
||||
forwardArgs.push(arg)
|
||||
}
|
||||
|
||||
return { forwardArgs, full, headed }
|
||||
}
|
||||
|
||||
const hasCustomTags = (forwardArgs: string[]) =>
|
||||
forwardArgs.some((arg) => arg === '--tags' || arg.startsWith('--tags='))
|
||||
|
||||
const fullNonExternalTags = 'not @prepared and not @external-model and not @external-tool'
|
||||
|
||||
const isTruthyEnv = (value: string | undefined) => value === '1' || value === 'true'
|
||||
|
||||
const shouldStartAgentBackend = () => {
|
||||
if (isTruthyEnv(process.env.E2E_START_AGENT_BACKEND)) return true
|
||||
|
||||
if (process.env.E2E_AGENT_BACKEND_URL || process.env.AGENT_BACKEND_BASE_URL) return false
|
||||
|
||||
return false
|
||||
}
|
||||
const seedCeleryQueues = 'dataset,priority_dataset,workflow_based_app_execution'
|
||||
|
||||
const readLogTail = async (logFilePath: string) => {
|
||||
const content = await readFile(logFilePath, 'utf8').catch(() => '')
|
||||
@@ -87,65 +47,41 @@ const waitForUnexpectedProcessExit = async (
|
||||
throw new Error(`${label} exited before becoming ready. See ${logFilePath}.${logTailMessage}`)
|
||||
}
|
||||
|
||||
const waitForManagedProcess = async ({
|
||||
errorMessage,
|
||||
managedProcess,
|
||||
url,
|
||||
}: {
|
||||
errorMessage: string
|
||||
managedProcess: ManagedProcess
|
||||
url: string
|
||||
}) => {
|
||||
let waiting = true
|
||||
try {
|
||||
await Promise.race([
|
||||
waitForUrl(url, 180_000, 1_000),
|
||||
waitForUnexpectedProcessExit(managedProcess, () => !waiting),
|
||||
])
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.message.includes('exited before becoming ready'))
|
||||
throw error
|
||||
|
||||
throw new Error(`${errorMessage} See ${managedProcess.logFilePath}.`)
|
||||
} finally {
|
||||
waiting = false
|
||||
}
|
||||
}
|
||||
|
||||
const main = async () => {
|
||||
const { forwardArgs, full, headed } = parseArgs(process.argv.slice(2))
|
||||
const startMiddlewareForRun = full
|
||||
const resetStateForRun = full
|
||||
const startAgentBackendForRun = shouldStartAgentBackend()
|
||||
|
||||
if (resetStateForRun) await resetState()
|
||||
|
||||
if (startMiddlewareForRun) await startMiddleware()
|
||||
|
||||
const { forwardArgs, full, headed, seed, seedOnly } = parseRunOptions(process.argv.slice(2))
|
||||
const startAgentBackendForRun = shouldStartManagedAgentBackend()
|
||||
const cucumberReportDir = path.join(e2eDir, 'cucumber-report')
|
||||
const logDir = path.join(e2eDir, '.logs')
|
||||
|
||||
await rm(cucumberReportDir, { force: true, recursive: true })
|
||||
await mkdir(logDir, { recursive: true })
|
||||
|
||||
const shellctlProcess = startAgentBackendForRun
|
||||
? await startLoggedProcess({
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/setup.ts', 'shellctl-sandbox'],
|
||||
cwd: e2eDir,
|
||||
label: 'shellctl sandbox',
|
||||
logFilePath: path.join(logDir, 'cucumber-shellctl-sandbox.log'),
|
||||
})
|
||||
: undefined
|
||||
|
||||
const difyAgentProcess = startAgentBackendForRun
|
||||
? await startLoggedProcess({
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/setup.ts', 'agent-backend'],
|
||||
cwd: e2eDir,
|
||||
env: {
|
||||
E2E_START_AGENT_BACKEND: '1',
|
||||
},
|
||||
label: 'agent backend',
|
||||
logFilePath: path.join(logDir, 'cucumber-agent-backend.log'),
|
||||
})
|
||||
: undefined
|
||||
|
||||
const apiProcess = await startLoggedProcess({
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/setup.ts', 'api'],
|
||||
cwd: e2eDir,
|
||||
env: startAgentBackendForRun
|
||||
? {
|
||||
E2E_START_AGENT_BACKEND: '1',
|
||||
}
|
||||
: undefined,
|
||||
label: 'api server',
|
||||
logFilePath: path.join(logDir, 'cucumber-api.log'),
|
||||
})
|
||||
|
||||
const celeryProcess = await startLoggedProcess({
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/setup.ts', 'celery'],
|
||||
cwd: e2eDir,
|
||||
label: 'celery worker',
|
||||
logFilePath: path.join(logDir, 'cucumber-celery.log'),
|
||||
})
|
||||
let apiProcess: ManagedProcess | undefined
|
||||
let celeryProcess: ManagedProcess | undefined
|
||||
let difyAgentProcess: ManagedProcess | undefined
|
||||
let middlewareStarted = false
|
||||
let shellctlProcess: ManagedProcess | undefined
|
||||
|
||||
let cleanupPromise: Promise<void> | undefined
|
||||
const cleanup = async () => {
|
||||
@@ -157,7 +93,7 @@ const main = async () => {
|
||||
{ label: 'Stop API server', run: () => stopManagedProcess(apiProcess) },
|
||||
{ label: 'Stop agent backend', run: () => stopManagedProcess(difyAgentProcess) },
|
||||
{ label: 'Stop shellctl sandbox', run: () => stopManagedProcess(shellctlProcess) },
|
||||
...(startMiddlewareForRun ? [{ label: 'Stop middleware', run: stopMiddleware }] : []),
|
||||
...(middlewareStarted ? [{ label: 'Stop middleware', run: stopMiddleware }] : []),
|
||||
])
|
||||
|
||||
if (cleanupErrors.length > 0)
|
||||
@@ -182,60 +118,73 @@ const main = async () => {
|
||||
process.once('SIGTERM', onTerminate)
|
||||
|
||||
try {
|
||||
if (shellctlProcess) {
|
||||
let waitingForShellctl = true
|
||||
try {
|
||||
if (full) await resetState()
|
||||
|
||||
if (full) {
|
||||
middlewareStarted = true
|
||||
await startMiddleware()
|
||||
}
|
||||
|
||||
if (!seedOnly) await rm(cucumberReportDir, { force: true, recursive: true })
|
||||
await mkdir(logDir, { recursive: true })
|
||||
|
||||
if (startAgentBackendForRun) {
|
||||
shellctlProcess = await startLoggedProcess({
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/setup.ts', 'shellctl-sandbox'],
|
||||
cwd: e2eDir,
|
||||
label: 'shellctl sandbox',
|
||||
logFilePath: path.join(logDir, 'cucumber-shellctl-sandbox.log'),
|
||||
})
|
||||
const shellctlPort = process.env.E2E_SHELLCTL_PORT || '5004'
|
||||
await Promise.race([
|
||||
waitForUrl(`http://127.0.0.1:${shellctlPort}/healthz`, 180_000, 1_000),
|
||||
waitForUnexpectedProcessExit(shellctlProcess, () => !waitingForShellctl),
|
||||
])
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.message.includes('exited before becoming ready'))
|
||||
throw error
|
||||
await waitForManagedProcess({
|
||||
errorMessage: 'Shellctl sandbox did not become ready.',
|
||||
managedProcess: shellctlProcess,
|
||||
url: `http://127.0.0.1:${shellctlPort}/healthz`,
|
||||
})
|
||||
|
||||
throw new Error(
|
||||
`Shellctl sandbox did not become ready. See ${shellctlProcess.logFilePath}.`,
|
||||
)
|
||||
} finally {
|
||||
waitingForShellctl = false
|
||||
}
|
||||
}
|
||||
|
||||
if (difyAgentProcess) {
|
||||
let waitingForAgentBackend = true
|
||||
try {
|
||||
difyAgentProcess = await startLoggedProcess({
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/setup.ts', 'agent-backend'],
|
||||
cwd: e2eDir,
|
||||
env: { E2E_START_AGENT_BACKEND: '1' },
|
||||
label: 'agent backend',
|
||||
logFilePath: path.join(logDir, 'cucumber-agent-backend.log'),
|
||||
})
|
||||
const agentBackendPort = process.env.E2E_AGENT_BACKEND_PORT || '5050'
|
||||
await Promise.race([
|
||||
waitForUrl(`http://127.0.0.1:${agentBackendPort}/openapi.json`, 180_000, 1_000),
|
||||
waitForUnexpectedProcessExit(difyAgentProcess, () => !waitingForAgentBackend),
|
||||
])
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.message.includes('exited before becoming ready'))
|
||||
throw error
|
||||
|
||||
throw new Error(`Agent backend did not become ready. See ${difyAgentProcess.logFilePath}.`)
|
||||
} finally {
|
||||
waitingForAgentBackend = false
|
||||
}
|
||||
await waitForManagedProcess({
|
||||
errorMessage: 'Agent backend did not become ready.',
|
||||
managedProcess: difyAgentProcess,
|
||||
url: `http://127.0.0.1:${agentBackendPort}/openapi.json`,
|
||||
})
|
||||
}
|
||||
|
||||
let waitingForApi = true
|
||||
try {
|
||||
await Promise.race([
|
||||
waitForUrl(`${apiURL}/health`, 180_000, 1_000),
|
||||
waitForUnexpectedProcessExit(apiProcess, () => !waitingForApi),
|
||||
])
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.message.includes('exited before becoming ready'))
|
||||
throw error
|
||||
apiProcess = await startLoggedProcess({
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/setup.ts', 'api'],
|
||||
cwd: e2eDir,
|
||||
env: startAgentBackendForRun ? { E2E_START_AGENT_BACKEND: '1' } : undefined,
|
||||
label: 'api server',
|
||||
logFilePath: path.join(logDir, 'cucumber-api.log'),
|
||||
})
|
||||
await waitForManagedProcess({
|
||||
errorMessage: `API did not become ready at ${apiURL}/health.`,
|
||||
managedProcess: apiProcess,
|
||||
url: `${apiURL}/health`,
|
||||
})
|
||||
|
||||
throw new Error(
|
||||
`API did not become ready at ${apiURL}/health. See ${apiProcess.logFilePath}.`,
|
||||
)
|
||||
} finally {
|
||||
waitingForApi = false
|
||||
}
|
||||
celeryProcess = await startLoggedProcess({
|
||||
command: 'npx',
|
||||
args: [
|
||||
'tsx',
|
||||
'./scripts/setup.ts',
|
||||
'celery',
|
||||
...(seed ? ['--queues', seedCeleryQueues] : []),
|
||||
],
|
||||
cwd: e2eDir,
|
||||
label: 'celery worker',
|
||||
logFilePath: path.join(logDir, 'cucumber-celery.log'),
|
||||
})
|
||||
|
||||
await startWebServer({
|
||||
baseURL,
|
||||
@@ -247,13 +196,15 @@ const main = async () => {
|
||||
timeoutMs: 300_000,
|
||||
})
|
||||
|
||||
if (seed) await runSeed(seed)
|
||||
|
||||
if (!seedOnly) {
|
||||
const cucumberEnv: NodeJS.ProcessEnv = {
|
||||
...process.env,
|
||||
CUCUMBER_HEADLESS: headed ? '0' : '1',
|
||||
}
|
||||
|
||||
if (startMiddlewareForRun && !hasCustomTags(forwardArgs))
|
||||
cucumberEnv.E2E_CUCUMBER_TAGS = fullNonExternalTags
|
||||
if (full && !hasCustomTags(forwardArgs)) cucumberEnv.E2E_CUCUMBER_TAGS = fullNonExternalTags
|
||||
|
||||
const result = await runCommand({
|
||||
command: 'npx',
|
||||
@@ -274,6 +225,7 @@ const main = async () => {
|
||||
}
|
||||
|
||||
process.exitCode = result.exitCode
|
||||
}
|
||||
} finally {
|
||||
process.off('SIGINT', onTerminate)
|
||||
process.off('SIGTERM', onTerminate)
|
||||
|
||||
@@ -6,7 +6,16 @@ const defaultExternalRuntimeTags = '@external-model or @external-tool'
|
||||
const main = async () => {
|
||||
await runForegroundProcess({
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/run-cucumber.ts', '--', '--tags', defaultExternalRuntimeTags],
|
||||
args: [
|
||||
'tsx',
|
||||
'./scripts/run-cucumber.ts',
|
||||
'--full',
|
||||
'--profile',
|
||||
'external-runtime',
|
||||
'--',
|
||||
'--tags',
|
||||
defaultExternalRuntimeTags,
|
||||
],
|
||||
cwd: e2eDir,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
import type { SeedOptions } from './seed-runner'
|
||||
|
||||
export type RunOptions = {
|
||||
forwardArgs: string[]
|
||||
full: boolean
|
||||
headed: boolean
|
||||
seed?: SeedOptions
|
||||
seedOnly: boolean
|
||||
}
|
||||
|
||||
const readOptionValue = (argv: string[], index: number, option: string) => {
|
||||
const value = argv[index + 1]
|
||||
if (!value || value.startsWith('--')) throw new Error(`${option} requires a value.`)
|
||||
|
||||
return value
|
||||
}
|
||||
|
||||
export const parseRunOptions = (argv: string[]): RunOptions => {
|
||||
let allowBlocked = false
|
||||
let dryRun = false
|
||||
let full = false
|
||||
let headed = false
|
||||
let pack = 'agent-v2'
|
||||
let profile: string | undefined
|
||||
let seedOnly = false
|
||||
const forwardArgs: string[] = []
|
||||
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index]!
|
||||
|
||||
if (arg === '--') {
|
||||
forwardArgs.push(...argv.slice(index + 1))
|
||||
break
|
||||
}
|
||||
|
||||
if (arg === '--full') {
|
||||
full = true
|
||||
continue
|
||||
}
|
||||
|
||||
if (arg === '--headed') {
|
||||
headed = true
|
||||
continue
|
||||
}
|
||||
|
||||
if (arg === '--seed-only') {
|
||||
seedOnly = true
|
||||
continue
|
||||
}
|
||||
|
||||
if (arg === '--allow-blocked') {
|
||||
allowBlocked = true
|
||||
continue
|
||||
}
|
||||
|
||||
if (arg === '--dry-run') {
|
||||
dryRun = true
|
||||
continue
|
||||
}
|
||||
|
||||
if (arg === '--pack') {
|
||||
pack = readOptionValue(argv, index, '--pack')
|
||||
index += 1
|
||||
continue
|
||||
}
|
||||
|
||||
if (arg.startsWith('--pack=')) {
|
||||
pack = arg.slice('--pack='.length)
|
||||
if (!pack) throw new Error('--pack requires a value.')
|
||||
continue
|
||||
}
|
||||
|
||||
if (arg === '--profile') {
|
||||
profile = readOptionValue(argv, index, '--profile')
|
||||
index += 1
|
||||
continue
|
||||
}
|
||||
|
||||
if (arg.startsWith('--profile=')) {
|
||||
profile = arg.slice('--profile='.length)
|
||||
if (!profile) throw new Error('--profile requires a value.')
|
||||
continue
|
||||
}
|
||||
|
||||
forwardArgs.push(arg)
|
||||
}
|
||||
|
||||
const shouldSeed = seedOnly || profile !== undefined
|
||||
if (!shouldSeed && (allowBlocked || dryRun || pack !== 'agent-v2'))
|
||||
throw new Error('Seed options require --seed-only or --profile.')
|
||||
if (dryRun && !seedOnly) throw new Error('--dry-run requires --seed-only.')
|
||||
|
||||
return {
|
||||
forwardArgs,
|
||||
full,
|
||||
headed,
|
||||
seed: shouldSeed
|
||||
? {
|
||||
allowBlocked,
|
||||
dryRun,
|
||||
pack,
|
||||
profile: profile ?? 'post-merge',
|
||||
}
|
||||
: undefined,
|
||||
seedOnly,
|
||||
}
|
||||
}
|
||||
|
||||
const isTruthyEnv = (value: string | undefined) => value === '1' || value === 'true'
|
||||
|
||||
export const shouldStartManagedAgentBackend = (env: NodeJS.ProcessEnv = process.env) => {
|
||||
const shouldStart = isTruthyEnv(env.E2E_START_AGENT_BACKEND)
|
||||
const externalUrl = env.E2E_AGENT_BACKEND_URL?.trim() || env.AGENT_BACKEND_BASE_URL?.trim()
|
||||
|
||||
if (shouldStart && externalUrl) {
|
||||
throw new Error(
|
||||
'E2E_START_AGENT_BACKEND cannot be enabled when E2E_AGENT_BACKEND_URL or AGENT_BACKEND_BASE_URL is set.',
|
||||
)
|
||||
}
|
||||
|
||||
return shouldStart
|
||||
}
|
||||
@@ -6,7 +6,16 @@ const postMergeTags = '@prepared or @external-model or @external-tool'
|
||||
const main = async () => {
|
||||
await runForegroundProcess({
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/run-cucumber.ts', '--', '--tags', postMergeTags],
|
||||
args: [
|
||||
'tsx',
|
||||
'./scripts/run-cucumber.ts',
|
||||
'--full',
|
||||
'--profile',
|
||||
'post-merge',
|
||||
'--',
|
||||
'--tags',
|
||||
postMergeTags,
|
||||
],
|
||||
cwd: e2eDir,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
import { e2eDir, isMainModule, runForegroundProcess } from './common'
|
||||
import './env-register'
|
||||
|
||||
const preparedTags = '@prepared'
|
||||
|
||||
const main = async () => {
|
||||
await runForegroundProcess({
|
||||
command: 'npx',
|
||||
args: [
|
||||
'tsx',
|
||||
'./scripts/run-cucumber.ts',
|
||||
'--full',
|
||||
'--profile',
|
||||
'prepared',
|
||||
'--',
|
||||
'--tags',
|
||||
preparedTags,
|
||||
],
|
||||
cwd: e2eDir,
|
||||
})
|
||||
}
|
||||
|
||||
if (isMainModule(import.meta.url)) void main()
|
||||
@@ -0,0 +1,54 @@
|
||||
import { chromium } from '@playwright/test'
|
||||
import { createAgentV2SeedTasks } from '../features/agent-v2/support/seed'
|
||||
import { ensureAuthenticatedState } from '../fixtures/auth'
|
||||
import { createStandaloneConsoleSession } from '../support/api/console-session'
|
||||
import { runSeedTasks, writeSeedReport } from '../support/seed'
|
||||
import { baseURL } from '../test-env'
|
||||
|
||||
export type SeedOptions = {
|
||||
allowBlocked: boolean
|
||||
dryRun: boolean
|
||||
pack: string
|
||||
profile: string
|
||||
}
|
||||
|
||||
const getTasks = (pack: string, profile: string) => {
|
||||
if (pack === 'agent-v2') return createAgentV2SeedTasks(profile)
|
||||
|
||||
throw new Error(`Unknown seed pack "${pack}".`)
|
||||
}
|
||||
|
||||
const ensureAuth = async () => {
|
||||
const browser = await chromium.launch({ headless: true })
|
||||
try {
|
||||
await ensureAuthenticatedState(browser, baseURL)
|
||||
} finally {
|
||||
await browser.close()
|
||||
}
|
||||
}
|
||||
|
||||
export const runSeed = async ({ allowBlocked, dryRun, pack, profile }: SeedOptions) => {
|
||||
console.warn(`[seed] bootstrapping auth state against ${baseURL}`)
|
||||
await ensureAuth()
|
||||
|
||||
const consoleSession = await createStandaloneConsoleSession()
|
||||
try {
|
||||
const results = await runSeedTasks(getTasks(pack, profile), {
|
||||
consoleClient: consoleSession.client,
|
||||
dryRun,
|
||||
resources: new Map(),
|
||||
})
|
||||
const reportName = `${pack}-${profile}`
|
||||
const reportPath = await writeSeedReport(reportName, results)
|
||||
const blockedCount = results.filter((result) => result.status === 'blocked').length
|
||||
|
||||
console.warn(`[seed] report ${reportPath}`)
|
||||
if (blockedCount > 0 && !allowBlocked) {
|
||||
throw new Error(
|
||||
`${blockedCount} seed task${blockedCount === 1 ? '' : 's'} blocked. Re-run with --allow-blocked only when partial readiness is intentional.`,
|
||||
)
|
||||
}
|
||||
} finally {
|
||||
await consoleSession.dispose()
|
||||
}
|
||||
}
|
||||
@@ -1,163 +0,0 @@
|
||||
import type { ManagedProcess } from '../support/process'
|
||||
import { mkdir } from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { chromium } from '@playwright/test'
|
||||
import { createAgentV2SeedTasks } from '../features/agent-v2/support/seed'
|
||||
import { ensureAuthenticatedState } from '../fixtures/auth'
|
||||
import { createStandaloneConsoleSession } from '../support/api/console-session'
|
||||
import { startLoggedProcess, stopManagedProcess, waitForUrl } from '../support/process'
|
||||
import { runSeedTasks, writeSeedReport } from '../support/seed'
|
||||
import { startWebServer, stopWebServer } from '../support/web-server'
|
||||
import { apiURL, baseURL, reuseExistingWebServer } from '../test-env'
|
||||
import { e2eDir, isMainModule } from './common'
|
||||
import './env-register'
|
||||
|
||||
type SeedOptions = {
|
||||
allowBlocked: boolean
|
||||
dryRun: boolean
|
||||
pack: string
|
||||
profile: string
|
||||
}
|
||||
|
||||
const parseArgs = (argv: string[]): SeedOptions => {
|
||||
const options: SeedOptions = {
|
||||
allowBlocked: false,
|
||||
dryRun: false,
|
||||
pack: 'agent-v2',
|
||||
profile: 'post-merge',
|
||||
}
|
||||
|
||||
for (const [index, arg] of argv.entries()) {
|
||||
if (arg === '--pack') {
|
||||
options.pack = argv[index + 1] || options.pack
|
||||
continue
|
||||
}
|
||||
if (arg.startsWith('--pack=')) {
|
||||
options.pack = arg.slice('--pack='.length)
|
||||
continue
|
||||
}
|
||||
if (arg === '--dry-run') options.dryRun = true
|
||||
if (arg === '--allow-blocked') options.allowBlocked = true
|
||||
if (arg === '--profile') {
|
||||
options.profile = argv[index + 1] || options.profile
|
||||
continue
|
||||
}
|
||||
if (arg.startsWith('--profile=')) {
|
||||
options.profile = arg.slice('--profile='.length)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
return options
|
||||
}
|
||||
|
||||
const getTasks = (pack: string, profile: string) => {
|
||||
if (pack === 'agent-v2') return createAgentV2SeedTasks(profile)
|
||||
|
||||
throw new Error(`Unknown seed pack "${pack}".`)
|
||||
}
|
||||
|
||||
const ensureAuth = async () => {
|
||||
const browser = await chromium.launch({ headless: true })
|
||||
try {
|
||||
await ensureAuthenticatedState(browser, baseURL)
|
||||
} finally {
|
||||
await browser.close()
|
||||
}
|
||||
}
|
||||
|
||||
const startApiProcess = async (logDir: string) => {
|
||||
try {
|
||||
await waitForUrl(`${apiURL}/health`, 1_000, 250, 1_000)
|
||||
return undefined
|
||||
} catch {
|
||||
// Start a local API process below.
|
||||
}
|
||||
|
||||
const apiProcess = await startLoggedProcess({
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/setup.ts', 'api'],
|
||||
cwd: e2eDir,
|
||||
label: 'api server',
|
||||
logFilePath: path.join(logDir, 'seed-api.log'),
|
||||
})
|
||||
|
||||
try {
|
||||
await waitForUrl(`${apiURL}/health`, 180_000, 1_000)
|
||||
return apiProcess
|
||||
} catch (error) {
|
||||
await stopManagedProcess(apiProcess)
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
const startCeleryProcess = async (logDir: string) =>
|
||||
startLoggedProcess({
|
||||
command: 'npx',
|
||||
args: [
|
||||
'tsx',
|
||||
'./scripts/setup.ts',
|
||||
'celery',
|
||||
'--queues',
|
||||
'dataset,priority_dataset,workflow_based_app_execution',
|
||||
],
|
||||
cwd: e2eDir,
|
||||
label: 'celery worker',
|
||||
logFilePath: path.join(logDir, 'seed-celery.log'),
|
||||
})
|
||||
|
||||
const main = async () => {
|
||||
const options = parseArgs(process.argv.slice(2))
|
||||
const logDir = path.join(e2eDir, '.logs')
|
||||
let apiProcess: ManagedProcess | undefined
|
||||
let celeryProcess: ManagedProcess | undefined
|
||||
let consoleSession: Awaited<ReturnType<typeof createStandaloneConsoleSession>> | undefined
|
||||
|
||||
await mkdir(logDir, { recursive: true })
|
||||
|
||||
try {
|
||||
apiProcess = await startApiProcess(logDir)
|
||||
celeryProcess = await startCeleryProcess(logDir)
|
||||
await startWebServer({
|
||||
baseURL,
|
||||
command: 'npx',
|
||||
args: ['tsx', './scripts/setup.ts', 'web'],
|
||||
cwd: e2eDir,
|
||||
logFilePath: path.join(logDir, 'seed-web.log'),
|
||||
reuseExistingServer: reuseExistingWebServer,
|
||||
timeoutMs: 300_000,
|
||||
})
|
||||
|
||||
console.warn(`[seed] bootstrapping auth state against ${baseURL}`)
|
||||
await ensureAuth()
|
||||
consoleSession = await createStandaloneConsoleSession()
|
||||
|
||||
const results = await runSeedTasks(getTasks(options.pack, options.profile), {
|
||||
consoleClient: consoleSession.client,
|
||||
dryRun: options.dryRun,
|
||||
resources: new Map(),
|
||||
})
|
||||
const reportName = `${options.pack}-${options.profile}`
|
||||
const reportPath = await writeSeedReport(reportName, results)
|
||||
const blockedCount = results.filter((result) => result.status === 'blocked').length
|
||||
|
||||
console.warn(`[seed] report ${reportPath}`)
|
||||
if (blockedCount > 0 && !options.allowBlocked) {
|
||||
throw new Error(
|
||||
`${blockedCount} seed task${blockedCount === 1 ? '' : 's'} blocked. Re-run with --allow-blocked only when partial readiness is intentional.`,
|
||||
)
|
||||
}
|
||||
} finally {
|
||||
await consoleSession?.dispose()
|
||||
await stopWebServer()
|
||||
await stopManagedProcess(celeryProcess)
|
||||
await stopManagedProcess(apiProcess)
|
||||
}
|
||||
}
|
||||
|
||||
if (isMainModule(import.meta.url)) {
|
||||
void main().catch((error) => {
|
||||
console.error(error instanceof Error ? error.message : String(error))
|
||||
process.exit(1)
|
||||
})
|
||||
}
|
||||
@@ -27,6 +27,7 @@ import {
|
||||
waitForCondition,
|
||||
webDir,
|
||||
} from './common'
|
||||
import './env-register'
|
||||
|
||||
const buildIdPath = path.join(webDir, '.next', 'BUILD_ID')
|
||||
const webBuildStampPath = path.join(webDir, '.next', 'e2e-web-build.sha256')
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import './scripts/env-register'
|
||||
|
||||
export const defaultBaseURL = 'http://127.0.0.1:3000'
|
||||
export const defaultApiURL = 'http://127.0.0.1:5001'
|
||||
export const defaultLocale = 'en-US'
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { parseRunOptions, shouldStartManagedAgentBackend } from '../scripts/run-options'
|
||||
|
||||
describe('E2E run options', () => {
|
||||
it('forwards Cucumber arguments without requesting seed data', () => {
|
||||
expect(parseRunOptions(['--tags', '@smoke'])).toEqual({
|
||||
forwardArgs: ['--tags', '@smoke'],
|
||||
full: false,
|
||||
headed: false,
|
||||
seed: undefined,
|
||||
seedOnly: false,
|
||||
})
|
||||
})
|
||||
|
||||
it('uses the post-merge profile for the default seed command', () => {
|
||||
expect(parseRunOptions(['--seed-only'])).toMatchObject({
|
||||
forwardArgs: [],
|
||||
seed: {
|
||||
allowBlocked: false,
|
||||
dryRun: false,
|
||||
pack: 'agent-v2',
|
||||
profile: 'post-merge',
|
||||
},
|
||||
seedOnly: true,
|
||||
})
|
||||
})
|
||||
|
||||
it('seeds a named profile before forwarding Cucumber arguments', () => {
|
||||
expect(parseRunOptions(['--profile', 'prepared', '--', '--tags', '@prepared'])).toMatchObject({
|
||||
forwardArgs: ['--tags', '@prepared'],
|
||||
seed: { profile: 'prepared' },
|
||||
seedOnly: false,
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects seed-only options when no seed was requested', () => {
|
||||
expect(() => parseRunOptions(['--allow-blocked'])).toThrow(
|
||||
'Seed options require --seed-only or --profile.',
|
||||
)
|
||||
expect(() => parseRunOptions(['--dry-run', '--profile', 'prepared'])).toThrow(
|
||||
'--dry-run requires --seed-only.',
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
describe('managed Agent backend selection', () => {
|
||||
it.each(['1', 'true'])('starts a managed backend for %s', (value) => {
|
||||
expect(shouldStartManagedAgentBackend({ E2E_START_AGENT_BACKEND: value })).toBe(true)
|
||||
})
|
||||
|
||||
it('uses an explicitly configured backend without starting a managed one', () => {
|
||||
expect(
|
||||
shouldStartManagedAgentBackend({ E2E_AGENT_BACKEND_URL: 'http://agent.example.test' }),
|
||||
).toBe(false)
|
||||
})
|
||||
|
||||
it('rejects two Agent backend owners', () => {
|
||||
expect(() =>
|
||||
shouldStartManagedAgentBackend({
|
||||
AGENT_BACKEND_BASE_URL: 'http://agent.example.test',
|
||||
E2E_START_AGENT_BACKEND: '1',
|
||||
}),
|
||||
).toThrow('E2E_START_AGENT_BACKEND cannot be enabled')
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user