Compare commits

...
Author SHA1 Message Date
Charles YaoandClaude Opus 4.7 585d292511 feat(web): add Forward-user-identity toggle to MCP provider modal (M3)
Exposes the M2 backend flags as a switch on the MCP provider create/edit
modal so workspace admins can opt in to enterprise SSO identity-forwarding
per provider. When the toggle flips on, the modal sends
forward_user_identity=true + identity_mode="idp_token" to the console API
(which the M2 backend persists on tool_mcp_providers).

- The toggle lives between Server Identifier and the Authentication tabs;
  it overrides the static Authorization (from Auth/Headers) at invoke time.
- The form-state hook hydrates from the GET response so editing preserves
  the previous choice across sessions.
- en-US + zh-Hans strings added; other locales fall back to en-US.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 19:51:36 -07:00
Charles YaoandClaude Opus 4.7 d3fa24d7e3 feat(api): add MCP user-identity forwarding (M2)
When an MCP provider has forward_user_identity enabled, MCPTool now asks
dify-enterprise to mint a short-lived per-user SSO id_token (via the M1
/inner/api/mcp/issue-token endpoint) and stamps it as the Authorization
Bearer on every outbound MCP request — so an MCP server can act on behalf
of the verified end user instead of seeing only "Dify is calling."

- Adds forward_user_identity (bool) + identity_mode ("off" | "idp_token")
  to tool_mcp_providers, plumbed through MCPProviderEntity, the controller,
  service-layer CRUD, and the tool/provider runtime.
- EnterpriseService.issue_mcp_token wraps the enterprise endpoint and maps
  428 → MCPNoRefreshTokenError, 401 → MCPIdentityRefreshError so workflows
  halt with a clear "please re-authenticate" instead of silently going
  anonymous.
- Identity_mode is intentionally an enum-shaped string column so future
  modes (e.g. RFC 8693 token exchange) land without UI/DB churn.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 19:51:20 -07:00
yyhandGitHub 91ac465982 fix(web): use default profile query cache (#36832) 2026-05-29 14:18:39 +00:00
yyhandGitHub 9490d63c50 refactor(web): remove app initializer and move auth boot logic to route boundaries (#36818) 2026-05-29 12:26:34 +00:00
非法操作andGitHub ae538ced47 chore: using single SSH_SCRIPT for saas dev (#36827) 2026-05-29 10:07:15 +00:00
EvanandGitHub 487249728b fix: remove unnecessary # type: ignore comments (#24494) (#36825) 2026-05-29 09:41:32 +00:00
EvanandGitHub 372a2e3e9c refactor: convert isinstance chains to match/case (part 7) (#35902) (#36826) 2026-05-29 09:40:33 +00:00
Yunlu WenandGitHub 4939a9c33d refactor: add ts common style check for web and cli (#36823) 2026-05-29 09:26:32 +00:00
Yunlu WenandGitHub b6f92f1dc4 fix(cli): fix style (#36821) 2026-05-29 08:34:36 +00:00
非法操作andGitHub ce276573a8 chore: deploy saas dev workflow (#36819) 2026-05-29 08:30:55 +00:00
Yunlu WenandGitHub 5070cc9668 refactor(cli): optimize error handling in flag parsing (#36810) 2026-05-29 07:39:26 +00:00
JoelandGitHub a392a72960 chore: not store search tag condition in url (#36814) 2026-05-29 07:30:35 +00:00
Xiyuan ChenandGitHub 30270b5c30 fix(device): surface SSO errors on /device and fix CLI null-account crash on external-SSO login (#36781) 2026-05-29 06:51:34 +00:00
JoelandGitHub 24715a9570 chore: unified plugin status icon position (#36816) 2026-05-29 06:45:25 +00:00
WOLIKIMCHENGGitHubrootautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
c530a5d272 fix(api): validate annotation list pagination query (#36807)
Co-authored-by: root <kinsonnee@gmail.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-05-29 06:25:48 +00:00
JoelandGitHub 418ee7398e fix: install failed plugin dose not show icon (#36811) 2026-05-29 06:07:43 +00:00
Escape0707andGitHub 78f40c0d25 test: stabilize modal context pricing test (#36524) 2026-05-29 05:19:37 +00:00
zyssyz123GitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2cc567c6a3 feat: add DTO for agent api (#36797)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-05-29 03:36:41 +00:00
Escape0707andGitHub a180ab19e4 chore: type check test container tests (#36790) 2026-05-29 01:54:25 +00:00
Escape0707andGitHub 13eaa436e7 test: isolate Redis state in container tests (#36740) 2026-05-28 12:42:25 +00:00
Yunlu WenandGitHub 3596d12e4c refactor(cli): use Store interface as token storage (#36726) 2026-05-28 10:02:51 +00:00
Xiyuan ChenandGitHub e8de10a3b5 feat(docker): add missing OPENAPI_* env vars to shared.env.example (#36752) 2026-05-28 08:52:03 +00:00
wangxiaoleiandGitHub f5ab5e7eb3 fix: fix cannot extract elements from a scalar (#36769) 2026-05-28 07:31:36 +00:00
BlackouttaGitHubXWautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
0c40e1c2a0 feat: add cross-environment app migration workflow (#36765)
Co-authored-by: XW <wei.xu1@wiz.ai>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-05-28 07:30:33 +00:00
Aladdin AliyevandGitHub c29d76757e docs(api): fix typo in vector migration docstrings (#36741) 2026-05-28 07:15:34 +00:00
CrazywoolaandGitHub 91c1d3ad81 fix: handle null plugin badges (#36767) 2026-05-28 07:00:32 +00:00
Nishchay MahorandGitHub 57b02e341c refactor: add @override decorators to storage backend subclasses (#36406) (#36755) 2026-05-28 06:04:47 +00:00
Xiyuan ChenandGitHub b94ff65e9f fix(docker): copy dify-agent source into production stage (#36757) 2026-05-28 06:01:11 +00:00
678260e34e test: migrate workspace members tests to containers (#36738)
Co-authored-by: jamesrayammons <63717587+jamesrayammons@users.noreply.github.com>
2026-05-28 06:01:05 +00:00
Kuroda KaynandGitHub 739e34d08a fix(docker): pin web docker node version (#36756) 2026-05-28 05:25:41 +00:00
-LAN-andGitHub 825fb9cb89 chore(codeowners): add Riskey for service API docs (#36731) 2026-05-28 05:06:12 +00:00
233 changed files with 10709 additions and 2699 deletions
+1
View File
@@ -166,6 +166,7 @@
# Frontend - App - API Documentation
/web/app/components/develop/ @JzoNgKVO @iamjoel
/web/app/components/develop/template/*.mdx @JzoNgKVO @iamjoel @RiskeyL
# Frontend - App - Logs and Annotations
/web/app/components/app/workflow-log/ @JzoNgKVO @iamjoel
@@ -1,4 +1,4 @@
name: Deploy Agent Dev
name: Deploy SaaS
permissions:
contents: read
@@ -7,7 +7,7 @@ on:
workflow_run:
workflows: ["Build and Push API & Web"]
branches:
- "deploy/agent-dev"
- "deploy/saas"
types:
- completed
@@ -16,13 +16,13 @@ jobs:
runs-on: depot-ubuntu-24.04
if: |
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.head_branch == 'deploy/agent-dev'
github.event.workflow_run.head_branch == 'deploy/saas'
steps:
- name: Deploy to server
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2 # v1.2.5
with:
host: ${{ secrets.AGENT_DEV_SSH_HOST }}
host: ${{ secrets.SAAS_DEV_SSH_HOST }}
username: ${{ secrets.SSH_USER }}
key: ${{ secrets.SSH_PRIVATE_KEY }}
script: |
${{ vars.SSH_SCRIPT || secrets.SSH_SCRIPT }}
${{ vars.SSH_SCRIPT_SAAS_DEV || secrets.SSH_SCRIPT_SAAS_DEV }}
+47 -16
View File
@@ -95,6 +95,51 @@ jobs:
if: steps.changed-files.outputs.any_changed == 'true'
uses: ./.github/actions/setup-web
- name: Web tsslint
if: steps.changed-files.outputs.any_changed == 'true'
env:
NODE_OPTIONS: --max-old-space-size=4096
run: vp run lint:tss
- name: Web dead code check
if: steps.changed-files.outputs.any_changed == 'true'
run: vp run knip
ts-common-style:
name: TS Common
runs-on: depot-ubuntu-24.04
permissions:
checks: write
pull-requests: read
steps:
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Check changed files
id: changed-files
uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6
with:
files: |
web/**
cli/**
e2e/**
sdks/nodejs-client/**
packages/**
package.json
pnpm-lock.yaml
pnpm-workspace.yaml
.nvmrc
eslint.config.mjs
.github/workflows/style.yml
.github/actions/setup-web/**
- name: Setup web environment
if: steps.changed-files.outputs.any_changed == 'true'
uses: ./.github/actions/setup-web
- name: Restore ESLint cache
if: steps.changed-files.outputs.any_changed == 'true'
id: eslint-cache-restore
@@ -105,28 +150,14 @@ jobs:
restore-keys: |
${{ runner.os }}-eslint-${{ hashFiles('pnpm-lock.yaml', 'eslint.config.mjs', 'web/eslint.config.mjs', 'web/eslint.constants.mjs', 'web/plugins/eslint/**') }}-
- name: Web style check
- name: Style check
if: steps.changed-files.outputs.any_changed == 'true'
working-directory: .
run: vp run lint:ci
- name: Web tsslint
- name: Type check
if: steps.changed-files.outputs.any_changed == 'true'
working-directory: ./web
env:
NODE_OPTIONS: --max-old-space-size=4096
run: vp run lint:tss
- name: Web type check
if: steps.changed-files.outputs.any_changed == 'true'
working-directory: .
run: vp run type-check
- name: Web dead code check
if: steps.changed-files.outputs.any_changed == 'true'
working-directory: ./web
run: vp run knip
- name: Save ESLint cache
if: steps.changed-files.outputs.any_changed == 'true' && success() && steps.eslint-cache-restore.outputs.cache-hit != 'true'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+1 -1
View File
@@ -27,7 +27,7 @@ COPY api/providers ./providers
COPY dify-agent/pyproject.toml dify-agent/README.md /app/dify-agent/
COPY dify-agent/src /app/dify-agent/src
# Trust the checked-in lock during image builds; local path sources are copied from the repository context.
RUN uv sync --frozen --no-dev
RUN uv sync --frozen --no-dev --no-editable
# production stage
FROM base AS production
+16 -1
View File
@@ -4,6 +4,12 @@ CLI command modules extracted from `commands.py`.
from .account import create_tenant, reset_email, reset_password
from .data_migrate import data_migrate, legacy_model_types
from .data_migration import (
export_migration_data,
export_migration_data_template,
import_migration_data,
migration_data_wizard,
)
from .plugin import (
extract_plugins,
extract_unique_plugins,
@@ -26,7 +32,12 @@ from .retention import (
restore_workflow_runs,
)
from .storage import clear_orphaned_file_records, file_usage, migrate_oss, remove_orphaned_files_on_storage
from .system import convert_to_agent_apps, fix_app_site_missing, reset_encrypt_key_pair, upgrade_db
from .system import (
convert_to_agent_apps,
fix_app_site_missing,
reset_encrypt_key_pair,
upgrade_db,
)
from .vector import (
add_qdrant_index,
migrate_annotation_vector_database,
@@ -48,10 +59,13 @@ __all__ = [
"data_migrate",
"delete_archived_workflow_runs",
"export_app_messages",
"export_migration_data",
"export_migration_data_template",
"extract_plugins",
"extract_unique_plugins",
"file_usage",
"fix_app_site_missing",
"import_migration_data",
"install_plugins",
"install_rag_pipeline_plugins",
"legacy_model_types",
@@ -59,6 +73,7 @@ __all__ = [
"migrate_data_for_plugin",
"migrate_knowledge_vector_database",
"migrate_oss",
"migration_data_wizard",
"old_metadata_migration",
"remove_orphaned_files_on_storage",
"reset_email",
+754
View File
@@ -0,0 +1,754 @@
from __future__ import annotations
import json
from datetime import datetime
from pathlib import Path
from typing import Any, cast
from uuid import UUID
import click
import sqlalchemy as sa
import yaml
from extensions.ext_database import db
from models import Tenant
from models.model import App
from models.tools import ApiToolProvider, MCPToolProvider, WorkflowToolProvider
from services.app_dsl_service import AppDslService
from services.data_migration.dependency_discovery_service import DependencyDiscoveryService
from services.data_migration.entities import (
DependencyKind,
ImportOptions,
MigrationDataError,
ReportContext,
ResourceReportItem,
)
from services.data_migration.export_service import ExportConfigParser, MigrationExportService
from services.data_migration.import_service import ImportRequest, MigrationImportService
from services.data_migration.package_service import MigrationPackageService
from services.data_migration.report_service import MigrationReportService
ID_STRATEGY_CHOICES = ["preserve-id", "generate-new-id"]
CONFLICT_STRATEGY_CHOICES = ["fail", "skip", "update"]
SUPPORTED_WIZARD_APP_MODES = ["workflow", "advanced-chat"]
WizardToolMap = dict[str, dict[str, str | None]]
WizardToolSelection = dict[str, list[str]]
def _scripted_export_template() -> dict[str, Any]:
return {
"source_tenant": {
"mode": "single",
"id": "",
"name": "admin's Workspace",
},
"apps": {
"modes": ["workflow", "advanced-chat"],
"ids": [],
"all": True,
},
"include_referenced_tools": True,
"additional_tools": {
"api_tools": [],
"workflow_tools": [],
"mcp_tools": [],
},
"include_secrets": False,
"import_options": {
"create_app_api_token_on_import": False,
"id_strategy": "preserve-id",
"conflict_strategy": "fail",
},
}
@click.command("app-migration-template", help="Print or write a scripted export config JSON template.")
@click.option(
"--output",
"output_file",
required=False,
type=click.Path(dir_okay=False),
help="Path to write the export config JSON template. Prints to stdout when omitted.",
)
@click.option("--overwrite", is_flag=True, default=False, help="Overwrite output if it already exists.")
def export_migration_data_template(output_file: str | None, overwrite: bool) -> None:
template_json = json.dumps(_scripted_export_template(), indent=2, ensure_ascii=False) + "\n"
if output_file is None:
click.echo(template_json, nl=False)
return
path = Path(output_file)
if path.exists() and not overwrite:
raise click.ClickException(f"Output file already exists: {output_file}")
path.write_text(template_json)
click.echo(click.style(f"Output written to {output_file}", fg="green"))
@click.command("export-app-migration", help="Export workflow migration data to a versioned JSON package.")
@click.option(
"--input",
"input_file",
required=False,
type=click.Path(exists=True, dir_okay=False),
help="Path to export config JSON.",
)
@click.option(
"--output",
"output_file",
required=False,
type=click.Path(dir_okay=False),
help="Path to migration package JSON.",
)
@click.option("--overwrite", is_flag=True, default=False, help="Overwrite output if it already exists.")
def export_migration_data(input_file: str | None, output_file: str | None, overwrite: bool) -> None:
try:
_require_options(("--input", input_file), ("--output", output_file))
assert input_file is not None
assert output_file is not None
raw_config = _load_json_object(input_file, "Export config")
selection = ExportConfigParser().parse(raw_config)
result = MigrationExportService().export(selection)
MigrationPackageService().save_package(result.package, output_file, overwrite=overwrite)
click.echo(click.style(f"Output written to {output_file}", fg="green"))
_render_report(result.report_items, context=_with_output_path(result.report_context, output_file))
except MigrationDataError as exc:
raise click.ClickException(str(exc)) from exc
@click.command("import-app-migration", help="Import a versioned migration data package.")
@click.option(
"--input",
"input_file",
required=False,
type=click.Path(exists=True, dir_okay=False),
help="Path to migration package JSON.",
)
@click.option("--target-tenant", default=None, help="Target tenant/workspace name. Overrides package metadata.")
@click.option("--operator-email", default=None, help="Operator account email in the target tenant.")
@click.option(
"--id-strategy",
default=None,
type=click.Choice(ID_STRATEGY_CHOICES),
help="Override package ID strategy.",
)
@click.option(
"--conflict-strategy",
default=None,
type=click.Choice(CONFLICT_STRATEGY_CHOICES),
help="Override package conflict strategy.",
)
@click.option(
"--create-app-api-token-on-import/--no-create-app-api-token-on-import",
default=None,
help="Override package app API token creation behavior.",
)
def import_migration_data(
input_file: str | None,
target_tenant: str | None,
operator_email: str | None,
id_strategy: str | None,
conflict_strategy: str | None,
create_app_api_token_on_import: bool | None,
) -> None:
try:
_require_options(("--input", input_file))
assert input_file is not None
package = MigrationPackageService().load_package(input_file)
result = MigrationImportService().import_package(
ImportRequest(
package=package,
cli_target_tenant=target_tenant,
operator_email=operator_email,
options_override=_build_options_override(
package.metadata.import_options,
id_strategy=id_strategy,
conflict_strategy=conflict_strategy,
create_app_api_token_on_import=create_app_api_token_on_import,
),
)
)
_render_report(result.report_items, context=result.report_context)
except MigrationDataError as exc:
raise click.ClickException(str(exc)) from exc
def parse_index_selection(raw: str, values: list[str]) -> list[str]:
normalized = raw.strip().lower()
if normalized == "all":
return values
selected: list[str] = []
for part in raw.split(","):
stripped = part.strip()
if not stripped:
continue
try:
index = int(stripped)
except ValueError as exc:
raise click.ClickException(f"Selection must be 'all' or comma-separated numbers: {raw}") from exc
if index < 1 or index > len(values):
raise click.ClickException(f"Selection index out of range: {index}")
selected.append(values[index - 1])
return list(dict.fromkeys(selected))
def _print_wizard_step(title: str) -> None:
click.echo("")
click.echo(f"==== {title} ====")
def _print_wizard_substep(title: str) -> None:
click.echo("")
click.echo(f"-- {title} --")
@click.command("app-migration-wizard", help="Interactively export workflow migration data.")
def migration_data_wizard() -> None:
try:
tenant = _prompt_source_tenant()
apps = _eligible_apps_for_tenant(tenant.id)
app_ids = _prompt_app_ids(apps)
_print_wizard_step("Referenced Tools")
include_referenced_tools = click.confirm(
"Automatically export tools referenced by selected apps? [y/n, default: y]",
default=True,
show_default=False,
)
auto_tools = _discover_auto_tools([app for app in apps if app.id in set(app_ids)], include_referenced_tools)
auto_tools = _resolve_auto_tool_names(tenant.id, auto_tools)
_print_auto_tools(auto_tools)
additional_tools = _prompt_additional_tools(tenant.id, auto_tools)
include_secrets, create_tokens, id_strategy, conflict_strategy = _prompt_import_options()
_print_wizard_step("Output")
output_file, overwrite = _prompt_output_file()
selection = ExportConfigParser().parse(
{
"source_tenant": {"mode": "single", "id": tenant.id, "name": tenant.name},
"apps": {"ids": app_ids, "all": False},
"include_referenced_tools": include_referenced_tools,
"additional_tools": additional_tools,
"include_secrets": include_secrets,
"import_options": {
"create_app_api_token_on_import": create_tokens,
"id_strategy": id_strategy,
"conflict_strategy": conflict_strategy,
},
}
)
_confirm_wizard_summary(
tenant_name=tenant.name,
app_names=[app.name for app in apps if app.id in set(app_ids)],
auto_tools=auto_tools,
additional_tools=additional_tools,
manual_labels=_selected_tool_labels_for_tenant(tenant.id, additional_tools),
include_referenced_tools=include_referenced_tools,
include_secrets=include_secrets,
create_tokens=create_tokens,
id_strategy=id_strategy,
conflict_strategy=conflict_strategy,
output_file=output_file,
)
result = MigrationExportService().export(selection)
MigrationPackageService().save_package(result.package, output_file, overwrite=overwrite)
click.echo(click.style(f"Output written to {output_file}", fg="green"))
_print_wizard_step("Report")
_render_report(result.report_items, context=_with_output_path(result.report_context, output_file))
except MigrationDataError as exc:
raise click.ClickException(str(exc)) from exc
def _load_json_object(path: str, label: str) -> dict[str, Any]:
try:
with Path(path).open(encoding="utf-8") as file:
raw = json.load(file)
except json.JSONDecodeError as exc:
raise MigrationDataError(f"{label} JSON is invalid: {exc.msg}") from exc
if not isinstance(raw, dict):
raise MigrationDataError(f"{label} JSON must be an object.")
return raw
def _require_options(*options: tuple[str, object | None]) -> None:
missing_options = [name for name, value in options if value is None]
if missing_options:
raise click.UsageError(f"Missing option(s): {', '.join(missing_options)}.")
def _build_options_override(
package_options: ImportOptions,
*,
id_strategy: str | None,
conflict_strategy: str | None,
create_app_api_token_on_import: bool | None,
) -> ImportOptions | None:
if id_strategy is None and conflict_strategy is None and create_app_api_token_on_import is None:
return None
return ImportOptions.from_mapping(
{
"id_strategy": id_strategy or package_options.id_strategy,
"conflict_strategy": conflict_strategy or package_options.conflict_strategy,
"create_app_api_token_on_import": (
create_app_api_token_on_import
if create_app_api_token_on_import is not None
else package_options.create_app_api_token_on_import
),
}
)
def _prompt_source_tenant() -> Tenant:
tenants = list(db.session.scalars(sa.select(Tenant).order_by(Tenant.name.asc())).all())
if not tenants:
raise MigrationDataError("No tenants found.")
_print_wizard_step("Source Tenant")
click.echo("Source tenants:")
for index, tenant in enumerate(tenants, 1):
click.echo(f"{index}. {tenant.name} ({tenant.id})")
tenant_index = click.prompt("Select one source tenant by number", type=int, default=1, show_default=True)
if tenant_index < 1 or tenant_index > len(tenants):
raise click.ClickException(f"Selection index out of range: {tenant_index}")
return tenants[tenant_index - 1]
def _eligible_apps_for_tenant(tenant_id: str) -> list[App]:
return list(
db.session.scalars(
sa.select(App)
.where(App.tenant_id == tenant_id, App.mode.in_(SUPPORTED_WIZARD_APP_MODES))
.order_by(App.name.asc())
).all()
)
def _prompt_app_ids(apps: list[App]) -> list[str]:
if not apps:
raise MigrationDataError("No workflow or advanced-chat apps found for the selected tenant.")
_print_wizard_step("App Selection")
click.echo("Currently supported app types: workflow and chatflow.")
click.echo("Workflow/chatflow apps:")
for index, app in enumerate(apps, 1):
mode = app.mode.value if hasattr(app.mode, "value") else app.mode
click.echo(f"{index}. {app.name} [{mode}] ({app.id})")
app_ids = parse_index_selection(
click.prompt("Select apps by number, comma-separated numbers, or all", default="all"),
[app.id for app in apps],
)
selected_apps = [app for app in apps if app.id in set(app_ids)]
click.echo("Selected apps:")
for app in selected_apps:
click.echo(f"- {app.name} ({app.id})")
return app_ids
def _prompt_import_options() -> tuple[bool, bool, str, str]:
_print_wizard_step("Import Options")
_print_wizard_substep("Secrets")
click.echo("Secrets include workflow/app DSL secret values, custom API tool credentials,")
click.echo("and full MCP provider connection data such as server URL, headers, authentication, and tool list.")
click.echo("If you choose no, credentials are omitted or masked,")
click.echo("and MCP providers are exported as dependency metadata only.")
click.echo("Treat the output JSON as sensitive if you choose yes.")
include_secrets = click.confirm(
"Include secrets in output JSON? [y/n, default: n]",
default=False,
show_default=False,
)
_print_wizard_substep("App API Tokens")
click.echo("When enabled, import will create an app API token if the imported app has none,")
click.echo("or reuse an existing app API token if one already exists.")
create_tokens = click.confirm(
"Create or reuse app API tokens during import? [y/n, default: n]",
default=False,
show_default=False,
)
_print_wizard_substep("ID Strategy")
click.echo("ID strategy controls whether imported app and tool IDs preserve source IDs")
click.echo("or use target-generated IDs.")
click.echo("preserve-id: keep source IDs where the target service supports it.")
click.echo("generate-new-id: let the target environment generate new IDs and rewrite references via mapping.")
id_strategy = click.prompt(
"Import ID strategy. Enter one of: preserve-id, generate-new-id",
type=click.Choice(ID_STRATEGY_CHOICES),
default="preserve-id",
show_default=True,
)
_print_wizard_substep("Conflict Strategy")
click.echo("Conflict strategy controls what import does when a target resource already exists.")
click.echo("fail: stop at the first conflict; previously committed resources are not rolled back.")
click.echo("skip: keep the existing target resource and skip importing that resource.")
click.echo("update: update the existing target resource in place.")
conflict_strategy = click.prompt(
"Import conflict strategy. Enter one of: fail, skip, update",
type=click.Choice(CONFLICT_STRATEGY_CHOICES),
default="update",
show_default=True,
)
return include_secrets, create_tokens, id_strategy, conflict_strategy
def _discover_auto_tools(apps: list[App], include_referenced_tools: bool) -> WizardToolMap:
auto_tools: WizardToolMap = {"api_tools": {}, "workflow_tools": {}, "mcp_tools": {}}
if not include_referenced_tools:
return auto_tools
discovery_service = DependencyDiscoveryService()
for app in apps:
dsl_content = AppDslService.export_dsl(app_model=app, include_secret=False)
raw_dsl = yaml.safe_load(dsl_content) if dsl_content else {}
dsl = raw_dsl if isinstance(raw_dsl, dict) else {}
for dependency in discovery_service.discover_from_dsl(dsl):
if dependency.kind == DependencyKind.API_TOOL:
auto_tools["api_tools"][dependency.provider_name or dependency.provider_id] = dependency.provider_id
elif dependency.kind == DependencyKind.WORKFLOW_TOOL:
auto_tools["workflow_tools"][dependency.provider_name or dependency.provider_id] = (
dependency.provider_id
)
elif dependency.kind == DependencyKind.MCP_TOOL:
auto_tools["mcp_tools"][dependency.provider_name or dependency.provider_id] = dependency.provider_id
return auto_tools
def _resolve_auto_tool_names(tenant_id: str, auto_tools: WizardToolMap) -> WizardToolMap:
return {
"api_tools": _resolve_api_tool_names(tenant_id, auto_tools["api_tools"]),
"workflow_tools": _resolve_workflow_tool_names(tenant_id, auto_tools["workflow_tools"]),
"mcp_tools": _resolve_mcp_tool_names(tenant_id, auto_tools["mcp_tools"]),
}
def _resolve_api_tool_names(tenant_id: str, tools: dict[str, str | None]) -> dict[str, str | None]:
resolved: dict[str, str | None] = {}
for name, identifier in tools.items():
predicates = [ApiToolProvider.name == name]
if _is_uuid_string(identifier):
predicates.append(ApiToolProvider.id == identifier)
provider = db.session.scalar(
sa.select(ApiToolProvider).where(
ApiToolProvider.tenant_id == tenant_id,
sa.or_(*predicates),
)
)
resolved[provider.name if provider else name] = provider.id if provider else identifier
return resolved
def _resolve_workflow_tool_names(tenant_id: str, tools: dict[str, str | None]) -> dict[str, str | None]:
resolved: dict[str, str | None] = {}
for name, identifier in tools.items():
predicates = [WorkflowToolProvider.name == name]
if _is_uuid_string(identifier):
predicates.append(WorkflowToolProvider.id == identifier)
provider = db.session.scalar(
sa.select(WorkflowToolProvider).where(
WorkflowToolProvider.tenant_id == tenant_id,
sa.or_(*predicates),
)
)
resolved[provider.name if provider else name] = provider.id if provider else identifier
return resolved
def _resolve_mcp_tool_names(tenant_id: str, tools: dict[str, str | None]) -> dict[str, str | None]:
resolved: dict[str, str | None] = {}
for name, identifier in tools.items():
predicates = [MCPToolProvider.name == name]
if identifier:
predicates.append(MCPToolProvider.server_identifier == identifier)
if _is_uuid_string(identifier):
predicates.append(MCPToolProvider.id == identifier)
provider = db.session.scalar(
sa.select(MCPToolProvider).where(
MCPToolProvider.tenant_id == tenant_id,
sa.or_(*predicates),
)
)
resolved[provider.name if provider else name] = provider.id if provider else identifier
return resolved
def _is_uuid_string(value: str | None) -> bool:
if not value:
return False
try:
UUID(value)
except ValueError:
return False
return True
def _print_auto_tools(auto_tools: WizardToolMap) -> None:
_print_wizard_step("Automatically Discovered Tools")
click.echo("Automatically discovered tools:")
_print_auto_tool_category("Custom API tools", auto_tools["api_tools"])
_print_auto_tool_category("Workflow tools", auto_tools["workflow_tools"])
_print_auto_tool_category("MCP tools", auto_tools["mcp_tools"])
def _print_auto_tool_category(label: str, values: dict[str, str | None]) -> None:
click.echo(label)
if not values:
click.echo("- none")
return
for name, identifier in sorted(values.items()):
click.echo(f"- {_format_tool_name_id(name, identifier)}")
def _prompt_additional_tools(tenant_id: str, auto_tools: WizardToolMap) -> WizardToolSelection:
selections: WizardToolSelection = {"api_tools": [], "workflow_tools": [], "mcp_tools": []}
_print_wizard_step("Additional Tools")
if not click.confirm(
"Export additional tools manually? [y/n, default: n]",
default=False,
show_default=False,
):
_print_final_tool_selection(auto_tools, selections, {})
return selections
manual_labels: dict[str, str] = {}
api_tool_options = [
(tool.name, tool.name, tool.id)
for tool in db.session.scalars(
sa.select(ApiToolProvider).where(ApiToolProvider.tenant_id == tenant_id).order_by(ApiToolProvider.name)
).all()
]
selections["api_tools"] = _prompt_tool_category(
"Custom API tools",
api_tool_options,
auto_tools=auto_tools["api_tools"],
)
manual_labels.update(_selected_tool_labels(api_tool_options, selections["api_tools"]))
workflow_tool_options = [
(tool.id, tool.name, tool.id)
for tool in db.session.scalars(
sa.select(WorkflowToolProvider)
.where(WorkflowToolProvider.tenant_id == tenant_id)
.order_by(WorkflowToolProvider.name)
).all()
]
selections["workflow_tools"] = _prompt_tool_category(
"Workflow tools",
workflow_tool_options,
auto_tools=auto_tools["workflow_tools"],
)
manual_labels.update(_selected_tool_labels(workflow_tool_options, selections["workflow_tools"]))
mcp_tool_options = [
(tool.id, tool.name, tool.server_identifier)
for tool in db.session.scalars(
sa.select(MCPToolProvider).where(MCPToolProvider.tenant_id == tenant_id).order_by(MCPToolProvider.name)
).all()
]
selections["mcp_tools"] = _prompt_tool_category(
"MCP tools",
mcp_tool_options,
auto_tools=auto_tools["mcp_tools"],
)
manual_labels.update(_selected_tool_labels(mcp_tool_options, selections["mcp_tools"]))
_print_final_tool_selection(auto_tools, selections, manual_labels)
return selections
def _selected_tool_labels_for_tenant(tenant_id: str, selected_tools: WizardToolSelection) -> dict[str, str]:
labels: dict[str, str] = {}
if selected_tools["api_tools"]:
labels.update(
_selected_tool_labels(
[
(tool.name, tool.name, tool.id)
for tool in db.session.scalars(
sa.select(ApiToolProvider)
.where(ApiToolProvider.tenant_id == tenant_id)
.order_by(ApiToolProvider.name)
).all()
],
selected_tools["api_tools"],
)
)
if selected_tools["workflow_tools"]:
labels.update(
_selected_tool_labels(
[
(tool.id, tool.name, tool.id)
for tool in db.session.scalars(
sa.select(WorkflowToolProvider)
.where(WorkflowToolProvider.tenant_id == tenant_id)
.order_by(WorkflowToolProvider.name)
).all()
],
selected_tools["workflow_tools"],
)
)
if selected_tools["mcp_tools"]:
labels.update(
_selected_tool_labels(
[
(tool.id, tool.name, tool.server_identifier)
for tool in db.session.scalars(
sa.select(MCPToolProvider)
.where(MCPToolProvider.tenant_id == tenant_id)
.order_by(MCPToolProvider.name)
).all()
],
selected_tools["mcp_tools"],
)
)
return labels
def _selected_tool_labels(options: list[tuple[str, str, str]], selected_values: list[str]) -> dict[str, str]:
selected = set(selected_values)
return {value: _format_tool_name_id(name, detail) for value, name, detail in options if value in selected}
def _prompt_tool_category(
label: str,
options: list[tuple[str, str, str]],
*,
auto_tools: dict[str, str | None],
) -> list[str]:
if not options:
click.echo(f"{label}: none")
return []
_print_wizard_step(label)
for index, (value, name, detail) in enumerate(options, 1):
marker = "[auto]" if _is_auto_tool(value, name, detail, auto_tools) else "[ ]"
click.echo(f"{index}. {marker} {name} ({detail})")
raw = click.prompt(
f"Select {label.lower()} by number, comma-separated numbers, all, or empty",
default="",
show_default=cast(Any, "empty"),
)
if not raw.strip():
return []
return parse_index_selection(raw, [value for value, _, _ in options])
def _is_auto_tool(value: str, name: str, detail: str, auto_tools: dict[str, str | None]) -> bool:
return name in auto_tools or value in auto_tools or value in auto_tools.values() or detail in auto_tools.values()
def _print_final_tool_selection(
auto_tools: WizardToolMap,
additional_tools: WizardToolSelection,
manual_labels: dict[str, str],
) -> None:
_print_wizard_step("Final Tool Selection")
_print_tool_selection_body(auto_tools, additional_tools, manual_labels)
def _print_tool_selection_body(
auto_tools: WizardToolMap,
additional_tools: WizardToolSelection,
manual_labels: dict[str, str],
) -> None:
click.echo("Final tools to export:")
_print_final_tool_category(
"Custom API tools",
auto_tools["api_tools"],
additional_tools["api_tools"],
manual_labels,
)
_print_final_tool_category(
"Workflow tools",
auto_tools["workflow_tools"],
additional_tools["workflow_tools"],
manual_labels,
)
_print_final_tool_category("MCP tools", auto_tools["mcp_tools"], additional_tools["mcp_tools"], manual_labels)
def _print_final_tool_category(
label: str,
auto_tools: dict[str, str | None],
manual_values: list[str],
manual_labels: dict[str, str],
) -> None:
click.echo(label)
lines = [f"- [auto] {_format_tool_name_id(name, identifier)}" for name, identifier in sorted(auto_tools.items())]
auto_identifiers = {identifier for identifier in auto_tools.values() if identifier}
lines.extend(
f"- [manual] {manual_labels.get(value, value)}"
for value in manual_values
if value not in auto_tools and value not in auto_identifiers
)
if not lines:
click.echo("- none")
return
for line in lines:
click.echo(line)
def _format_tool_name_id(name: str, identifier: str | None) -> str:
if identifier and identifier != name:
return f"{name}: {identifier}"
return name
def _confirm_wizard_summary(
*,
tenant_name: str,
app_names: list[str],
auto_tools: WizardToolMap,
additional_tools: WizardToolSelection,
manual_labels: dict[str, str],
include_referenced_tools: bool,
include_secrets: bool,
create_tokens: bool,
id_strategy: str,
conflict_strategy: str,
output_file: str,
) -> None:
_print_wizard_step("Summary")
click.echo("Migration export summary:")
click.echo(f"source tenant: {tenant_name}")
click.echo(f"selected apps: {len(app_names)}")
for app_name in app_names:
click.echo(f"- {app_name}")
click.echo(f"auto referenced tools: {str(include_referenced_tools).lower()}")
_print_tool_selection_body(auto_tools, additional_tools, manual_labels)
click.echo(f"include secrets: {str(include_secrets).lower()}")
click.echo(f"create app api token on import: {str(create_tokens).lower()}")
click.echo(f"id strategy: {id_strategy}")
click.echo(f"conflict strategy: {conflict_strategy}")
click.echo(f"output path: {output_file}")
if not click.confirm("Write migration package? [y/n, default: y]", default=True, show_default=False):
raise click.Abort()
def _prompt_output_file() -> tuple[str, bool]:
default_output = f"migration-data-{datetime.now().strftime('%Y%m%d-%H%M%S')}.json"
output_file = click.prompt("Output path", default=default_output, show_default=True)
if output_file.lower() in {"y", "yes", "n", "no"}:
raise click.ClickException("Output path must be a file path. Press Enter to use the default path.")
overwrite = False
if Path(output_file).exists():
overwrite = click.confirm(
"Output file exists. Overwrite? [y/n, default: n]",
default=False,
show_default=False,
)
if not overwrite:
raise click.ClickException(f"Output file already exists: {output_file}")
return output_file, overwrite
def _with_output_path(context: ReportContext | None, output_path: str) -> ReportContext:
if context is None:
return ReportContext(output_path=output_path)
return ReportContext(
output_path=output_path,
source_scope=context.source_scope,
selected_app_count=context.selected_app_count,
include_secrets=context.include_secrets,
target_tenant=context.target_tenant,
operator_email=context.operator_email,
app_api_tokens_created=context.app_api_tokens_created,
app_api_tokens_reused=context.app_api_tokens_reused,
id_mapping_count=context.id_mapping_count,
id_mappings=context.id_mappings,
)
def _render_report(report_items: list[ResourceReportItem], *, context: ReportContext | None = None) -> None:
for line in MigrationReportService().render(report_items, context=context):
click.echo(line)
+2 -2
View File
@@ -30,7 +30,7 @@ def vdb_migrate(scope: str):
def migrate_annotation_vector_database():
"""
Migrate annotation datas to target vector database .
Migrate annotation data to target vector database.
"""
click.echo(click.style("Starting annotation data migration.", fg="green"))
create_count = 0
@@ -140,7 +140,7 @@ def migrate_annotation_vector_database():
def migrate_knowledge_vector_database():
"""
Migrate vector database datas to target vector database .
Migrate vector database data to target vector database.
"""
click.echo(click.style("Starting vector database migration.", fg="green"))
create_count = 0
+96 -29
View File
@@ -1,9 +1,17 @@
from flask_restx import Resource
from controllers.common.schema import register_schema_models
from controllers.common.schema import register_response_schema_models, register_schema_models
from controllers.console import console_ns
from controllers.console.app.wraps import get_app_model
from controllers.console.wraps import account_initialization_required, edit_permission_required, setup_required
from fields.agent_fields import (
AgentAppComposerResponse,
AgentComposerCandidatesResponse,
AgentComposerImpactResponse,
AgentComposerValidateResponse,
WorkflowAgentComposerResponse,
)
from libs.helper import dump_response
from libs.login import current_account_with_tenant, login_required
from models.model import App, AppMode
from services.agent.composer_service import AgentComposerService
@@ -11,23 +19,40 @@ from services.agent.composer_validator import ComposerConfigValidator
from services.entities.agent_entities import ComposerSavePayload
register_schema_models(console_ns, ComposerSavePayload)
register_response_schema_models(
console_ns,
AgentAppComposerResponse,
AgentComposerCandidatesResponse,
AgentComposerImpactResponse,
AgentComposerValidateResponse,
WorkflowAgentComposerResponse,
)
@console_ns.route("/apps/<uuid:app_id>/workflows/draft/nodes/<string:node_id>/agent-composer")
class WorkflowAgentComposerApi(Resource):
@console_ns.response(
200, "Workflow agent composer state", console_ns.models[WorkflowAgentComposerResponse.__name__]
)
@setup_required
@login_required
@account_initialization_required
@get_app_model(mode=[AppMode.WORKFLOW, AppMode.ADVANCED_CHAT])
def get(self, app_model: App, node_id: str):
_, tenant_id = current_account_with_tenant()
return AgentComposerService.load_workflow_composer(
tenant_id=tenant_id,
app_id=app_model.id,
node_id=node_id,
return dump_response(
WorkflowAgentComposerResponse,
AgentComposerService.load_workflow_composer(
tenant_id=tenant_id,
app_id=app_model.id,
node_id=node_id,
),
)
@console_ns.expect(console_ns.models[ComposerSavePayload.__name__])
@console_ns.response(
200, "Workflow agent composer saved", console_ns.models[WorkflowAgentComposerResponse.__name__]
)
@setup_required
@login_required
@account_initialization_required
@@ -36,18 +61,24 @@ class WorkflowAgentComposerApi(Resource):
def put(self, app_model: App, node_id: str):
account, tenant_id = current_account_with_tenant()
payload = ComposerSavePayload.model_validate(console_ns.payload or {})
return AgentComposerService.save_workflow_composer(
tenant_id=tenant_id,
app_id=app_model.id,
node_id=node_id,
account_id=account.id,
payload=payload,
return dump_response(
WorkflowAgentComposerResponse,
AgentComposerService.save_workflow_composer(
tenant_id=tenant_id,
app_id=app_model.id,
node_id=node_id,
account_id=account.id,
payload=payload,
),
)
@console_ns.route("/apps/<uuid:app_id>/workflows/draft/nodes/<string:node_id>/agent-composer/validate")
class WorkflowAgentComposerValidateApi(Resource):
@console_ns.expect(console_ns.models[ComposerSavePayload.__name__])
@console_ns.response(
200, "Workflow agent composer validation result", console_ns.models[AgentComposerValidateResponse.__name__]
)
@setup_required
@login_required
@account_initialization_required
@@ -55,21 +86,29 @@ class WorkflowAgentComposerValidateApi(Resource):
def post(self, app_model: App, node_id: str):
payload = ComposerSavePayload.model_validate(console_ns.payload or {})
ComposerConfigValidator.validate_save_payload(payload)
return {"result": "success", "errors": []}
return dump_response(AgentComposerValidateResponse, {"result": "success", "errors": []})
@console_ns.route("/apps/<uuid:app_id>/workflows/draft/nodes/<string:node_id>/agent-composer/candidates")
class WorkflowAgentComposerCandidatesApi(Resource):
@console_ns.response(
200, "Workflow agent composer candidates", console_ns.models[AgentComposerCandidatesResponse.__name__]
)
@setup_required
@login_required
@account_initialization_required
@get_app_model(mode=[AppMode.WORKFLOW, AppMode.ADVANCED_CHAT])
def get(self, app_model: App, node_id: str):
return AgentComposerService.get_workflow_candidates(app_id=app_model.id)
return dump_response(
AgentComposerCandidatesResponse,
AgentComposerService.get_workflow_candidates(app_id=app_model.id),
)
@console_ns.route("/apps/<uuid:app_id>/workflows/draft/nodes/<string:node_id>/agent-composer/impact")
class WorkflowAgentComposerImpactApi(Resource):
@console_ns.expect(console_ns.models[ComposerSavePayload.__name__])
@console_ns.response(200, "Workflow agent composer impact", console_ns.models[AgentComposerImpactResponse.__name__])
@setup_required
@login_required
@account_initialization_required
@@ -79,13 +118,21 @@ class WorkflowAgentComposerImpactApi(Resource):
payload = ComposerSavePayload.model_validate(console_ns.payload or {})
current_snapshot_id = payload.binding.current_snapshot_id if payload.binding else None
if not current_snapshot_id:
return {"current_snapshot_id": None, "workflow_node_count": 0, "bindings": []}
return AgentComposerService.calculate_impact(tenant_id=tenant_id, current_snapshot_id=current_snapshot_id)
return dump_response(
AgentComposerImpactResponse, {"current_snapshot_id": None, "workflow_node_count": 0, "bindings": []}
)
return dump_response(
AgentComposerImpactResponse,
AgentComposerService.calculate_impact(tenant_id=tenant_id, current_snapshot_id=current_snapshot_id),
)
@console_ns.route("/apps/<uuid:app_id>/workflows/draft/nodes/<string:node_id>/agent-composer/save-to-roster")
class WorkflowAgentComposerSaveToRosterApi(Resource):
@console_ns.expect(console_ns.models[ComposerSavePayload.__name__])
@console_ns.response(
200, "Workflow agent composer saved to roster", console_ns.models[WorkflowAgentComposerResponse.__name__]
)
@setup_required
@login_required
@account_initialization_required
@@ -94,26 +141,34 @@ class WorkflowAgentComposerSaveToRosterApi(Resource):
def post(self, app_model: App, node_id: str):
account, tenant_id = current_account_with_tenant()
payload = ComposerSavePayload.model_validate(console_ns.payload or {})
return AgentComposerService.save_workflow_composer(
tenant_id=tenant_id,
app_id=app_model.id,
node_id=node_id,
account_id=account.id,
payload=payload,
return dump_response(
WorkflowAgentComposerResponse,
AgentComposerService.save_workflow_composer(
tenant_id=tenant_id,
app_id=app_model.id,
node_id=node_id,
account_id=account.id,
payload=payload,
),
)
@console_ns.route("/apps/<uuid:app_id>/agent-composer")
class AgentAppComposerApi(Resource):
@console_ns.response(200, "Agent app composer state", console_ns.models[AgentAppComposerResponse.__name__])
@setup_required
@login_required
@account_initialization_required
@get_app_model()
def get(self, app_model: App):
_, tenant_id = current_account_with_tenant()
return AgentComposerService.load_agent_app_composer(tenant_id=tenant_id, app_id=app_model.id)
return dump_response(
AgentAppComposerResponse,
AgentComposerService.load_agent_app_composer(tenant_id=tenant_id, app_id=app_model.id),
)
@console_ns.expect(console_ns.models[ComposerSavePayload.__name__])
@console_ns.response(200, "Agent app composer saved", console_ns.models[AgentAppComposerResponse.__name__])
@setup_required
@login_required
@account_initialization_required
@@ -122,17 +177,23 @@ class AgentAppComposerApi(Resource):
def put(self, app_model: App):
account, tenant_id = current_account_with_tenant()
payload = ComposerSavePayload.model_validate(console_ns.payload or {})
return AgentComposerService.save_agent_app_composer(
tenant_id=tenant_id,
app_id=app_model.id,
account_id=account.id,
payload=payload,
return dump_response(
AgentAppComposerResponse,
AgentComposerService.save_agent_app_composer(
tenant_id=tenant_id,
app_id=app_model.id,
account_id=account.id,
payload=payload,
),
)
@console_ns.route("/apps/<uuid:app_id>/agent-composer/validate")
class AgentAppComposerValidateApi(Resource):
@console_ns.expect(console_ns.models[ComposerSavePayload.__name__])
@console_ns.response(
200, "Agent app composer validation result", console_ns.models[AgentComposerValidateResponse.__name__]
)
@setup_required
@login_required
@account_initialization_required
@@ -140,14 +201,20 @@ class AgentAppComposerValidateApi(Resource):
def post(self, app_model: App):
payload = ComposerSavePayload.model_validate(console_ns.payload or {})
ComposerConfigValidator.validate_save_payload(payload)
return {"result": "success", "errors": []}
return dump_response(AgentComposerValidateResponse, {"result": "success", "errors": []})
@console_ns.route("/apps/<uuid:app_id>/agent-composer/candidates")
class AgentAppComposerCandidatesApi(Resource):
@console_ns.response(
200, "Agent app composer candidates", console_ns.models[AgentComposerCandidatesResponse.__name__]
)
@setup_required
@login_required
@account_initialization_required
@get_app_model()
def get(self, app_model: App):
return AgentComposerService.get_agent_app_candidates(app_id=app_model.id)
return dump_response(
AgentComposerCandidatesResponse,
AgentComposerService.get_agent_app_candidates(app_id=app_model.id),
)
+65 -18
View File
@@ -4,10 +4,18 @@ from flask import request
from flask_restx import Resource
from pydantic import BaseModel, Field
from controllers.common.schema import register_schema_models
from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models
from controllers.console import console_ns
from controllers.console.wraps import account_initialization_required, edit_permission_required, setup_required
from extensions.ext_database import db
from fields.agent_fields import (
AgentConfigSnapshotDetailResponse,
AgentConfigSnapshotListResponse,
AgentInviteOptionsResponse,
AgentRosterListResponse,
AgentRosterResponse,
)
from libs.helper import dump_response
from libs.login import current_account_with_tenant, login_required
from services.agent.roster_service import AgentRosterService
from services.entities.agent_entities import RosterAgentCreatePayload, RosterAgentUpdatePayload, RosterListQuery
@@ -29,6 +37,14 @@ register_schema_models(
RosterAgentUpdatePayload,
RosterListQuery,
)
register_response_schema_models(
console_ns,
AgentConfigSnapshotDetailResponse,
AgentConfigSnapshotListResponse,
AgentInviteOptionsResponse,
AgentRosterListResponse,
AgentRosterResponse,
)
def _agent_roster_service() -> AgentRosterService:
@@ -37,17 +53,23 @@ def _agent_roster_service() -> AgentRosterService:
@console_ns.route("/agents")
class AgentRosterListApi(Resource):
@console_ns.doc(params=query_params_from_model(RosterListQuery))
@console_ns.response(200, "Agent roster list", console_ns.models[AgentRosterListResponse.__name__])
@setup_required
@login_required
@account_initialization_required
def get(self):
_, tenant_id = current_account_with_tenant()
query = RosterListQuery.model_validate(request.args.to_dict(flat=True))
return _agent_roster_service().list_roster_agents(
tenant_id=tenant_id, page=query.page, limit=query.limit, keyword=query.keyword
return dump_response(
AgentRosterListResponse,
_agent_roster_service().list_roster_agents(
tenant_id=tenant_id, page=query.page, limit=query.limit, keyword=query.keyword
),
)
@console_ns.expect(console_ns.models[RosterAgentCreatePayload.__name__])
@console_ns.response(201, "Agent created", console_ns.models[AgentRosterResponse.__name__])
@setup_required
@login_required
@account_initialization_required
@@ -57,36 +79,49 @@ class AgentRosterListApi(Resource):
payload = RosterAgentCreatePayload.model_validate(console_ns.payload or {})
service = _agent_roster_service()
agent = service.create_roster_agent(tenant_id=tenant_id, account_id=account.id, payload=payload)
return service.get_roster_agent_detail(tenant_id=tenant_id, agent_id=agent.id), 201
return dump_response(
AgentRosterResponse,
service.get_roster_agent_detail(tenant_id=tenant_id, agent_id=agent.id),
), 201
@console_ns.route("/agents/invite-options")
class AgentInviteOptionsApi(Resource):
@console_ns.doc(params=query_params_from_model(AgentInviteOptionsQuery))
@console_ns.response(200, "Agent invite options", console_ns.models[AgentInviteOptionsResponse.__name__])
@setup_required
@login_required
@account_initialization_required
def get(self):
_, tenant_id = current_account_with_tenant()
query = AgentInviteOptionsQuery.model_validate(request.args.to_dict(flat=True))
return _agent_roster_service().list_invite_options(
tenant_id=tenant_id,
page=query.page,
limit=query.limit,
keyword=query.keyword,
app_id=query.app_id,
return dump_response(
AgentInviteOptionsResponse,
_agent_roster_service().list_invite_options(
tenant_id=tenant_id,
page=query.page,
limit=query.limit,
keyword=query.keyword,
app_id=query.app_id,
),
)
@console_ns.route("/agents/<uuid:agent_id>")
class AgentRosterDetailApi(Resource):
@console_ns.response(200, "Agent detail", console_ns.models[AgentRosterResponse.__name__])
@setup_required
@login_required
@account_initialization_required
def get(self, agent_id: UUID):
_, tenant_id = current_account_with_tenant()
return _agent_roster_service().get_roster_agent_detail(tenant_id=tenant_id, agent_id=str(agent_id))
return dump_response(
AgentRosterResponse,
_agent_roster_service().get_roster_agent_detail(tenant_id=tenant_id, agent_id=str(agent_id)),
)
@console_ns.expect(console_ns.models[RosterAgentUpdatePayload.__name__])
@console_ns.response(200, "Agent updated", console_ns.models[AgentRosterResponse.__name__])
@setup_required
@login_required
@account_initialization_required
@@ -94,10 +129,14 @@ class AgentRosterDetailApi(Resource):
def patch(self, agent_id: UUID):
account, tenant_id = current_account_with_tenant()
payload = RosterAgentUpdatePayload.model_validate(console_ns.payload or {})
return _agent_roster_service().update_roster_agent(
tenant_id=tenant_id, agent_id=str(agent_id), account_id=account.id, payload=payload
return dump_response(
AgentRosterResponse,
_agent_roster_service().update_roster_agent(
tenant_id=tenant_id, agent_id=str(agent_id), account_id=account.id, payload=payload
),
)
@console_ns.response(204, "Agent archived")
@setup_required
@login_required
@account_initialization_required
@@ -110,23 +149,31 @@ class AgentRosterDetailApi(Resource):
@console_ns.route("/agents/<uuid:agent_id>/versions")
class AgentRosterVersionsApi(Resource):
@console_ns.response(200, "Agent versions", console_ns.models[AgentConfigSnapshotListResponse.__name__])
@setup_required
@login_required
@account_initialization_required
def get(self, agent_id: UUID):
_, tenant_id = current_account_with_tenant()
return {"data": _agent_roster_service().list_agent_versions(tenant_id=tenant_id, agent_id=str(agent_id))}
return dump_response(
AgentConfigSnapshotListResponse,
{"data": _agent_roster_service().list_agent_versions(tenant_id=tenant_id, agent_id=str(agent_id))},
)
@console_ns.route("/agents/<uuid:agent_id>/versions/<uuid:version_id>")
class AgentRosterVersionDetailApi(Resource):
@console_ns.response(200, "Agent version detail", console_ns.models[AgentConfigSnapshotDetailResponse.__name__])
@setup_required
@login_required
@account_initialization_required
def get(self, agent_id: UUID, version_id: UUID):
_, tenant_id = current_account_with_tenant()
return _agent_roster_service().get_agent_version_detail(
tenant_id=tenant_id,
agent_id=str(agent_id),
version_id=str(version_id),
return dump_response(
AgentConfigSnapshotDetailResponse,
_agent_roster_service().get_agent_version_detail(
tenant_id=tenant_id,
agent_id=str(agent_id),
version_id=str(version_id),
),
)
@@ -5,7 +5,7 @@ from uuid import UUID
from flask import request
from flask_restx import Resource, marshal
from pydantic import BaseModel, Field
from sqlalchemy import String, cast, func, or_, select
from sqlalchemy import String, case, cast, func, literal, or_, select
from sqlalchemy.dialects.postgresql import JSONB
from werkzeug.exceptions import Forbidden, NotFound
@@ -169,12 +169,17 @@ class DatasetDocumentSegmentListApi(Resource):
# Use database-specific methods for JSON array search
if dify_config.SQLALCHEMY_DATABASE_URI_SCHEME == "postgresql":
# PostgreSQL: Use jsonb_array_elements_text to properly handle Unicode/Chinese text
# Guard with jsonb_typeof to avoid "cannot extract elements from a scalar" error
# when keywords is null or a non-array JSON value.
# Feed the set-returning function a JSON array in every row. Filtering in
# the subquery is not enough because PostgreSQL can still evaluate the
# SRF on scalar JSON before applying the predicate.
keywords_jsonb = cast(DocumentSegment.keywords, JSONB)
keywords_array = case(
(func.jsonb_typeof(keywords_jsonb) == "array", keywords_jsonb),
else_=cast(literal("[]"), JSONB),
)
keywords_condition = func.array_to_string(
func.array(
select(func.jsonb_array_elements_text(cast(DocumentSegment.keywords, JSONB)))
.where(func.jsonb_typeof(cast(DocumentSegment.keywords, JSONB)) == "array")
select(func.jsonb_array_elements_text(keywords_array))
.correlate(DocumentSegment)
.scalar_subquery()
),
@@ -209,6 +209,11 @@ class MCPProviderBasePayload(BaseModel):
configuration: dict[str, Any] | None = Field(default_factory=dict)
headers: dict[str, Any] | None = Field(default_factory=dict)
authentication: dict[str, Any] | None = Field(default_factory=dict)
# M3 — user-identity forwarding (M2 backend already supports these on the
# service layer). Defaults preserve pre-M3 behavior for clients that don't
# send the fields yet.
forward_user_identity: bool = False
identity_mode: Literal["off", "idp_token"] = "off"
class MCPProviderCreatePayload(MCPProviderBasePayload):
@@ -985,6 +990,8 @@ class ToolProviderMCPApi(Resource):
headers=payload.headers or {},
configuration=configuration,
authentication=authentication,
forward_user_identity=payload.forward_user_identity,
identity_mode=payload.identity_mode,
)
# 2) Try to fetch tools immediately after creation so they appear without a second save.
@@ -1052,6 +1059,8 @@ class ToolProviderMCPApi(Resource):
configuration=configuration,
authentication=authentication,
validation_result=validation_result,
forward_user_identity=payload.forward_user_identity,
identity_mode=payload.identity_mode,
)
return {"result": "success"}
+21 -9
View File
@@ -6,7 +6,7 @@ from flask_restx import Resource
from flask_restx.api import HTTPStatus
from pydantic import BaseModel, Field, TypeAdapter
from controllers.common.schema import register_schema_models
from controllers.common.schema import query_params_from_model, register_schema_models
from controllers.console.wraps import edit_permission_required
from controllers.service_api import service_api_ns
from controllers.service_api.wraps import validate_app_token
@@ -32,8 +32,19 @@ class AnnotationReplyActionPayload(BaseModel):
embedding_model_name: str = Field(description="Embedding model name")
class AnnotationListQuery(BaseModel):
page: int = Field(default=1, ge=1, description="Page number")
limit: int = Field(default=20, ge=1, description="Number of annotations per page")
keyword: str = Field(default="", description="Keyword to search annotations")
register_schema_models(
service_api_ns, AnnotationCreatePayload, AnnotationReplyActionPayload, Annotation, AnnotationList
service_api_ns,
AnnotationCreatePayload,
AnnotationReplyActionPayload,
AnnotationListQuery,
Annotation,
AnnotationList,
)
@@ -100,6 +111,7 @@ class AnnotationReplyActionStatusApi(Resource):
class AnnotationListApi(Resource):
@service_api_ns.doc("list_annotations")
@service_api_ns.doc(description="List annotations for the application")
@service_api_ns.doc(params=query_params_from_model(AnnotationListQuery))
@service_api_ns.doc(
responses={
200: "Annotations retrieved successfully",
@@ -114,18 +126,18 @@ class AnnotationListApi(Resource):
@validate_app_token
def get(self, app_model: App):
"""List annotations for the application."""
page = request.args.get("page", default=1, type=int)
limit = request.args.get("limit", default=20, type=int)
keyword = request.args.get("keyword", default="", type=str)
query = AnnotationListQuery.model_validate(request.args.to_dict(flat=True))
annotation_list, total = AppAnnotationService.get_annotation_list_by_app_id(app_model.id, page, limit, keyword)
annotation_list, total = AppAnnotationService.get_annotation_list_by_app_id(
app_model.id, query.page, query.limit, query.keyword
)
annotation_models = TypeAdapter(list[Annotation]).validate_python(annotation_list, from_attributes=True)
response = AnnotationList(
data=annotation_models,
has_more=len(annotation_list) == limit,
limit=limit,
has_more=len(annotation_list) == query.limit,
limit=query.limit,
total=total,
page=page,
page=query.page,
)
return response.model_dump(mode="json")
@@ -562,15 +562,16 @@ class WorkflowResponseConverter:
outputs, outputs_truncated = self._truncate_mapping(encoded_outputs)
metadata = self._merge_metadata(event.execution_metadata, snapshot)
if isinstance(event, QueueNodeSucceededEvent):
status = WorkflowNodeExecutionStatus.SUCCEEDED
error_message = event.error
elif isinstance(event, QueueNodeFailedEvent):
status = WorkflowNodeExecutionStatus.FAILED
error_message = event.error
else:
status = WorkflowNodeExecutionStatus.EXCEPTION
error_message = event.error
match event:
case QueueNodeSucceededEvent():
status = WorkflowNodeExecutionStatus.SUCCEEDED
error_message = event.error
case QueueNodeFailedEvent():
status = WorkflowNodeExecutionStatus.FAILED
error_message = event.error
case _:
status = WorkflowNodeExecutionStatus.EXCEPTION
error_message = event.error
return NodeFinishStreamResponse(
task_id=task_id,
@@ -91,26 +91,28 @@ class AppGeneratorTTSPublisher:
)
future_queue.put(futures_result)
break
elif isinstance(message.event, QueueAgentMessageEvent | QueueLLMChunkEvent):
message_content = message.event.chunk.delta.message.content
if not message_content:
continue
match message_content:
case str():
self.msg_text += message_content
case list():
for content in message_content:
if not isinstance(content, TextPromptMessageContent):
continue
self.msg_text += content.data
elif isinstance(message.event, QueueTextChunkEvent):
self.msg_text += message.event.text
elif isinstance(message.event, QueueNodeSucceededEvent):
if message.event.outputs is None:
continue
output = message.event.outputs.get("output", "")
if isinstance(output, str):
self.msg_text += output
else:
match message.event:
case QueueAgentMessageEvent() | QueueLLMChunkEvent():
message_content = message.event.chunk.delta.message.content
if not message_content:
continue
match message_content:
case str():
self.msg_text += message_content
case list():
for content in message_content:
if not isinstance(content, TextPromptMessageContent):
continue
self.msg_text += content.data
case QueueTextChunkEvent():
self.msg_text += message.event.text
case QueueNodeSucceededEvent():
if message.event.outputs is None:
continue
output = message.event.outputs.get("output", "")
if isinstance(output, str):
self.msg_text += output
self.last_message = message
sentence_arr, text_tmp = self._extract_sentence(self.msg_text)
if len(sentence_arr) >= min(self.max_sentence, 7):
+13 -1
View File
@@ -3,7 +3,7 @@ from __future__ import annotations
import json
from datetime import datetime
from enum import StrEnum
from typing import TYPE_CHECKING, Any
from typing import TYPE_CHECKING, Any, Literal
from urllib.parse import urlparse
from pydantic import BaseModel
@@ -76,6 +76,14 @@ class MCPProviderEntity(BaseModel):
created_at: datetime
updated_at: datetime
# M2 — user-identity forwarding. When forward_user_identity is True AND
# identity_mode is "idp_token", the MCP tool runtime asks dify-enterprise
# to mint a fresh SSO id_token for the calling user and stamps it on the
# outbound MCP request as `Authorization: Bearer <token>`. Defaults keep
# pre-M2 providers unchanged (no forwarding).
forward_user_identity: bool = False
identity_mode: Literal["off", "idp_token"] = "off"
@classmethod
def from_db_model(cls, db_provider: MCPToolProvider) -> MCPProviderEntity:
"""Create entity from database model with decryption"""
@@ -96,6 +104,8 @@ class MCPProviderEntity(BaseModel):
icon=db_provider.icon or "",
created_at=db_provider.created_at,
updated_at=db_provider.updated_at,
forward_user_identity=db_provider.forward_user_identity,
identity_mode=db_provider.identity_mode, # type: ignore[arg-type]
)
@property
@@ -170,6 +180,8 @@ class MCPProviderEntity(BaseModel):
"updated_at": int(self.updated_at.timestamp()),
"label": I18nObject(en_US=self.name, zh_Hans=self.name).to_dict(),
"description": I18nObject(en_US="", zh_Hans="").to_dict(),
"forward_user_identity": self.forward_user_identity,
"identity_mode": self.identity_mode,
}
# Add configuration
+26 -23
View File
@@ -54,36 +54,39 @@ class Blob(BaseModel):
def as_string(self) -> str:
"""Read data as a string."""
if self.data is None and self.path:
return Path(str(self.path)).read_text(encoding=self.encoding)
elif isinstance(self.data, bytes):
return self.data.decode(self.encoding)
elif isinstance(self.data, str):
return self.data
else:
raise ValueError(f"Unable to get string for blob {self}")
match self.data:
case None if self.path:
return Path(str(self.path)).read_text(encoding=self.encoding)
case bytes():
return self.data.decode(self.encoding)
case str():
return self.data
case _:
raise ValueError(f"Unable to get string for blob {self}")
def as_bytes(self) -> bytes:
"""Read data as bytes."""
if isinstance(self.data, bytes):
return self.data
elif isinstance(self.data, str):
return self.data.encode(self.encoding)
elif self.data is None and self.path:
return Path(str(self.path)).read_bytes()
else:
raise ValueError(f"Unable to get bytes for blob {self}")
match self.data:
case bytes():
return self.data
case str():
return self.data.encode(self.encoding)
case None if self.path:
return Path(str(self.path)).read_bytes()
case _:
raise ValueError(f"Unable to get bytes for blob {self}")
@contextlib.contextmanager
def as_bytes_io(self) -> Generator[BytesIO | BufferedReader, None, None]:
"""Read data as a byte stream."""
if isinstance(self.data, bytes):
yield BytesIO(self.data)
elif self.data is None and self.path:
with open(str(self.path), "rb") as f:
yield f
else:
raise NotImplementedError(f"Unable to convert blob {self}")
match self.data:
case bytes():
yield BytesIO(self.data)
case None if self.path:
with open(str(self.path), "rb") as f:
yield f
case _:
raise NotImplementedError(f"Unable to convert blob {self}")
@classmethod
def from_path(
+12
View File
@@ -54,6 +54,14 @@ class ToolProviderApiEntity(BaseModel):
configuration: MCPConfiguration | None = Field(
default=None, description="The timeout and sse_read_timeout of the MCP tool"
)
# M3 — user-identity forwarding flags. Round-tripped through the console
# API so the create/edit modal can hydrate the toggle state.
forward_user_identity: bool = Field(
default=False, description="Whether Dify forwards the calling user's SSO identity to this MCP server"
)
identity_mode: str = Field(
default="off", description="Identity-forwarding mechanism: 'off' or 'idp_token'"
)
# Workflow
workflow_app_id: str | None = Field(default=None, description="The app id of the workflow tool")
@@ -92,6 +100,10 @@ class ToolProviderApiEntity(BaseModel):
optional_fields.update(self.optional_field("is_dynamic_registration", self.is_dynamic_registration))
optional_fields.update(self.optional_field("masked_headers", self.masked_headers))
optional_fields.update(self.optional_field("original_headers", self.original_headers))
# M3 — forwarding flags. Always emit (False/"off" are valid
# values that the UI must hydrate, not skip).
optional_fields["forward_user_identity"] = self.forward_user_identity
optional_fields["identity_mode"] = self.identity_mode
case ToolProviderType.WORKFLOW:
optional_fields.update(self.optional_field("workflow_app_id", self.workflow_app_id))
case _:
+11 -1
View File
@@ -1,4 +1,4 @@
from typing import Any, Self
from typing import Any, Literal, Self
from core.entities.mcp_provider import MCPProviderEntity
from core.mcp.types import Tool as RemoteMCPTool
@@ -28,6 +28,8 @@ class MCPToolProviderController(ToolProviderController):
headers: dict[str, str] | None = None,
timeout: float | None = None,
sse_read_timeout: float | None = None,
forward_user_identity: bool = False,
identity_mode: Literal["off", "idp_token"] = "off",
):
super().__init__(entity)
self.entity: ToolProviderEntityWithPlugin = entity
@@ -37,6 +39,8 @@ class MCPToolProviderController(ToolProviderController):
self.headers = headers or {}
self.timeout = timeout
self.sse_read_timeout = sse_read_timeout
self.forward_user_identity = forward_user_identity
self.identity_mode: Literal["off", "idp_token"] = identity_mode
@property
def provider_type(self) -> ToolProviderType:
@@ -105,6 +109,8 @@ class MCPToolProviderController(ToolProviderController):
headers=entity.headers,
timeout=entity.timeout,
sse_read_timeout=entity.sse_read_timeout,
forward_user_identity=entity.forward_user_identity,
identity_mode=entity.identity_mode,
)
def _validate_credentials(self, user_id: str, credentials: dict[str, Any]):
@@ -134,6 +140,8 @@ class MCPToolProviderController(ToolProviderController):
headers=self.headers,
timeout=self.timeout,
sse_read_timeout=self.sse_read_timeout,
forward_user_identity=self.forward_user_identity,
identity_mode=self.identity_mode,
)
def get_tools(self) -> list[MCPTool]:
@@ -151,6 +159,8 @@ class MCPToolProviderController(ToolProviderController):
headers=self.headers,
timeout=self.timeout,
sse_read_timeout=self.sse_read_timeout,
forward_user_identity=self.forward_user_identity,
identity_mode=self.identity_mode,
)
for tool_entity in self.entity.tools
]
+50 -3
View File
@@ -4,7 +4,7 @@ import base64
import json
import logging
from collections.abc import Generator, Mapping
from typing import Any, cast
from typing import Any, Literal, cast
from core.mcp.auth_client import MCPClientWithAuthRetry
from core.mcp.error import MCPConnectionError
@@ -38,6 +38,8 @@ class MCPTool(Tool):
headers: dict[str, str] | None = None,
timeout: float | None = None,
sse_read_timeout: float | None = None,
forward_user_identity: bool = False,
identity_mode: Literal["off", "idp_token"] = "off",
):
super().__init__(entity, runtime)
self.tenant_id = tenant_id
@@ -47,6 +49,8 @@ class MCPTool(Tool):
self.headers = headers or {}
self.timeout = timeout
self.sse_read_timeout = sse_read_timeout
self.forward_user_identity = forward_user_identity
self.identity_mode: Literal["off", "idp_token"] = identity_mode
self._latest_usage = LLMUsage.empty_usage()
def tool_provider_type(self) -> ToolProviderType:
@@ -60,7 +64,7 @@ class MCPTool(Tool):
app_id: str | None = None,
message_id: str | None = None,
) -> Generator[ToolInvokeMessage, None, None]:
result = self.invoke_remote_mcp_tool(tool_parameters)
result = self.invoke_remote_mcp_tool(tool_parameters, user_id=user_id, app_id=app_id)
# Extract usage metadata from MCP protocol's _meta field
self._latest_usage = self._derive_usage_from_result(result)
@@ -234,6 +238,8 @@ class MCPTool(Tool):
headers=self.headers,
timeout=self.timeout,
sse_read_timeout=self.sse_read_timeout,
forward_user_identity=self.forward_user_identity,
identity_mode=self.identity_mode,
)
def _handle_none_parameter(self, parameter: dict[str, Any]) -> dict[str, Any]:
@@ -246,7 +252,12 @@ class MCPTool(Tool):
if value is not None and not (isinstance(value, str) and value.strip() == "")
}
def invoke_remote_mcp_tool(self, tool_parameters: dict[str, Any]) -> CallToolResult:
def invoke_remote_mcp_tool(
self,
tool_parameters: dict[str, Any],
user_id: str | None = None,
app_id: str | None = None,
) -> CallToolResult:
headers = self.headers.copy() if self.headers else {}
tool_parameters = self._handle_none_parameter(tool_parameters)
@@ -271,6 +282,14 @@ class MCPTool(Tool):
if tokens and tokens.access_token:
headers["Authorization"] = f"{tokens.token_type.capitalize()} {tokens.access_token}"
# User-identity forwarding: if enabled on this provider, ask the
# enterprise side to mint a fresh SSO id_token (audience-scoped to
# the MCP server's URL per RFC 8707) and stamp it as Authorization.
# This OVERRIDES any Authorization already on the request — the
# forwarded identity is what the MCP server should trust.
if self.forward_user_identity and self.identity_mode == "idp_token" and user_id:
self._inject_forwarded_identity(headers, user_id=user_id, app_id=app_id, audience=server_url)
# Step 2: Session is now closed, perform network operations without holding database connection
# MCPClientWithAuthRetry will create a new session lazily only if auth retry is needed
try:
@@ -286,3 +305,31 @@ class MCPTool(Tool):
raise ToolInvokeError(f"Failed to connect to MCP server: {e}") from e
except Exception as e:
raise ToolInvokeError(f"Failed to invoke tool: {e}") from e
def _inject_forwarded_identity(
self,
headers: dict[str, str],
*,
user_id: str,
app_id: str | None,
audience: str,
) -> None:
"""Call the enterprise IssueMCPToken endpoint and stamp Authorization.
Errors are surfaced as ToolInvokeError so the workflow halts with a
clear message instead of silently dropping identity and hitting the
MCP server unauthenticated.
"""
from services.enterprise.base import MCPTokenError
from services.enterprise.enterprise_service import EnterpriseService
try:
token, _expires_at = EnterpriseService.issue_mcp_token(
user_id=user_id,
tenant_id=self.tenant_id,
app_id=app_id,
audience=audience,
)
except MCPTokenError as e:
raise ToolInvokeError(f"Failed to obtain forwarded identity token: {e}") from e
headers["Authorization"] = f"Bearer {token}"
+5 -3
View File
@@ -863,7 +863,7 @@ class ToolManager:
return controller
@classmethod
def user_get_api_provider(cls, provider: str, tenant_id: str):
def user_get_api_provider(cls, provider: str, tenant_id: str, mask: bool = True):
"""
get api provider
"""
@@ -902,8 +902,10 @@ class ToolManager:
tenant_id=tenant_id,
controller=controller,
)
masked_credentials = encrypter.mask_plugin_credentials(encrypter.decrypt(credentials))
if mask:
masked_credentials = encrypter.mask_plugin_credentials(encrypter.decrypt(credentials))
else:
masked_credentials = encrypter.decrypt(credentials)
try:
icon = emoji_icon_adapter.validate_json(provider_obj.icon)
+2 -2
View File
@@ -6,7 +6,7 @@ from json.decoder import JSONDecodeError
from typing import Any, TypedDict
import httpx
from flask import request
from flask import has_request_context, request
from yaml import YAMLError, safe_load
from core.tools.entities.common_entities import I18nObject
@@ -44,7 +44,7 @@ class ApiBasedToolSchemaParser:
raise ToolProviderNotFoundError("No server found in the openapi yaml.")
server_url = openapi["servers"][0]["url"]
request_env = request.headers.get("X-Request-Env")
request_env = request.headers.get("X-Request-Env") if has_request_context() else None
if request_env:
matched_servers = [server["url"] for server in openapi["servers"] if server["env"] == request_env]
server_url = matched_servers[0] if matched_servers else server_url
@@ -1,6 +1,7 @@
import logging
from core.tools.entities.tool_entities import ToolProviderType
from core.tools.errors import ToolProviderNotFoundError
from core.tools.tool_manager import ToolManager
from core.tools.utils.configuration import ToolParameterConfigurationManager
from core.workflow.human_input_adapter import adapt_node_config_for_graph
@@ -38,6 +39,14 @@ def handle(sender, **kwargs):
identity_id=f"WORKFLOW.{app.id}.{node_data.get('id')}",
)
manager.delete_tool_parameters_cache()
except ToolProviderNotFoundError as exc:
logger.info(
"Skipped deleting tool parameters cache for workflow %s node %s "
"because tool provider is missing: %s",
app.id,
node_data.get("id"),
exc,
)
except Exception:
# tool dose not exist
logger.exception(
+8
View File
@@ -15,14 +15,18 @@ def init_app(app: DifyApp):
data_migrate,
delete_archived_workflow_runs,
export_app_messages,
export_migration_data,
export_migration_data_template,
extract_plugins,
extract_unique_plugins,
file_usage,
fix_app_site_missing,
import_migration_data,
install_plugins,
install_rag_pipeline_plugins,
migrate_data_for_plugin,
migrate_oss,
migration_data_wizard,
old_metadata_migration,
remove_orphaned_files_on_storage,
reset_email,
@@ -70,6 +74,10 @@ def init_app(app: DifyApp):
clean_workflow_runs,
clean_expired_messages,
export_app_messages,
export_migration_data,
export_migration_data_template,
import_migration_data,
migration_data_wizard,
]
for cmd in cmds_to_register:
app.cli.add_command(cmd)
@@ -1,5 +1,6 @@
import posixpath
from collections.abc import Generator
from typing import override
import oss2 as aliyun_s3
@@ -29,9 +30,11 @@ class AliyunOssStorage(BaseStorage):
cloudbox_id=dify_config.ALIYUN_CLOUDBOX_ID,
)
@override
def save(self, filename, data):
self.client.put_object(self.__wrapper_folder_filename(filename), data)
@override
def load_once(self, filename: str) -> bytes:
obj = self.client.get_object(self.__wrapper_folder_filename(filename))
data = obj.read()
@@ -39,17 +42,21 @@ class AliyunOssStorage(BaseStorage):
return b""
return data
@override
def load_stream(self, filename: str) -> Generator:
obj = self.client.get_object(self.__wrapper_folder_filename(filename))
while chunk := obj.read(4096):
yield chunk
@override
def download(self, filename: str, target_filepath):
self.client.get_object_to_file(self.__wrapper_folder_filename(filename), target_filepath)
@override
def exists(self, filename: str):
return self.client.object_exists(self.__wrapper_folder_filename(filename))
@override
def delete(self, filename: str):
self.client.delete_object(self.__wrapper_folder_filename(filename))
+7
View File
@@ -1,5 +1,6 @@
import logging
from collections.abc import Generator
from typing import override
import boto3
from botocore.client import Config
@@ -48,9 +49,11 @@ class AwsS3Storage(BaseStorage):
# other error, raise exception
raise
@override
def save(self, filename, data):
self.client.put_object(Bucket=self.bucket_name, Key=filename, Body=data)
@override
def load_once(self, filename: str) -> bytes:
try:
data: bytes = self.client.get_object(Bucket=self.bucket_name, Key=filename)["Body"].read()
@@ -61,6 +64,7 @@ class AwsS3Storage(BaseStorage):
raise
return data
@override
def load_stream(self, filename: str) -> Generator:
try:
response = self.client.get_object(Bucket=self.bucket_name, Key=filename)
@@ -73,9 +77,11 @@ class AwsS3Storage(BaseStorage):
else:
raise
@override
def download(self, filename, target_filepath):
self.client.download_file(self.bucket_name, filename, target_filepath)
@override
def exists(self, filename):
try:
self.client.head_object(Bucket=self.bucket_name, Key=filename)
@@ -83,5 +89,6 @@ class AwsS3Storage(BaseStorage):
except:
return False
@override
def delete(self, filename: str):
self.client.delete_object(Bucket=self.bucket_name, Key=filename)
@@ -1,5 +1,6 @@
from collections.abc import Generator
from datetime import timedelta
from typing import override
from azure.identity import ChainedTokenCredential, DefaultAzureCredential
from azure.storage.blob import AccountSasPermissions, BlobServiceClient, ResourceTypes, generate_account_sas
@@ -26,6 +27,7 @@ class AzureBlobStorage(BaseStorage):
else:
self.credential = None
@override
def save(self, filename, data):
if not self.bucket_name:
return
@@ -34,6 +36,7 @@ class AzureBlobStorage(BaseStorage):
blob_container = client.get_container_client(container=self.bucket_name)
blob_container.upload_blob(filename, data)
@override
def load_once(self, filename: str) -> bytes:
if not self.bucket_name:
raise FileNotFoundError("Azure bucket name is not configured.")
@@ -46,6 +49,7 @@ class AzureBlobStorage(BaseStorage):
raise TypeError(f"Expected bytes from blob.readall(), got {type(data).__name__}")
return data
@override
def load_stream(self, filename: str) -> Generator:
if not self.bucket_name:
raise FileNotFoundError("Azure bucket name is not configured.")
@@ -55,6 +59,7 @@ class AzureBlobStorage(BaseStorage):
blob_data = blob.download_blob()
yield from blob_data.chunks()
@override
def download(self, filename, target_filepath):
if not self.bucket_name:
return
@@ -66,6 +71,7 @@ class AzureBlobStorage(BaseStorage):
blob_data = blob.download_blob()
blob_data.readinto(my_blob)
@override
def exists(self, filename):
if not self.bucket_name:
return False
@@ -75,6 +81,7 @@ class AzureBlobStorage(BaseStorage):
blob = client.get_blob_client(container=self.bucket_name, blob=filename)
return blob.exists()
@override
def delete(self, filename: str):
if not self.bucket_name:
return
@@ -1,6 +1,7 @@
import base64
import hashlib
from collections.abc import Generator
from typing import override
from baidubce.auth.bce_credentials import BceCredentials
from baidubce.bce_client_configuration import BceClientConfiguration
@@ -26,6 +27,7 @@ class BaiduObsStorage(BaseStorage):
self.client = BosClient(config=client_config)
@override
def save(self, filename, data):
md5 = hashlib.md5()
md5.update(data)
@@ -34,24 +36,29 @@ class BaiduObsStorage(BaseStorage):
bucket_name=self.bucket_name, key=filename, data=data, content_length=len(data), content_md5=content_md5
)
@override
def load_once(self, filename: str) -> bytes:
response = self.client.get_object(bucket_name=self.bucket_name, key=filename)
data: bytes = response.data.read()
return data
@override
def load_stream(self, filename: str) -> Generator:
response = self.client.get_object(bucket_name=self.bucket_name, key=filename).data
while chunk := response.read(4096):
yield chunk
@override
def download(self, filename, target_filepath):
self.client.get_object_to_file(bucket_name=self.bucket_name, key=filename, file_name=target_filepath)
@override
def exists(self, filename):
res = self.client.get_object_meta_data(bucket_name=self.bucket_name, key=filename)
if res is None:
return False
return True
@override
def delete(self, filename: str):
self.client.delete_object(bucket_name=self.bucket_name, key=filename)
@@ -10,7 +10,7 @@ import tempfile
from collections.abc import Generator
from io import BytesIO
from pathlib import Path
from typing import Any
from typing import Any, override
import clickzetta
from pydantic import BaseModel, model_validator
@@ -251,6 +251,7 @@ class ClickZettaVolumeStorage(BaseStorage):
# Don't raise exception, let the operation continue
# The table might exist but not be visible due to permissions
@override
def save(self, filename: str, data: bytes):
"""Save data to ClickZetta Volume.
@@ -304,6 +305,7 @@ class ClickZettaVolumeStorage(BaseStorage):
# Clean up temporary file
Path(temp_file_path).unlink(missing_ok=True)
@override
def load_once(self, filename: str) -> bytes:
"""Load file content from ClickZetta Volume.
@@ -364,6 +366,7 @@ class ClickZettaVolumeStorage(BaseStorage):
logger.debug("File %s loaded from ClickZetta Volume", filename)
return content
@override
def load_stream(self, filename: str) -> Generator:
"""Load file as stream from ClickZetta Volume.
@@ -382,6 +385,7 @@ class ClickZettaVolumeStorage(BaseStorage):
logger.debug("File %s loaded as stream from ClickZetta Volume", filename)
@override
def download(self, filename: str, target_filepath: str):
"""Download file from ClickZetta Volume to local path.
@@ -395,6 +399,7 @@ class ClickZettaVolumeStorage(BaseStorage):
logger.debug("File %s downloaded from ClickZetta Volume to %s", filename, target_filepath)
@override
def exists(self, filename: str) -> bool:
"""Check if file exists in ClickZetta Volume.
@@ -436,6 +441,7 @@ class ClickZettaVolumeStorage(BaseStorage):
logger.warning("Error checking file existence for %s: %s", filename, e)
return False
@override
def delete(self, filename: str):
"""Delete file from ClickZetta Volume.
@@ -472,6 +478,7 @@ class ClickZettaVolumeStorage(BaseStorage):
logger.debug("File %s deleted from ClickZetta Volume", filename)
@override
def scan(self, path: str, files: bool = True, directories: bool = False) -> list[str]:
"""Scan files and directories in ClickZetta Volume.
@@ -1,7 +1,7 @@
import base64
import io
from collections.abc import Generator
from typing import Any
from typing import Any, override
from google.cloud import storage as google_cloud_storage # type: ignore
from pydantic import TypeAdapter
@@ -29,12 +29,14 @@ class GoogleCloudStorage(BaseStorage):
else:
self.client = google_cloud_storage.Client()
@override
def save(self, filename, data):
bucket = self.client.get_bucket(self.bucket_name)
blob = bucket.blob(filename)
with io.BytesIO(data) as stream:
blob.upload_from_file(stream)
@override
def load_once(self, filename: str) -> bytes:
bucket = self.client.get_bucket(self.bucket_name)
blob = bucket.get_blob(filename)
@@ -43,6 +45,7 @@ class GoogleCloudStorage(BaseStorage):
data: bytes = blob.download_as_bytes()
return data
@override
def load_stream(self, filename: str) -> Generator:
bucket = self.client.get_bucket(self.bucket_name)
blob = bucket.get_blob(filename)
@@ -52,6 +55,7 @@ class GoogleCloudStorage(BaseStorage):
while chunk := blob_stream.read(4096):
yield chunk
@override
def download(self, filename, target_filepath):
bucket = self.client.get_bucket(self.bucket_name)
blob = bucket.get_blob(filename)
@@ -59,11 +63,13 @@ class GoogleCloudStorage(BaseStorage):
raise FileNotFoundError("File not found")
blob.download_to_filename(target_filepath)
@override
def exists(self, filename):
bucket = self.client.get_bucket(self.bucket_name)
blob = bucket.blob(filename)
return blob.exists()
@override
def delete(self, filename: str):
bucket = self.client.get_bucket(self.bucket_name)
bucket.delete_blob(filename)
@@ -1,4 +1,5 @@
from collections.abc import Generator
from typing import override
from obs import ObsClient
@@ -20,27 +21,33 @@ class HuaweiObsStorage(BaseStorage):
path_style=dify_config.HUAWEI_OBS_PATH_STYLE,
)
@override
def save(self, filename, data):
self.client.putObject(bucketName=self.bucket_name, objectKey=filename, content=data)
@override
def load_once(self, filename: str) -> bytes:
data: bytes = self.client.getObject(bucketName=self.bucket_name, objectKey=filename)["body"].response.read()
return data
@override
def load_stream(self, filename: str) -> Generator:
response = self.client.getObject(bucketName=self.bucket_name, objectKey=filename)["body"].response
while chunk := response.read(4096):
yield chunk
@override
def download(self, filename, target_filepath):
self.client.getObject(bucketName=self.bucket_name, objectKey=filename, downloadPath=target_filepath)
@override
def exists(self, filename):
res = self._get_meta(filename)
if res is None:
return False
return True
@override
def delete(self, filename: str):
self.client.deleteObject(bucketName=self.bucket_name, objectKey=filename)
+8 -1
View File
@@ -2,7 +2,7 @@ import logging
import os
from collections.abc import Generator
from pathlib import Path
from typing import Any
from typing import Any, override
import opendal
from dotenv import dotenv_values
@@ -41,10 +41,12 @@ class OpenDALStorage(BaseStorage):
logger.debug("opendal operator created with scheme %s", scheme)
logger.debug("added retry layer to opendal operator")
@override
def save(self, filename: str, data: bytes):
self.op.write(path=filename, bs=data)
logger.debug("file %s saved", filename)
@override
def load_once(self, filename: str) -> bytes:
if not self.exists(filename):
raise FileNotFoundError("File not found")
@@ -53,6 +55,7 @@ class OpenDALStorage(BaseStorage):
logger.debug("file %s loaded", filename)
return content
@override
def load_stream(self, filename: str) -> Generator:
if not self.exists(filename):
raise FileNotFoundError("File not found")
@@ -67,6 +70,7 @@ class OpenDALStorage(BaseStorage):
yield chunk
logger.debug("file %s loaded as stream", filename)
@override
def download(self, filename: str, target_filepath: str):
if not self.exists(filename):
raise FileNotFoundError("File not found")
@@ -74,9 +78,11 @@ class OpenDALStorage(BaseStorage):
Path(target_filepath).write_bytes(self.op.read(path=filename))
logger.debug("file %s downloaded to %s", filename, target_filepath)
@override
def exists(self, filename: str) -> bool:
return self.op.exists(path=filename)
@override
def delete(self, filename: str):
if self.exists(filename):
self.op.delete(path=filename)
@@ -84,6 +90,7 @@ class OpenDALStorage(BaseStorage):
return
logger.debug("file %s not found, skip delete", filename)
@override
def scan(self, path: str, files: bool = True, directories: bool = False) -> list[str]:
if not self.exists(path):
raise FileNotFoundError("Path not found")
@@ -1,4 +1,5 @@
from collections.abc import Generator
from typing import override
import boto3
from botocore.exceptions import ClientError
@@ -22,9 +23,11 @@ class OracleOCIStorage(BaseStorage):
region_name=dify_config.OCI_REGION,
)
@override
def save(self, filename, data):
self.client.put_object(Bucket=self.bucket_name, Key=filename, Body=data)
@override
def load_once(self, filename: str) -> bytes:
try:
data: bytes = self.client.get_object(Bucket=self.bucket_name, Key=filename)["Body"].read()
@@ -35,6 +38,7 @@ class OracleOCIStorage(BaseStorage):
raise
return data
@override
def load_stream(self, filename: str) -> Generator:
try:
response = self.client.get_object(Bucket=self.bucket_name, Key=filename)
@@ -45,9 +49,11 @@ class OracleOCIStorage(BaseStorage):
else:
raise
@override
def download(self, filename, target_filepath):
self.client.download_file(self.bucket_name, filename, target_filepath)
@override
def exists(self, filename):
try:
self.client.head_object(Bucket=self.bucket_name, Key=filename)
@@ -55,5 +61,6 @@ class OracleOCIStorage(BaseStorage):
except:
return False
@override
def delete(self, filename: str):
self.client.delete_object(Bucket=self.bucket_name, Key=filename)
@@ -1,6 +1,7 @@
import io
from collections.abc import Generator
from pathlib import Path
from typing import override
from supabase import Client
@@ -28,29 +29,35 @@ class SupabaseStorage(BaseStorage):
if not self.bucket_exists():
self.client.storage.create_bucket(id=id, name=bucket_name)
@override
def save(self, filename, data):
self.client.storage.from_(self.bucket_name).upload(filename, data)
@override
def load_once(self, filename: str) -> bytes:
content: bytes = self.client.storage.from_(self.bucket_name).download(filename)
return content
@override
def load_stream(self, filename: str) -> Generator:
result = self.client.storage.from_(self.bucket_name).download(filename)
byte_stream = io.BytesIO(result)
while chunk := byte_stream.read(4096): # Read in chunks of 4KB
yield chunk
@override
def download(self, filename, target_filepath):
result = self.client.storage.from_(self.bucket_name).download(filename)
Path(target_filepath).write_bytes(result)
@override
def exists(self, filename):
result = self.client.storage.from_(self.bucket_name).list(path=filename)
if len(result) > 0:
return True
return False
@override
def delete(self, filename: str):
self.client.storage.from_(self.bucket_name).remove([filename])
@@ -1,4 +1,5 @@
from collections.abc import Generator
from typing import override
from qcloud_cos import CosConfig, CosS3Client
@@ -29,23 +30,29 @@ class TencentCosStorage(BaseStorage):
)
self.client = CosS3Client(config)
@override
def save(self, filename, data):
self.client.put_object(Bucket=self.bucket_name, Body=data, Key=filename)
@override
def load_once(self, filename: str) -> bytes:
data: bytes = self.client.get_object(Bucket=self.bucket_name, Key=filename)["Body"].get_raw_stream().read()
return data
@override
def load_stream(self, filename: str) -> Generator:
response = self.client.get_object(Bucket=self.bucket_name, Key=filename)
yield from response["Body"].get_stream(chunk_size=4096)
@override
def download(self, filename, target_filepath):
response = self.client.get_object(Bucket=self.bucket_name, Key=filename)
response["Body"].get_stream_to_file(target_filepath)
@override
def exists(self, filename):
return self.client.object_exists(Bucket=self.bucket_name, Key=filename)
@override
def delete(self, filename: str):
self.client.delete_object(Bucket=self.bucket_name, Key=filename)
@@ -1,4 +1,5 @@
from collections.abc import Generator
from typing import override
import tos
@@ -27,11 +28,13 @@ class VolcengineTosStorage(BaseStorage):
region=dify_config.VOLCENGINE_TOS_REGION,
)
@override
def save(self, filename, data):
if not self.bucket_name:
raise ValueError("VOLCENGINE_TOS_BUCKET_NAME is not set")
self.client.put_object(bucket=self.bucket_name, key=filename, content=data)
@override
def load_once(self, filename: str) -> bytes:
if not self.bucket_name:
raise FileNotFoundError("VOLCENGINE_TOS_BUCKET_NAME is not set")
@@ -40,6 +43,7 @@ class VolcengineTosStorage(BaseStorage):
raise TypeError(f"Expected bytes, got {type(data).__name__}")
return data
@override
def load_stream(self, filename: str) -> Generator:
if not self.bucket_name:
raise FileNotFoundError("VOLCENGINE_TOS_BUCKET_NAME is not set")
@@ -47,11 +51,13 @@ class VolcengineTosStorage(BaseStorage):
while chunk := response.read(4096):
yield chunk
@override
def download(self, filename, target_filepath):
if not self.bucket_name:
raise ValueError("VOLCENGINE_TOS_BUCKET_NAME is not set")
self.client.get_object_to_file(bucket=self.bucket_name, key=filename, file_path=target_filepath)
@override
def exists(self, filename):
if not self.bucket_name:
return False
@@ -60,6 +66,7 @@ class VolcengineTosStorage(BaseStorage):
return False
return True
@override
def delete(self, filename: str):
if not self.bucket_name:
return
+192
View File
@@ -0,0 +1,192 @@
from typing import Any, Literal
from pydantic import Field
from fields.base import ResponseModel
from models.agent import (
AgentConfigRevisionOperation,
AgentIconType,
AgentKind,
AgentScope,
AgentSource,
AgentStatus,
WorkflowAgentBindingType,
)
from models.agent_config_entities import (
AgentSoulConfig,
DeclaredOutputConfig,
DeclaredOutputType,
WorkflowNodeJobConfig,
)
from services.entities.agent_entities import (
ComposerCandidateCapabilities,
ComposerSaveStrategy,
ComposerVariant,
)
class AgentConfigSnapshotSummaryResponse(ResponseModel):
id: str
agent_id: str | None = None
version: int
summary: str | None = None
version_note: str | None = None
created_by: str | None = None
created_at: str | None = None
class AgentRosterResponse(ResponseModel):
id: str
name: str
description: str
icon_type: AgentIconType | None = None
icon: str | None = None
icon_background: str | None = None
agent_kind: AgentKind
scope: AgentScope
source: AgentSource
app_id: str | None = None
workflow_id: str | None = None
workflow_node_id: str | None = None
active_config_snapshot_id: str | None = None
active_config_snapshot: AgentConfigSnapshotSummaryResponse | None = None
status: AgentStatus
created_by: str | None = None
updated_by: str | None = None
archived_by: str | None = None
archived_at: str | None = None
created_at: str | None = None
updated_at: str | None = None
class AgentInviteOptionResponse(AgentRosterResponse):
is_in_current_workflow: bool = False
in_current_workflow_count: int = 0
existing_node_ids: list[str] = Field(default_factory=list)
class AgentRosterListResponse(ResponseModel):
data: list[AgentRosterResponse]
page: int
limit: int
total: int
has_more: bool
class AgentInviteOptionsResponse(ResponseModel):
data: list[AgentInviteOptionResponse]
page: int
limit: int
total: int
has_more: bool
class AgentConfigRevisionResponse(ResponseModel):
id: str
previous_snapshot_id: str | None = None
current_snapshot_id: str
revision: int
operation: AgentConfigRevisionOperation
summary: str | None = None
version_note: str | None = None
created_by: str | None = None
created_at: str | None = None
class AgentConfigSnapshotDetailResponse(AgentConfigSnapshotSummaryResponse):
config_snapshot: AgentSoulConfig
revisions: list[AgentConfigRevisionResponse] = Field(default_factory=list)
class AgentConfigSnapshotListResponse(ResponseModel):
data: list[AgentConfigSnapshotSummaryResponse]
class AgentComposerAgentResponse(ResponseModel):
id: str
name: str
description: str
scope: AgentScope
status: AgentStatus
active_config_snapshot_id: str | None = None
class AgentComposerBindingResponse(ResponseModel):
id: str
binding_type: WorkflowAgentBindingType
agent_id: str | None = None
current_snapshot_id: str | None = None
workflow_id: str
node_id: str
class AgentComposerSoulLockResponse(ResponseModel):
locked: bool
can_unlock: bool = False
reason: str | None = None
class AgentComposerImpactBindingResponse(ResponseModel):
app_id: str
workflow_id: str
node_id: str
class AgentComposerImpactResponse(ResponseModel):
current_snapshot_id: str | None = None
workflow_node_count: int
bindings: list[AgentComposerImpactBindingResponse] = Field(default_factory=list)
class WorkflowAgentComposerResponse(ResponseModel):
variant: Literal[ComposerVariant.WORKFLOW]
agent: AgentComposerAgentResponse | None = None
active_config_snapshot: AgentConfigSnapshotSummaryResponse | None = None
binding: AgentComposerBindingResponse | None = None
soul_lock: AgentComposerSoulLockResponse
agent_soul: AgentSoulConfig
node_job: WorkflowNodeJobConfig
effective_declared_outputs: list[DeclaredOutputConfig] = Field(default_factory=list)
save_options: list[ComposerSaveStrategy]
impact_summary: AgentComposerImpactResponse | None = None
app_id: str | None = None
workflow_id: str | None = None
node_id: str | None = None
class AgentAppComposerResponse(ResponseModel):
variant: Literal[ComposerVariant.AGENT_APP]
agent: AgentComposerAgentResponse
active_config_snapshot: AgentConfigSnapshotSummaryResponse
agent_soul: AgentSoulConfig
save_options: list[ComposerSaveStrategy]
class AgentComposerValidateResponse(ResponseModel):
result: Literal["success"]
errors: list[str] = Field(default_factory=list)
class AgentComposerNodeJobCandidatesResponse(ResponseModel):
previous_node_outputs: list[dict[str, Any]] = Field(default_factory=list)
declare_output_types: list[DeclaredOutputType] = Field(default_factory=list)
human_contacts: list[dict[str, Any]] = Field(default_factory=list)
class AgentComposerSoulCandidatesResponse(ResponseModel):
skills_files: list[dict[str, Any]] = Field(default_factory=list)
dify_tools: list[dict[str, Any]] = Field(default_factory=list)
cli_tools: list[dict[str, Any]] = Field(default_factory=list)
knowledge_datasets: list[dict[str, Any]] = Field(default_factory=list)
human_contacts: list[dict[str, Any]] = Field(default_factory=list)
class AgentComposerCandidatesResponse(ResponseModel):
variant: ComposerVariant
allowed_node_job_candidates: AgentComposerNodeJobCandidatesResponse = Field(
default_factory=AgentComposerNodeJobCandidatesResponse
)
allowed_soul_candidates: AgentComposerSoulCandidatesResponse = Field(
default_factory=AgentComposerSoulCandidatesResponse
)
capabilities: ComposerCandidateCapabilities = Field(default_factory=ComposerCandidateCapabilities)
@@ -0,0 +1,56 @@
"""add identity mode to mcp tool provider
Revision ID: 3df4dbcc1e21
Revises: 7885bd53f9a9
Create Date: 2026-05-29 15:00:00.000000
Adds two columns to `tool_mcp_providers` that drive the M2 MCP user-identity
forwarding feature:
* `forward_user_identity` (bool, default false) — master switch per provider.
* `identity_mode` (string, default "off") — which forwarding mechanism to use:
"off" — no header forwarded (default; pre-M2 behaviour).
"idp_token" — call dify-enterprise /inner/api/mcp/issue-token, stamp
the returned id_token on the outbound MCP request as
`Authorization: Bearer <token>`.
The columns are filled with safe defaults for existing rows so older providers
keep their current behaviour (no identity forwarding) until an admin opts in.
"""
import sqlalchemy as sa
from alembic import op
import models as models
# revision identifiers, used by Alembic.
revision = "3df4dbcc1e21"
down_revision = "7885bd53f9a9"
branch_labels = None
depends_on = None
def upgrade():
op.add_column(
"tool_mcp_providers",
sa.Column(
"forward_user_identity",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
),
)
op.add_column(
"tool_mcp_providers",
sa.Column(
"identity_mode",
sa.String(length=32),
nullable=False,
server_default=sa.text("'off'"),
),
)
def downgrade():
op.drop_column("tool_mcp_providers", "identity_mode")
op.drop_column("tool_mcp_providers", "forward_user_identity")
+15
View File
@@ -343,6 +343,21 @@ class MCPToolProvider(TypeBase):
# encrypted headers for MCP server requests
encrypted_headers: Mapped[str | None] = mapped_column(LongText, nullable=True, default=None)
# M2 (MCP user-identity forwarding) — master switch per provider. When True
# AND identity_mode is "idp_token", workflows that invoke tools on this
# provider will have the caller's SSO id_token stamped on the outbound
# request as `Authorization: Bearer …`. Off by default so existing
# providers retain pre-M2 behaviour.
forward_user_identity: Mapped[bool] = mapped_column(
sa.Boolean, nullable=False, server_default=sa.text("false"), default=False
)
# M2 — which identity-forwarding mechanism to use. Reserved values:
# "off" — no forwarding (default).
# "idp_token" — forward a Bearer id_token minted by dify-enterprise.
identity_mode: Mapped[str] = mapped_column(
sa.String(32), nullable=False, server_default=sa.text("'off'"), default="off"
)
def load_user(self) -> Account | None:
return db.session.scalar(select(Account).where(Account.id == self.user_id))
+356 -52
View File
@@ -343,11 +343,19 @@ Check if activation token is valid
### /agents
#### GET
##### Parameters
| Name | Located in | Description | Required | Schema |
| ---- | ---------- | ----------- | -------- | ------ |
| keyword | query | | No | string |
| limit | query | | No | integer |
| page | query | | No | integer |
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Agent roster list | [AgentRosterListResponse](#agentrosterlistresponse) |
#### POST
##### Parameters
@@ -358,18 +366,27 @@ Check if activation token is valid
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 201 | Agent created | [AgentRosterResponse](#agentrosterresponse) |
### /agents/invite-options
#### GET
##### Parameters
| Name | Located in | Description | Required | Schema |
| ---- | ---------- | ----------- | -------- | ------ |
| app_id | query | Workflow app id for in-current-workflow markers | No | string |
| keyword | query | | No | string |
| limit | query | | No | integer |
| page | query | | No | integer |
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Agent invite options | [AgentInviteOptionsResponse](#agentinviteoptionsresponse) |
### /agents/{agent_id}
@@ -384,7 +401,7 @@ Check if activation token is valid
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| 204 | Agent archived |
#### GET
##### Parameters
@@ -395,9 +412,9 @@ Check if activation token is valid
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Agent detail | [AgentRosterResponse](#agentrosterresponse) |
#### PATCH
##### Parameters
@@ -409,9 +426,9 @@ Check if activation token is valid
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Agent updated | [AgentRosterResponse](#agentrosterresponse) |
### /agents/{agent_id}/versions
@@ -424,9 +441,9 @@ Check if activation token is valid
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Agent versions | [AgentConfigSnapshotListResponse](#agentconfigsnapshotlistresponse) |
### /agents/{agent_id}/versions/{version_id}
@@ -440,9 +457,9 @@ Check if activation token is valid
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Agent version detail | [AgentConfigSnapshotDetailResponse](#agentconfigsnapshotdetailresponse) |
### /all-workspaces
@@ -978,9 +995,9 @@ Run draft workflow for advanced chat application
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Agent app composer state | [AgentAppComposerResponse](#agentappcomposerresponse) |
#### PUT
##### Parameters
@@ -992,9 +1009,9 @@ Run draft workflow for advanced chat application
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Agent app composer saved | [AgentAppComposerResponse](#agentappcomposerresponse) |
### /apps/{app_id}/agent-composer/candidates
@@ -1007,9 +1024,9 @@ Run draft workflow for advanced chat application
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Agent app composer candidates | [AgentComposerCandidatesResponse](#agentcomposercandidatesresponse) |
### /apps/{app_id}/agent-composer/validate
@@ -1023,9 +1040,9 @@ Run draft workflow for advanced chat application
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Agent app composer validation result | [AgentComposerValidateResponse](#agentcomposervalidateresponse) |
### /apps/{app_id}/agent/logs
@@ -3224,9 +3241,9 @@ Run draft workflow loop node
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Workflow agent composer state | [WorkflowAgentComposerResponse](#workflowagentcomposerresponse) |
#### PUT
##### Parameters
@@ -3239,9 +3256,9 @@ Run draft workflow loop node
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Workflow agent composer saved | [WorkflowAgentComposerResponse](#workflowagentcomposerresponse) |
### /apps/{app_id}/workflows/draft/nodes/{node_id}/agent-composer/candidates
@@ -3255,9 +3272,9 @@ Run draft workflow loop node
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Workflow agent composer candidates | [AgentComposerCandidatesResponse](#agentcomposercandidatesresponse) |
### /apps/{app_id}/workflows/draft/nodes/{node_id}/agent-composer/impact
@@ -3268,12 +3285,13 @@ Run draft workflow loop node
| ---- | ---------- | ----------- | -------- | ------ |
| app_id | path | | Yes | string |
| node_id | path | | Yes | string |
| payload | body | | Yes | [ComposerSavePayload](#composersavepayload) |
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Workflow agent composer impact | [AgentComposerImpactResponse](#agentcomposerimpactresponse) |
### /apps/{app_id}/workflows/draft/nodes/{node_id}/agent-composer/save-to-roster
@@ -3288,9 +3306,9 @@ Run draft workflow loop node
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Workflow agent composer saved to roster | [WorkflowAgentComposerResponse](#workflowagentcomposerresponse) |
### /apps/{app_id}/workflows/draft/nodes/{node_id}/agent-composer/validate
@@ -3305,9 +3323,9 @@ Run draft workflow loop node
##### Responses
| Code | Description |
| ---- | ----------- |
| 200 | Success |
| Code | Description | Schema |
| ---- | ----------- | ------ |
| 200 | Workflow agent composer validation result | [AgentComposerValidateResponse](#agentcomposervalidateresponse) |
### /apps/{app_id}/workflows/draft/nodes/{node_id}/last-run
@@ -10651,6 +10669,150 @@ Get banner list
| model_mode | string | Model mode | Yes |
| model_name | string | Model name | Yes |
#### AgentAppComposerResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| active_config_snapshot | [AgentConfigSnapshotSummaryResponse](#agentconfigsnapshotsummaryresponse) | | Yes |
| agent | [AgentComposerAgentResponse](#agentcomposeragentresponse) | | Yes |
| agent_soul | [AgentSoulConfig](#agentsoulconfig) | | Yes |
| save_options | [ [ComposerSaveStrategy](#composersavestrategy) ] | | Yes |
| variant | string | | Yes |
#### AgentComposerAgentResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| active_config_snapshot_id | string | | No |
| description | string | | Yes |
| id | string | | Yes |
| name | string | | Yes |
| scope | [AgentScope](#agentscope) | | Yes |
| status | [AgentStatus](#agentstatus) | | Yes |
#### AgentComposerBindingResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| agent_id | string | | No |
| binding_type | [WorkflowAgentBindingType](#workflowagentbindingtype) | | Yes |
| current_snapshot_id | string | | No |
| id | string | | Yes |
| node_id | string | | Yes |
| workflow_id | string | | Yes |
#### AgentComposerCandidatesResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| allowed_node_job_candidates | [AgentComposerNodeJobCandidatesResponse](#agentcomposernodejobcandidatesresponse) | | No |
| allowed_soul_candidates | [AgentComposerSoulCandidatesResponse](#agentcomposersoulcandidatesresponse) | | No |
| capabilities | [ComposerCandidateCapabilities](#composercandidatecapabilities) | | No |
| variant | [ComposerVariant](#composervariant) | | Yes |
#### AgentComposerImpactBindingResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| app_id | string | | Yes |
| node_id | string | | Yes |
| workflow_id | string | | Yes |
#### AgentComposerImpactResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| bindings | [ [AgentComposerImpactBindingResponse](#agentcomposerimpactbindingresponse) ] | | No |
| current_snapshot_id | string | | No |
| workflow_node_count | integer | | Yes |
#### AgentComposerNodeJobCandidatesResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| declare_output_types | [ [DeclaredOutputType](#declaredoutputtype) ] | | No |
| human_contacts | [ object ] | | No |
| previous_node_outputs | [ object ] | | No |
#### AgentComposerSoulCandidatesResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| cli_tools | [ object ] | | No |
| dify_tools | [ object ] | | No |
| human_contacts | [ object ] | | No |
| knowledge_datasets | [ object ] | | No |
| skills_files | [ object ] | | No |
#### AgentComposerSoulLockResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| can_unlock | boolean | | No |
| locked | boolean | | Yes |
| reason | string | | No |
#### AgentComposerValidateResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| errors | [ string ] | | No |
| result | string | | Yes |
#### AgentConfigRevisionOperation
Audit operation recorded for Agent Soul version/revision changes.
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| AgentConfigRevisionOperation | string | Audit operation recorded for Agent Soul version/revision changes. | |
#### AgentConfigRevisionResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| created_at | string | | No |
| created_by | string | | No |
| current_snapshot_id | string | | Yes |
| id | string | | Yes |
| operation | [AgentConfigRevisionOperation](#agentconfigrevisionoperation) | | Yes |
| previous_snapshot_id | string | | No |
| revision | integer | | Yes |
| summary | string | | No |
| version_note | string | | No |
#### AgentConfigSnapshotDetailResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| agent_id | string | | No |
| config_snapshot | [AgentSoulConfig](#agentsoulconfig) | | Yes |
| created_at | string | | No |
| created_by | string | | No |
| id | string | | Yes |
| revisions | [ [AgentConfigRevisionResponse](#agentconfigrevisionresponse) ] | | No |
| summary | string | | No |
| version | integer | | Yes |
| version_note | string | | No |
#### AgentConfigSnapshotListResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| data | [ [AgentConfigSnapshotSummaryResponse](#agentconfigsnapshotsummaryresponse) ] | | Yes |
#### AgentConfigSnapshotSummaryResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| agent_id | string | | No |
| created_at | string | | No |
| created_by | string | | No |
| id | string | | Yes |
| summary | string | | No |
| version | integer | | Yes |
| version_note | string | | No |
#### AgentIconType
Supported icon storage formats for Agent roster entries.
@@ -10665,6 +10827,35 @@ Supported icon storage formats for Agent roster entries.
| ---- | ---- | ----------- | -------- |
| agent_id | string | | Yes |
#### AgentInviteOptionResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| active_config_snapshot | [AgentConfigSnapshotSummaryResponse](#agentconfigsnapshotsummaryresponse) | | No |
| active_config_snapshot_id | string | | No |
| agent_kind | [AgentKind](#agentkind) | | Yes |
| app_id | string | | No |
| archived_at | string | | No |
| archived_by | string | | No |
| created_at | string | | No |
| created_by | string | | No |
| description | string | | Yes |
| existing_node_ids | [ string ] | | No |
| icon | string | | No |
| icon_background | string | | No |
| icon_type | [AgentIconType](#agenticontype) | | No |
| id | string | | Yes |
| in_current_workflow_count | integer | | No |
| is_in_current_workflow | boolean | | No |
| name | string | | Yes |
| scope | [AgentScope](#agentscope) | | Yes |
| source | [AgentSource](#agentsource) | | Yes |
| status | [AgentStatus](#agentstatus) | | Yes |
| updated_at | string | | No |
| updated_by | string | | No |
| workflow_id | string | | No |
| workflow_node_id | string | | No |
#### AgentInviteOptionsQuery
| Name | Type | Description | Required |
@@ -10674,6 +10865,27 @@ Supported icon storage formats for Agent roster entries.
| limit | integer | | No |
| page | integer | | No |
#### AgentInviteOptionsResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| data | [ [AgentInviteOptionResponse](#agentinviteoptionresponse) ] | | Yes |
| has_more | boolean | | Yes |
| limit | integer | | Yes |
| page | integer | | Yes |
| total | integer | | Yes |
#### AgentKind
Agent implementation family.
This leaves room for future non-Dify agent implementations while keeping
the current roster/workflow APIs scoped to Dify Agent.
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| AgentKind | string | Agent implementation family. This leaves room for future non-Dify agent implementations while keeping the current roster/workflow APIs scoped to Dify Agent. | |
#### AgentKnowledgeQueryMode
| Name | Type | Description | Required |
@@ -10687,6 +10899,50 @@ Supported icon storage formats for Agent roster entries.
| conversation_id | string | Conversation UUID | Yes |
| message_id | string | Message UUID | Yes |
#### AgentRosterListResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| data | [ [AgentRosterResponse](#agentrosterresponse) ] | | Yes |
| has_more | boolean | | Yes |
| limit | integer | | Yes |
| page | integer | | Yes |
| total | integer | | Yes |
#### AgentRosterResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| active_config_snapshot | [AgentConfigSnapshotSummaryResponse](#agentconfigsnapshotsummaryresponse) | | No |
| active_config_snapshot_id | string | | No |
| agent_kind | [AgentKind](#agentkind) | | Yes |
| app_id | string | | No |
| archived_at | string | | No |
| archived_by | string | | No |
| created_at | string | | No |
| created_by | string | | No |
| description | string | | Yes |
| icon | string | | No |
| icon_background | string | | No |
| icon_type | [AgentIconType](#agenticontype) | | No |
| id | string | | Yes |
| name | string | | Yes |
| scope | [AgentScope](#agentscope) | | Yes |
| source | [AgentSource](#agentsource) | | Yes |
| status | [AgentStatus](#agentstatus) | | Yes |
| updated_at | string | | No |
| updated_by | string | | No |
| workflow_id | string | | No |
| workflow_node_id | string | | No |
#### AgentScope
Visibility and lifecycle scope of an Agent record.
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| AgentScope | string | Visibility and lifecycle scope of an Agent record. | |
#### AgentSoulConfig
| Name | Type | Description | Required |
@@ -10821,6 +11077,22 @@ Reference to model credentials resolved only at runtime.
| cli_tools | [ object ] | | No |
| dify_tools | [ [AgentSoulDifyToolConfig](#agentsouldifytoolconfig) ] | | No |
#### AgentSource
Origin that created or imported the Agent.
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| AgentSource | string | Origin that created or imported the Agent. | |
#### AgentStatus
Soft lifecycle state for Agent records.
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| AgentStatus | string | Soft lifecycle state for Agent records. | |
#### AgentThought
| Name | Type | Description | Required |
@@ -11528,6 +11800,12 @@ Button styles for user actions.
| binding_type | string | *Enum:* `"inline_agent"`, `"roster_agent"` | Yes |
| current_snapshot_id | string | | No |
#### ComposerCandidateCapabilities
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| human_roster_available | boolean | | No |
#### ComposerSavePayload
| Name | Type | Description | Required |
@@ -15369,6 +15647,32 @@ in form definiton, or a variable while the workflow is running.
| embedding_provider_name | string | | Yes |
| vector_weight | number | | Yes |
#### WorkflowAgentBindingType
How a workflow node is bound to an Agent.
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| WorkflowAgentBindingType | string | How a workflow node is bound to an Agent. | |
#### WorkflowAgentComposerResponse
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| active_config_snapshot | [AgentConfigSnapshotSummaryResponse](#agentconfigsnapshotsummaryresponse) | | No |
| agent | [AgentComposerAgentResponse](#agentcomposeragentresponse) | | No |
| agent_soul | [AgentSoulConfig](#agentsoulconfig) | | Yes |
| app_id | string | | No |
| binding | [AgentComposerBindingResponse](#agentcomposerbindingresponse) | | No |
| effective_declared_outputs | [ [DeclaredOutputConfig](#declaredoutputconfig) ] | | No |
| impact_summary | [AgentComposerImpactResponse](#agentcomposerimpactresponse) | | No |
| node_id | string | | No |
| node_job | [WorkflowNodeJobConfig](#workflownodejobconfig) | | Yes |
| save_options | [ [ComposerSaveStrategy](#composersavestrategy) ] | | Yes |
| soul_lock | [AgentComposerSoulLockResponse](#agentcomposersoullockresponse) | | Yes |
| variant | string | | Yes |
| workflow_id | string | | No |
#### WorkflowAppLogPaginationResponse
| Name | Type | Description | Required |
+16
View File
@@ -112,6 +112,14 @@ List annotations for the application
List annotations for the application
##### Parameters
| Name | Located in | Description | Required | Schema |
| ---- | ---------- | ----------- | -------- | ------ |
| keyword | query | Keyword to search annotations | No | string |
| limit | query | Number of annotations per page | No | integer |
| page | query | Page number | No | integer |
##### Responses
| Code | Description | Schema |
@@ -2169,6 +2177,14 @@ Returns a list of available models for the specified model type.
| page | integer | | Yes |
| total | integer | | Yes |
#### AnnotationListQuery
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| keyword | string | Keyword to search annotations | No |
| limit | integer | Number of annotations per page | No |
| page | integer | Page number | No |
#### AnnotationReplyActionPayload
| Name | Type | Description | Required |
+4 -1
View File
@@ -97,6 +97,7 @@ class AppDslService:
icon: str | None = None,
icon_background: str | None = None,
app_id: str | None = None,
import_app_id: str | None = None,
) -> Import:
"""Import an app from YAML content or URL."""
import_id = str(uuid.uuid4())
@@ -262,6 +263,7 @@ class AppDslService:
icon=icon,
icon_background=icon_background,
dependencies=check_dependencies_pending_data,
import_app_id=import_app_id,
)
draft_var_srv = WorkflowDraftVariableService(session=self._session)
@@ -385,6 +387,7 @@ class AppDslService:
icon: str | None = None,
icon_background: str | None = None,
dependencies: list[PluginDependency] | None = None,
import_app_id: str | None = None,
) -> App:
"""Create a new app or update an existing one."""
app_data = data.get("app", {})
@@ -417,7 +420,7 @@ class AppDslService:
# Create new app
app = App()
app.id = str(uuid4())
app.id = import_app_id or str(uuid4())
app.tenant_id = account.current_tenant_id
app.mode = app_mode
app.name = name or app_data.get("name", "")
+17
View File
@@ -0,0 +1,17 @@
from services.data_migration.entities import (
ConflictStrategy,
ExportSelection,
IdStrategy,
ImportOptions,
MigrationDataError,
MigrationPackage,
)
__all__ = [
"ConflictStrategy",
"ExportSelection",
"IdStrategy",
"ImportOptions",
"MigrationDataError",
"MigrationPackage",
]
@@ -0,0 +1,92 @@
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
from services.data_migration.entities import DependencyKind
@dataclass(frozen=True)
class DiscoveredDependency:
kind: DependencyKind
provider_id: str
provider_name: str | None = None
source: str | None = None
class DependencyDiscoveryService:
def discover_from_dsl(self, dsl: dict[str, Any]) -> list[DiscoveredDependency]:
seen: set[tuple[DependencyKind, str]] = set()
result: list[DiscoveredDependency] = []
for node in self._nodes_from_dsl(dsl):
data = node.get("data", {}) if isinstance(node, dict) else {}
for dependency in self._dependencies_from_node(data):
key = (dependency.kind, dependency.provider_id)
if dependency.provider_id and key not in seen:
seen.add(key)
result.append(dependency)
return result
def _nodes_from_dsl(self, dsl: dict[str, Any]) -> list[dict[str, Any]]:
nodes: list[dict[str, Any]] = []
graph = dsl.get("graph") if isinstance(dsl, dict) else None
if isinstance(graph, dict) and isinstance(graph.get("nodes"), list):
nodes.extend(node for node in graph["nodes"] if isinstance(node, dict))
workflow = dsl.get("workflow") if isinstance(dsl, dict) else None
workflow_graph = workflow.get("graph") if isinstance(workflow, dict) else None
if isinstance(workflow_graph, dict) and isinstance(workflow_graph.get("nodes"), list):
nodes.extend(node for node in workflow_graph["nodes"] if isinstance(node, dict))
return nodes
def _dependencies_from_node(self, data: dict[str, Any]) -> list[DiscoveredDependency]:
dependencies: list[DiscoveredDependency] = []
node_type = data.get("type")
if node_type == "tool":
dependency = self._from_tool_config(data, source="tool_node")
if dependency:
dependencies.append(dependency)
if node_type == "agent":
for tool_config in self._agent_tool_configs(data):
if isinstance(tool_config, dict):
dependency = self._from_tool_config(tool_config, source="agent_node")
if dependency:
dependencies.append(dependency)
return dependencies
def _agent_tool_configs(self, data: dict[str, Any]) -> list[dict[str, Any]]:
configs = data.get("tools")
if isinstance(configs, list):
return [config for config in configs if isinstance(config, dict)]
agent_parameters = data.get("agent_parameters")
if not isinstance(agent_parameters, dict):
return []
tools_parameter = agent_parameters.get("tools")
if not isinstance(tools_parameter, dict):
return []
value = tools_parameter.get("value", [])
if not isinstance(value, list):
return []
return [config for config in value if isinstance(config, dict)]
def _from_tool_config(self, config: dict[str, Any], *, source: str) -> DiscoveredDependency | None:
provider_id = config.get("provider_id") or config.get("provider_name") or config.get("provider")
if not provider_id:
return None
provider_type = str(config.get("provider_type") or config.get("type") or "")
kind = self._kind_from_provider_type(provider_type)
return DiscoveredDependency(
kind=kind,
provider_id=str(provider_id),
provider_name=config.get("provider_name"),
source=source,
)
def _kind_from_provider_type(self, provider_type: str) -> DependencyKind:
normalized = provider_type.lower()
if normalized in {"api", "custom", "api_tool"}:
return DependencyKind.API_TOOL
if normalized in {"workflow", "workflow_tool"}:
return DependencyKind.WORKFLOW_TOOL
if normalized == "mcp":
return DependencyKind.MCP_TOOL
return DependencyKind.BUILTIN_OR_PLUGIN_TOOL
+241
View File
@@ -0,0 +1,241 @@
"""Typed entities for versioned cross-environment migration packages.
This module is intentionally side-effect free. It owns only value objects and
validation for migration package/config shapes; command output and database I/O
belong in adapter and service modules built on top of these entities.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import StrEnum
from typing import Any, Literal, TypedDict
class MigrationDataError(ValueError):
"""Raised when migration config or package data is invalid."""
class IdStrategy(StrEnum):
PRESERVE_ID = "preserve-id"
GENERATE_NEW_ID = "generate-new-id"
class ConflictStrategy(StrEnum):
FAIL = "fail"
SKIP = "skip"
UPDATE = "update"
class ResourceType(StrEnum):
WORKFLOW = "workflow"
API_TOOL = "api_tool"
WORKFLOW_TOOL = "workflow_tool"
MCP_TOOL = "mcp_tool"
DEPENDENCY = "dependency"
class DependencyKind(StrEnum):
API_TOOL = "api_tool"
WORKFLOW_TOOL = "workflow_tool"
MCP_TOOL = "mcp_tool"
BUILTIN_OR_PLUGIN_TOOL = "builtin_or_plugin_tool"
UNRESOLVED = "unresolved"
class TargetTenantSelector(TypedDict, total=False):
id: str
name: str
def _parse_target_tenant(value: Any) -> TargetTenantSelector | None:
if value is None:
return None
if not isinstance(value, dict):
raise MigrationDataError("metadata.target_tenant must be an object when provided.")
target: TargetTenantSelector = {}
target_id = value.get("id")
if target_id is not None:
if not isinstance(target_id, str):
raise MigrationDataError("metadata.target_tenant.id must be a string.")
target["id"] = target_id
target_name = value.get("name")
if target_name is not None:
if not isinstance(target_name, str):
raise MigrationDataError("metadata.target_tenant.name must be a string.")
target["name"] = target_name
unsupported_keys = sorted(set(value.keys()) - {"id", "name"})
if unsupported_keys:
raise MigrationDataError(f"metadata.target_tenant contains unsupported fields: {unsupported_keys}")
return target
def _parse_package_section(value: Any, section: str) -> list[dict[str, Any]]:
if value is None:
return []
if not isinstance(value, list):
raise MigrationDataError(f"Migration package field '{section}' must be a list.")
for item in value:
if not isinstance(item, dict):
raise MigrationDataError(f"Migration package field '{section}' must contain only objects.")
return value
@dataclass(frozen=True)
class SourceTenant:
id: str
name: str
@classmethod
def from_mapping(cls, value: dict[str, Any]) -> SourceTenant:
return cls(id=str(value.get("id", "")), name=str(value.get("name", "")))
@dataclass(frozen=True)
class ImportOptions:
create_app_api_token_on_import: bool = False
id_strategy: IdStrategy = IdStrategy.PRESERVE_ID
conflict_strategy: ConflictStrategy = ConflictStrategy.FAIL
@classmethod
def from_mapping(cls, value: dict[str, Any] | None) -> ImportOptions:
value = value or {}
try:
id_strategy = IdStrategy(value.get("id_strategy", IdStrategy.PRESERVE_ID))
except ValueError as exc:
raise MigrationDataError(f"Unsupported import_options.id_strategy: {value.get('id_strategy')}") from exc
try:
conflict_strategy = ConflictStrategy(value.get("conflict_strategy", ConflictStrategy.FAIL))
except ValueError as exc:
raise MigrationDataError(
f"Unsupported import_options.conflict_strategy: {value.get('conflict_strategy')}"
) from exc
return cls(
create_app_api_token_on_import=bool(value.get("create_app_api_token_on_import", False)),
id_strategy=id_strategy,
conflict_strategy=conflict_strategy,
)
@dataclass(frozen=True)
class MigrationMetadata:
version: str
source_scope: Literal["single"]
source_tenants: list[SourceTenant]
target_tenant: TargetTenantSelector | None = None
created_at: str | None = None
include_secrets: bool = False
import_options: ImportOptions = field(default_factory=ImportOptions)
@classmethod
def from_mapping(cls, value: dict[str, Any]) -> MigrationMetadata:
version = value.get("version")
if not version:
raise MigrationDataError("Migration package must include metadata.version.")
source_scope = value.get("source_scope", "single")
if source_scope != "single":
raise MigrationDataError(f"Unsupported source_scope: {source_scope}")
source_tenants = [
SourceTenant.from_mapping(item) for item in value.get("source_tenants", []) if isinstance(item, dict)
]
return cls(
version=str(version),
source_scope="single",
source_tenants=source_tenants,
target_tenant=_parse_target_tenant(value.get("target_tenant")),
created_at=value.get("created_at"),
include_secrets=bool(value.get("include_secrets", False)),
import_options=ImportOptions.from_mapping(value.get("import_options")),
)
@dataclass(frozen=True)
class MigrationPackage:
metadata: MigrationMetadata
workflows: list[dict[str, Any]] = field(default_factory=list)
tools: list[dict[str, Any]] = field(default_factory=list)
workflow_tools: list[dict[str, Any]] = field(default_factory=list)
mcp_tools: list[dict[str, Any]] = field(default_factory=list)
dependencies: list[dict[str, Any]] = field(default_factory=list)
@classmethod
def from_mapping(cls, value: dict[str, Any]) -> MigrationPackage:
metadata_value = value.get("metadata")
if not isinstance(metadata_value, dict):
raise MigrationDataError("Migration package must include metadata.version.")
return cls(
metadata=MigrationMetadata.from_mapping(metadata_value),
workflows=_parse_package_section(value.get("workflows"), "workflows"),
tools=_parse_package_section(value.get("tools"), "tools"),
workflow_tools=_parse_package_section(value.get("workflow_tools"), "workflow_tools"),
mcp_tools=_parse_package_section(value.get("mcp_tools"), "mcp_tools"),
dependencies=_parse_package_section(value.get("dependencies"), "dependencies"),
)
@dataclass(frozen=True)
class ExportSelection:
source_tenant_name: str
app_ids: list[str]
source_tenant_id: str | None = None
export_all_apps: bool = False
include_referenced_tools: bool = True
additional_api_tools: list[str] = field(default_factory=list)
additional_workflow_tools: list[str] = field(default_factory=list)
additional_mcp_tools: list[str] = field(default_factory=list)
include_secrets: bool = False
import_options: ImportOptions = field(default_factory=ImportOptions)
@dataclass(frozen=True)
class ResourceReportItem:
resource_type: ResourceType
identifier: str
name: str | None
status: str
message: str | None = None
@dataclass(frozen=True)
class ResourceIdMapping:
resource_type: ResourceType
name: str | None
source_id: str
target_id: str
@dataclass(frozen=True)
class ExportResult:
package: MigrationPackage
report_items: list[ResourceReportItem]
report_context: ReportContext | None = None
@dataclass(frozen=True)
class ImportTarget:
tenant_id: str
tenant_name: str
operator_id: str
operator_email: str | None
@dataclass(frozen=True)
class ImportResult:
report_items: list[ResourceReportItem]
id_mapping: dict[str, str] = field(default_factory=dict)
report_context: ReportContext | None = None
@dataclass(frozen=True)
class ReportContext:
output_path: str | None = None
source_scope: str | None = None
selected_app_count: int | None = None
include_secrets: bool | None = None
target_tenant: str | None = None
operator_email: str | None = None
app_api_tokens_created: int = 0
app_api_tokens_reused: int = 0
id_mapping_count: int = 0
id_mappings: dict[str, str] = field(default_factory=dict)
id_mapping_details: list[ResourceIdMapping] = field(default_factory=list)
@@ -0,0 +1,492 @@
from __future__ import annotations
from collections.abc import Iterable
from typing import Any
from uuid import UUID
import sqlalchemy as sa
import yaml
from core.tools.tool_manager import ToolManager
from extensions.ext_database import db
from graphon.model_runtime.utils.encoders import jsonable_encoder
from models import Account, Tenant
from models.account import TenantAccountJoin
from models.model import App
from models.tools import MCPToolProvider
from services.app_dsl_service import AppDslService
from services.data_migration.dependency_discovery_service import DependencyDiscoveryService, DiscoveredDependency
from services.data_migration.entities import (
DependencyKind,
ExportResult,
ExportSelection,
ImportOptions,
MigrationDataError,
ReportContext,
ResourceReportItem,
ResourceType,
)
from services.data_migration.package_service import MigrationPackageService
from services.tools.workflow_tools_manage_service import WorkflowToolManageService
SUPPORTED_APP_MODES = {"workflow", "advanced-chat"}
class ExportConfigParser:
def parse(self, data: dict[str, Any]) -> ExportSelection:
if not isinstance(data, dict):
raise MigrationDataError("Export config JSON must be an object.")
source_tenant = self._source_tenant(data)
source_tenant_name = self._source_tenant_name(source_tenant, data)
apps = self._mapping(data.get("apps"), field_name="apps")
self._validate_source_scope(data)
self._validate_app_modes(apps.get("modes", []))
additional_tools = self._mapping(data.get("additional_tools"), field_name="additional_tools")
return ExportSelection(
source_tenant_name=source_tenant_name,
app_ids=self._string_list(apps.get("ids", data.get("workflows", [])), field_name="apps.ids"),
source_tenant_id=source_tenant.get("id"),
export_all_apps=bool(apps.get("all", data.get("export_all_workflows", False))),
include_referenced_tools=bool(data.get("include_referenced_tools", True)),
additional_api_tools=self._string_list(
additional_tools.get("api_tools", data.get("tools", [])), field_name="additional_tools.api_tools"
),
additional_workflow_tools=self._string_list(
additional_tools.get("workflow_tools", data.get("workflow_tools", [])),
field_name="additional_tools.workflow_tools",
),
additional_mcp_tools=self._string_list(
additional_tools.get("mcp_tools", data.get("mcp_tools", [])),
field_name="additional_tools.mcp_tools",
),
include_secrets=bool(data.get("include_secrets", False)),
import_options=ImportOptions.from_mapping(data.get("import_options")),
)
def _source_tenant(self, data: dict[str, Any]) -> dict[str, Any]:
if "source_tenant" in data:
return self._mapping(data.get("source_tenant"), field_name="source_tenant")
return {}
def _source_tenant_name(self, source_tenant: dict[str, Any], data: dict[str, Any]) -> str:
if source_tenant:
source_tenant_name = source_tenant.get("name")
if not source_tenant_name:
raise MigrationDataError("Export config must include source_tenant.name.")
return str(source_tenant_name)
source_tenant_name = data.get("tenant_name")
if not source_tenant_name:
raise MigrationDataError("Export config must include source_tenant.name.")
return str(source_tenant_name)
def _validate_source_scope(self, data: dict[str, Any]) -> None:
source_tenant = data.get("source_tenant")
if not isinstance(source_tenant, dict):
return
mode = source_tenant.get("mode", "single")
if mode != "single":
raise MigrationDataError(f"Unsupported source_tenant.mode: {mode}")
def _validate_app_modes(self, modes: Any) -> None:
app_modes = self._string_list(modes, field_name="apps.modes") if modes else []
unsupported_modes = sorted(set(app_modes) - SUPPORTED_APP_MODES)
if unsupported_modes:
raise MigrationDataError(f"Unsupported app modes for export: {unsupported_modes}")
def _mapping(self, value: Any, *, field_name: str) -> dict[str, Any]:
if value is None:
return {}
if not isinstance(value, dict):
raise MigrationDataError(f"Export config field '{field_name}' must be an object.")
return value
def _string_list(self, value: Any, *, field_name: str) -> list[str]:
if value is None:
return []
if not isinstance(value, list):
raise MigrationDataError(f"Export config field '{field_name}' must be a list.")
return [str(item) for item in value]
class MigrationExportService:
def __init__(
self,
*,
package_service: MigrationPackageService | None = None,
dependency_discovery_service: DependencyDiscoveryService | None = None,
) -> None:
self.package_service = package_service or MigrationPackageService()
self.dependency_discovery_service = dependency_discovery_service or DependencyDiscoveryService()
def export(self, selection: ExportSelection) -> ExportResult:
tenant = self._get_tenant(selection)
package = self.package_service.build_empty_package(
source_tenant_id=tenant.id,
source_tenant_name=tenant.name,
include_secrets=selection.include_secrets,
import_options=selection.import_options,
)
report_items: list[ResourceReportItem] = []
discovered_dependencies: list[DiscoveredDependency] = []
apps = self._selected_apps(tenant.id, selection)
exported_app_ids = {app.id for app in apps}
for app in apps:
dsl_content = AppDslService.export_dsl(app_model=app, include_secret=selection.include_secrets)
package.workflows.append(
{
"id": app.id,
"name": app.name,
"mode": app.mode.value if hasattr(app.mode, "value") else app.mode,
"dsl": dsl_content,
"source_tenant_id": tenant.id,
"create_app_api_token_on_import": selection.import_options.create_app_api_token_on_import,
}
)
report_items.append(ResourceReportItem(ResourceType.WORKFLOW, app.id, app.name, "exported"))
if selection.include_referenced_tools:
discovered_dependencies.extend(self._discover_dependencies(dsl_content))
self._export_api_tools(
tenant.id,
self._provider_ids(selection.additional_api_tools, discovered_dependencies, DependencyKind.API_TOOL),
include_secrets=selection.include_secrets,
exported_tools=package.tools,
report_items=report_items,
)
self._export_workflow_tools(
tenant,
self._provider_ids(
selection.additional_workflow_tools, discovered_dependencies, DependencyKind.WORKFLOW_TOOL
),
exported_app_ids=exported_app_ids,
exported_workflow_tools=package.workflow_tools,
dependencies=package.dependencies,
report_items=report_items,
)
self._export_mcp_tools(
tenant_id=tenant.id,
provider_ids=self._provider_ids(
selection.additional_mcp_tools,
discovered_dependencies,
DependencyKind.MCP_TOOL,
),
include_secrets=selection.include_secrets,
exported_mcp_tools=package.mcp_tools,
dependencies=package.dependencies,
report_items=report_items,
)
self._record_dependency_metadata(
self._dependencies_by_kind(discovered_dependencies, DependencyKind.BUILTIN_OR_PLUGIN_TOOL),
package.dependencies,
report_items,
)
return ExportResult(
package=package,
report_items=report_items,
report_context=ReportContext(
source_scope=package.metadata.source_scope,
selected_app_count=len(apps),
include_secrets=selection.include_secrets,
),
)
def _get_tenant(self, selection: ExportSelection) -> Tenant:
if selection.source_tenant_id:
tenant = db.session.get(Tenant, selection.source_tenant_id)
if tenant is None:
raise MigrationDataError(f"Source tenant not found: {selection.source_tenant_id}")
if tenant.name != selection.source_tenant_name:
raise MigrationDataError(
f"Source tenant id/name mismatch: {selection.source_tenant_id} / {selection.source_tenant_name}"
)
return tenant
tenants = list(db.session.scalars(sa.select(Tenant).where(Tenant.name == selection.source_tenant_name)).all())
if not tenants:
raise MigrationDataError(f"Source tenant not found: {selection.source_tenant_name}")
if len(tenants) > 1:
raise MigrationDataError(
f"Source tenant name is ambiguous; use source_tenant.id: {selection.source_tenant_name}"
)
return tenants[0]
def _selected_apps(self, tenant_id: str, selection: ExportSelection) -> list[App]:
query = sa.select(App).where(App.tenant_id == tenant_id, App.mode.in_(SUPPORTED_APP_MODES))
if not selection.export_all_apps:
if not selection.app_ids:
return []
query = query.where(App.id.in_(selection.app_ids))
apps = list(db.session.scalars(query).all())
if not selection.export_all_apps and len(apps) != len(set(selection.app_ids)):
found_ids = {app.id for app in apps}
missing_ids = [app_id for app_id in selection.app_ids if app_id not in found_ids]
raise MigrationDataError(
f"Selected app IDs not found in source tenant or unsupported app mode: {missing_ids}"
)
return apps
def _discover_dependencies(self, dsl_content: str | dict[str, Any]) -> list[DiscoveredDependency]:
if isinstance(dsl_content, dict):
dsl = dsl_content
else:
raw_dsl = yaml.safe_load(dsl_content) if dsl_content else {}
dsl = raw_dsl if isinstance(raw_dsl, dict) else {}
return self.dependency_discovery_service.discover_from_dsl(dsl)
def _export_api_tools(
self,
tenant_id: str,
provider_ids: Iterable[str],
*,
include_secrets: bool,
exported_tools: list[dict[str, Any]],
report_items: list[ResourceReportItem],
) -> None:
for provider_id in self._dedupe(provider_ids):
try:
tool_data = ToolManager.user_get_api_provider(
provider=provider_id,
tenant_id=tenant_id,
mask=not include_secrets,
)
if not include_secrets:
tool_data.pop("credentials", None)
tool_data.pop("tools", None)
tool_data["provider_name"] = provider_id
tool_data["source_tenant_id"] = tenant_id
exported_tools.append(tool_data)
report_items.append(ResourceReportItem(ResourceType.API_TOOL, provider_id, provider_id, "exported"))
except Exception as exc:
report_items.append(
ResourceReportItem(ResourceType.API_TOOL, provider_id, provider_id, "unresolved", str(exc))
)
def _export_workflow_tools(
self,
tenant: Tenant,
provider_ids: Iterable[str],
*,
exported_app_ids: set[str],
exported_workflow_tools: list[dict[str, Any]],
dependencies: list[dict[str, Any]],
report_items: list[ResourceReportItem],
) -> None:
provider_ids = self._dedupe(provider_ids)
if not provider_ids:
return
owner = self._get_tenant_owner(tenant.id)
if owner is None:
for provider_id in provider_ids:
report_items.append(
ResourceReportItem(
ResourceType.WORKFLOW_TOOL,
provider_id,
provider_id,
"unresolved",
f"No owner account found for source tenant: {tenant.name}",
)
)
return
for provider_id in provider_ids:
try:
tool_data = WorkflowToolManageService.get_workflow_tool_by_tool_id(
user_id=owner.id,
tenant_id=tenant.id,
workflow_tool_id=provider_id,
)
tool_info = jsonable_encoder(tool_data)
tool_info["id"] = provider_id
tool_info["app_id"] = tool_info.get("workflow_app_id")
tool_info["source_tenant_id"] = tenant.id
for field_name in ("workflow_tool_id", "workflow_app_id", "tool"):
tool_info.pop(field_name, None)
exported_workflow_tools.append(tool_info)
if tool_info.get("app_id") not in exported_app_ids:
workflow_app_id = str(tool_info.get("app_id") or "")
workflow_app = db.session.get(App, workflow_app_id) if workflow_app_id else None
self._record_dependency_metadata(
[
DiscoveredDependency(
DependencyKind.WORKFLOW_TOOL,
workflow_app_id,
provider_name=workflow_app.name if workflow_app else tool_info.get("name"),
source="workflow_tool_app",
)
],
dependencies,
report_items,
)
report_items.append(
ResourceReportItem(ResourceType.WORKFLOW_TOOL, provider_id, tool_info.get("name"), "exported")
)
except Exception as exc:
report_items.append(
ResourceReportItem(ResourceType.WORKFLOW_TOOL, provider_id, provider_id, "unresolved", str(exc))
)
def _get_tenant_owner(self, tenant_id: str) -> Account | None:
return db.session.scalar(
sa.select(Account)
.join(TenantAccountJoin, Account.id == TenantAccountJoin.account_id)
.where(TenantAccountJoin.tenant_id == tenant_id, TenantAccountJoin.role == "owner")
.order_by(TenantAccountJoin.created_at.asc())
.limit(1)
)
def _export_mcp_tools(
self,
*,
tenant_id: str,
provider_ids: Iterable[str],
include_secrets: bool,
exported_mcp_tools: list[dict[str, Any]],
dependencies: list[dict[str, Any]],
report_items: list[ResourceReportItem],
) -> None:
for provider_id in self._dedupe(provider_ids):
if not include_secrets:
self._record_dependency_metadata(
[DiscoveredDependency(DependencyKind.MCP_TOOL, provider_id, source="mcp_provider")],
dependencies,
report_items,
)
continue
try:
provider = self._get_mcp_provider(tenant_id, provider_id)
exported_mcp_tools.append(self._serialize_mcp_provider(provider))
report_items.append(ResourceReportItem(ResourceType.MCP_TOOL, provider_id, provider.name, "exported"))
except Exception as exc:
report_items.append(
ResourceReportItem(ResourceType.MCP_TOOL, provider_id, provider_id, "unresolved", str(exc))
)
def _get_mcp_provider(self, tenant_id: str, provider_id: str) -> MCPToolProvider:
predicates = [MCPToolProvider.server_identifier == provider_id]
if self._is_uuid_string(provider_id):
predicates.append(MCPToolProvider.id == provider_id)
provider = db.session.scalar(
sa.select(MCPToolProvider).where(MCPToolProvider.tenant_id == tenant_id, sa.or_(*predicates))
)
if provider is None:
raise MigrationDataError(f"MCP provider not found: {provider_id}")
return provider
def _is_uuid_string(self, value: str) -> bool:
try:
UUID(value)
except ValueError:
return False
return True
def _serialize_mcp_provider(self, provider: MCPToolProvider) -> dict[str, Any]:
provider_entity = provider.to_entity()
provider_icon = provider_entity.provider_icon
if isinstance(provider_icon, dict):
icon = provider_icon.get("content")
icon_background = provider_icon.get("background")
icon_type = "emoji"
else:
icon = provider_icon
icon_background = None
icon_type = "url"
return {
"id": provider.id,
"name": provider.name,
"server_url": provider_entity.decrypt_server_url(),
"server_identifier": provider.server_identifier,
"icon": icon,
"icon_background": icon_background,
"icon_type": icon_type,
"configuration": {"timeout": provider.timeout, "sse_read_timeout": provider.sse_read_timeout},
"headers": provider_entity.decrypt_headers(),
"authentication": self._serialize_mcp_authentication(provider_entity.decrypt_authentication()),
"tools": provider.tool_dict,
"source_tenant_id": provider.tenant_id,
}
def _serialize_mcp_authentication(self, authentication: dict[str, Any] | None) -> dict[str, Any] | None:
if not authentication or not authentication.get("client_id"):
return None
return {
"client_id": authentication["client_id"],
"client_secret": authentication.get("client_secret"),
}
def _record_dependency_metadata(
self,
dependencies_to_record: Iterable[DiscoveredDependency],
dependencies: list[dict[str, Any]],
report_items: list[ResourceReportItem],
) -> None:
existing = {(item.get("kind"), item.get("provider_id")) for item in dependencies}
for dependency in dependencies_to_record:
key = (dependency.kind.value, dependency.provider_id)
if key in existing:
continue
existing.add(key)
dependencies.append(
{
"kind": dependency.kind.value,
"provider_id": dependency.provider_id,
"provider_name": dependency.provider_name,
"source": dependency.source,
}
)
report_items.append(
ResourceReportItem(
ResourceType.DEPENDENCY,
dependency.provider_id,
self._dependency_report_name(dependency),
"dependency-only",
self._dependency_message(dependency.kind),
)
)
def _provider_ids(
self,
manual_provider_ids: Iterable[str],
discovered_dependencies: Iterable[DiscoveredDependency],
kind: DependencyKind,
) -> list[str]:
provider_ids = list(manual_provider_ids)
provider_ids.extend(
self._provider_export_identifier(dependency)
for dependency in discovered_dependencies
if dependency.kind == kind
)
return self._dedupe(provider_ids)
def _provider_export_identifier(self, dependency: DiscoveredDependency) -> str:
if dependency.kind == DependencyKind.API_TOOL and dependency.provider_name:
return dependency.provider_name
return dependency.provider_id
def _dependencies_by_kind(
self, discovered_dependencies: Iterable[DiscoveredDependency], kind: DependencyKind
) -> list[DiscoveredDependency]:
return [dependency for dependency in discovered_dependencies if dependency.kind == kind]
def _dedupe(self, values: Iterable[str]) -> list[str]:
seen: set[str] = set()
result: list[str] = []
for value in values:
if value and value not in seen:
seen.add(value)
result.append(value)
return result
def _dependency_message(self, kind: DependencyKind) -> str:
if kind == DependencyKind.MCP_TOOL:
return "Configure MCP provider manually in the target tenant unless exporting with secrets enabled."
if kind == DependencyKind.BUILTIN_OR_PLUGIN_TOOL:
return "Ensure the built-in or plugin tool exists in the target environment."
return "Dependency metadata only; ensure the resource exists in the target environment."
def _dependency_report_name(self, dependency: DiscoveredDependency) -> str:
name = dependency.provider_name or dependency.provider_id
if dependency.kind == DependencyKind.WORKFLOW_TOOL:
return f"workflow {name}"
return f"{dependency.kind.value} {name}"
@@ -0,0 +1,938 @@
"""Apply versioned migration packages to an explicitly resolved target tenant.
Import target resolution is deliberately performed before any resource import
work. The service does not write Click output; callers receive structured
report items and can decide how to render them.
"""
from __future__ import annotations
import json
from collections.abc import Iterable
from dataclasses import dataclass
from typing import Any, cast
from uuid import UUID
import sqlalchemy as sa
import yaml
from sqlalchemy import or_
from sqlalchemy.orm import Session, sessionmaker
from core.entities.mcp_provider import MCPAuthentication, MCPConfiguration
from core.tools.entities.tool_entities import ApiProviderSchemaType, WorkflowToolParameterConfiguration
from extensions.ext_database import db
from libs.datetime_utils import naive_utc_now
from models import Account, ApiToken, Tenant, TenantAccountJoin, TenantAccountRole
from models.enums import ApiTokenType
from models.model import App
from models.tools import ApiToolProvider, MCPToolProvider, WorkflowToolProvider
from services.app_dsl_service import AppDslService
from services.data_migration.dependency_discovery_service import DependencyDiscoveryService
from services.data_migration.entities import (
ConflictStrategy,
DependencyKind,
IdStrategy,
ImportOptions,
ImportResult,
ImportTarget,
MigrationDataError,
MigrationPackage,
ReportContext,
ResourceIdMapping,
ResourceReportItem,
ResourceType,
)
from services.entities.dsl_entities import ImportStatus
from services.tools.api_tools_manage_service import ApiToolManageService
from services.tools.mcp_tools_manage_service import MCPToolManageService
from services.tools.workflow_tools_manage_service import WorkflowToolManageService
from services.workflow_service import WorkflowService
@dataclass(frozen=True)
class ImportRequest:
"""Structured input for package import.
`cli_target_tenant` and `config_target_tenant` are target tenant names from
outer adapters. They intentionally override package metadata, because a
migration package may be reused across environments.
"""
package: MigrationPackage
cli_target_tenant: str | None = None
config_target_tenant: str | None = None
operator_email: str | None = None
options_override: ImportOptions | None = None
class ImportTargetResolver:
"""Resolve the target tenant and operator before import side effects begin."""
def select_target_tenant_name(self, request: ImportRequest) -> str:
if request.cli_target_tenant:
return request.cli_target_tenant
if request.config_target_tenant:
return request.config_target_tenant
package_target = request.package.metadata.target_tenant or {}
if package_target.get("name"):
return package_target["name"]
if package_target.get("id"):
return package_target["id"]
raise MigrationDataError(
"Target tenant must be provided by --target-tenant, import config, or package metadata."
)
def resolve(self, request: ImportRequest) -> ImportTarget:
target_tenant_name = self.select_target_tenant_name(request)
package_target = request.package.metadata.target_tenant or {}
if request.cli_target_tenant or request.config_target_tenant:
tenant = self._resolve_tenant_by_id_or_name(target_tenant_name)
elif package_target.get("id") and self._is_uuid(package_target["id"]):
tenant = db.session.get(Tenant, package_target["id"])
if tenant is not None and package_target.get("name") and tenant.name != package_target.get("name"):
raise MigrationDataError(
f"Target tenant id/name mismatch: {package_target['id']} / {package_target['name']}"
)
else:
tenant = self._resolve_tenant_by_id_or_name(target_tenant_name)
if tenant is None:
raise MigrationDataError(f"Target tenant not found: {target_tenant_name}")
account_query = (
db.session.query(Account)
.join(TenantAccountJoin, Account.id == TenantAccountJoin.account_id)
.filter(TenantAccountJoin.tenant_id == tenant.id)
)
if request.operator_email:
account_query = account_query.filter(Account.email == request.operator_email)
identity = request.operator_email
else:
account_query = account_query.filter(TenantAccountJoin.role == TenantAccountRole.OWNER).order_by(
TenantAccountJoin.created_at.asc()
)
identity = "earliest owner"
account = account_query.first()
if account is None:
raise MigrationDataError(f"No operator account found for target tenant {target_tenant_name}: {identity}")
return ImportTarget(
tenant_id=tenant.id,
tenant_name=tenant.name,
operator_id=account.id,
operator_email=account.email,
)
def _resolve_tenant_by_id_or_name(self, value: str) -> Tenant | None:
if self._is_uuid(value):
tenant = db.session.get(Tenant, value)
if tenant is not None:
return tenant
tenants = list(db.session.scalars(sa.select(Tenant).where(Tenant.name == value)).all())
if len(tenants) > 1:
raise MigrationDataError(f"Target tenant name is ambiguous; use target_tenant.id: {value}")
return tenants[0] if tenants else None
def _is_uuid(self, value: str) -> bool:
try:
UUID(value)
except ValueError:
return False
return True
class MigrationImportService:
"""Apply package resources using Dify service APIs and structured reporting."""
target_resolver: ImportTargetResolver
def __init__(self, *, target_resolver: ImportTargetResolver | None = None) -> None:
self.target_resolver = target_resolver or ImportTargetResolver()
def import_package(self, request: ImportRequest) -> ImportResult:
target = self.target_resolver.resolve(request)
options = request.options_override or request.package.metadata.import_options
report_items = [
ResourceReportItem(
resource_type=ResourceType.DEPENDENCY,
identifier=target.tenant_id,
name=target.tenant_name,
status="resolved",
message=f"operator: {target.operator_email or target.operator_id}",
)
]
id_mapping: dict[str, str] = {}
id_mapping_details: list[ResourceIdMapping] = []
self._import_api_tools(
request.package,
target,
options,
report_items,
id_mapping,
id_mapping_details,
self._source_api_provider_ids_by_name(request.package),
)
self._import_mcp_tools(request.package, target, options, report_items, id_mapping, id_mapping_details)
self._preflight_dependency_only_mcp(request.package, target, report_items)
workflow_tool_app_ids = self._workflow_tool_source_app_ids(request.package)
imported_workflow_ids: set[str] = set()
if workflow_tool_app_ids:
self._import_workflows(
request.package,
target,
options,
report_items,
id_mapping,
id_mapping_details=id_mapping_details,
imported_workflow_ids=imported_workflow_ids,
only_app_ids=workflow_tool_app_ids,
)
self._import_workflow_tools(request.package, target, options, id_mapping, id_mapping_details, report_items)
self._import_workflows(
request.package,
target,
options,
report_items,
id_mapping,
id_mapping_details=id_mapping_details,
imported_workflow_ids=imported_workflow_ids,
skip_app_ids=imported_workflow_ids,
)
return ImportResult(
report_items=report_items,
id_mapping=id_mapping,
report_context=ReportContext(
target_tenant=target.tenant_name,
operator_email=target.operator_email,
id_mapping_count=len(id_mapping),
id_mappings=dict(id_mapping),
id_mapping_details=id_mapping_details,
),
)
def _import_workflows(
self,
package: MigrationPackage,
target: ImportTarget,
options: ImportOptions,
report_items: list[ResourceReportItem],
id_mapping: dict[str, str],
id_mapping_details: list[ResourceIdMapping],
imported_workflow_ids: set[str] | None = None,
only_app_ids: set[str] | None = None,
skip_app_ids: set[str] | None = None,
) -> None:
account = db.session.get(Account, target.operator_id)
tenant = db.session.get(Tenant, target.tenant_id)
if account is None:
raise MigrationDataError(f"Operator account not found: {target.operator_id}")
if tenant is None:
raise MigrationDataError(f"Target tenant not found: {target.tenant_id}")
account.current_tenant = tenant
for workflow_data in package.workflows:
app_id = self._optional_string(workflow_data.get("id"))
if only_app_ids and app_id not in only_app_ids:
continue
if skip_app_ids and app_id in skip_app_ids:
continue
dsl_content = self._rewrite_workflow_dsl_provider_ids(
self._required_string(workflow_data, "dsl", "workflow"),
id_mapping,
)
existing_app = (
self._find_existing_app(app_id, target.tenant_id)
if options.id_strategy == IdStrategy.PRESERVE_ID
else None
)
if existing_app is not None and options.conflict_strategy == ConflictStrategy.FAIL:
raise MigrationDataError(f"App already exists and conflict_strategy=fail: {app_id}")
if existing_app is not None and options.conflict_strategy == ConflictStrategy.SKIP:
if app_id:
self._record_id_mappings(
id_mapping,
id_mapping_details,
ResourceType.WORKFLOW,
workflow_data.get("name") if isinstance(workflow_data.get("name"), str) else None,
{app_id},
existing_app.id,
)
report_items.append(
ResourceReportItem(ResourceType.WORKFLOW, str(app_id), workflow_data.get("name"), "skipped")
)
continue
imported_app_id = self._import_workflow_app(
account=account,
workflow_data=workflow_data,
dsl_content=dsl_content,
app_id=app_id,
existing_app=existing_app,
options=options,
)
if app_id:
self._record_id_mappings(
id_mapping,
id_mapping_details,
ResourceType.WORKFLOW,
workflow_data.get("name") if isinstance(workflow_data.get("name"), str) else None,
{app_id},
imported_app_id,
)
if imported_workflow_ids is not None:
imported_workflow_ids.add(app_id)
if options.create_app_api_token_on_import:
self._create_or_reuse_app_api_token(imported_app_id, target.tenant_id)
report_items.append(
ResourceReportItem(
ResourceType.WORKFLOW,
imported_app_id,
workflow_data.get("name"),
"updated" if existing_app is not None else "created",
)
)
def _workflow_tool_source_app_ids(self, package: MigrationPackage) -> set[str]:
app_ids: set[str] = set()
for workflow_tool_data in package.workflow_tools:
app_id = self._optional_string(workflow_tool_data.get("app_id"))
if app_id:
app_ids.add(app_id)
return app_ids
def _import_workflow_app(
self,
*,
account: Account,
workflow_data: dict[str, object],
dsl_content: str,
app_id: str | None,
existing_app: App | None,
options: ImportOptions,
) -> str:
import_service = AppDslService(cast(Session, db.session))
if existing_app is not None:
import_result = import_service.import_app(
account=account,
import_mode="yaml-content",
yaml_content=dsl_content,
app_id=existing_app.id,
)
else:
import_app_id = app_id if self._should_preserve_source_app_id(options) else None
import_result = import_service.import_app(
account=account,
import_mode="yaml-content",
yaml_content=dsl_content,
import_app_id=import_app_id,
)
if import_result.status not in {ImportStatus.COMPLETED, ImportStatus.COMPLETED_WITH_WARNINGS}:
error = import_result.error or f"unexpected import status {import_result.status}"
raise MigrationDataError(f"Workflow import failed: {error}")
if import_result.app_id is None:
raise MigrationDataError(f"Workflow import did not return an app id: {workflow_data.get('name')}")
db.session.commit()
return import_result.app_id
def _rewrite_workflow_dsl_provider_ids(self, dsl_content: str, id_mapping: dict[str, str]) -> str:
if not id_mapping:
return dsl_content
parsed = yaml.safe_load(dsl_content) if dsl_content else {}
if not isinstance(parsed, dict):
return dsl_content
for node in self._workflow_nodes(parsed):
data = node.get("data") if isinstance(node, dict) else None
if not isinstance(data, dict):
continue
self._rewrite_tool_config_provider_id(data, id_mapping)
for tool_config in self._agent_tool_configs(data):
self._rewrite_tool_config_provider_id(tool_config, id_mapping)
return yaml.safe_dump(parsed, sort_keys=False, allow_unicode=True)
def _rewrite_tool_config_provider_id(self, tool_config: dict[str, Any], id_mapping: dict[str, str]) -> None:
provider_id = self._optional_string(tool_config.get("provider_id"))
if provider_id and provider_id in id_mapping:
tool_config["provider_id"] = id_mapping[provider_id]
def _source_api_provider_ids_by_name(self, package: MigrationPackage) -> dict[str, set[str]]:
provider_ids_by_name: dict[str, set[str]] = {}
discovery_service = DependencyDiscoveryService()
for workflow_data in package.workflows:
dsl_content = self._optional_string(workflow_data.get("dsl"))
if not dsl_content:
continue
parsed = yaml.safe_load(dsl_content) if dsl_content else {}
if not isinstance(parsed, dict):
continue
for dependency in discovery_service.discover_from_dsl(parsed):
if dependency.kind != DependencyKind.API_TOOL or not dependency.provider_name:
continue
provider_ids_by_name.setdefault(dependency.provider_name, set()).add(dependency.provider_id)
return provider_ids_by_name
def _workflow_nodes(self, dsl: dict[str, Any]) -> list[dict[str, Any]]:
nodes: list[dict[str, Any]] = []
graph = dsl.get("graph")
if isinstance(graph, dict) and isinstance(graph.get("nodes"), list):
nodes.extend(node for node in graph["nodes"] if isinstance(node, dict))
workflow = dsl.get("workflow")
workflow_graph = workflow.get("graph") if isinstance(workflow, dict) else None
if isinstance(workflow_graph, dict) and isinstance(workflow_graph.get("nodes"), list):
nodes.extend(node for node in workflow_graph["nodes"] if isinstance(node, dict))
return nodes
def _agent_tool_configs(self, data: dict[str, Any]) -> list[dict[str, Any]]:
configs = data.get("tools")
if isinstance(configs, list):
return [config for config in configs if isinstance(config, dict)]
agent_parameters = data.get("agent_parameters")
if not isinstance(agent_parameters, dict):
return []
tools_parameter = agent_parameters.get("tools")
if not isinstance(tools_parameter, dict):
return []
value = tools_parameter.get("value", [])
if not isinstance(value, list):
return []
return [config for config in value if isinstance(config, dict)]
def _should_preserve_source_app_id(self, options: ImportOptions) -> bool:
return options.id_strategy == IdStrategy.PRESERVE_ID
def _find_existing_app(self, app_id: str | None, tenant_id: str) -> App | None:
if not self._is_uuid_string(app_id):
return None
return db.session.scalar(sa.select(App).where(App.id == app_id, App.tenant_id == tenant_id))
def _create_or_reuse_app_api_token(self, app_id: str, tenant_id: str) -> None:
existing = db.session.scalar(
sa.select(ApiToken).where(
ApiToken.type == ApiTokenType.APP,
ApiToken.app_id == app_id,
ApiToken.tenant_id == tenant_id,
)
)
if existing is not None:
return
api_token = ApiToken()
api_token.app_id = app_id
api_token.tenant_id = tenant_id
api_token.token = ApiToken.generate_api_key("app", 24)
api_token.type = ApiTokenType.APP
db.session.add(api_token)
db.session.commit()
def _import_api_tools(
self,
package: MigrationPackage,
target: ImportTarget,
options: ImportOptions,
report_items: list[ResourceReportItem],
id_mapping: dict[str, str],
id_mapping_details: list[ResourceIdMapping],
source_provider_ids_by_name: dict[str, set[str]],
) -> None:
for tool_data in package.tools:
provider_name = self._required_string(tool_data, "provider_name", "api_tool")
schema = self._required_string(tool_data, "schema", "api_tool")
existing = db.session.scalar(
sa.select(ApiToolProvider).where(
ApiToolProvider.tenant_id == target.tenant_id,
ApiToolProvider.name == provider_name,
)
)
if existing is not None and options.conflict_strategy == ConflictStrategy.FAIL:
raise MigrationDataError(f"API tool already exists and conflict_strategy=fail: {provider_name}")
if existing is not None and options.conflict_strategy == ConflictStrategy.SKIP:
self._record_id_mappings(
id_mapping,
id_mapping_details,
ResourceType.API_TOOL,
provider_name,
self._api_tool_source_ids(provider_name, tool_data, source_provider_ids_by_name),
existing.id,
)
report_items.append(ResourceReportItem(ResourceType.API_TOOL, provider_name, provider_name, "skipped"))
continue
schema_info = ApiToolManageService.parser_api_schema(schema=schema)
schema_type = cast(ApiProviderSchemaType, schema_info["schema_type"])
credentials = (
cast(dict[str, Any], tool_data.get("credentials"))
if isinstance(tool_data.get("credentials"), dict)
else {}
)
credentials = credentials or {"auth_type": "none"}
raw_icon = tool_data.get("icon")
icon = (
cast(dict[str, Any], raw_icon)
if isinstance(raw_icon, dict)
else {"content": "tool", "background": "#FEF7C3"}
)
raw_labels = tool_data.get("labels")
labels = [str(label) for label in raw_labels] if isinstance(raw_labels, list) else []
if existing is not None:
ApiToolManageService.update_api_tool_provider(
user_id=target.operator_id,
tenant_id=target.tenant_id,
provider_name=provider_name,
original_provider=existing.name,
_schema_type=schema_type,
schema=schema,
privacy_policy=self._optional_string(tool_data.get("privacy_policy")) or "",
credentials=credentials,
custom_disclaimer=self._optional_string(tool_data.get("custom_disclaimer")) or "",
labels=labels,
icon=icon,
)
status = "updated"
else:
ApiToolManageService.create_api_tool_provider(
user_id=target.operator_id,
tenant_id=target.tenant_id,
provider_name=provider_name,
schema_type=schema_type,
schema=schema,
privacy_policy=self._optional_string(tool_data.get("privacy_policy")) or "",
credentials=credentials,
custom_disclaimer=self._optional_string(tool_data.get("custom_disclaimer")) or "",
labels=labels,
icon=icon,
)
status = "created"
target_provider = self._find_api_tool_provider(target.tenant_id, provider_name)
if target_provider is not None:
self._record_id_mappings(
id_mapping,
id_mapping_details,
ResourceType.API_TOOL,
provider_name,
self._api_tool_source_ids(provider_name, tool_data, source_provider_ids_by_name),
target_provider.id,
)
report_items.append(ResourceReportItem(ResourceType.API_TOOL, provider_name, provider_name, status))
def _find_api_tool_provider(self, tenant_id: str, provider_name: str) -> ApiToolProvider | None:
return db.session.scalar(
sa.select(ApiToolProvider).where(
ApiToolProvider.tenant_id == tenant_id,
ApiToolProvider.name == provider_name,
)
)
def _api_tool_source_ids(
self,
provider_name: str,
tool_data: dict[str, Any],
source_provider_ids_by_name: dict[str, set[str]],
) -> set[str]:
source_ids = set(source_provider_ids_by_name.get(provider_name, set()))
source_id = self._optional_string(tool_data.get("id"))
if source_id:
source_ids.add(source_id)
return source_ids
def _record_id_mappings(
self,
id_mapping: dict[str, str],
id_mapping_details: list[ResourceIdMapping],
resource_type: ResourceType,
name: str | None,
source_ids: Iterable[str],
target_id: str,
) -> None:
for source_id in source_ids:
id_mapping[source_id] = target_id
id_mapping_details[:] = [item for item in id_mapping_details if item.source_id != source_id]
id_mapping_details.append(ResourceIdMapping(resource_type, name, source_id, target_id))
def _import_workflow_tools(
self,
package: MigrationPackage,
target: ImportTarget,
options: ImportOptions,
id_mapping: dict[str, str],
id_mapping_details: list[ResourceIdMapping],
report_items: list[ResourceReportItem],
) -> None:
if not package.workflow_tools:
return
account = db.session.get(Account, target.operator_id)
if account is None:
raise MigrationDataError(f"Operator account not found: {target.operator_id}")
for workflow_tool_data in package.workflow_tools:
app_id = self._optional_string(workflow_tool_data.get("app_id"))
resolved_app_id = id_mapping.get(app_id or "", app_id)
if not resolved_app_id or self._find_existing_app(resolved_app_id, target.tenant_id) is None:
report_items.append(
ResourceReportItem(
ResourceType.WORKFLOW_TOOL,
str(workflow_tool_data.get("id", workflow_tool_data.get("name", ""))),
self._optional_string(workflow_tool_data.get("name")),
"unresolved",
"Referenced workflow app was not found in the target tenant; workflow tool was skipped.",
)
)
continue
try:
self._ensure_workflow_app_is_published(target, account, resolved_app_id)
except Exception as exc:
report_items.append(
ResourceReportItem(
ResourceType.WORKFLOW_TOOL,
str(workflow_tool_data.get("id", workflow_tool_data.get("name", ""))),
self._optional_string(workflow_tool_data.get("name")),
"unresolved",
f"Referenced workflow app could not be published: {exc}",
)
)
continue
workflow_tool_id = self._optional_string(workflow_tool_data.get("id"))
tool_name = self._required_string(workflow_tool_data, "name", "workflow_tool")
lookup_workflow_tool_id = workflow_tool_id if options.id_strategy == IdStrategy.PRESERVE_ID else None
existing = self._find_existing_workflow_tool(
target.tenant_id, lookup_workflow_tool_id, tool_name, resolved_app_id
)
if existing is not None and options.conflict_strategy == ConflictStrategy.FAIL:
raise MigrationDataError(f"Workflow tool already exists and conflict_strategy=fail: {tool_name}")
if existing is not None and options.conflict_strategy == ConflictStrategy.SKIP:
if workflow_tool_id:
self._record_id_mappings(
id_mapping,
id_mapping_details,
ResourceType.WORKFLOW_TOOL,
tool_name,
{workflow_tool_id},
existing.id,
)
report_items.append(ResourceReportItem(ResourceType.WORKFLOW_TOOL, existing.id, tool_name, "skipped"))
continue
raw_icon = workflow_tool_data.get("icon")
icon = (
cast(dict[str, Any], raw_icon)
if isinstance(raw_icon, dict)
else {"content": "🤖", "background": "#FFEAD5"}
)
raw_parameters = workflow_tool_data.get("parameters")
parameters = [
parameter
if isinstance(parameter, WorkflowToolParameterConfiguration)
else WorkflowToolParameterConfiguration(**parameter)
for parameter in (raw_parameters if isinstance(raw_parameters, list) else [])
if isinstance(parameter, dict | WorkflowToolParameterConfiguration)
]
raw_labels = workflow_tool_data.get("labels")
labels = [str(label) for label in raw_labels] if isinstance(raw_labels, list) else []
label = self._optional_string(workflow_tool_data.get("label")) or tool_name
description = self._optional_string(workflow_tool_data.get("description")) or ""
privacy_policy = self._optional_string(workflow_tool_data.get("privacy_policy")) or ""
if existing is not None:
WorkflowToolManageService.update_workflow_tool(
user_id=account.id,
tenant_id=target.tenant_id,
workflow_tool_id=existing.id,
name=tool_name,
label=label,
icon=icon,
description=description,
parameters=parameters,
privacy_policy=privacy_policy,
labels=labels,
)
status = "updated"
identifier = existing.id
else:
import_id = workflow_tool_id if options.id_strategy == IdStrategy.PRESERVE_ID else ""
WorkflowToolManageService.create_workflow_tool(
user_id=account.id,
tenant_id=target.tenant_id,
workflow_app_id=resolved_app_id,
name=tool_name,
label=label,
icon=icon,
description=description,
parameters=parameters,
privacy_policy=privacy_policy,
labels=labels,
import_id=import_id or "",
)
status = "created"
target_provider = self._find_existing_workflow_tool(
target.tenant_id, import_id or None, tool_name, resolved_app_id
)
if target_provider is None:
raise MigrationDataError(f"Workflow tool was not created: {tool_name}")
identifier = target_provider.id
if workflow_tool_id:
self._record_id_mappings(
id_mapping,
id_mapping_details,
ResourceType.WORKFLOW_TOOL,
tool_name,
{workflow_tool_id},
identifier,
)
report_items.append(ResourceReportItem(ResourceType.WORKFLOW_TOOL, identifier, tool_name, status))
def _ensure_workflow_app_is_published(self, target: ImportTarget, account: Account, app_id: str) -> None:
app = self._find_existing_app(app_id, target.tenant_id)
if app is None:
raise MigrationDataError(f"Referenced workflow app was not found in target tenant: {app_id}")
if app.workflow_id:
return
workflow_service = WorkflowService()
with sessionmaker(db.engine).begin() as session:
app_in_session = session.get(App, app_id)
account_in_session = session.get(Account, account.id)
if app_in_session is None:
raise MigrationDataError(f"Referenced workflow app was not found in target tenant: {app_id}")
if account_in_session is None:
raise MigrationDataError(f"Operator account not found: {account.id}")
workflow = workflow_service.publish_workflow(
session=session,
app_model=app_in_session,
account=account_in_session,
marked_name="Migration import",
marked_comment="Published automatically for workflow tool import.",
)
app_in_session.workflow_id = workflow.id
app_in_session.updated_by = account.id
app_in_session.updated_at = naive_utc_now()
def _import_mcp_tools(
self,
package: MigrationPackage,
target: ImportTarget,
options: ImportOptions,
report_items: list[ResourceReportItem],
id_mapping: dict[str, str],
id_mapping_details: list[ResourceIdMapping],
) -> None:
for mcp_data in package.mcp_tools:
name = self._required_string(mcp_data, "name", "mcp_tool")
server_identifier = self._required_string(mcp_data, "server_identifier", "mcp_tool")
provider_id = self._optional_string(mcp_data.get("id"))
lookup_provider_id = provider_id if options.id_strategy == IdStrategy.PRESERVE_ID else None
existing = self._find_existing_mcp_tool(target.tenant_id, lookup_provider_id, server_identifier)
if existing is not None and options.conflict_strategy == ConflictStrategy.FAIL:
raise MigrationDataError(f"MCP tool already exists and conflict_strategy=fail: {name}")
if existing is not None and options.conflict_strategy == ConflictStrategy.SKIP:
if provider_id:
self._record_id_mappings(
id_mapping,
id_mapping_details,
ResourceType.MCP_TOOL,
name,
{provider_id},
existing.id,
)
report_items.append(ResourceReportItem(ResourceType.MCP_TOOL, existing.id, name, "skipped"))
continue
service = MCPToolManageService(session=cast(Session, db.session))
configuration = MCPConfiguration.model_validate(mcp_data.get("configuration") or {})
authentication = (
MCPAuthentication.model_validate(mcp_data["authentication"]) if mcp_data.get("authentication") else None
)
if existing is not None:
service.update_provider(
tenant_id=target.tenant_id,
provider_id=existing.id,
server_url=self._required_string(mcp_data, "server_url", "mcp_tool"),
name=name,
icon=self._optional_string(mcp_data.get("icon")) or "",
icon_type=self._optional_string(mcp_data.get("icon_type")) or "emoji",
icon_background=self._optional_string(mcp_data.get("icon_background")) or "",
server_identifier=server_identifier,
headers=mcp_data.get("headers") if isinstance(mcp_data.get("headers"), dict) else {},
configuration=configuration,
authentication=authentication,
)
db.session.commit()
status = "updated"
identifier = existing.id
provider = existing
else:
service.create_provider(
tenant_id=target.tenant_id,
user_id=target.operator_id,
server_url=self._required_string(mcp_data, "server_url", "mcp_tool"),
name=name,
icon=self._optional_string(mcp_data.get("icon")) or "",
icon_type=self._optional_string(mcp_data.get("icon_type")) or "emoji",
icon_background=self._optional_string(mcp_data.get("icon_background")) or "",
server_identifier=server_identifier,
headers=mcp_data.get("headers") if isinstance(mcp_data.get("headers"), dict) else {},
configuration=configuration,
authentication=authentication,
)
created_provider = self._find_existing_mcp_tool(target.tenant_id, lookup_provider_id, server_identifier)
if created_provider is None:
raise MigrationDataError(f"MCP provider was not created: {name}")
status = "created"
provider = created_provider
identifier = provider.id
self._restore_mcp_provider_tools(provider, mcp_data)
db.session.commit()
if provider_id:
self._record_id_mappings(
id_mapping,
id_mapping_details,
ResourceType.MCP_TOOL,
name,
{provider_id},
identifier,
)
report_items.append(ResourceReportItem(ResourceType.MCP_TOOL, identifier, name, status))
def _restore_mcp_provider_tools(self, provider: MCPToolProvider, mcp_data: dict[str, object]) -> None:
tools = mcp_data.get("tools")
if not isinstance(tools, list):
return
provider.tools = json.dumps(tools)
provider.authed = True
def _find_existing_mcp_tool(
self, tenant_id: str, provider_id: str | None, server_identifier: str
) -> MCPToolProvider | None:
predicates = [MCPToolProvider.server_identifier == server_identifier]
if self._is_uuid_string(provider_id):
predicates.append(MCPToolProvider.id == provider_id)
return db.session.scalar(
sa.select(MCPToolProvider).where(MCPToolProvider.tenant_id == tenant_id, or_(*predicates)).limit(1)
)
def _is_uuid_string(self, value: str | None) -> bool:
if not value:
return False
try:
UUID(value)
except ValueError:
return False
return True
def _find_existing_workflow_tool(
self, tenant_id: str, workflow_tool_id: str | None, tool_name: str, app_id: str
) -> WorkflowToolProvider | None:
predicates = [WorkflowToolProvider.name == tool_name, WorkflowToolProvider.app_id == app_id]
if self._is_uuid_string(workflow_tool_id):
predicates.append(WorkflowToolProvider.id == workflow_tool_id)
return db.session.scalar(
sa.select(WorkflowToolProvider)
.where(WorkflowToolProvider.tenant_id == tenant_id, or_(*predicates))
.limit(1)
)
def _preflight_dependency_only_mcp(
self, package: MigrationPackage, target: ImportTarget, report_items: list[ResourceReportItem]
) -> None:
for dependency in package.dependencies:
if dependency.get("kind") != DependencyKind.MCP_TOOL.value:
continue
provider_id = str(dependency.get("provider_id", dependency.get("id", "")))
provider_name = self._optional_string(dependency.get("provider_name") or dependency.get("name"))
existing = self._find_dependency_only_mcp_provider(target.tenant_id, provider_id, provider_name)
report_name = f"mcp_tool {provider_name or getattr(existing, 'name', None) or provider_id}"
if existing is not None:
report_items.append(
ResourceReportItem(
ResourceType.DEPENDENCY,
provider_id,
report_name,
"available",
"MCP provider exists in target tenant.",
)
)
continue
reference_summary = self._dependency_only_mcp_reference_summary(package, provider_id, provider_name)
message = "missing in target tenant"
if reference_summary:
message = f"{message}; referenced by {reference_summary}"
message = f"{message}; configure it manually before running the workflow."
report_items.append(
ResourceReportItem(
ResourceType.DEPENDENCY,
provider_id,
report_name,
"skipped",
message,
)
)
def _find_dependency_only_mcp_provider(
self, tenant_id: str, provider_id: str, provider_name: str | None
) -> MCPToolProvider | None:
predicates = [MCPToolProvider.server_identifier == provider_id]
if self._is_uuid_string(provider_id):
predicates.append(MCPToolProvider.id == provider_id)
return db.session.scalar(
sa.select(MCPToolProvider).where(MCPToolProvider.tenant_id == tenant_id, or_(*predicates)).limit(1)
)
def _dependency_only_mcp_reference_summary(
self, package: MigrationPackage, provider_id: str, provider_name: str | None
) -> str:
references = self._dependency_only_mcp_references(package, provider_id, provider_name)
return "; ".join(references)
def _dependency_only_mcp_references(
self, package: MigrationPackage, provider_id: str, provider_name: str | None
) -> list[str]:
references: list[str] = []
seen: set[str] = set()
for workflow_data in package.workflows:
workflow_name = self._optional_string(workflow_data.get("name"))
workflow_id = self._optional_string(workflow_data.get("id"))
workflow_label = workflow_name or workflow_id or "unknown workflow"
dsl_content = self._optional_string(workflow_data.get("dsl"))
if not dsl_content:
continue
parsed = yaml.safe_load(dsl_content) if dsl_content else {}
if not isinstance(parsed, dict):
continue
for node in self._workflow_nodes(parsed):
data = node.get("data") if isinstance(node, dict) else None
if not isinstance(data, dict):
continue
for tool_config in [data, *self._agent_tool_configs(data)]:
if not self._is_mcp_dependency_reference(tool_config, provider_id, provider_name):
continue
tool_label = self._mcp_tool_reference_label(node, tool_config)
reference = f"{workflow_label} / {tool_label}"
if reference not in seen:
seen.add(reference)
references.append(reference)
return references
def _is_mcp_dependency_reference(
self, tool_config: dict[str, Any], provider_id: str, provider_name: str | None
) -> bool:
provider_type = str(tool_config.get("provider_type") or tool_config.get("type") or "").lower()
if provider_type != "mcp":
return False
config_provider_id = self._optional_string(
tool_config.get("provider_id") or tool_config.get("provider_name") or tool_config.get("provider")
)
if config_provider_id == provider_id:
return True
return bool(provider_name and config_provider_id == provider_name)
def _mcp_tool_reference_label(self, node: dict[str, Any], tool_config: dict[str, Any]) -> str:
for key in ("tool_name", "tool", "name"):
value = self._optional_string(tool_config.get(key))
if value:
return value
node_id = self._optional_string(node.get("id"))
return node_id or "unknown tool"
def _required_string(self, value: dict[str, object], field_name: str, resource_name: str) -> str:
field_value = value.get(field_name)
if not isinstance(field_value, str) or not field_value:
raise MigrationDataError(f"Missing required {resource_name} field: {field_name}")
return field_value
def _optional_string(self, value: object) -> str | None:
if isinstance(value, str) and value:
return value
return None
@@ -0,0 +1,71 @@
"""JSON persistence for versioned cross-environment migration packages.
The package service validates file shape and serializes only structured package
entities. It does not perform CLI rendering or database access, keeping it safe
to reuse from Click adapters, tests, and future import/export services.
"""
from __future__ import annotations
import json
from dataclasses import asdict
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from services.data_migration.entities import (
ImportOptions,
MigrationDataError,
MigrationMetadata,
MigrationPackage,
SourceTenant,
TargetTenantSelector,
)
PACKAGE_VERSION = "1"
class MigrationPackageService:
def load_package(self, path: str | Path) -> MigrationPackage:
package_path = Path(path)
with package_path.open(encoding="utf-8") as file:
raw = json.load(file)
if not isinstance(raw, dict):
raise MigrationDataError("Migration package JSON must be an object.")
package = MigrationPackage.from_mapping(raw)
if package.metadata.version != PACKAGE_VERSION:
raise MigrationDataError(f"Unsupported migration package version: {package.metadata.version}")
return package
def save_package(self, package: MigrationPackage, path: str | Path, *, overwrite: bool) -> None:
package_path = Path(path)
if package_path.exists() and not overwrite:
raise MigrationDataError(f"Output file already exists: {package_path}")
package_path.parent.mkdir(parents=True, exist_ok=True)
with package_path.open("w", encoding="utf-8") as file:
json.dump(self.to_mapping(package), file, ensure_ascii=False, indent=2)
file.write("\n")
def build_empty_package(
self,
*,
source_tenant_id: str,
source_tenant_name: str,
include_secrets: bool,
import_options: ImportOptions | None = None,
target_tenant: TargetTenantSelector | None = None,
) -> MigrationPackage:
return MigrationPackage(
metadata=MigrationMetadata(
version=PACKAGE_VERSION,
source_scope="single",
source_tenants=[SourceTenant(id=source_tenant_id, name=source_tenant_name)],
target_tenant=target_tenant,
created_at=datetime.now(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z"),
include_secrets=include_secrets,
import_options=import_options or ImportOptions(),
)
)
def to_mapping(self, package: MigrationPackage) -> dict[str, Any]:
return asdict(package)
@@ -0,0 +1,76 @@
from __future__ import annotations
from collections import Counter
from services.data_migration.entities import ReportContext, ResourceIdMapping, ResourceReportItem
class MigrationReportService:
"""Render structured migration resource results into CLI-friendly summary lines."""
def render(self, items: list[ResourceReportItem], *, context: ReportContext | None = None) -> list[str]:
counts = Counter((item.resource_type.value, item.status) for item in items)
lines = self._render_context(context)
lines.extend(
[f"{resource_type} {status}: {count}" for (resource_type, status), count in sorted(counts.items())]
)
actionable_items = [
item for item in items if item.status in {"dependency-only", "skipped", "unresolved"} and item.message
]
for item in actionable_items:
lines.append(self._render_actionable_detail(item))
return lines
def _render_context(self, context: ReportContext | None) -> list[str]:
if context is None:
return []
lines: list[str] = []
if context.output_path:
lines.append(f"output: {context.output_path}")
if context.source_scope:
lines.append(f"source scope: {context.source_scope}")
if context.selected_app_count is not None:
lines.append(f"selected apps: {context.selected_app_count}")
if context.include_secrets is not None:
lines.append(f"include secrets: {str(context.include_secrets).lower()}")
if context.target_tenant:
lines.append(f"target tenant: {context.target_tenant}")
if context.operator_email:
lines.append(f"operator: {context.operator_email}")
if context.app_api_tokens_created or context.app_api_tokens_reused:
lines.append(
f"app api tokens: {context.app_api_tokens_created} created, {context.app_api_tokens_reused} reused"
)
if context.id_mappings:
lines.append(f"resource references resolved: {len(context.id_mappings)}")
if context.id_mapping_details:
lines.extend(
self._render_id_mapping_detail(item)
for item in sorted(
context.id_mapping_details,
key=lambda item: (item.resource_type.value, item.name or "", item.source_id),
)
)
else:
lines.extend(
f"- {source_id} -> {target_id}" for source_id, target_id in sorted(context.id_mappings.items())
)
elif context.id_mapping_count:
lines.append(f"resource references resolved: {context.id_mapping_count}")
return lines
def _render_id_mapping_detail(self, item: ResourceIdMapping) -> str:
label = item.resource_type.value
if item.name:
label = f"{label} {item.name}"
return f"- {label}: {item.source_id} -> {item.target_id}"
def _render_actionable_detail(self, item: ResourceReportItem) -> str:
if item.resource_type.value == "dependency" and item.name and self._has_dependency_type_prefix(item.name):
if item.identifier and item.identifier not in item.name:
return f"dependency {item.name}: {item.identifier}: {item.message}"
return f"dependency {item.name}: {item.message}"
return f"{item.resource_type.value} {item.identifier}: {item.message}"
def _has_dependency_type_prefix(self, name: str) -> bool:
return name.startswith(("workflow ", "api_tool ", "workflow_tool ", "mcp_tool ", "builtin_or_plugin_tool "))
+18 -18
View File
@@ -2329,15 +2329,15 @@ class DocumentService:
# if knowledge_config.data_source:
# if knowledge_config.data_source.info_list.data_source_type == "upload_file":
# upload_file_list = knowledge_config.data_source.info_list.file_info_list.file_ids
# # type: ignore
#
# count = len(upload_file_list)
# elif knowledge_config.data_source.info_list.data_source_type == "notion_import":
# notion_info_list = knowledge_config.data_source.info_list.notion_info_list
# for notion_info in notion_info_list: # type: ignore
# for notion_info in notion_info_list:
# count = count + len(notion_info.pages)
# elif knowledge_config.data_source.info_list.data_source_type == "website_crawl":
# website_info = knowledge_config.data_source.info_list.website_info_list
# count = len(website_info.urls) # type: ignore
# count = len(website_info.urls)
# batch_upload_limit = int(dify_config.BATCH_UPLOAD_LIMIT)
# if features.billing.subscription.plan == CloudPlan.SANDBOX and count > 1:
@@ -2349,7 +2349,7 @@ class DocumentService:
# # if dataset is empty, update dataset data_source_type
# if not dataset.data_source_type:
# dataset.data_source_type = knowledge_config.data_source.info_list.data_source_type # type: ignore
# dataset.data_source_type = knowledge_config.data_source.info_list.data_source_type
# if not dataset.indexing_technique:
# if knowledge_config.indexing_technique not in Dataset.INDEXING_TECHNIQUE_LIST:
@@ -2386,7 +2386,7 @@ class DocumentService:
# knowledge_config.retrieval_model.model_dump()
# if knowledge_config.retrieval_model
# else default_retrieval_model
# ) # type: ignore
# )
# documents = []
# if knowledge_config.original_document_id:
@@ -2425,8 +2425,8 @@ class DocumentService:
# position = DocumentService.get_documents_position(dataset.id)
# document_ids = []
# duplicate_document_ids = []
# if knowledge_config.data_source.info_list.data_source_type == "upload_file": # type: ignore
# upload_file_list = knowledge_config.data_source.info_list.file_info_list.file_ids # type: ignore
# if knowledge_config.data_source.info_list.data_source_type == "upload_file":
# upload_file_list = knowledge_config.data_source.info_list.file_info_list.file_ids
# for file_id in upload_file_list:
# file = (
# db.session.query(UploadFile)
@@ -2452,7 +2452,7 @@ class DocumentService:
# name=file_name,
# ).first()
# if document:
# document.dataset_process_rule_id = dataset_process_rule.id # type: ignore
# document.dataset_process_rule_id = dataset_process_rule.id
# document.updated_at = datetime.datetime.now(datetime.UTC).replace(tzinfo=None)
# document.created_from = created_from
# document.doc_form = knowledge_config.doc_form
@@ -2466,8 +2466,8 @@ class DocumentService:
# continue
# document = DocumentService.build_document(
# dataset,
# dataset_process_rule.id, # type: ignore
# knowledge_config.data_source.info_list.data_source_type, # type: ignore
# dataset_process_rule.id,
# knowledge_config.data_source.info_list.data_source_type,
# knowledge_config.doc_form,
# knowledge_config.doc_language,
# data_source_info,
@@ -2482,8 +2482,8 @@ class DocumentService:
# document_ids.append(document.id)
# documents.append(document)
# position += 1
# elif knowledge_config.data_source.info_list.data_source_type == "notion_import": # type: ignore
# notion_info_list = knowledge_config.data_source.info_list.notion_info_list # type: ignore
# elif knowledge_config.data_source.info_list.data_source_type == "notion_import":
# notion_info_list = knowledge_config.data_source.info_list.notion_info_list
# if not notion_info_list:
# raise ValueError("No notion info list found.")
# exist_page_ids = []
@@ -2523,8 +2523,8 @@ class DocumentService:
# truncated_page_name = page.page_name[:255] if page.page_name else "nopagename"
# document = DocumentService.build_document(
# dataset,
# dataset_process_rule.id, # type: ignore
# knowledge_config.data_source.info_list.data_source_type, # type: ignore
# dataset_process_rule.id,
# knowledge_config.data_source.info_list.data_source_type,
# knowledge_config.doc_form,
# knowledge_config.doc_language,
# data_source_info,
@@ -2544,8 +2544,8 @@ class DocumentService:
# # delete not selected documents
# if len(exist_document) > 0:
# clean_notion_document_task.delay(list(exist_document.values()), dataset.id)
# elif knowledge_config.data_source.info_list.data_source_type == "website_crawl": # type: ignore
# website_info = knowledge_config.data_source.info_list.website_info_list # type: ignore
# elif knowledge_config.data_source.info_list.data_source_type == "website_crawl":
# website_info = knowledge_config.data_source.info_list.website_info_list
# if not website_info:
# raise ValueError("No website info list found.")
# urls = website_info.urls
@@ -2563,8 +2563,8 @@ class DocumentService:
# document_name = url
# document = DocumentService.build_document(
# dataset,
# dataset_process_rule.id, # type: ignore
# knowledge_config.data_source.info_list.data_source_type, # type: ignore
# dataset_process_rule.id,
# knowledge_config.data_source.info_list.data_source_type,
# knowledge_config.doc_form,
# knowledge_config.doc_language,
# data_source_info,
+25
View File
@@ -12,8 +12,33 @@ from services.errors.enterprise import (
EnterpriseAPIForbiddenError,
EnterpriseAPINotFoundError,
EnterpriseAPIUnauthorizedError,
EnterpriseServiceError,
)
# M2 — IssueMCPToken specific errors. Co-located here (rather than in
# services/errors/enterprise.py) because services.enterprise.base is part of
# the leaf-mounted file set the local dev override applies; the errors module
# stays at the EE image's baked-in version.
class MCPTokenError(EnterpriseServiceError):
"""Generic failure of the IssueMCPToken RPC."""
class MCPNoRefreshTokenError(MCPTokenError):
"""The user has no stored SSO refresh_token on the enterprise side.
The workflow should ask them to re-authenticate."""
def __init__(self, description: str = ""):
super().__init__(description, status_code=428)
class MCPIdentityRefreshError(MCPTokenError):
"""The enterprise side tried to refresh the user's SSO refresh_token
against the IdP and failed (revoked/expired/IdP error)."""
def __init__(self, description: str = ""):
super().__init__(description, status_code=401)
logger = logging.getLogger(__name__)
+66 -1
View File
@@ -11,7 +11,16 @@ from pydantic import BaseModel, ConfigDict, Field, model_validator
from configs import dify_config
from extensions.ext_redis import redis_client
from services.enterprise.base import EnterpriseRequest
from services.enterprise.base import (
EnterpriseRequest,
MCPIdentityRefreshError,
MCPNoRefreshTokenError,
MCPTokenError,
)
from services.errors.enterprise import (
EnterpriseAPIError,
EnterpriseAPIUnauthorizedError,
)
if TYPE_CHECKING:
from services.feature_service import LicenseStatus
@@ -121,6 +130,62 @@ class EnterpriseService:
def get_workspace_info(cls, tenant_id: str):
return EnterpriseRequest.send_request("GET", f"/workspace/{tenant_id}/info")
@classmethod
def issue_mcp_token(
cls,
user_id: str,
tenant_id: str,
app_id: str | None,
audience: str,
) -> tuple[str, int]:
"""Mint a short-lived SSO id_token (or OAuth2 access_token) representing
the calling Dify user, audience-scoped to the given MCP server identifier.
Used by MCPTool.invoke_remote_mcp_tool to stamp `Authorization: Bearer
<token>` on outbound MCP requests when the provider has
forward_user_identity=True and identity_mode="idp_token".
Returns:
(token, expires_at_unix_seconds)
Raises:
MCPNoRefreshTokenError: user has no stored SSO refresh_token on the
enterprise side; surface to the workflow as "please log in via SSO".
MCPIdentityRefreshError: enterprise tried to refresh against the IdP
and the IdP rejected (revoked/expired session).
MCPTokenError: any other failure of the enterprise endpoint.
"""
try:
response = EnterpriseRequest.send_request(
"POST",
"/mcp/issue-token",
json={
"user_id": user_id,
"tenant_id": tenant_id,
"app_id": app_id or "",
"audience": audience,
},
)
except EnterpriseAPIUnauthorizedError as e:
# Enterprise side returns 401 when the IdP rejected the refresh.
raise MCPIdentityRefreshError(str(e) or "identity refresh failed; please re-authenticate") from e
except EnterpriseAPIError as e:
# Map the 428 PreconditionRequired we emit on no-stored-refresh-token.
if getattr(e, "status_code", None) == 428:
raise MCPNoRefreshTokenError(
str(e) or "user has no stored SSO refresh token; please re-authenticate"
) from e
raise MCPTokenError(f"issue_mcp_token failed: {e}") from e
if not isinstance(response, dict):
raise MCPTokenError("invalid response shape from enterprise /mcp/issue-token")
token = response.get("token")
expires_at = response.get("expires_at")
if not token or not isinstance(token, str) or not isinstance(expires_at, int):
raise MCPTokenError(f"missing token/expires_at in enterprise response: {response}")
return token, expires_at
@classmethod
def initiate_device_flow_sso(cls, signed_state: str) -> dict:
return EnterpriseRequest.send_request(
+15 -1
View File
@@ -4,7 +4,7 @@ import logging
from collections.abc import Mapping
from datetime import datetime
from enum import StrEnum
from typing import Any
from typing import Any, Literal
from urllib.parse import urlparse
from pydantic import BaseModel, Field
@@ -136,6 +136,8 @@ class MCPToolManageService:
configuration: MCPConfiguration,
authentication: MCPAuthentication | None = None,
headers: dict[str, str] | None = None,
forward_user_identity: bool = False,
identity_mode: Literal["off", "idp_token"] = "off",
) -> ToolProviderApiEntity:
"""Create a new MCP provider."""
# Validate URL format
@@ -171,6 +173,8 @@ class MCPToolManageService:
sse_read_timeout=configuration.sse_read_timeout,
encrypted_headers=encrypted_headers,
encrypted_credentials=encrypted_credentials,
forward_user_identity=forward_user_identity,
identity_mode=identity_mode,
)
self._session.add(mcp_tool)
@@ -194,6 +198,8 @@ class MCPToolManageService:
configuration: MCPConfiguration,
authentication: MCPAuthentication | None = None,
validation_result: ServerUrlValidationResult | None = None,
forward_user_identity: bool | None = None,
identity_mode: Literal["off", "idp_token"] | None = None,
) -> None:
"""
Update an MCP provider.
@@ -255,6 +261,14 @@ class MCPToolManageService:
if authentication and authentication.client_id:
mcp_provider.encrypted_credentials = self._process_credentials(authentication, mcp_provider, tenant_id)
# Update user-identity forwarding settings if provided.
# None means "leave unchanged" so this stays backwards-compatible
# with existing callers that don't know about M2.
if forward_user_identity is not None:
mcp_provider.forward_user_identity = forward_user_identity
if identity_mode is not None:
mcp_provider.identity_mode = identity_mode
# Flush changes to database
self._session.flush()
@@ -41,6 +41,7 @@ class WorkflowToolManageService:
parameters: list[WorkflowToolParameterConfiguration],
privacy_policy: str = "",
labels: list[str] | None = None,
import_id: str = "",
):
# check if the name is unique
existing_workflow_tool_provider: WorkflowToolProvider | None = None
@@ -92,7 +93,8 @@ class WorkflowToolManageService:
privacy_policy=privacy_policy,
version=workflow.version,
)
if import_id:
workflow_tool_provider.id = import_id
try:
WorkflowToolProviderController.from_db(workflow_tool_provider)
except Exception as e:
@@ -505,7 +505,7 @@ def _truncate_container_database(app: Flask) -> None:
session_factory-created sessions. Truncating after each test gives the suite
a central DB isolation contract that does not depend on which session a test used.
This only covers SQLAlchemy application tables in db.metadata for now;
Redis, object storage, and custom ad hoc metadata still need their own cleanup.
object storage and custom ad hoc metadata still need their own cleanup.
"""
with app.app_context():
db.session.remove()
@@ -524,13 +524,27 @@ def _truncate_container_database(app: Flask) -> None:
db.session.remove()
def _flush_container_redis(app: Flask) -> None:
"""
Reset Redis after a container integration test.
Tests in this package share one Redis container for performance. Application
code stores temporary tokens, rate-limit counters, locks, and cache entries
there, so flushing after each test gives Redis-backed state the same
isolation contract as the PostgreSQL container.
"""
with app.app_context():
app.extensions["redis"].flushdb()
@pytest.fixture(autouse=True)
def isolate_container_database(request: pytest.FixtureRequest) -> Generator[None, None, None]:
"""
Clean DB state after tests that use the containerized Flask app.
Clean DB and Redis state after tests that use the containerized Flask app.
This fixture intentionally does not depend on flask_app_with_containers so
non-DB tests under this package do not start the full app/container stack.
tests under this package do not start the full app/container stack just to
run state cleanup.
"""
yield
@@ -538,7 +552,10 @@ def isolate_container_database(request: pytest.FixtureRequest) -> Generator[None
return
app = request.getfixturevalue("flask_app_with_containers")
_truncate_container_database(app)
try:
_truncate_container_database(app)
finally:
_flush_container_redis(app)
@pytest.fixture(scope="package", autouse=True)
@@ -0,0 +1,298 @@
from __future__ import annotations
from unittest.mock import patch
from uuid import uuid4
import pytest
from werkzeug.exceptions import HTTPException
import services
from controllers.console.auth.error import MemberNotInTenantError
from controllers.console.workspace import members as members_module
from controllers.console.workspace.members import MemberCancelInviteApi, MemberUpdateRoleApi, OwnerTransfer
from models.account import Account, Tenant, TenantAccountJoin, TenantAccountRole, TenantStatus
def unwrap(func):
while hasattr(func, "__wrapped__"):
func = func.__wrapped__
return func
class WorkspaceMembersIntegrationFactory:
@staticmethod
def create_tenant(db_session_with_containers) -> Tenant:
tenant = Tenant(name=f"Tenant {uuid4()}", plan="basic", status=TenantStatus.NORMAL)
db_session_with_containers.add(tenant)
db_session_with_containers.commit()
return tenant
@staticmethod
def create_account(
db_session_with_containers,
*,
email_prefix: str,
tenant: Tenant | None = None,
role: TenantAccountRole = TenantAccountRole.NORMAL,
current: bool = False,
) -> Account:
account = Account(
name=f"Account {uuid4()}",
email=f"{email_prefix}-{uuid4()}@example.com",
password="hashed-password",
password_salt="salt",
interface_language="en-US",
timezone="UTC",
)
db_session_with_containers.add(account)
db_session_with_containers.commit()
if tenant is not None:
join = TenantAccountJoin(
tenant_id=tenant.id,
account_id=account.id,
role=role,
current=current,
)
db_session_with_containers.add(join)
db_session_with_containers.commit()
account.current_tenant = tenant
return account
@staticmethod
def create_owner_workspace(db_session_with_containers) -> tuple[Tenant, Account]:
tenant = WorkspaceMembersIntegrationFactory.create_tenant(db_session_with_containers)
owner = WorkspaceMembersIntegrationFactory.create_account(
db_session_with_containers,
email_prefix="owner",
tenant=tenant,
role=TenantAccountRole.OWNER,
current=True,
)
return tenant, owner
@staticmethod
def create_owner_transfer_token(account: Account) -> str:
_, token = members_module.AccountService.generate_owner_transfer_token(
account.email,
account=account,
code="123456",
additional_data={},
)
return token
@staticmethod
def get_join(db_session_with_containers, *, tenant: Tenant, account: Account) -> TenantAccountJoin:
tenant_id = tenant.id
account_id = account.id
db_session_with_containers.expire_all()
join = (
db_session_with_containers.query(TenantAccountJoin)
.filter_by(tenant_id=tenant_id, account_id=account_id)
.one()
)
return join
class TestMemberCancelInviteApiWithContainers:
def test_cancel_success(self, flask_app_with_containers, db_session_with_containers):
api = MemberCancelInviteApi()
method = unwrap(api.delete)
factory = WorkspaceMembersIntegrationFactory
tenant, current_user = factory.create_owner_workspace(db_session_with_containers)
member = factory.create_account(db_session_with_containers, email_prefix="member")
with (
flask_app_with_containers.test_request_context("/"),
patch.object(members_module, "current_account_with_tenant", return_value=(current_user, tenant.id)),
patch.object(members_module.TenantService, "remove_member_from_tenant") as mock_remove_member,
):
result, status = method(api, member.id)
assert status == 200
assert result["result"] == "success"
mock_remove_member.assert_called_once()
called_tenant, called_member, called_current_user = mock_remove_member.call_args.args
assert called_tenant.id == tenant.id
assert called_member.id == member.id
assert called_current_user.id == current_user.id
def test_cancel_not_found(self, flask_app_with_containers, db_session_with_containers):
api = MemberCancelInviteApi()
method = unwrap(api.delete)
factory = WorkspaceMembersIntegrationFactory
tenant, current_user = factory.create_owner_workspace(db_session_with_containers)
with (
flask_app_with_containers.test_request_context("/"),
patch.object(members_module, "current_account_with_tenant", return_value=(current_user, tenant.id)),
):
with pytest.raises(HTTPException):
method(api, str(uuid4()))
def test_cancel_cannot_operate_self(self, flask_app_with_containers, db_session_with_containers):
api = MemberCancelInviteApi()
method = unwrap(api.delete)
factory = WorkspaceMembersIntegrationFactory
tenant, current_user = factory.create_owner_workspace(db_session_with_containers)
member = factory.create_account(db_session_with_containers, email_prefix="member")
with (
flask_app_with_containers.test_request_context("/"),
patch.object(members_module, "current_account_with_tenant", return_value=(current_user, tenant.id)),
patch.object(
members_module.TenantService,
"remove_member_from_tenant",
side_effect=services.errors.account.CannotOperateSelfError("x"),
),
):
result, status = method(api, member.id)
assert status == 400
assert result["code"] == "cannot-operate-self"
def test_cancel_no_permission(self, flask_app_with_containers, db_session_with_containers):
api = MemberCancelInviteApi()
method = unwrap(api.delete)
factory = WorkspaceMembersIntegrationFactory
tenant, current_user = factory.create_owner_workspace(db_session_with_containers)
member = factory.create_account(db_session_with_containers, email_prefix="member")
with (
flask_app_with_containers.test_request_context("/"),
patch.object(members_module, "current_account_with_tenant", return_value=(current_user, tenant.id)),
patch.object(
members_module.TenantService,
"remove_member_from_tenant",
side_effect=services.errors.account.NoPermissionError("x"),
),
):
result, status = method(api, member.id)
assert status == 403
assert result["code"] == "forbidden"
def test_cancel_member_not_in_tenant(self, flask_app_with_containers, db_session_with_containers):
api = MemberCancelInviteApi()
method = unwrap(api.delete)
factory = WorkspaceMembersIntegrationFactory
tenant, current_user = factory.create_owner_workspace(db_session_with_containers)
member = factory.create_account(db_session_with_containers, email_prefix="member")
with (
flask_app_with_containers.test_request_context("/"),
patch.object(members_module, "current_account_with_tenant", return_value=(current_user, tenant.id)),
patch.object(
members_module.TenantService,
"remove_member_from_tenant",
side_effect=services.errors.account.MemberNotInTenantError(),
),
):
result, status = method(api, member.id)
assert status == 404
assert result["code"] == "member-not-found"
class TestMemberUpdateRoleApiWithContainers:
def test_update_success(self, flask_app_with_containers, db_session_with_containers):
api = MemberUpdateRoleApi()
method = unwrap(api.put)
factory = WorkspaceMembersIntegrationFactory
tenant, current_user = factory.create_owner_workspace(db_session_with_containers)
member = factory.create_account(
db_session_with_containers,
email_prefix="member",
tenant=tenant,
role=TenantAccountRole.EDITOR,
)
with (
flask_app_with_containers.test_request_context("/", json={"role": "normal"}),
patch.object(members_module, "current_account_with_tenant", return_value=(current_user, tenant.id)),
):
result = method(api, member.id)
if isinstance(result, tuple):
result = result[0]
assert result["result"] == "success"
assert (
factory.get_join(db_session_with_containers, tenant=tenant, account=member).role == TenantAccountRole.NORMAL
)
def test_update_member_not_found(self, flask_app_with_containers, db_session_with_containers):
api = MemberUpdateRoleApi()
method = unwrap(api.put)
factory = WorkspaceMembersIntegrationFactory
tenant, current_user = factory.create_owner_workspace(db_session_with_containers)
with (
flask_app_with_containers.test_request_context("/", json={"role": "normal"}),
patch.object(members_module, "current_account_with_tenant", return_value=(current_user, tenant.id)),
):
with pytest.raises(HTTPException):
method(api, str(uuid4()))
class TestOwnerTransferApiWithContainers:
def test_member_not_in_tenant(self, flask_app_with_containers, db_session_with_containers):
api = OwnerTransfer()
method = unwrap(api.post)
factory = WorkspaceMembersIntegrationFactory
tenant, current_user = factory.create_owner_workspace(db_session_with_containers)
member = factory.create_account(db_session_with_containers, email_prefix="member")
token = factory.create_owner_transfer_token(current_user)
with (
flask_app_with_containers.test_request_context("/", json={"token": token}),
patch.object(members_module, "current_account_with_tenant", return_value=(current_user, tenant.id)),
):
with pytest.raises(MemberNotInTenantError):
method(api, member.id)
def test_member_not_found(self, flask_app_with_containers, db_session_with_containers):
api = OwnerTransfer()
method = unwrap(api.post)
factory = WorkspaceMembersIntegrationFactory
tenant, current_user = factory.create_owner_workspace(db_session_with_containers)
token = factory.create_owner_transfer_token(current_user)
with (
flask_app_with_containers.test_request_context("/", json={"token": token}),
patch.object(members_module, "current_account_with_tenant", return_value=(current_user, tenant.id)),
):
with pytest.raises(HTTPException):
method(api, str(uuid4()))
def test_transfer_success(self, flask_app_with_containers, db_session_with_containers):
api = OwnerTransfer()
method = unwrap(api.post)
factory = WorkspaceMembersIntegrationFactory
tenant, current_user = factory.create_owner_workspace(db_session_with_containers)
member = factory.create_account(
db_session_with_containers,
email_prefix="member",
tenant=tenant,
role=TenantAccountRole.NORMAL,
)
token = factory.create_owner_transfer_token(current_user)
with (
flask_app_with_containers.test_request_context("/", json={"token": token}),
patch.object(members_module, "current_account_with_tenant", return_value=(current_user, tenant.id)),
patch.object(members_module.AccountService, "send_new_owner_transfer_notify_email") as mock_new_owner_email,
patch.object(members_module.AccountService, "send_old_owner_transfer_notify_email") as mock_old_owner_email,
):
result = method(api, member.id)
assert result["result"] == "success"
assert (
factory.get_join(db_session_with_containers, tenant=tenant, account=member).role == TenantAccountRole.OWNER
)
assert (
factory.get_join(db_session_with_containers, tenant=tenant, account=current_user).role
== TenantAccountRole.ADMIN
)
mock_new_owner_email.assert_called_once()
mock_old_owner_email.assert_called_once()
@@ -0,0 +1,182 @@
preset = "strict"
project-includes = ["."]
search-path = ["../.."]
# Verify project-excludes from the repo root:
# tmp_config=$(mktemp --tmpdir=api/tests/test_containers_integration_tests pyrefly-no-excludes.XXXXXX.toml)
# awk 'BEGIN {skip=0} /^project-excludes = \[/ {skip=1; next} skip && /^\]/ {skip=0; next} !skip {print}' api/tests/test_containers_integration_tests/pyrefly.toml > "$tmp_config"
# tmp_name=$(basename "$tmp_config")
# comm -3 <(sed -n 's/^ "\(.*\)",$/\1/p' api/tests/test_containers_integration_tests/pyrefly.toml | sort) <(uv --directory api run pyrefly check --config "tests/test_containers_integration_tests/$tmp_name" --summary=none --output-format=min-text 2>/dev/null | rg '^ERROR ' | sed -E 's#^ERROR (tests/test_containers_integration_tests/[^:]+):.*#\1#' | sed 's#^tests/test_containers_integration_tests/##' | sort -u)
# rm --force "$tmp_config"
project-excludes = [
"commands/test_legacy_model_type_migration.py",
"conftest.py",
"controllers/console/app/test_app_apis.py",
"controllers/console/app/test_app_import_api.py",
"controllers/console/app/test_chat_conversation_status_count_api.py",
"controllers/console/app/test_conversation_read_timestamp.py",
"controllers/console/app/test_workflow_draft_variable.py",
"controllers/console/auth/test_email_register.py",
"controllers/console/auth/test_forgot_password.py",
"controllers/console/auth/test_oauth.py",
"controllers/console/auth/test_password_reset.py",
"controllers/console/datasets/rag_pipeline/test_rag_pipeline.py",
"controllers/console/datasets/rag_pipeline/test_rag_pipeline_datasets.py",
"controllers/console/datasets/rag_pipeline/test_rag_pipeline_import.py",
"controllers/console/datasets/rag_pipeline/test_rag_pipeline_workflow.py",
"controllers/console/datasets/test_data_source.py",
"controllers/console/explore/test_conversation.py",
"controllers/console/test_api_based_extension.py",
"controllers/console/test_apikey.py",
"controllers/console/workspace/test_members.py",
"controllers/console/workspace/test_tool_provider.py",
"controllers/console/workspace/test_trigger_providers.py",
"controllers/console/workspace/test_workspace_wraps.py",
"controllers/mcp/test_mcp.py",
"controllers/service_api/dataset/test_dataset.py",
"controllers/service_api/test_site.py",
"controllers/web/test_conversation.py",
"controllers/web/test_site.py",
"controllers/web/test_web_forgot_password.py",
"controllers/web/test_wraps.py",
"core/app/layers/test_pause_state_persist_layer.py",
"core/rag/pipeline/test_queue_integration.py",
"core/rag/retrieval/test_dataset_retrieval_integration.py",
"core/workflow/test_human_input_resume_node_execution.py",
"factories/test_storage_key_loader.py",
"helpers/__init__.py",
"helpers/execution_extra_content.py",
"libs/broadcast_channel/redis/test_channel.py",
"libs/broadcast_channel/redis/test_sharded_channel.py",
"libs/broadcast_channel/redis/test_streams_channel.py",
"libs/test_auto_renew_redis_lock_integration.py",
"libs/test_rate_limiter_integration.py",
"models/test_account.py",
"models/test_conversation_message_inputs.py",
"models/test_types_enum_text.py",
"repositories/test_sqlalchemy_api_workflow_node_execution_repository.py",
"repositories/test_sqlalchemy_api_workflow_run_repository.py",
"repositories/test_sqlalchemy_execution_extra_content_repository.py",
"repositories/test_sqlalchemy_workflow_node_execution_repository.py",
"repositories/test_workflow_run_repository.py",
"services/auth/test_api_key_auth_service.py",
"services/auth/test_auth_integration.py",
"services/dataset_collection_binding.py",
"services/dataset_service_update_delete.py",
"services/document_service_status.py",
"services/enterprise/test_account_deletion_sync.py",
"services/plugin/test_plugin_parameter_service.py",
"services/plugin/test_plugin_permission_service.py",
"services/plugin/test_plugin_service.py",
"services/rag_pipeline/test_rag_pipeline_service_db.py",
"services/recommend_app/test_database_retrieval.py",
"services/test_account_service.py",
"services/test_advanced_prompt_template_service.py",
"services/test_agent_service.py",
"services/test_annotation_service.py",
"services/test_api_based_extension_service.py",
"services/test_api_token_service.py",
"services/test_app_dsl_service.py",
"services/test_app_generate_service.py",
"services/test_app_service.py",
"services/test_attachment_service.py",
"services/test_audio_service_db.py",
"services/test_billing_service.py",
"services/test_conversation_service.py",
"services/test_conversation_service_variables.py",
"services/test_conversation_variable_updater.py",
"services/test_credit_pool_service.py",
"services/test_dataset_permission_service.py",
"services/test_dataset_service.py",
"services/test_dataset_service_batch_update_document_status.py",
"services/test_dataset_service_create_dataset.py",
"services/test_dataset_service_delete_dataset.py",
"services/test_dataset_service_document.py",
"services/test_dataset_service_get_segments.py",
"services/test_dataset_service_permissions.py",
"services/test_dataset_service_retrieval.py",
"services/test_dataset_service_update_dataset.py",
"services/test_delete_archived_workflow_run.py",
"services/test_document_service_display_status.py",
"services/test_document_service_rename_document.py",
"services/test_end_user_service.py",
"services/test_feature_service.py",
"services/test_feedback_service.py",
"services/test_file_service.py",
"services/test_hit_testing_service.py",
"services/test_human_input_delivery_test.py",
"services/test_human_input_delivery_test_service.py",
"services/test_message_export_service.py",
"services/test_message_service.py",
"services/test_message_service_execution_extra_content.py",
"services/test_message_service_extra_contents.py",
"services/test_messages_clean_service.py",
"services/test_metadata_partial_update.py",
"services/test_metadata_service.py",
"services/test_model_load_balancing_service.py",
"services/test_model_provider_service.py",
"services/test_oauth_server_service.py",
"services/test_ops_service.py",
"services/test_recommended_app_service.py",
"services/test_restore_archived_workflow_run.py",
"services/test_saved_message_service.py",
"services/test_schedule_service.py",
"services/test_tag_service.py",
"services/test_trigger_provider_service.py",
"services/test_web_conversation_service.py",
"services/test_webapp_auth_service.py",
"services/test_webhook_service.py",
"services/test_webhook_service_relationships.py",
"services/test_workflow_app_service.py",
"services/test_workflow_draft_variable_service.py",
"services/test_workflow_run_service.py",
"services/test_workflow_service.py",
"services/test_workspace_service.py",
"services/tools/test_api_tools_manage_service.py",
"services/tools/test_mcp_tools_manage_service.py",
"services/tools/test_tools_transform_service.py",
"services/tools/test_workflow_tools_manage_service.py",
"services/workflow/test_workflow_converter.py",
"services/workflow/test_workflow_deletion.py",
"services/workflow/test_workflow_node_execution_service_repository.py",
"tasks/test_add_document_to_index_task.py",
"tasks/test_batch_clean_document_task.py",
"tasks/test_batch_create_segment_to_index_task.py",
"tasks/test_clean_dataset_task.py",
"tasks/test_clean_notion_document_task.py",
"tasks/test_create_segment_to_index_task.py",
"tasks/test_dataset_indexing_task.py",
"tasks/test_deal_dataset_vector_index_task.py",
"tasks/test_delete_account_task.py",
"tasks/test_delete_segment_from_index_task.py",
"tasks/test_disable_segment_from_index_task.py",
"tasks/test_disable_segments_from_index_task.py",
"tasks/test_document_indexing_sync_task.py",
"tasks/test_document_indexing_task.py",
"tasks/test_document_indexing_update_task.py",
"tasks/test_duplicate_document_indexing_task.py",
"tasks/test_enable_segments_to_index_task.py",
"tasks/test_mail_account_deletion_task.py",
"tasks/test_mail_change_mail_task.py",
"tasks/test_mail_email_code_login_task.py",
"tasks/test_mail_human_input_delivery_task.py",
"tasks/test_mail_inner_task.py",
"tasks/test_mail_invite_member_task.py",
"tasks/test_mail_owner_transfer_task.py",
"tasks/test_mail_register_task.py",
"tasks/test_rag_pipeline_run_tasks.py",
"tasks/test_remove_app_and_related_data_task.py",
"test_container_state_isolation.py",
"test_opendal_fs_default_root.py",
"test_workflow_pause_integration.py",
"trigger/conftest.py",
"trigger/test_trigger_e2e.py",
"workflow/nodes/code_executor/test_code_executor.py",
"workflow/nodes/code_executor/test_code_javascript.py",
"workflow/nodes/code_executor/test_code_jinja2.py",
"workflow/nodes/code_executor/test_code_python3.py",
"workflow/nodes/code_executor/test_utils.py",
]
[errors]
unannotated-return = true
@@ -0,0 +1,39 @@
from __future__ import annotations
from uuid import uuid4
from extensions.ext_redis import redis_client
from models.account import Account
ACCOUNT_EMAIL = f"container-state-isolation-{uuid4()}@example.com"
REDIS_KEY = f"container-state-isolation:{uuid4()}"
def test_1_container_state_can_be_written(
flask_app_with_containers,
db_session_with_containers,
) -> None:
account = Account(
name="Container State Isolation",
email=ACCOUNT_EMAIL,
password="hashed-password",
password_salt="salt",
interface_language="en-US",
timezone="UTC",
)
db_session_with_containers.add(account)
db_session_with_containers.commit()
with flask_app_with_containers.app_context():
redis_client.set(REDIS_KEY, "leaked")
assert redis_client.get(REDIS_KEY) == b"leaked"
def test_2_container_state_is_flushed_between_tests(
flask_app_with_containers,
db_session_with_containers,
) -> None:
assert db_session_with_containers.query(Account).filter_by(email=ACCOUNT_EMAIL).one_or_none() is None
with flask_app_with_containers.app_context():
assert redis_client.get(REDIS_KEY) is None
@@ -0,0 +1,71 @@
import json
from click.testing import CliRunner
from commands.data_migration import (
ID_STRATEGY_CHOICES,
export_migration_data,
export_migration_data_template,
import_migration_data,
)
def test_export_command_requires_input_and_output():
result = CliRunner().invoke(export_migration_data, [])
assert result.exit_code != 0
assert export_migration_data.name == "export-app-migration"
assert "--input" in result.output
assert "--output" in result.output
def test_import_command_requires_input_and_target_tenant_or_package_metadata():
result = CliRunner().invoke(import_migration_data, [])
assert result.exit_code != 0
assert import_migration_data.name == "import-app-migration"
assert "--input" in result.output
def test_import_command_does_not_expose_unimplemented_map_id_strategy():
assert ID_STRATEGY_CHOICES == ["preserve-id", "generate-new-id"]
def test_export_template_command_prints_scripted_json_template():
result = CliRunner().invoke(export_migration_data_template, [])
assert result.exit_code == 0
assert export_migration_data_template.name == "app-migration-template"
template = json.loads(result.output)
assert template == {
"source_tenant": {"mode": "single", "id": "", "name": "admin's Workspace"},
"apps": {"modes": ["workflow", "advanced-chat"], "ids": [], "all": True},
"include_referenced_tools": True,
"additional_tools": {"api_tools": [], "workflow_tools": [], "mcp_tools": []},
"include_secrets": False,
"import_options": {
"create_app_api_token_on_import": False,
"id_strategy": "preserve-id",
"conflict_strategy": "fail",
},
}
def test_export_template_command_writes_output_file(tmp_path):
output_file = tmp_path / "export-template.json"
result = CliRunner().invoke(export_migration_data_template, ["--output", str(output_file)])
assert result.exit_code == 0
assert f"Output written to {output_file}" in result.output
assert json.loads(output_file.read_text())["apps"]["all"] is True
def test_export_template_command_requires_overwrite_for_existing_output(tmp_path):
output_file = tmp_path / "export-template.json"
output_file.write_text("{}")
result = CliRunner().invoke(export_migration_data_template, ["--output", str(output_file)])
assert result.exit_code != 0
assert "already exists" in result.output
@@ -0,0 +1,384 @@
from commands.data_migration import (
CONFLICT_STRATEGY_CHOICES,
ID_STRATEGY_CHOICES,
_confirm_wizard_summary,
_print_auto_tools,
_print_final_tool_selection,
_print_wizard_step,
_prompt_additional_tools,
_prompt_output_file,
_prompt_tool_category,
_resolve_mcp_tool_names,
migration_data_wizard,
parse_index_selection,
)
def test_parse_index_selection_supports_all():
assert parse_index_selection("all", ["a", "b", "c"]) == ["a", "b", "c"]
def test_wizard_command_uses_app_migration_name():
assert migration_data_wizard.name == "app-migration-wizard"
def test_parse_index_selection_supports_comma_indexes():
assert parse_index_selection("1, 3", ["a", "b", "c"]) == ["a", "c"]
def test_print_wizard_step_adds_separator(monkeypatch):
output_lines = []
monkeypatch.setattr("commands.data_migration.click.echo", output_lines.append)
_print_wizard_step("App Selection")
assert output_lines == ["", "==== App Selection ===="]
def test_conflict_strategy_choices_exclude_replace():
assert CONFLICT_STRATEGY_CHOICES == ["fail", "skip", "update"]
def test_prompt_app_ids_explains_comma_selection_and_default(monkeypatch):
from commands.data_migration import _prompt_app_ids
prompts = []
output_lines = []
apps = [
type("App", (), {"id": "app-1", "name": "embedded", "mode": "workflow"})(),
type("App", (), {"id": "app-2", "name": "main", "mode": "advanced-chat"})(),
]
def capture_prompt(text, **kwargs):
prompts.append((text, kwargs))
return "1,2"
monkeypatch.setattr("commands.data_migration.click.echo", output_lines.append)
monkeypatch.setattr("commands.data_migration.click.prompt", capture_prompt)
assert _prompt_app_ids(apps) == ["app-1", "app-2"]
assert prompts == [("Select apps by number, comma-separated numbers, or all", {"default": "all"})]
assert "Currently supported app types: workflow and chatflow." in output_lines
def test_prompt_tool_category_marks_auto_discovered_tools(monkeypatch):
output_lines = []
monkeypatch.setattr("commands.data_migration.click.echo", output_lines.append)
monkeypatch.setattr("commands.data_migration.click.prompt", lambda *args, **kwargs: "")
selected = _prompt_tool_category(
"Custom API tools",
[("weather", "weather", "tool-id"), ("calendar", "calendar", "calendar-id")],
auto_tools={"weather": "tool-id"},
)
assert selected == []
assert "1. [auto] weather (tool-id)" in output_lines
assert "2. [ ] calendar (calendar-id)" in output_lines
assert output_lines[:2] == ["", "==== Custom API tools ===="]
def test_prompt_tool_category_explains_comma_selection_and_default(monkeypatch):
prompts = []
def capture_prompt(text, **kwargs):
prompts.append((text, kwargs))
return ""
monkeypatch.setattr("commands.data_migration.click.echo", lambda *_args, **_kwargs: None)
monkeypatch.setattr("commands.data_migration.click.prompt", capture_prompt)
selected = _prompt_tool_category(
"Custom API tools",
[("weather", "weather", "tool-id")],
auto_tools={},
)
assert selected == []
assert prompts == [
(
"Select custom api tools by number, comma-separated numbers, all, or empty",
{"default": "", "show_default": "empty"},
)
]
def test_prompt_output_file_shows_default(monkeypatch):
prompts = []
def capture_prompt(text, **kwargs):
prompts.append((text, kwargs))
return "migration-data.json"
monkeypatch.setattr("commands.data_migration.click.prompt", capture_prompt)
assert _prompt_output_file() == ("migration-data.json", False)
assert prompts[0][0] == "Output path"
assert prompts[0][1]["show_default"] is True
def test_prompt_tool_category_marks_auto_by_detail_and_supports_multi_select(monkeypatch):
monkeypatch.setattr("commands.data_migration.click.echo", lambda *_args, **_kwargs: None)
monkeypatch.setattr("commands.data_migration.click.prompt", lambda *args, **kwargs: "1,2")
selected = _prompt_tool_category(
"Workflow tools",
[("tool-1", "embedded", "app-1"), ("tool-2", "other", "app-2")],
auto_tools={"embedded": "app-1"},
)
assert selected == ["tool-1", "tool-2"]
def test_prompt_tool_category_marks_auto_by_value():
output_lines = []
from commands import data_migration
original_echo = data_migration.click.echo
original_prompt = data_migration.click.prompt
data_migration.click.echo = output_lines.append
data_migration.click.prompt = lambda *args, **kwargs: ""
try:
_prompt_tool_category(
"Workflow tools",
[("tool-1", "embedded_workflow_as_tool", "tool-1")],
auto_tools={"embedded_workflow_as_tool": "tool-1"},
)
finally:
data_migration.click.echo = original_echo
data_migration.click.prompt = original_prompt
assert "1. [auto] embedded_workflow_as_tool (tool-1)" in output_lines
def test_print_auto_tools_lists_each_category(monkeypatch):
output_lines = []
monkeypatch.setattr("commands.data_migration.click.echo", output_lines.append)
_print_auto_tools(
{
"api_tools": {"weather": "3bac3aa9-dd87-4351-9459-a7099137b028"},
"workflow_tools": {"embedded_workflow_as_tool": "e6024578-41b7-4fb5-a81f-9201358e5835"},
"mcp_tools": {},
}
)
assert "Automatically discovered tools:" in output_lines
assert "Custom API tools" in output_lines
assert "- weather: 3bac3aa9-dd87-4351-9459-a7099137b028" in output_lines
assert "Workflow tools" in output_lines
assert "- embedded_workflow_as_tool: e6024578-41b7-4fb5-a81f-9201358e5835" in output_lines
assert "MCP tools" in output_lines
assert "- none" in output_lines
def test_resolve_mcp_tool_names_does_not_compare_non_uuid_identifier_to_uuid_id(monkeypatch):
statements = []
def capture_scalar(statement):
statements.append(str(statement))
monkeypatch.setattr("commands.data_migration.db.session.scalar", capture_scalar)
assert _resolve_mcp_tool_names("49a99e46-bc2c-4885-91fa-47615f6192b5", {"my-test-mcp": "my-test-mcp"}) == {
"my-test-mcp": "my-test-mcp"
}
assert "tool_mcp_providers.id =" not in statements[0]
assert "tool_mcp_providers.server_identifier =" in statements[0]
def test_prompt_additional_tools_prints_final_selection_when_skipped(monkeypatch):
output_lines = []
confirm_prompts = []
def capture_confirm(prompt, **kwargs):
confirm_prompts.append((prompt, kwargs))
return False
monkeypatch.setattr("commands.data_migration.click.confirm", capture_confirm)
monkeypatch.setattr("commands.data_migration.click.echo", output_lines.append)
selected = _prompt_additional_tools(
"tenant-id",
{
"api_tools": {"weather": "3bac3aa9-dd87-4351-9459-a7099137b028"},
"workflow_tools": {},
"mcp_tools": {},
},
)
assert selected == {"api_tools": [], "workflow_tools": [], "mcp_tools": []}
assert confirm_prompts == [
("Export additional tools manually? [y/n, default: n]", {"default": False, "show_default": False})
]
assert "Final tools to export:" in output_lines
assert "- [auto] weather: 3bac3aa9-dd87-4351-9459-a7099137b028" in output_lines
def test_final_tool_selection_deduplicates_manual_tool_already_auto(monkeypatch):
output_lines = []
monkeypatch.setattr("commands.data_migration.click.echo", output_lines.append)
_print_final_tool_selection(
{
"api_tools": {},
"workflow_tools": {"embedded_workflow_as_tool": "e6024578-41b7-4fb5-a81f-9201358e5835"},
"mcp_tools": {},
},
{
"api_tools": [],
"workflow_tools": ["e6024578-41b7-4fb5-a81f-9201358e5835"],
"mcp_tools": [],
},
{"e6024578-41b7-4fb5-a81f-9201358e5835": "embedded_workflow_as_tool: e6024578"},
)
assert "- [auto] embedded_workflow_as_tool: e6024578-41b7-4fb5-a81f-9201358e5835" in output_lines
assert not any(line.startswith("- [manual]") for line in output_lines)
def test_prompt_output_file_rejects_yes_no_typo(monkeypatch):
import click
import pytest
monkeypatch.setattr("commands.data_migration.click.prompt", lambda *args, **kwargs: "y")
with pytest.raises(click.ClickException, match="Output path must be a file path"):
_prompt_output_file()
def test_confirm_wizard_summary_shows_conflict_strategy(monkeypatch):
output_lines = []
confirm_prompts = []
monkeypatch.setattr("commands.data_migration.click.echo", output_lines.append)
monkeypatch.setattr(
"commands.data_migration.click.confirm",
lambda prompt, **kwargs: confirm_prompts.append((prompt, kwargs)) or True,
)
_confirm_wizard_summary(
tenant_name="admin's Workspace",
app_names=["main_chatflow"],
auto_tools={"api_tools": {}, "workflow_tools": {}, "mcp_tools": {}},
additional_tools={"api_tools": [], "workflow_tools": [], "mcp_tools": []},
manual_labels={},
include_referenced_tools=True,
include_secrets=False,
create_tokens=True,
id_strategy="preserve-id",
conflict_strategy="fail",
output_file="migration-data.json",
)
assert "id strategy: preserve-id" in output_lines
assert "conflict strategy: fail" in output_lines
assert confirm_prompts == [("Write migration package? [y/n, default: y]", {"default": True, "show_default": False})]
def test_confirm_wizard_summary_shows_final_deduplicated_tool_selection(monkeypatch):
output_lines = []
monkeypatch.setattr("commands.data_migration.click.echo", output_lines.append)
monkeypatch.setattr("commands.data_migration.click.confirm", lambda *args, **kwargs: True)
_confirm_wizard_summary(
tenant_name="admin's Workspace",
app_names=["main_chatflow"],
auto_tools={
"api_tools": {"weather": "weather-id"},
"workflow_tools": {"embedded_workflow_as_tool": "workflow-tool-id"},
"mcp_tools": {},
},
additional_tools={
"api_tools": ["weather-id", "calendar"],
"workflow_tools": [],
"mcp_tools": ["mcp-id"],
},
manual_labels={
"calendar": "calendar: calendar-id",
"mcp-id": "my-test-mcp: mcp-id",
},
include_referenced_tools=True,
include_secrets=False,
create_tokens=False,
id_strategy="preserve-id",
conflict_strategy="update",
output_file="migration-data.json",
)
assert "Final tools to export:" in output_lines
assert "Custom API tools" in output_lines
assert "- [auto] weather: weather-id" in output_lines
assert "- [manual] calendar: calendar-id" in output_lines
assert "Workflow tools" in output_lines
assert "- [auto] embedded_workflow_as_tool: workflow-tool-id" in output_lines
assert "MCP tools" in output_lines
assert "- [manual] my-test-mcp: mcp-id" in output_lines
assert not any(line.startswith("additional api tools:") for line in output_lines)
assert not any(line.startswith("additional workflow tools:") for line in output_lines)
assert not any(line.startswith("additional mcp tools:") for line in output_lines)
assert "- [manual] weather-id" not in output_lines
def test_import_options_prompts_explain_secrets_reuse_and_conflicts(monkeypatch):
from commands.data_migration import _prompt_import_options
output_lines = []
confirm_prompts = []
prompt_calls = []
def capture_confirm(prompt, **kwargs):
confirm_prompts.append((prompt, kwargs))
return False
def capture_prompt(prompt, **kwargs):
prompt_calls.append((prompt, kwargs))
return kwargs["default"]
monkeypatch.setattr("commands.data_migration.click.echo", output_lines.append)
monkeypatch.setattr("commands.data_migration.click.confirm", capture_confirm)
monkeypatch.setattr("commands.data_migration.click.prompt", capture_prompt)
include_secrets, create_tokens, id_strategy, conflict_strategy = _prompt_import_options()
assert include_secrets is False
assert create_tokens is False
assert id_strategy == "preserve-id"
assert conflict_strategy == "update"
assert "Secrets include workflow/app DSL secret values, custom API tool credentials," in output_lines
assert "-- Secrets --" in output_lines
assert "If you choose no, credentials are omitted or masked," in output_lines
assert "-- App API Tokens --" in output_lines
assert "When enabled, import will create an app API token if the imported app has none," in output_lines
assert "or reuse an existing app API token if one already exists." in output_lines
assert "-- ID Strategy --" in output_lines
assert "ID strategy controls whether imported app and tool IDs preserve source IDs" in output_lines
assert "or use target-generated IDs." in output_lines
assert "preserve-id: keep source IDs where the target service supports it." in output_lines
assert (
"generate-new-id: let the target environment generate new IDs and rewrite references via mapping."
in output_lines
)
assert "-- Conflict Strategy --" in output_lines
assert "Conflict strategy controls what import does when a target resource already exists." in output_lines
assert "fail: stop at the first conflict; previously committed resources are not rolled back." in output_lines
assert "skip: keep the existing target resource and skip importing that resource." in output_lines
assert "update: update the existing target resource in place." in output_lines
assert confirm_prompts == [
("Include secrets in output JSON? [y/n, default: n]", {"default": False, "show_default": False}),
("Create or reuse app API tokens during import? [y/n, default: n]", {"default": False, "show_default": False}),
]
assert prompt_calls[0][0] == "Import ID strategy. Enter one of: preserve-id, generate-new-id"
assert prompt_calls[0][1]["default"] == "preserve-id"
assert prompt_calls[0][1]["show_default"] is True
assert prompt_calls[0][1]["type"].choices == ID_STRATEGY_CHOICES
assert prompt_calls[1][0] == "Import conflict strategy. Enter one of: fail, skip, update"
assert prompt_calls[1][1]["default"] == "update"
assert prompt_calls[1][1]["show_default"] is True
assert prompt_calls[1][1]["type"].choices == CONFLICT_STRATEGY_CHOICES
@@ -30,6 +30,90 @@ def _unwrap(method):
return method
def _agent_response(agent_id: str = "agent-1") -> dict:
return {
"id": agent_id,
"name": "Analyst",
"description": "",
"icon_type": None,
"icon": None,
"icon_background": None,
"agent_kind": "dify_agent",
"scope": "roster",
"source": "agent_app",
"app_id": None,
"workflow_id": None,
"workflow_node_id": None,
"active_config_snapshot_id": "version-1",
"active_config_snapshot": _version_response(),
"status": "active",
"created_by": "account-1",
"updated_by": "account-1",
"archived_by": None,
"archived_at": None,
"created_at": None,
"updated_at": None,
}
def _version_response(version_id: str = "version-1") -> dict:
return {
"id": version_id,
"agent_id": "agent-1",
"version": 1,
"summary": None,
"version_note": None,
"created_by": "account-1",
"created_at": None,
}
def _workflow_composer_response(**overrides) -> dict:
response = {
"variant": "workflow",
"agent": None,
"active_config_snapshot": None,
"binding": None,
"soul_lock": {"locked": False, "can_unlock": False, "reason": "workflow_only_empty"},
"agent_soul": {},
"node_job": {},
"effective_declared_outputs": [],
"save_options": ["node_job_only"],
"impact_summary": None,
"app_id": "app-1",
"workflow_id": "workflow-1",
"node_id": "node-1",
}
response.update(overrides)
return response
def _agent_app_composer_response() -> dict:
return {
"variant": "agent_app",
"agent": {
"id": "agent-1",
"name": "Analyst",
"description": "",
"scope": "roster",
"status": "active",
"active_config_snapshot_id": "version-1",
},
"active_config_snapshot": _version_response(),
"agent_soul": {},
"save_options": ["save_to_current_version", "save_as_new_version"],
}
def _candidates_response(variant: str) -> dict:
return {
"variant": variant,
"allowed_node_job_candidates": {},
"allowed_soul_candidates": {},
"capabilities": {"human_roster_available": False},
}
@pytest.fixture
def account():
return SimpleNamespace(id="account-1")
@@ -67,14 +151,15 @@ def test_roster_list_post_creates_agent_and_returns_detail(app, monkeypatch):
monkeypatch.setattr(
roster_controller.AgentRosterService,
"get_roster_agent_detail",
lambda _self, **kwargs: {"id": kwargs["agent_id"], "tenant_id": kwargs["tenant_id"]},
lambda _self, **kwargs: _agent_response(kwargs["agent_id"]),
)
with app.test_request_context(json={"name": "Analyst", "agent_soul": {"prompt": {"system_prompt": "x"}}}):
result, status = _unwrap(AgentRosterListApi.post)(AgentRosterListApi())
assert status == 201
assert result == {"id": "agent-1", "tenant_id": "tenant-1"}
assert result["id"] == "agent-1"
assert result["agent_kind"] == "dify_agent"
def test_invite_options_get_parses_app_id(app, monkeypatch):
@@ -82,14 +167,14 @@ def test_invite_options_get_parses_app_id(app, monkeypatch):
def list_invite_options(_self, **kwargs):
captured.update(kwargs)
return {"data": []}
return {"data": [], "page": kwargs["page"], "limit": kwargs["limit"], "total": 0, "has_more": False}
monkeypatch.setattr(roster_controller.AgentRosterService, "list_invite_options", list_invite_options)
with app.test_request_context("/console/api/agents/invite-options?page=1&limit=10&app_id=app-1"):
result = _unwrap(AgentInviteOptionsApi.get)(AgentInviteOptionsApi())
assert result == {"data": []}
assert result == {"data": [], "page": 1, "limit": 10, "total": 0, "has_more": False}
assert captured == {"tenant_id": "tenant-1", "page": 1, "limit": 10, "keyword": None, "app_id": "app-1"}
@@ -100,12 +185,12 @@ def test_roster_detail_patch_delete_and_versions_call_services(app, monkeypatch)
monkeypatch.setattr(
roster_controller.AgentRosterService,
"get_roster_agent_detail",
lambda _self, **kwargs: {"id": kwargs["agent_id"]},
lambda _self, **kwargs: _agent_response(kwargs["agent_id"]),
)
monkeypatch.setattr(
roster_controller.AgentRosterService,
"update_roster_agent",
lambda _self, **kwargs: {"id": kwargs["agent_id"], "description": kwargs["payload"].description},
lambda _self, **kwargs: {**_agent_response(kwargs["agent_id"]), "description": kwargs["payload"].description},
)
monkeypatch.setattr(
roster_controller.AgentRosterService,
@@ -115,12 +200,29 @@ def test_roster_detail_patch_delete_and_versions_call_services(app, monkeypatch)
monkeypatch.setattr(
roster_controller.AgentRosterService,
"list_agent_versions",
lambda _self, **kwargs: [{"id": "version-1"}],
lambda _self, **kwargs: [_version_response()],
)
monkeypatch.setattr(
roster_controller.AgentRosterService,
"get_agent_version_detail",
lambda _self, **kwargs: {"id": kwargs["version_id"], "agent_id": kwargs["agent_id"]},
lambda _self, **kwargs: {
**_version_response(kwargs["version_id"]),
"agent_id": kwargs["agent_id"],
"config_snapshot": {},
"revisions": [
{
"id": "revision-1",
"previous_snapshot_id": None,
"current_snapshot_id": kwargs["version_id"],
"revision": 1,
"operation": "create_version",
"summary": None,
"version_note": None,
"created_by": "account-1",
"created_at": None,
}
],
},
)
assert _unwrap(AgentRosterDetailApi.get)(AgentRosterDetailApi(), agent_id)["id"] == agent_id
@@ -128,11 +230,10 @@ def test_roster_detail_patch_delete_and_versions_call_services(app, monkeypatch)
assert _unwrap(AgentRosterDetailApi.patch)(AgentRosterDetailApi(), agent_id)["description"] == "updated"
assert _unwrap(AgentRosterDetailApi.delete)(AgentRosterDetailApi(), agent_id) == ("", 204)
assert archived["account_id"] == "account-1"
assert _unwrap(AgentRosterVersionsApi.get)(AgentRosterVersionsApi(), agent_id) == {"data": [{"id": "version-1"}]}
assert _unwrap(AgentRosterVersionDetailApi.get)(AgentRosterVersionDetailApi(), agent_id, version_id) == {
"id": version_id,
"agent_id": agent_id,
}
assert _unwrap(AgentRosterVersionsApi.get)(AgentRosterVersionsApi(), agent_id)["data"][0]["id"] == "version-1"
version_detail = _unwrap(AgentRosterVersionDetailApi.get)(AgentRosterVersionDetailApi(), agent_id, version_id)
assert version_detail["id"] == version_id
assert version_detail["agent_id"] == agent_id
def test_workflow_composer_get_put_validate_candidates_impact_and_save(app, monkeypatch):
@@ -145,50 +246,52 @@ def test_workflow_composer_get_put_validate_candidates_impact_and_save(app, monk
monkeypatch.setattr(
composer_controller.AgentComposerService,
"load_workflow_composer",
lambda **kwargs: {"node_id": kwargs["node_id"]},
lambda **kwargs: _workflow_composer_response(node_id=kwargs["node_id"]),
)
monkeypatch.setattr(
composer_controller.AgentComposerService,
"save_workflow_composer",
lambda **kwargs: {"saved": kwargs["payload"].save_strategy.value, "account_id": kwargs["account_id"]},
lambda **kwargs: _workflow_composer_response(save_options=[kwargs["payload"].save_strategy.value]),
)
monkeypatch.setattr(composer_controller.ComposerConfigValidator, "validate_save_payload", lambda payload: None)
monkeypatch.setattr(
composer_controller.AgentComposerService,
"get_workflow_candidates",
lambda **kwargs: {"data": []},
lambda **kwargs: _candidates_response("workflow"),
)
monkeypatch.setattr(
composer_controller.AgentComposerService,
"calculate_impact",
lambda **kwargs: {"current_snapshot_id": kwargs["current_snapshot_id"], "workflow_node_count": 1},
lambda **kwargs: {
"current_snapshot_id": kwargs["current_snapshot_id"],
"workflow_node_count": 1,
"bindings": [],
},
)
assert _unwrap(WorkflowAgentComposerApi.get)(WorkflowAgentComposerApi(), app_model, "node-1") == {
"node_id": "node-1"
}
workflow_state = _unwrap(WorkflowAgentComposerApi.get)(WorkflowAgentComposerApi(), app_model, "node-1")
assert workflow_state["node_id"] == "node-1"
with app.test_request_context(json=payload):
assert _unwrap(WorkflowAgentComposerApi.put)(WorkflowAgentComposerApi(), app_model, "node-1") == {
"saved": "node_job_only",
"account_id": "account-1",
}
saved_state = _unwrap(WorkflowAgentComposerApi.put)(WorkflowAgentComposerApi(), app_model, "node-1")
assert saved_state["save_options"] == ["node_job_only"]
assert _unwrap(WorkflowAgentComposerValidateApi.post)(
WorkflowAgentComposerValidateApi(), app_model, "node-1"
) == {"result": "success", "errors": []}
assert _unwrap(WorkflowAgentComposerCandidatesApi.get)(
WorkflowAgentComposerCandidatesApi(), app_model, "node-1"
) == {"data": []}
assert (
_unwrap(WorkflowAgentComposerCandidatesApi.get)(WorkflowAgentComposerCandidatesApi(), app_model, "node-1")[
"variant"
]
== "workflow"
)
with app.test_request_context(json=payload):
assert _unwrap(WorkflowAgentComposerImpactApi.post)(WorkflowAgentComposerImpactApi(), app_model, "node-1") == {
"current_snapshot_id": "version-1",
"workflow_node_count": 1,
"bindings": [],
}
assert (
_unwrap(WorkflowAgentComposerSaveToRosterApi.post)(
WorkflowAgentComposerSaveToRosterApi(), app_model, "node-1"
)["saved"]
== "node_job_only"
)
assert _unwrap(WorkflowAgentComposerSaveToRosterApi.post)(
WorkflowAgentComposerSaveToRosterApi(), app_model, "node-1"
)["save_options"] == ["node_job_only"]
def test_workflow_impact_returns_empty_without_version(app):
@@ -212,28 +315,26 @@ def test_agent_app_composer_get_put_validate_and_candidates(app, monkeypatch):
monkeypatch.setattr(
composer_controller.AgentComposerService,
"load_agent_app_composer",
lambda **kwargs: {"loaded": True},
lambda **kwargs: _agent_app_composer_response(),
)
monkeypatch.setattr(
composer_controller.AgentComposerService,
"save_agent_app_composer",
lambda **kwargs: {"saved": kwargs["payload"].variant.value, "account_id": kwargs["account_id"]},
lambda **kwargs: _agent_app_composer_response(),
)
monkeypatch.setattr(composer_controller.ComposerConfigValidator, "validate_save_payload", lambda payload: None)
monkeypatch.setattr(
composer_controller.AgentComposerService,
"get_agent_app_candidates",
lambda **kwargs: {"data": []},
lambda **kwargs: _candidates_response("agent_app"),
)
assert _unwrap(AgentAppComposerApi.get)(AgentAppComposerApi(), app_model) == {"loaded": True}
assert _unwrap(AgentAppComposerApi.get)(AgentAppComposerApi(), app_model)["variant"] == "agent_app"
with app.test_request_context(json=payload):
assert _unwrap(AgentAppComposerApi.put)(AgentAppComposerApi(), app_model) == {
"saved": "agent_app",
"account_id": "account-1",
}
assert _unwrap(AgentAppComposerApi.put)(AgentAppComposerApi(), app_model)["variant"] == "agent_app"
assert _unwrap(AgentAppComposerValidateApi.post)(AgentAppComposerValidateApi(), app_model) == {
"result": "success",
"errors": [],
}
assert _unwrap(AgentAppComposerCandidatesApi.get)(AgentAppComposerCandidatesApi(), app_model) == {"data": []}
agent_app_candidates = _unwrap(AgentAppComposerCandidatesApi.get)(AgentAppComposerCandidatesApi(), app_model)
assert agent_app_candidates["variant"] == "agent_app"
@@ -1036,6 +1036,48 @@ class TestSegmentListAdvancedCases:
assert status == 200
assert response["total"] == 1
def test_segment_list_postgres_keyword_filter_handles_scalar_keywords(self, app: Flask):
api = DatasetDocumentSegmentListApi()
method = unwrap(api.get)
dataset = MagicMock()
document = MagicMock()
pagination = MagicMock(items=[], total=0, pages=0)
with (
app.test_request_context("/?keyword=test"),
patch(
"controllers.console.datasets.datasets_segments.current_account_with_tenant",
return_value=(MagicMock(), "11111111-1111-1111-1111-111111111111"),
),
patch(
"controllers.console.datasets.datasets_segments.DatasetService.get_dataset",
return_value=dataset,
),
patch(
"controllers.console.datasets.datasets_segments.DatasetService.check_dataset_permission",
return_value=None,
),
patch(
"controllers.console.datasets.datasets_segments.DocumentService.get_document",
return_value=document,
),
patch(
"controllers.console.datasets.datasets_segments.dify_config",
SimpleNamespace(SQLALCHEMY_DATABASE_URI_SCHEME="postgresql"),
),
patch(
"controllers.console.datasets.datasets_segments.db.paginate",
return_value=pagination,
) as paginate_mock,
):
method(api, "22222222-2222-2222-2222-222222222222", "33333333-3333-3333-3333-333333333333")
query = paginate_mock.call_args.kwargs["select"]
sql = str(query.compile(compile_kwargs={"literal_binds": True}))
assert "jsonb_array_elements_text(CASE" in sql
assert "ELSE CAST('[]' AS JSONB)" in sql
def test_segment_list_permission_denied(self, app: Flask):
"""Test segment list with permission denied"""
api = DatasetDocumentSegmentListApi()
@@ -3,22 +3,18 @@ from unittest.mock import MagicMock, patch
import pytest
from flask import Flask
from werkzeug.exceptions import HTTPException
import services
from controllers.console.auth.error import (
CannotTransferOwnerToSelfError,
EmailCodeError,
InvalidEmailError,
InvalidTokenError,
MemberNotInTenantError,
NotOwnerError,
OwnerTransferLimitError,
)
from controllers.console.error import EmailSendIpLimitError, WorkspaceMembersLimitExceeded
from controllers.console.workspace.members import (
DatasetOperatorMemberListApi,
MemberCancelInviteApi,
MemberInviteEmailApi,
MemberListApi,
MemberUpdateRoleApi,
@@ -251,135 +247,7 @@ class TestMemberInviteEmailApi:
assert result["invitation_results"][0]["status"] == "failed"
class TestMemberCancelInviteApi:
def test_cancel_success(self, app: Flask):
api = MemberCancelInviteApi()
method = unwrap(api.delete)
tenant = MagicMock(id="t1")
user = MagicMock(current_tenant=tenant)
member = MagicMock()
with (
app.test_request_context("/"),
patch("controllers.console.workspace.members.current_account_with_tenant", return_value=(user, "t1")),
patch("controllers.console.workspace.members.db.session.get") as get_mock,
patch("controllers.console.workspace.members.TenantService.remove_member_from_tenant"),
):
get_mock.return_value = member
result, status = method(api, member.id)
assert status == 200
assert result["result"] == "success"
def test_cancel_not_found(self, app: Flask):
api = MemberCancelInviteApi()
method = unwrap(api.delete)
tenant = MagicMock(id="t1")
user = MagicMock(current_tenant=tenant)
with (
app.test_request_context("/"),
patch("controllers.console.workspace.members.current_account_with_tenant", return_value=(user, "t1")),
patch("controllers.console.workspace.members.db.session.get") as get_mock,
):
get_mock.return_value = None
with pytest.raises(HTTPException):
method(api, "x")
def test_cancel_cannot_operate_self(self, app: Flask):
api = MemberCancelInviteApi()
method = unwrap(api.delete)
tenant = MagicMock(id="t1")
user = MagicMock(current_tenant=tenant)
member = MagicMock()
with (
app.test_request_context("/"),
patch("controllers.console.workspace.members.current_account_with_tenant", return_value=(user, "t1")),
patch("controllers.console.workspace.members.db.session.get") as get_mock,
patch(
"controllers.console.workspace.members.TenantService.remove_member_from_tenant",
side_effect=services.errors.account.CannotOperateSelfError("x"),
),
):
get_mock.return_value = member
result, status = method(api, member.id)
assert status == 400
def test_cancel_no_permission(self, app: Flask):
api = MemberCancelInviteApi()
method = unwrap(api.delete)
tenant = MagicMock(id="t1")
user = MagicMock(current_tenant=tenant)
member = MagicMock()
with (
app.test_request_context("/"),
patch("controllers.console.workspace.members.current_account_with_tenant", return_value=(user, "t1")),
patch("controllers.console.workspace.members.db.session.get") as get_mock,
patch(
"controllers.console.workspace.members.TenantService.remove_member_from_tenant",
side_effect=services.errors.account.NoPermissionError("x"),
),
):
get_mock.return_value = member
result, status = method(api, member.id)
assert status == 403
def test_cancel_member_not_in_tenant(self, app: Flask):
api = MemberCancelInviteApi()
method = unwrap(api.delete)
tenant = MagicMock(id="t1")
user = MagicMock(current_tenant=tenant)
member = MagicMock()
with (
app.test_request_context("/"),
patch("controllers.console.workspace.members.current_account_with_tenant", return_value=(user, "t1")),
patch("controllers.console.workspace.members.db.session.get") as get_mock,
patch(
"controllers.console.workspace.members.TenantService.remove_member_from_tenant",
side_effect=services.errors.account.MemberNotInTenantError(),
),
):
get_mock.return_value = member
result, status = method(api, member.id)
assert status == 404
class TestMemberUpdateRoleApi:
def test_update_success(self, app: Flask):
api = MemberUpdateRoleApi()
method = unwrap(api.put)
tenant = MagicMock()
user = MagicMock(current_tenant=tenant)
member = MagicMock()
payload = {"role": "normal"}
with (
app.test_request_context("/", json=payload),
patch("controllers.console.workspace.members.current_account_with_tenant", return_value=(user, "t1")),
patch("controllers.console.workspace.members.db.session.get", return_value=member),
patch("controllers.console.workspace.members.TenantService.update_member_role"),
):
result = method(api, "id")
if isinstance(result, tuple):
result = result[0]
assert result["result"] == "success"
def test_update_invalid_role(self, app: Flask):
api = MemberUpdateRoleApi()
method = unwrap(api.put)
@@ -391,23 +259,6 @@ class TestMemberUpdateRoleApi:
assert status == 400
def test_update_member_not_found(self, app: Flask):
api = MemberUpdateRoleApi()
method = unwrap(api.put)
payload = {"role": "normal"}
with (
app.test_request_context("/", json=payload),
patch(
"controllers.console.workspace.members.current_account_with_tenant",
return_value=(MagicMock(current_tenant=MagicMock()), "t1"),
),
patch("controllers.console.workspace.members.db.session.get", return_value=None),
):
with pytest.raises(HTTPException):
method(api, "id")
class TestDatasetOperatorMemberListApi:
def test_get_success(self, app: Flask):
@@ -637,27 +488,3 @@ class TestOwnerTransferApi:
):
with pytest.raises(InvalidTokenError):
method(api, "2")
def test_member_not_in_tenant(self, app: Flask):
api = OwnerTransfer()
method = unwrap(api.post)
tenant = MagicMock()
user = MagicMock(id="1", email="a@test.com", current_tenant=tenant)
member = MagicMock()
payload = {"token": "t"}
with (
app.test_request_context("/", json=payload),
patch("controllers.console.workspace.members.current_account_with_tenant", return_value=(user, "t1")),
patch("controllers.console.workspace.members.TenantService.is_owner", return_value=True),
patch(
"controllers.console.workspace.members.AccountService.get_owner_transfer_data",
return_value={"email": "a@test.com"},
),
patch("controllers.console.workspace.members.db.session.get", return_value=member),
patch("controllers.console.workspace.members.TenantService.is_member", return_value=False),
):
with pytest.raises(MemberNotInTenantError):
method(api, "2")
@@ -19,10 +19,12 @@ from unittest.mock import Mock
import pytest
from flask import Flask
from flask_restx.api import HTTPStatus
from pydantic import ValidationError
from controllers.service_api.app.annotation import (
AnnotationCreatePayload,
AnnotationListApi,
AnnotationListQuery,
AnnotationReplyActionApi,
AnnotationReplyActionPayload,
AnnotationReplyActionStatusApi,
@@ -106,6 +108,28 @@ class TestAnnotationReplyActionPayload:
assert payload.score_threshold == 0.0
class TestAnnotationListQuery:
def test_defaults(self) -> None:
query = AnnotationListQuery.model_validate({})
assert query.page == 1
assert query.limit == 20
assert query.keyword == ""
def test_valid_numeric_strings(self) -> None:
query = AnnotationListQuery.model_validate({"page": "2", "limit": "5", "keyword": "refund"})
assert query.page == 2
assert query.limit == 5
assert query.keyword == "refund"
@pytest.mark.parametrize("field", ["page", "limit"])
@pytest.mark.parametrize("value", ["abc", "1.5", "1e2", "", "0", "-1"])
def test_invalid_explicit_pagination_value(self, field: str, value: str) -> None:
with pytest.raises(ValidationError):
AnnotationListQuery.model_validate({field: value})
# ---------------------------------------------------------------------------
# Model and Error Pattern Tests
# ---------------------------------------------------------------------------
@@ -232,22 +256,55 @@ class TestAnnotationReplyActionStatusApi:
class TestAnnotationListApi:
def test_get(self, app: Flask, monkeypatch: pytest.MonkeyPatch) -> None:
def test_get_uses_defaults(self, app: Flask, monkeypatch: pytest.MonkeyPatch) -> None:
annotation = SimpleNamespace(id="a1", question="q", content="a", created_at=0)
monkeypatch.setattr(
AppAnnotationService,
"get_annotation_list_by_app_id",
lambda *_args, **_kwargs: ([annotation], 1),
)
get_mock = Mock(return_value=([annotation], 1))
monkeypatch.setattr(AppAnnotationService, "get_annotation_list_by_app_id", get_mock)
api = AnnotationListApi()
handler = _unwrap(api.get)
app_model = SimpleNamespace(id="app")
with app.test_request_context("/apps/annotations?page=1&limit=1", method="GET"):
with app.test_request_context("/apps/annotations", method="GET"):
response = handler(api, app_model=app_model)
assert response["page"] == 1
assert response["limit"] == 20
get_mock.assert_called_once_with("app", 1, 20, "")
def test_get_accepts_valid_numeric_strings(self, app: Flask, monkeypatch: pytest.MonkeyPatch) -> None:
annotation = SimpleNamespace(id="a1", question="q", content="a", created_at=0)
get_mock = Mock(return_value=([annotation], 1))
monkeypatch.setattr(AppAnnotationService, "get_annotation_list_by_app_id", get_mock)
api = AnnotationListApi()
handler = _unwrap(api.get)
app_model = SimpleNamespace(id="app")
with app.test_request_context("/apps/annotations?page=2&limit=5&keyword=refund", method="GET"):
response = handler(api, app_model=app_model)
assert response["total"] == 1
assert response["page"] == 2
assert response["limit"] == 5
get_mock.assert_called_once_with("app", 2, 5, "refund")
@pytest.mark.parametrize("query_string", ["page=abc&limit=5", "page=1&limit=abc", "page=&limit=5", "limit=0"])
def test_get_rejects_invalid_explicit_pagination_value(
self, app: Flask, monkeypatch: pytest.MonkeyPatch, query_string: str
) -> None:
get_mock = Mock(return_value=([], 0))
monkeypatch.setattr(AppAnnotationService, "get_annotation_list_by_app_id", get_mock)
api = AnnotationListApi()
handler = _unwrap(api.get)
app_model = SimpleNamespace(id="app")
with app.test_request_context(f"/apps/annotations?{query_string}", method="GET"):
with pytest.raises(ValidationError):
handler(api, app_model=app_model)
get_mock.assert_not_called()
def test_create(self, app: Flask, monkeypatch: pytest.MonkeyPatch) -> None:
annotation = SimpleNamespace(id="a1", question="q", content="a", created_at=0)
@@ -0,0 +1,42 @@
import logging
from types import SimpleNamespace
from core.tools.errors import ToolProviderNotFoundError
from events.event_handlers import delete_tool_parameters_cache_when_sync_draft_workflow as handler_module
def test_missing_tool_provider_does_not_log_error_traceback(monkeypatch, caplog):
app = SimpleNamespace(id="workflow-id", tenant_id="tenant-id")
workflow = SimpleNamespace(
graph_dict={
"nodes": [
{
"id": "node-id",
"data": {
"type": "tool",
},
}
]
}
)
tool_entity = SimpleNamespace(
provider_type=SimpleNamespace(value="mcp"),
provider_id="my-test-mcp-server",
provider_name="my-test-mcp-server",
tool_name="echo",
credential_id=None,
)
monkeypatch.setattr(handler_module, "adapt_node_config_for_graph", lambda node_data: {"data": node_data["data"]})
monkeypatch.setattr(handler_module.ToolEntity, "model_validate", lambda data: tool_entity)
monkeypatch.setattr(
handler_module.ToolManager,
"get_tool_runtime",
lambda **kwargs: (_ for _ in ()).throw(ToolProviderNotFoundError("mcp provider not found")),
)
with caplog.at_level(logging.INFO, logger=handler_module.logger.name):
handler_module.handle(app, synced_draft_workflow=workflow)
assert not [record for record in caplog.records if record.levelno >= logging.ERROR]
assert "Skipped deleting tool parameters cache" in caplog.text
@@ -0,0 +1,115 @@
from services.data_migration.dependency_discovery_service import DependencyDiscoveryService
from services.data_migration.entities import DependencyKind
def test_discovers_and_deduplicates_standalone_tool_nodes():
graph = {
"graph": {
"nodes": [
{"data": {"type": "tool", "provider_type": "api", "provider_id": "weather"}},
{"data": {"type": "tool", "provider_type": "api", "provider_id": "weather"}},
{"data": {"type": "tool", "provider_type": "workflow", "provider_id": "wf-tool-1"}},
{"data": {"type": "tool", "provider_type": "builtin", "provider_id": "google_search"}},
]
}
}
dependencies = DependencyDiscoveryService().discover_from_dsl(graph)
assert [(item.kind, item.provider_id) for item in dependencies] == [
(DependencyKind.API_TOOL, "weather"),
(DependencyKind.WORKFLOW_TOOL, "wf-tool-1"),
(DependencyKind.BUILTIN_OR_PLUGIN_TOOL, "google_search"),
]
def test_discovers_agent_node_tools():
graph = {
"graph": {
"nodes": [
{
"data": {
"type": "agent",
"tools": [
{"provider_type": "mcp", "provider_id": "mcp-1"},
{"provider_type": "api", "provider_id": "api-1"},
],
}
}
]
}
}
dependencies = DependencyDiscoveryService().discover_from_dsl(graph)
assert [(item.kind, item.provider_id) for item in dependencies] == [
(DependencyKind.MCP_TOOL, "mcp-1"),
(DependencyKind.API_TOOL, "api-1"),
]
def test_discovers_tool_nodes_from_exported_workflow_dsl_shape():
dsl = {
"workflow": {
"graph": {
"nodes": [
{
"data": {
"type": "tool",
"provider_type": "api",
"provider_id": "api-provider-id",
"provider_name": "weather",
}
},
{
"data": {
"type": "tool",
"provider_type": "workflow",
"provider_id": "workflow-tool-id",
"provider_name": "embedded_workflow",
}
},
]
}
}
}
dependencies = DependencyDiscoveryService().discover_from_dsl(dsl)
assert [(item.kind, item.provider_id, item.provider_name) for item in dependencies] == [
(DependencyKind.API_TOOL, "api-provider-id", "weather"),
(DependencyKind.WORKFLOW_TOOL, "workflow-tool-id", "embedded_workflow"),
]
def test_discovers_agent_tools_from_exported_agent_parameter_shape():
dsl = {
"workflow": {
"graph": {
"nodes": [
{
"data": {
"type": "agent",
"agent_parameters": {
"tools": {
"value": [
{
"provider_type": "api",
"provider_id": "api-provider-id",
"provider_name": "weather",
}
]
}
},
}
}
]
}
}
}
dependencies = DependencyDiscoveryService().discover_from_dsl(dsl)
assert [(item.kind, item.provider_id, item.provider_name) for item in dependencies] == [
(DependencyKind.API_TOOL, "api-provider-id", "weather"),
]
@@ -0,0 +1,71 @@
import pytest
from services.data_migration.entities import (
ConflictStrategy,
IdStrategy,
ImportOptions,
MigrationDataError,
MigrationMetadata,
MigrationPackage,
SourceTenant,
)
def test_import_options_defaults_match_spec():
options = ImportOptions()
assert options.create_app_api_token_on_import is False
assert options.id_strategy == IdStrategy.PRESERVE_ID
assert options.conflict_strategy == ConflictStrategy.FAIL
def test_metadata_requires_version():
with pytest.raises(MigrationDataError, match="metadata.version"):
MigrationMetadata.from_mapping({})
def test_metadata_parses_source_tenants_and_import_options():
metadata = MigrationMetadata.from_mapping(
{
"version": "1",
"source_scope": "single",
"source_tenants": [{"id": "tenant-1", "name": "source"}],
"target_tenant": {"name": "prod"},
"include_secrets": False,
"import_options": {"conflict_strategy": "update"},
}
)
assert metadata.version == "1"
assert metadata.source_scope == "single"
assert metadata.source_tenants == [SourceTenant(id="tenant-1", name="source")]
assert metadata.target_tenant == {"name": "prod"}
assert metadata.import_options.conflict_strategy == ConflictStrategy.UPDATE
assert metadata.import_options.id_strategy == IdStrategy.PRESERVE_ID
def test_import_options_invalid_strategy_raises_domain_error():
with pytest.raises(MigrationDataError, match="id_strategy"):
ImportOptions.from_mapping({"id_strategy": "unknown"})
with pytest.raises(MigrationDataError, match="id_strategy"):
ImportOptions.from_mapping({"id_strategy": "map-id"})
with pytest.raises(MigrationDataError, match="conflict_strategy"):
ImportOptions.from_mapping({"conflict_strategy": "unknown"})
with pytest.raises(MigrationDataError, match="conflict_strategy"):
ImportOptions.from_mapping({"conflict_strategy": "replace"})
def test_metadata_rejects_invalid_target_tenant_shape():
with pytest.raises(MigrationDataError, match="target_tenant"):
MigrationMetadata.from_mapping({"version": "1", "target_tenant": "prod"})
def test_migration_package_sections_must_be_lists_of_objects():
with pytest.raises(MigrationDataError, match="workflows"):
MigrationPackage.from_mapping({"metadata": {"version": "1"}, "workflows": {"id": "app-1"}})
with pytest.raises(MigrationDataError, match="tools"):
MigrationPackage.from_mapping({"metadata": {"version": "1"}, "tools": ["weather"]})
@@ -0,0 +1,201 @@
import pytest
from services.data_migration.dependency_discovery_service import DiscoveredDependency
from services.data_migration.entities import (
ConflictStrategy,
DependencyKind,
IdStrategy,
MigrationDataError,
ResourceType,
)
from services.data_migration.export_service import ExportConfigParser, MigrationExportService
def test_export_config_parser_accepts_new_scripted_shape():
selection = ExportConfigParser().parse(
{
"source_tenant": {"mode": "single", "name": "admin's Workspace"},
"apps": {"modes": ["workflow", "advanced-chat"], "ids": ["app-1"], "all": False},
"include_referenced_tools": True,
"additional_tools": {
"api_tools": ["weather"],
"workflow_tools": ["workflow-tool-1"],
"mcp_tools": ["mcp-1"],
},
"include_secrets": False,
"import_options": {
"create_app_api_token_on_import": True,
"id_strategy": "preserve-id",
"conflict_strategy": "fail",
},
}
)
assert selection.source_tenant_name == "admin's Workspace"
assert selection.app_ids == ["app-1"]
assert selection.export_all_apps is False
assert selection.additional_api_tools == ["weather"]
assert selection.additional_workflow_tools == ["workflow-tool-1"]
assert selection.additional_mcp_tools == ["mcp-1"]
assert selection.include_secrets is False
assert selection.import_options.create_app_api_token_on_import is True
assert selection.import_options.id_strategy == IdStrategy.PRESERVE_ID
assert selection.import_options.conflict_strategy == ConflictStrategy.FAIL
def test_export_config_parser_defaults_to_secret_free_all_apps():
selection = ExportConfigParser().parse(
{
"source_tenant": {"name": "source"},
"apps": {"all": True},
}
)
assert selection.export_all_apps is True
assert selection.include_referenced_tools is True
assert selection.include_secrets is False
def test_export_config_parser_requires_explicit_source_tenant_name_for_new_shape():
with pytest.raises(MigrationDataError, match="source_tenant.name"):
ExportConfigParser().parse({"source_tenant": {"mode": "single"}, "apps": {"all": True}})
def test_export_config_parser_accepts_limited_backwards_draft_shape():
selection = ExportConfigParser().parse(
{
"tenant_name": "legacy-source",
"workflows": ["app-1"],
"tools": ["weather"],
"workflow_tools": ["wf-tool-1"],
"mcp_tools": ["mcp-1"],
"export_all_workflows": False,
}
)
assert selection.source_tenant_name == "legacy-source"
assert selection.app_ids == ["app-1"]
assert selection.additional_api_tools == ["weather"]
assert selection.additional_workflow_tools == ["wf-tool-1"]
assert selection.additional_mcp_tools == ["mcp-1"]
assert selection.include_secrets is False
def test_export_config_parser_rejects_unsupported_app_modes():
with pytest.raises(MigrationDataError, match="Unsupported app modes"):
ExportConfigParser().parse(
{
"source_tenant": {"name": "source"},
"apps": {"modes": ["chat"], "all": True},
}
)
def test_secret_free_api_tool_export_uses_masking_and_omits_credentials(monkeypatch):
calls = []
def fake_get_api_provider(provider: str, tenant_id: str, mask: bool = True):
calls.append((provider, tenant_id, mask))
return {"credentials": {"api_key": "masked"}, "schema": {"openapi": "3.0.0"}, "tools": ["unused"]}
monkeypatch.setattr(
"services.data_migration.export_service.ToolManager.user_get_api_provider",
fake_get_api_provider,
)
service = MigrationExportService()
tools: list[dict] = []
report_items = []
service._export_api_tools(
"tenant-1",
["weather"],
include_secrets=False,
exported_tools=tools,
report_items=report_items,
)
assert calls == [("weather", "tenant-1", True)]
assert tools == [{"schema": {"openapi": "3.0.0"}, "provider_name": "weather", "source_tenant_id": "tenant-1"}]
assert report_items[0].resource_type == ResourceType.API_TOOL
def test_secret_free_mcp_dependencies_are_dependency_only():
service = MigrationExportService()
dependencies: list[dict] = []
mcp_tools: list[dict] = []
report_items = []
service._export_mcp_tools(
tenant_id="tenant-1",
provider_ids=["mcp-1"],
include_secrets=False,
exported_mcp_tools=mcp_tools,
dependencies=dependencies,
report_items=report_items,
)
assert mcp_tools == []
assert dependencies == [
{
"kind": DependencyKind.MCP_TOOL.value,
"provider_id": "mcp-1",
"provider_name": None,
"source": "mcp_provider",
}
]
assert report_items[0].status == "dependency-only"
assert report_items[0].name == "mcp_tool mcp-1"
def test_get_mcp_provider_does_not_compare_non_uuid_identifier_to_uuid_id(monkeypatch):
statements = []
def capture_scalar(statement):
statements.append(str(statement))
monkeypatch.setattr("services.data_migration.export_service.db.session.scalar", capture_scalar)
with pytest.raises(MigrationDataError, match="MCP provider not found"):
MigrationExportService()._get_mcp_provider("tenant-1", "my-test-mcp")
assert len(statements) == 1
assert "tool_mcp_providers.id =" not in statements[0]
assert "tool_mcp_providers.server_identifier =" in statements[0]
def test_dependency_ids_are_deduplicated_with_manual_selection_first():
service = MigrationExportService()
provider_ids = service._provider_ids(
manual_provider_ids=["weather", "weather", "manual"],
discovered_dependencies=[
DiscoveredDependency(DependencyKind.API_TOOL, "weather"),
DiscoveredDependency(DependencyKind.API_TOOL, "forecast"),
DiscoveredDependency(DependencyKind.WORKFLOW_TOOL, "workflow-tool"),
],
kind=DependencyKind.API_TOOL,
)
assert provider_ids == ["weather", "manual", "forecast"]
def test_api_provider_ids_use_provider_name_from_discovered_dependencies():
service = MigrationExportService()
provider_ids = service._provider_ids(
manual_provider_ids=[],
discovered_dependencies=[
DiscoveredDependency(DependencyKind.API_TOOL, "api-provider-id", provider_name="weather"),
],
kind=DependencyKind.API_TOOL,
)
assert provider_ids == ["weather"]
def test_mcp_authentication_export_omits_runtime_header_shape():
service = MigrationExportService()
assert service._serialize_mcp_authentication({"Authorization": "Bearer token"}) is None
assert service._serialize_mcp_authentication({"client_id": "id", "client_secret": "secret"}) == {
"client_id": "id",
"client_secret": "secret",
}
@@ -0,0 +1,973 @@
import pytest
import yaml
from models.tools import MCPToolProvider, WorkflowToolProvider
from services.app_dsl_service import Import
from services.data_migration.entities import (
ConflictStrategy,
IdStrategy,
ImportOptions,
ImportTarget,
MigrationDataError,
MigrationPackage,
ResourceIdMapping,
ResourceReportItem,
ResourceType,
)
from services.data_migration.import_service import ImportRequest, ImportTargetResolver, MigrationImportService
from services.entities.dsl_entities import ImportStatus
def test_target_tenant_precedence_cli_then_config_then_package():
package = MigrationPackage.from_mapping(
{
"metadata": {
"version": "1",
"source_scope": "single",
"source_tenants": [],
"target_tenant": {"name": "from-package"},
}
}
)
resolver = ImportTargetResolver()
assert (
resolver.select_target_tenant_name(
ImportRequest(package=package, cli_target_tenant="from-cli", config_target_tenant="from-config")
)
== "from-cli"
)
assert (
resolver.select_target_tenant_name(
ImportRequest(package=package, cli_target_tenant=None, config_target_tenant="from-config")
)
== "from-config"
)
assert (
resolver.select_target_tenant_name(
ImportRequest(package=package, cli_target_tenant=None, config_target_tenant=None)
)
== "from-package"
)
def test_target_tenant_missing_fails_before_import():
package = MigrationPackage.from_mapping({"metadata": {"version": "1", "source_scope": "single"}})
with pytest.raises(MigrationDataError, match="Target tenant"):
ImportTargetResolver().select_target_tenant_name(
ImportRequest(package=package, cli_target_tenant=None, config_target_tenant=None)
)
def test_package_target_tenant_id_can_be_used_without_name():
package = MigrationPackage.from_mapping(
{"metadata": {"version": "1", "source_scope": "single", "target_tenant": {"id": "tenant-id"}}}
)
assert ImportTargetResolver().select_target_tenant_name(ImportRequest(package=package)) == "tenant-id"
def test_target_tenant_name_is_not_treated_as_uuid():
resolver = ImportTargetResolver()
assert resolver._is_uuid("admin's Workspace") is False
assert resolver._is_uuid("49a99e46-bc2c-4885-91fa-47615f6192b5") is True
def test_package_target_tenant_id_ignores_invalid_uuid(monkeypatch):
package = MigrationPackage.from_mapping(
{"metadata": {"version": "1", "source_scope": "single", "target_tenant": {"id": "not-a-uuid"}}}
)
class StubSession:
def get(self, model, identifier):
raise AssertionError("invalid UUID should not be passed to session.get")
def scalars(self, statement):
class EmptyResult:
def all(self):
return []
return EmptyResult()
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
with pytest.raises(MigrationDataError, match="Target tenant not found"):
ImportTargetResolver().resolve(ImportRequest(package=package))
def test_options_override_replaces_package_defaults():
package = MigrationPackage.from_mapping(
{
"metadata": {
"version": "1",
"source_scope": "single",
"target_tenant": {"name": "target"},
"import_options": {
"create_app_api_token_on_import": True,
"conflict_strategy": "update",
},
}
}
)
captured_options: list[ImportOptions] = []
class StubResolver(ImportTargetResolver):
def resolve(self, request: ImportRequest) -> ImportTarget:
return ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
)
class CapturingImportService(MigrationImportService):
def _import_workflows(
self,
package: MigrationPackage,
target: ImportTarget,
options: ImportOptions,
report_items: list[ResourceReportItem],
id_mapping: dict[str, str],
**kwargs,
) -> None:
captured_options.append(options)
override = ImportOptions(create_app_api_token_on_import=False, conflict_strategy=ConflictStrategy.SKIP)
CapturingImportService(target_resolver=StubResolver()).import_package(
ImportRequest(package=package, options_override=override)
)
assert captured_options == [override]
def test_only_preserve_id_strategy_reuses_source_app_id():
service = MigrationImportService()
assert service._should_preserve_source_app_id(ImportOptions(id_strategy=IdStrategy.PRESERVE_ID)) is True
assert service._should_preserve_source_app_id(ImportOptions(id_strategy=IdStrategy.GENERATE_NEW_ID)) is False
def test_find_existing_app_ignores_invalid_uuid(monkeypatch):
class StubSession:
def scalar(self, statement):
raise AssertionError("invalid UUID should not be queried against App.id")
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
assert MigrationImportService()._find_existing_app("not-a-uuid", "tenant-1") is None
def test_find_existing_workflow_tool_does_not_compare_invalid_uuid(monkeypatch):
captured = []
class StubSession:
def scalar(self, statement):
captured.append(statement)
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
MigrationImportService()._find_existing_workflow_tool("tenant-1", "not-a-uuid", "tool-name", "app-id")
where_clause = str(captured[0].whereclause)
assert f"{WorkflowToolProvider.__tablename__}.id" not in where_clause
def test_find_existing_mcp_tool_does_not_compare_invalid_uuid(monkeypatch):
captured = []
class StubSession:
def scalar(self, statement):
captured.append(statement)
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
MigrationImportService()._find_existing_mcp_tool("tenant-1", "my-test-mcp", "my-test-mcp")
where_clause = str(captured[0].whereclause)
assert f"{MCPToolProvider.__tablename__}.id" not in where_clause
assert f"{MCPToolProvider.__tablename__}.name" not in where_clause
def test_workflow_app_import_does_not_wrap_app_dsl_import_in_nested_transaction(monkeypatch):
class FailingNestedTransaction:
def __enter__(self):
raise AssertionError("nested transaction should not be opened")
def __exit__(self, exc_type, exc_value, traceback):
return False
class StubSession:
def begin_nested(self):
return FailingNestedTransaction()
def commit(self):
return None
class StubAppDslService:
def __init__(self, session):
self.session = session
def import_app(self, **kwargs):
return Import(id="import-id", status=ImportStatus.COMPLETED, app_id="imported-app-id")
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
monkeypatch.setattr(import_service, "AppDslService", StubAppDslService)
imported_app_id = MigrationImportService()._import_workflow_app(
account=object(),
workflow_data={"name": "main_chatflow"},
dsl_content="app:\n mode: workflow\n",
app_id="source-app-id",
existing_app=None,
options=ImportOptions(id_strategy=IdStrategy.PRESERVE_ID),
)
assert imported_app_id == "imported-app-id"
def test_rewrite_workflow_dsl_replaces_tool_provider_ids():
dsl_content = yaml.safe_dump(
{
"app": {"mode": "workflow"},
"workflow": {
"graph": {
"nodes": [
{
"data": {
"type": "tool",
"provider_id": "source-api-provider-id",
"provider_name": "weather",
}
},
{
"data": {
"type": "agent",
"agent_parameters": {
"tools": {
"value": [
{
"provider_id": "source-agent-provider-id",
"provider_name": "agent_weather",
}
]
}
},
}
},
]
}
},
}
)
rewritten = MigrationImportService()._rewrite_workflow_dsl_provider_ids(
dsl_content,
{
"source-api-provider-id": "target-api-provider-id",
"source-agent-provider-id": "target-agent-provider-id",
},
)
graph = yaml.safe_load(rewritten)["workflow"]["graph"]
assert graph["nodes"][0]["data"]["provider_id"] == "target-api-provider-id"
assert (
graph["nodes"][1]["data"]["agent_parameters"]["tools"]["value"][0]["provider_id"] == "target-agent-provider-id"
)
def test_source_api_provider_ids_are_discovered_from_workflow_dsl():
package = MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"workflows": [
{
"dsl": yaml.safe_dump(
{
"workflow": {
"graph": {
"nodes": [
{
"data": {
"type": "tool",
"provider_id": "source-api-provider-id",
"provider_name": "weather",
"provider_type": "api",
}
}
]
}
}
}
)
}
],
}
)
assert MigrationImportService()._source_api_provider_ids_by_name(package) == {"weather": {"source-api-provider-id"}}
def test_workflow_tool_import_publishes_referenced_app_before_create(monkeypatch):
events = []
account = type("Account", (), {"id": "account-1"})()
class StubSession:
def get(self, model, identifier):
return account
class PublishingImportService(MigrationImportService):
def _find_existing_app(self, app_id, tenant_id):
return object()
def _find_existing_workflow_tool(self, tenant_id, workflow_tool_id, tool_name, app_id):
if ("created", app_id) in events:
return type("WorkflowToolProvider", (), {"id": workflow_tool_id or "created-workflow-tool-id"})()
return None
def _ensure_workflow_app_is_published(self, target, account, app_id):
events.append(("published", app_id))
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
monkeypatch.setattr(
import_service.WorkflowToolManageService,
"create_workflow_tool",
lambda **kwargs: events.append(("created", kwargs["workflow_app_id"])),
)
PublishingImportService()._import_workflow_tools(
MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"workflow_tools": [
{
"id": "workflow-tool-1",
"name": "embedded_workflow_as_tool",
"app_id": "workflow-app-1",
}
],
}
),
ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
),
ImportOptions(),
{},
[],
[],
)
assert events == [("published", "workflow-app-1"), ("created", "workflow-app-1")]
@pytest.mark.parametrize(
("id_strategy", "expected_import_id"),
[
(IdStrategy.PRESERVE_ID, "source-workflow-tool-id"),
(IdStrategy.GENERATE_NEW_ID, ""),
],
)
def test_workflow_tool_import_id_follows_id_strategy(monkeypatch, id_strategy, expected_import_id):
created_kwargs = []
target_provider = type("WorkflowToolProvider", (), {"id": "target-workflow-tool-id"})()
account = type("Account", (), {"id": "account-1"})()
id_mapping = {"source-app-id": "target-app-id"}
id_mapping_details = []
class StubSession:
def get(self, model, identifier):
return account
class StrategyImportService(MigrationImportService):
def _find_existing_app(self, app_id, tenant_id):
return object()
def _find_existing_workflow_tool(self, tenant_id, workflow_tool_id, tool_name, app_id):
return target_provider if created_kwargs else None
def _ensure_workflow_app_is_published(self, target, account, app_id):
return None
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
monkeypatch.setattr(
import_service.WorkflowToolManageService,
"create_workflow_tool",
lambda **kwargs: created_kwargs.append(kwargs),
)
StrategyImportService()._import_workflow_tools(
MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"workflow_tools": [
{
"id": "source-workflow-tool-id",
"name": "embedded_workflow_as_tool",
"app_id": "source-app-id",
}
],
}
),
ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
),
ImportOptions(id_strategy=id_strategy),
id_mapping,
id_mapping_details,
[],
)
assert created_kwargs[0]["import_id"] == expected_import_id
assert id_mapping["source-workflow-tool-id"] == "target-workflow-tool-id"
assert id_mapping_details == [
ResourceIdMapping(
ResourceType.WORKFLOW_TOOL,
"embedded_workflow_as_tool",
"source-workflow-tool-id",
"target-workflow-tool-id",
)
]
def test_workflow_tool_skip_records_id_mapping(monkeypatch):
account = type("Account", (), {"id": "account-1"})()
existing_provider = type("WorkflowToolProvider", (), {"id": "existing-workflow-tool-id"})()
id_mapping = {"source-app-id": "target-app-id"}
class StubSession:
def get(self, model, identifier):
return account
class SkipImportService(MigrationImportService):
def _find_existing_app(self, app_id, tenant_id):
return object()
def _find_existing_workflow_tool(self, tenant_id, workflow_tool_id, tool_name, app_id):
return existing_provider
def _ensure_workflow_app_is_published(self, target, account, app_id):
return None
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
SkipImportService()._import_workflow_tools(
MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"workflow_tools": [
{
"id": "source-workflow-tool-id",
"name": "embedded_workflow_as_tool",
"app_id": "source-app-id",
}
],
}
),
ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
),
ImportOptions(conflict_strategy=ConflictStrategy.SKIP, id_strategy=IdStrategy.GENERATE_NEW_ID),
id_mapping,
[],
[],
)
assert id_mapping["source-workflow-tool-id"] == "existing-workflow-tool-id"
@pytest.mark.parametrize("conflict_strategy", [ConflictStrategy.SKIP, ConflictStrategy.UPDATE])
def test_api_tool_existing_provider_records_id_mapping(monkeypatch, conflict_strategy):
target_provider = type("ApiToolProvider", (), {"id": "target-api-provider-id", "name": "weather"})()
id_mapping = {}
id_mapping_details = []
report_items = []
class ExistingApiImportService(MigrationImportService):
def _find_api_tool_provider(self, tenant_id, provider_name):
return target_provider
from services.data_migration import import_service
monkeypatch.setattr(import_service.db.session, "scalar", lambda statement: target_provider)
monkeypatch.setattr(
import_service.ApiToolManageService, "parser_api_schema", lambda schema: {"schema_type": "openapi"}
)
monkeypatch.setattr(import_service.ApiToolManageService, "update_api_tool_provider", lambda **kwargs: None)
ExistingApiImportService()._import_api_tools(
MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"tools": [{"id": "source-api-provider-id", "provider_name": "weather", "schema": "openapi: 3.0.0"}],
}
),
ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
),
ImportOptions(conflict_strategy=conflict_strategy),
report_items,
id_mapping,
id_mapping_details,
{"weather": {"source-api-provider-id-from-dsl"}},
)
assert id_mapping == {
"source-api-provider-id": "target-api-provider-id",
"source-api-provider-id-from-dsl": "target-api-provider-id",
}
assert (
ResourceIdMapping(ResourceType.API_TOOL, "weather", "source-api-provider-id", "target-api-provider-id")
in id_mapping_details
)
def test_api_tool_create_records_id_mapping(monkeypatch):
target_provider = type("ApiToolProvider", (), {"id": "target-api-provider-id", "name": "weather"})()
id_mapping = {}
class StubSession:
def scalar(self, statement):
return None
class CreatedApiImportService(MigrationImportService):
def _find_api_tool_provider(self, tenant_id, provider_name):
return target_provider
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
monkeypatch.setattr(
import_service.ApiToolManageService, "parser_api_schema", lambda schema: {"schema_type": "openapi"}
)
monkeypatch.setattr(import_service.ApiToolManageService, "create_api_tool_provider", lambda **kwargs: None)
CreatedApiImportService()._import_api_tools(
MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"tools": [{"id": "source-api-provider-id", "provider_name": "weather", "schema": "openapi: 3.0.0"}],
}
),
ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
),
ImportOptions(),
[],
id_mapping,
[],
{},
)
assert id_mapping["source-api-provider-id"] == "target-api-provider-id"
def test_mcp_tool_import_restores_exported_tool_list(monkeypatch):
provider = type("Provider", (), {"id": "target-provider-id", "tools": "[]", "authed": False})()
report_items = []
class StubSession:
def scalar(self, statement):
return provider
def commit(self):
return None
class StubMCPToolManageService:
def __init__(self, session):
self.session = session
def update_provider(self, **kwargs):
return None
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
monkeypatch.setattr(import_service, "MCPToolManageService", StubMCPToolManageService)
MigrationImportService()._import_mcp_tools(
MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"mcp_tools": [
{
"id": "source-provider-id",
"name": "my-test-mcp",
"server_identifier": "my-test-mcp",
"server_url": "http://localhost:3000/mcp",
"configuration": {},
"tools": [{"name": "echo"}],
}
],
}
),
ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
),
ImportOptions(conflict_strategy=ConflictStrategy.UPDATE),
report_items,
{},
[],
)
assert provider.tools == '[{"name": "echo"}]'
assert provider.authed is True
@pytest.mark.parametrize("conflict_strategy", [ConflictStrategy.SKIP, ConflictStrategy.UPDATE])
def test_mcp_tool_existing_provider_records_id_mapping(monkeypatch, conflict_strategy):
provider = type("Provider", (), {"id": "target-mcp-provider-id", "tools": "[]", "authed": False})()
id_mapping = {}
id_mapping_details = []
class StubSession:
def commit(self):
return None
class ExistingMCPImportService(MigrationImportService):
def _find_existing_mcp_tool(self, tenant_id, provider_id, server_identifier):
return provider
class StubMCPToolManageService:
def __init__(self, session):
self.session = session
def update_provider(self, **kwargs):
return None
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
monkeypatch.setattr(import_service, "MCPToolManageService", StubMCPToolManageService)
ExistingMCPImportService()._import_mcp_tools(
MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"mcp_tools": [
{
"id": "source-mcp-provider-id",
"name": "my-test-mcp",
"server_identifier": "my-test-mcp",
"server_url": "http://localhost:3000/mcp",
"configuration": {},
"tools": [{"name": "echo"}],
}
],
}
),
ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
),
ImportOptions(conflict_strategy=conflict_strategy),
[],
id_mapping,
id_mapping_details,
)
assert id_mapping["source-mcp-provider-id"] == "target-mcp-provider-id"
assert "my-test-mcp" not in id_mapping
assert id_mapping_details == [
ResourceIdMapping(ResourceType.MCP_TOOL, "my-test-mcp", "source-mcp-provider-id", "target-mcp-provider-id")
]
def test_mcp_tool_create_records_id_mapping(monkeypatch):
provider = type("Provider", (), {"id": "target-mcp-provider-id", "tools": "[]", "authed": False})()
id_mapping = {}
provider_created = False
class StubSession:
def commit(self):
return None
class CreatedMCPImportService(MigrationImportService):
def _find_existing_mcp_tool(self, tenant_id, provider_id, server_identifier):
return provider if provider_created else None
class StubMCPToolManageService:
def __init__(self, session):
self.session = session
def create_provider(self, **kwargs):
nonlocal provider_created
provider_created = True
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
monkeypatch.setattr(import_service, "MCPToolManageService", StubMCPToolManageService)
CreatedMCPImportService()._import_mcp_tools(
MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"mcp_tools": [
{
"id": "source-mcp-provider-id",
"name": "my-test-mcp",
"server_identifier": "my-test-mcp",
"server_url": "http://localhost:3000/mcp",
"configuration": {},
}
],
}
),
ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
),
ImportOptions(),
[],
id_mapping,
[],
)
assert id_mapping["source-mcp-provider-id"] == "target-mcp-provider-id"
def test_dependency_only_mcp_preflight_reports_missing_target_provider_with_workflow_context(monkeypatch):
report_items = []
package = MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"dependencies": [
{
"kind": "mcp_tool",
"provider_id": "my-test-mcp-server",
"provider_name": "my-test-mcp",
}
],
"workflows": [
{
"name": "workflow2",
"dsl": yaml.safe_dump(
{
"workflow": {
"graph": {
"nodes": [
{
"id": "node-1",
"data": {
"type": "tool",
"provider_type": "mcp",
"provider_id": "my-test-mcp-server",
"tool_name": "echo",
},
}
]
}
}
}
),
}
],
}
)
from services.data_migration import import_service
monkeypatch.setattr(import_service.db.session, "scalar", lambda statement: None)
MigrationImportService()._preflight_dependency_only_mcp(
package,
ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
),
report_items,
)
assert report_items == [
ResourceReportItem(
ResourceType.DEPENDENCY,
"my-test-mcp-server",
"mcp_tool my-test-mcp",
"skipped",
"missing in target tenant; referenced by workflow2 / echo; "
"configure it manually before running the workflow.",
)
]
def test_dependency_only_mcp_lookup_does_not_compare_non_uuid_identifier_to_uuid_id(monkeypatch):
captured = []
class StubSession:
def scalar(self, statement):
captured.append(statement)
from services.data_migration import import_service
monkeypatch.setattr(import_service.db, "session", StubSession())
MigrationImportService()._find_dependency_only_mcp_provider(
"tenant-1",
"my-test-mcp-server",
"my-test-mcp",
)
where_clause = str(captured[0].whereclause)
assert f"{MCPToolProvider.__tablename__}.id" not in where_clause
def test_dependency_only_mcp_preflight_reports_available_target_provider(monkeypatch):
report_items = []
package = MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"dependencies": [{"kind": "mcp_tool", "provider_id": "my-test-mcp-server"}],
}
)
provider = type(
"Provider",
(),
{"id": "target-provider-id", "name": "my-test-mcp", "server_identifier": "my-test-mcp-server"},
)()
from services.data_migration import import_service
monkeypatch.setattr(import_service.db.session, "scalar", lambda statement: provider)
MigrationImportService()._preflight_dependency_only_mcp(
package,
ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
),
report_items,
)
assert report_items == [
ResourceReportItem(
ResourceType.DEPENDENCY,
"my-test-mcp-server",
"mcp_tool my-test-mcp",
"available",
"MCP provider exists in target tenant.",
)
]
def test_import_package_imports_workflow_tool_provider_apps_before_consumers():
events = []
class StubResolver(ImportTargetResolver):
def resolve(self, request):
return ImportTarget(
tenant_id="tenant-1",
tenant_name="target",
operator_id="account-1",
operator_email="owner@example.com",
)
class OrderedImportService(MigrationImportService):
def _import_api_tools(
self,
package,
target,
options,
report_items,
id_mapping,
id_mapping_details,
source_provider_ids_by_name,
):
events.append(("api_tools", "imported"))
def _import_workflows(
self,
package,
target,
options,
report_items,
id_mapping,
id_mapping_details,
*,
imported_workflow_ids=None,
only_app_ids=None,
skip_app_ids=None,
):
only_app_ids = set(only_app_ids or [])
skip_app_ids = set(skip_app_ids or [])
for workflow_data in package.workflows:
app_id = workflow_data["id"]
if only_app_ids and app_id not in only_app_ids:
continue
if app_id in skip_app_ids:
continue
events.append(("workflow", app_id))
id_mapping[app_id] = app_id
if imported_workflow_ids is not None:
imported_workflow_ids.add(app_id)
def _import_workflow_tools(self, package, target, options, id_mapping, id_mapping_details, report_items):
events.append(("workflow_tool", package.workflow_tools[0]["id"]))
def _import_mcp_tools(self, package, target, options, report_items, id_mapping, id_mapping_details):
events.append(("mcp_tools", "imported"))
package = MigrationPackage.from_mapping(
{
"metadata": {"version": "1", "source_scope": "single"},
"workflows": [
{"id": "provider-app", "name": "embedded", "dsl": "app:\n mode: workflow\n"},
{"id": "consumer-app", "name": "main", "dsl": "app:\n mode: advanced-chat\n"},
],
"workflow_tools": [{"id": "workflow-tool", "name": "embedded_tool", "app_id": "provider-app"}],
}
)
OrderedImportService(target_resolver=StubResolver()).import_package(ImportRequest(package=package))
assert events == [
("api_tools", "imported"),
("mcp_tools", "imported"),
("workflow", "provider-app"),
("workflow_tool", "workflow-tool"),
("workflow", "consumer-app"),
]
@@ -0,0 +1,53 @@
import json
import pytest
from services.data_migration.entities import MigrationDataError
from services.data_migration.package_service import MigrationPackageService
def test_load_package_rejects_branch_draft_shape(tmp_path):
package_file = tmp_path / "draft.json"
package_file.write_text(json.dumps({"workflows": [], "tools": []}), encoding="utf-8")
with pytest.raises(MigrationDataError, match="metadata.version"):
MigrationPackageService().load_package(package_file)
def test_load_package_rejects_unsupported_version(tmp_path):
package_file = tmp_path / "future.json"
package_file.write_text(json.dumps({"metadata": {"version": "999", "source_scope": "single"}}), encoding="utf-8")
with pytest.raises(MigrationDataError, match="Unsupported migration package version"):
MigrationPackageService().load_package(package_file)
def test_save_package_writes_versioned_shape(tmp_path):
output_file = tmp_path / "migration-data.json"
package = MigrationPackageService().build_empty_package(
source_tenant_id="tenant-1",
source_tenant_name="source",
include_secrets=False,
)
MigrationPackageService().save_package(package, output_file, overwrite=False)
data = json.loads(output_file.read_text(encoding="utf-8"))
assert data["metadata"]["version"] == "1"
assert data["metadata"]["source_tenants"] == [{"id": "tenant-1", "name": "source"}]
assert data["metadata"]["include_secrets"] is False
assert data["workflows"] == []
assert data["dependencies"] == []
def test_save_package_without_overwrite_fails_when_file_exists(tmp_path):
output_file = tmp_path / "migration-data.json"
output_file.write_text("{}", encoding="utf-8")
package = MigrationPackageService().build_empty_package(
source_tenant_id="tenant-1",
source_tenant_name="source",
include_secrets=False,
)
with pytest.raises(MigrationDataError, match="already exists"):
MigrationPackageService().save_package(package, output_file, overwrite=False)
@@ -0,0 +1,111 @@
from services.data_migration.entities import ReportContext, ResourceIdMapping, ResourceReportItem, ResourceType
from services.data_migration.report_service import MigrationReportService
def test_report_summarizes_by_resource_type_and_status():
lines = MigrationReportService().render(
[
ResourceReportItem(ResourceType.WORKFLOW, "app-1", "App", "exported"),
ResourceReportItem(ResourceType.API_TOOL, "weather", "Weather", "exported"),
ResourceReportItem(ResourceType.DEPENDENCY, "mcp-1", "MCP", "dependency-only"),
]
)
assert "workflow exported: 1" in lines
assert "api_tool exported: 1" in lines
assert "dependency dependency-only: 1" in lines
def test_report_includes_actionable_lines_for_skipped_and_unresolved_items():
lines = MigrationReportService().render(
[
ResourceReportItem(ResourceType.WORKFLOW, "app-1", "App", "skipped", "App already exists"),
ResourceReportItem(ResourceType.DEPENDENCY, "mcp-1", None, "unresolved", "MCP provider not found"),
ResourceReportItem(ResourceType.API_TOOL, "weather", "Weather", "exported"),
]
)
assert "workflow skipped: 1" in lines
assert "dependency unresolved: 1" in lines
assert "workflow app-1: App already exists" in lines
assert "dependency mcp-1: MCP provider not found" in lines
def test_report_dependency_detail_uses_type_and_name_when_available():
lines = MigrationReportService().render(
[
ResourceReportItem(
ResourceType.DEPENDENCY,
"785e52f1-06bf-483c-8dcf-712e59fd43b9",
"workflow embedded_workflow",
"dependency-only",
"Dependency metadata only; ensure the resource exists in the target environment.",
),
ResourceReportItem(
ResourceType.DEPENDENCY,
"my-test-mcp",
"mcp_tool my-test-mcp",
"dependency-only",
"Configure MCP provider manually in the target tenant unless exporting with secrets enabled.",
),
]
)
assert (
"dependency workflow embedded_workflow: 785e52f1-06bf-483c-8dcf-712e59fd43b9: "
"Dependency metadata only; ensure the resource exists in the target environment."
) in lines
assert (
"dependency mcp_tool my-test-mcp: "
"Configure MCP provider manually in the target tenant unless exporting with secrets enabled."
) in lines
def test_report_includes_dependency_only_detail_lines():
lines = MigrationReportService().render(
[
ResourceReportItem(
ResourceType.DEPENDENCY,
"mcp-1",
"MCP",
"dependency-only",
"Configure manually in target tenant.",
),
]
)
assert "dependency dependency-only: 1" in lines
assert "dependency mcp-1: Configure manually in target tenant." in lines
def test_report_includes_export_and_import_context():
lines = MigrationReportService().render(
[],
context=ReportContext(
output_path="migration-data.json",
source_scope="single",
selected_app_count=2,
include_secrets=False,
target_tenant="prod",
operator_email="admin@example.com",
app_api_tokens_created=1,
app_api_tokens_reused=2,
id_mapping_count=3,
id_mappings={"source-app": "target-app", "source-tool": "target-tool"},
id_mapping_details=[
ResourceIdMapping(ResourceType.WORKFLOW, "Main workflow", "source-app", "target-app"),
ResourceIdMapping(ResourceType.API_TOOL, "weather", "source-tool", "target-tool"),
],
),
)
assert "output: migration-data.json" in lines
assert "source scope: single" in lines
assert "selected apps: 2" in lines
assert "include secrets: false" in lines
assert "target tenant: prod" in lines
assert "operator: admin@example.com" in lines
assert "app api tokens: 1 created, 2 reused" in lines
assert "resource references resolved: 2" in lines
assert "- workflow Main workflow: source-app -> target-app" in lines
assert "- api_tool weather: source-tool -> target-tool" in lines
+13 -6
View File
@@ -6,8 +6,8 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { startMock } from '../../test/fixtures/dify-mock/server.js'
import { loadAppInfoCache } from '../cache/app-info.js'
import { createClient } from '../http/client.js'
import { CACHE_APP_INFO, cachePath } from '../store/manager.js'
import { YamlStore } from '../store/store.js'
import { ENV_CACHE_DIR } from '../store/dir.js'
import { CACHE_APP_INFO, getCache } from '../store/manager.js'
import { FieldInfo, FieldParameters } from '../types/app-meta.js'
import { AppMetaClient } from './app-meta.js'
import { AppsClient } from './apps.js'
@@ -15,17 +15,24 @@ import { AppsClient } from './apps.js'
describe('AppMetaClient', () => {
let mock: DifyMock
let dir: string
let prevCacheDir: string | undefined
beforeEach(async () => {
mock = await startMock({ scenario: 'happy' })
dir = await mkdtemp(join(tmpdir(), 'difyctl-meta-'))
prevCacheDir = process.env[ENV_CACHE_DIR]
process.env[ENV_CACHE_DIR] = dir
})
afterEach(async () => {
if (prevCacheDir === undefined)
delete process.env[ENV_CACHE_DIR]
else
process.env[ENV_CACHE_DIR] = prevCacheDir
await mock.stop()
await rm(dir, { recursive: true, force: true })
})
it('cache miss → fetch → populate; warm hit skips network', async () => {
const cache = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const cache = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
const apps = new AppsClient(createClient({ host: mock.url, bearer: 'dfoa_test' }))
const spy = vi.spyOn(apps, 'describe')
const client = new AppMetaClient({ apps, host: mock.url, cache })
@@ -40,7 +47,7 @@ describe('AppMetaClient', () => {
})
it('slim hit + full request triggers fresh fetch + merges', async () => {
const cache = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const cache = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
const apps = new AppsClient(createClient({ host: mock.url, bearer: 'dfoa_test' }))
const spy = vi.spyOn(apps, 'describe')
const client = new AppMetaClient({ apps, host: mock.url, cache })
@@ -54,7 +61,7 @@ describe('AppMetaClient', () => {
})
it('expired cache entry refetches', async () => {
const cache = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)), ttlMs: 100, now: () => new Date('2026-05-09T00:00:00Z') })
const cache = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO), ttlMs: 100, now: () => new Date('2026-05-09T00:00:00Z') })
const apps = new AppsClient(createClient({ host: mock.url, bearer: 'dfoa_test' }))
const spy = vi.spyOn(apps, 'describe')
const client = new AppMetaClient({ apps, host: mock.url, cache, now: () => new Date('2026-05-09T00:00:00Z') })
@@ -68,7 +75,7 @@ describe('AppMetaClient', () => {
})
it('invalidate forces next get to fetch', async () => {
const cache = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const cache = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
const apps = new AppsClient(createClient({ host: mock.url, bearer: 'dfoa_test' }))
const spy = vi.spyOn(apps, 'describe')
const client = new AppMetaClient({ apps, host: mock.url, cache })
+1 -1
View File
@@ -40,7 +40,7 @@ export type PollSuccess = {
subject_type?: string
subject_email?: string
subject_issuer?: string
account?: PollAccount
account?: PollAccount | null
workspaces?: readonly PollWorkspace[]
default_workspace_id?: string
token_id?: string
-101
View File
@@ -1,101 +0,0 @@
import { mkdtemp, rm, stat, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { FILE_PERM } from '../store/dir.js'
import { FileBackend, TOKENS_FILE_NAME } from './file-backend.js'
describe('FileBackend', () => {
let dir: string
let backend: FileBackend
beforeEach(async () => {
dir = await mkdtemp(join(tmpdir(), 'difyctl-tokens-'))
backend = new FileBackend(dir)
})
afterEach(async () => {
await rm(dir, { recursive: true, force: true })
})
it('returns undefined when file is missing', async () => {
expect(await backend.get('cloud.dify.ai', 'acct-1')).toBeUndefined()
})
it('returns empty list when file is missing', async () => {
expect(await backend.list('cloud.dify.ai')).toEqual([])
})
it('round-trips put/get for a single token', async () => {
await backend.put('cloud.dify.ai', 'acct-1', 'dfoa_abc')
expect(await backend.get('cloud.dify.ai', 'acct-1')).toBe('dfoa_abc')
})
it('list returns accountIds for the given host', async () => {
await backend.put('cloud.dify.ai', 'acct-1', 'dfoa_a')
await backend.put('cloud.dify.ai', 'acct-2', 'dfoa_b')
await backend.put('self.example.com', 'acct-3', 'dfoa_c')
const ids = await backend.list('cloud.dify.ai')
expect([...ids].sort()).toEqual(['acct-1', 'acct-2'])
})
it('list returns empty array for unknown host', async () => {
await backend.put('cloud.dify.ai', 'acct-1', 'dfoa_a')
expect(await backend.list('other.example.com')).toEqual([])
})
it('delete removes the entry', async () => {
await backend.put('cloud.dify.ai', 'acct-1', 'dfoa_a')
await backend.delete('cloud.dify.ai', 'acct-1')
expect(await backend.get('cloud.dify.ai', 'acct-1')).toBeUndefined()
})
it('delete is a no-op for missing entries', async () => {
await expect(backend.delete('cloud.dify.ai', 'missing')).resolves.toBeUndefined()
})
it('delete prunes empty host entries', async () => {
await backend.put('cloud.dify.ai', 'acct-1', 'dfoa_a')
await backend.delete('cloud.dify.ai', 'acct-1')
expect(await backend.list('cloud.dify.ai')).toEqual([])
})
it('overwrites existing token for same host+accountId', async () => {
await backend.put('cloud.dify.ai', 'acct-1', 'dfoa_old')
await backend.put('cloud.dify.ai', 'acct-1', 'dfoa_new')
expect(await backend.get('cloud.dify.ai', 'acct-1')).toBe('dfoa_new')
})
it('writes file with mode 0600', async () => {
await backend.put('cloud.dify.ai', 'acct-1', 'dfoa_a')
const info = await stat(join(dir, TOKENS_FILE_NAME))
expect(info.mode & 0o777).toBe(FILE_PERM)
})
it('rewrites existing file with mode 0600 even if previously permissive', async () => {
const path = join(dir, TOKENS_FILE_NAME)
await writeFile(path, 'hosts: {}\n', { mode: 0o644 })
await backend.put('cloud.dify.ai', 'acct-1', 'dfoa_a')
const info = await stat(path)
expect(info.mode & 0o777).toBe(FILE_PERM)
})
it('writes valid YAML readable by a fresh backend', async () => {
await backend.put('cloud.dify.ai', 'acct-1', 'dfoa_a')
const fresh = new FileBackend(dir)
expect(await fresh.get('cloud.dify.ai', 'acct-1')).toBe('dfoa_a')
})
it('persists multiple hosts simultaneously', async () => {
await backend.put('cloud.dify.ai', 'acct-1', 'dfoa_a')
await backend.put('self.example.com', 'acct-2', 'dfoa_b')
expect(await backend.get('cloud.dify.ai', 'acct-1')).toBe('dfoa_a')
expect(await backend.get('self.example.com', 'acct-2')).toBe('dfoa_b')
})
it('treats malformed YAML as empty', async () => {
const path = join(dir, TOKENS_FILE_NAME)
await writeFile(path, 'not: valid: yaml: [\n', { mode: FILE_PERM })
expect(await backend.get('cloud.dify.ai', 'acct-1')).toBeUndefined()
})
})
-99
View File
@@ -1,99 +0,0 @@
import type { TokenStore } from './store.js'
import { mkdir, readFile, rename, stat, unlink, writeFile } from 'node:fs/promises'
import { join } from 'node:path'
import yaml from 'js-yaml'
import { DIR_PERM, FILE_PERM } from '../store/dir.js'
export const TOKENS_FILE_NAME = 'tokens.yml'
type AccountMap = Record<string, string>
type HostMap = Record<string, AccountMap>
type TokensFile = { hosts?: HostMap }
export class FileBackend implements TokenStore {
private readonly dir: string
private readonly path: string
constructor(dir: string) {
this.dir = dir
this.path = join(dir, TOKENS_FILE_NAME)
}
async put(host: string, accountId: string, token: string): Promise<void> {
const file = await this.read()
const hosts = file.hosts ?? {}
const accounts = hosts[host] ?? {}
accounts[accountId] = token
hosts[host] = accounts
await this.write({ hosts })
}
async get(host: string, accountId: string): Promise<string | undefined> {
const file = await this.read()
return file.hosts?.[host]?.[accountId]
}
async delete(host: string, accountId: string): Promise<void> {
const file = await this.read()
const accounts = file.hosts?.[host]
if (accounts === undefined || !(accountId in accounts))
return
delete accounts[accountId]
if (Object.keys(accounts).length === 0 && file.hosts !== undefined)
delete file.hosts[host]
await this.write(file)
}
async list(host: string): Promise<readonly string[]> {
const file = await this.read()
const accounts = file.hosts?.[host]
return accounts === undefined ? [] : Object.keys(accounts)
}
private async read(): Promise<TokensFile> {
let raw: string
try {
raw = await readFile(this.path, 'utf8')
}
catch (err) {
if ((err as NodeJS.ErrnoException).code === 'ENOENT')
return {}
throw err
}
let parsed: unknown
try {
parsed = yaml.load(raw)
}
catch {
return {}
}
if (parsed === null || typeof parsed !== 'object')
return {}
return parsed as TokensFile
}
private async write(file: TokensFile): Promise<void> {
await mkdir(this.dir, { recursive: true, mode: DIR_PERM })
const body = yaml.dump(file, { lineWidth: -1, noRefs: true })
const tmp = `${this.path}.tmp.${process.pid}.${Date.now()}`
try {
await writeFile(tmp, body, { mode: FILE_PERM })
await rename(tmp, this.path)
}
catch (err) {
try {
await unlink(tmp)
}
catch { /* tmp may not exist */ }
throw err
}
try {
const info = await stat(this.path)
if ((info.mode & 0o777) !== FILE_PERM) {
const { chmod } = await import('node:fs/promises')
await chmod(this.path, FILE_PERM)
}
}
catch { /* best-effort permission tighten */ }
}
}
+57 -57
View File
@@ -1,9 +1,9 @@
import { mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises'
import { mkdtemp, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { FILE_PERM } from '../store/dir.js'
import { HOSTS_FILE_NAME, HostsBundleSchema, loadHosts, saveHosts } from './hosts.js'
import { ENV_CONFIG_DIR } from '../store/dir.js'
import { HostsBundleSchema, loadHosts, saveHosts } from './hosts.js'
describe('HostsBundleSchema', () => {
it('parses a minimal logged-out bundle', () => {
@@ -46,86 +46,86 @@ describe('HostsBundleSchema', () => {
})
expect(parsed.available_workspaces).toHaveLength(2)
})
it('drops unknown top-level fields on parse', () => {
const parsed = HostsBundleSchema.parse({
current_host: 'cloud.dify.ai',
future_field: 42,
token_storage: 'file',
})
expect(parsed.current_host).toBe('cloud.dify.ai')
expect((parsed as Record<string, unknown>).future_field).toBeUndefined()
})
})
describe('loadHosts/saveHosts', () => {
let dir: string
let prevConfigDir: string | undefined
beforeEach(async () => {
dir = await mkdtemp(join(tmpdir(), 'difyctl-hosts-'))
prevConfigDir = process.env[ENV_CONFIG_DIR]
process.env[ENV_CONFIG_DIR] = dir
})
afterEach(async () => {
if (prevConfigDir === undefined)
delete process.env[ENV_CONFIG_DIR]
else
process.env[ENV_CONFIG_DIR] = prevConfigDir
await rm(dir, { recursive: true, force: true })
})
it('returns undefined when file is missing', async () => {
expect(await loadHosts(dir)).toBeUndefined()
it('returns undefined when nothing was saved', () => {
expect(loadHosts()).toBeUndefined()
})
it('round-trips bundle through YAML', async () => {
await saveHosts(dir, {
it('round-trips a fully-populated bundle', () => {
saveHosts({
current_host: 'cloud.dify.ai',
scheme: 'https',
account: { id: 'acct-1', email: 'a@b.c', name: 'A' },
workspace: { id: 'ws-1', name: 'My Space', role: 'owner' },
available_workspaces: [
{ id: 'ws-1', name: 'My Space', role: 'owner' },
{ id: 'ws-2', name: 'Other', role: 'normal' },
],
token_storage: 'keychain',
token_id: 'tok_xyz',
})
const loaded = await loadHosts(dir)
const loaded = loadHosts()
expect(loaded?.current_host).toBe('cloud.dify.ai')
expect(loaded?.scheme).toBe('https')
expect(loaded?.account?.email).toBe('a@b.c')
expect(loaded?.workspace?.id).toBe('ws-1')
expect(loaded?.available_workspaces).toHaveLength(2)
expect(loaded?.token_storage).toBe('keychain')
expect(loaded?.token_id).toBe('tok_xyz')
})
it('round-trips a file-mode bundle with bearer token', () => {
saveHosts({
current_host: 'self.example.com',
token_storage: 'file',
tokens: { bearer: 'dfoa_test' },
})
const loaded = loadHosts()
expect(loaded?.tokens?.bearer).toBe('dfoa_test')
expect(loaded?.token_storage).toBe('file')
})
it('overwrites previous bundle on save', () => {
saveHosts({ current_host: 'old.example.com', token_storage: 'file' })
saveHosts({ current_host: 'new.example.com', token_storage: 'keychain' })
const loaded = loadHosts()
expect(loaded?.current_host).toBe('new.example.com')
expect(loaded?.token_storage).toBe('keychain')
})
it('writes file with mode 0600', async () => {
await saveHosts(dir, { current_host: 'cloud.dify.ai', token_storage: 'file' })
const info = await stat(join(dir, HOSTS_FILE_NAME))
expect(info.mode & 0o777).toBe(FILE_PERM)
})
it('rewrites permissive existing file with mode 0600', async () => {
const path = join(dir, HOSTS_FILE_NAME)
await writeFile(path, 'current_host: ""\ntoken_storage: file\n', { mode: 0o644 })
await saveHosts(dir, { current_host: 'cloud.dify.ai', token_storage: 'file' })
const info = await stat(path)
expect(info.mode & 0o777).toBe(FILE_PERM)
})
it('atomic write: temp file does not survive on success', async () => {
await saveHosts(dir, { current_host: 'cloud.dify.ai', token_storage: 'file' })
const { readdir } = await import('node:fs/promises')
const entries = await readdir(dir)
expect(entries.filter(n => n.includes('.tmp.'))).toHaveLength(0)
})
it('drops unknown top-level fields', async () => {
const path = join(dir, HOSTS_FILE_NAME)
await writeFile(path, 'current_host: cloud.dify.ai\nfuture_field: 42\ntoken_storage: file\n', { mode: FILE_PERM })
const loaded = await loadHosts(dir)
expect(loaded?.current_host).toBe('cloud.dify.ai')
expect((loaded as Record<string, unknown> | undefined)?.future_field).toBeUndefined()
})
it('throws on malformed YAML', async () => {
const path = join(dir, HOSTS_FILE_NAME)
await writeFile(path, ': : :\n', { mode: FILE_PERM })
await expect(loadHosts(dir)).rejects.toThrow()
})
it('throws when YAML contradicts schema', async () => {
const path = join(dir, HOSTS_FILE_NAME)
await writeFile(path, 'token_storage: cloud\n', { mode: FILE_PERM })
await expect(loadHosts(dir)).rejects.toThrow()
})
it('produces YAML with stable keys', async () => {
await saveHosts(dir, {
it('rejects invalid input at save time', () => {
expect(() => saveHosts({
current_host: 'cloud.dify.ai',
token_storage: 'file',
tokens: { bearer: 'dfoa_x' },
})
const raw = await readFile(join(dir, HOSTS_FILE_NAME), 'utf8')
expect(raw).toContain('current_host: cloud.dify.ai')
expect(raw).toContain('bearer: dfoa_x')
token_storage: 'cloud',
} as never)).toThrow()
})
})
+15 -49
View File
@@ -1,10 +1,6 @@
import { mkdir, readFile, rename, unlink, writeFile } from 'node:fs/promises'
import { join } from 'node:path'
import yaml from 'js-yaml'
import type { Store } from '../store/store.js'
import { z } from 'zod'
import { DIR_PERM, FILE_PERM } from '../store/dir.js'
export const HOSTS_FILE_NAME = 'hosts.yml'
import { getHostStore, tokenKey } from '../store/manager.js'
const StorageModeSchema = z.enum(['keychain', 'file'])
export type StorageMode = z.infer<typeof StorageModeSchema>
@@ -48,53 +44,23 @@ export const HostsBundleSchema = z.object({
})
export type HostsBundle = z.infer<typeof HostsBundleSchema>
export async function loadHosts(dir: string): Promise<HostsBundle | undefined> {
const path = join(dir, HOSTS_FILE_NAME)
let raw: string
try {
raw = await readFile(path, 'utf8')
}
catch (err) {
if ((err as NodeJS.ErrnoException).code === 'ENOENT')
return undefined
throw err
}
const parsed = yaml.load(raw)
return HostsBundleSchema.parse(parsed ?? {})
export function loadHosts(): HostsBundle | undefined {
const raw = getHostStore().getTyped<Record<string, unknown>>()
if (raw === null)
return undefined
return HostsBundleSchema.parse(raw)
}
export async function saveHosts(dir: string, bundle: HostsBundle): Promise<void> {
await mkdir(dir, { recursive: true, mode: DIR_PERM })
export function saveHosts(bundle: HostsBundle): void {
const validated = HostsBundleSchema.parse(bundle)
const body = yaml.dump(stripUndefined(validated), { lineWidth: -1, noRefs: true, sortKeys: false })
const target = join(dir, HOSTS_FILE_NAME)
const tmp = `${target}.tmp.${process.pid}.${Date.now()}`
getHostStore().setTyped(validated)
}
export function clearLocal(bundle: HostsBundle, store: Store): void {
const accountId = bundle.account?.id ?? bundle.external_subject?.email ?? 'default'
try {
await writeFile(tmp, body, { mode: FILE_PERM })
await rename(tmp, target)
}
catch (err) {
try {
await unlink(tmp)
}
catch { /* tmp may not exist */ }
throw err
}
const { chmod, stat } = await import('node:fs/promises')
try {
const info = await stat(target)
if ((info.mode & 0o777) !== FILE_PERM)
await chmod(target, FILE_PERM)
store.unset(tokenKey(bundle.current_host, accountId))
}
catch { /* best-effort */ }
}
function stripUndefined<T extends Record<string, unknown>>(input: T): Record<string, unknown> {
const out: Record<string, unknown> = {}
for (const [k, v] of Object.entries(input)) {
if (v === undefined)
continue
out[k] = v
}
return out
getHostStore().rm()
}
-111
View File
@@ -1,111 +0,0 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
const passwords = new Map<string, string>()
const setPassword = vi.fn()
const getPassword = vi.fn()
const deletePassword = vi.fn()
class FakeAsyncEntry {
private readonly key: string
constructor(service: string, username: string) {
this.key = `${service}::${username}`
}
async setPassword(value: string): Promise<void> {
setPassword(this.key, value)
passwords.set(this.key, value)
}
async getPassword(): Promise<string | undefined> {
getPassword(this.key)
return passwords.get(this.key)
}
async deletePassword(): Promise<boolean> {
deletePassword(this.key)
if (!passwords.has(this.key))
return false
passwords.delete(this.key)
return true
}
}
vi.mock('@napi-rs/keyring', () => ({
AsyncEntry: FakeAsyncEntry,
}))
const { KEYRING_SERVICE, KeyringBackend } = await import('./keyring-backend.js')
beforeEach(() => {
passwords.clear()
setPassword.mockClear()
getPassword.mockClear()
deletePassword.mockClear()
})
describe('KeyringBackend', () => {
it('uses service name "difyctl"', () => {
expect(KEYRING_SERVICE).toBe('difyctl')
})
it('returns undefined when no password is stored', async () => {
const k = new KeyringBackend()
expect(await k.get('cloud.dify.ai', 'acct-1')).toBeUndefined()
})
it('round-trips put/get', async () => {
const k = new KeyringBackend()
await k.put('cloud.dify.ai', 'acct-1', 'dfoa_x')
expect(await k.get('cloud.dify.ai', 'acct-1')).toBe('dfoa_x')
})
it('keys by host::accountId', async () => {
const k = new KeyringBackend()
await k.put('cloud.dify.ai', 'acct-1', 'A')
await k.put('cloud.dify.ai', 'acct-2', 'B')
expect(await k.get('cloud.dify.ai', 'acct-1')).toBe('A')
expect(await k.get('cloud.dify.ai', 'acct-2')).toBe('B')
})
it('delete removes the entry', async () => {
const k = new KeyringBackend()
await k.put('cloud.dify.ai', 'acct-1', 'A')
await k.delete('cloud.dify.ai', 'acct-1')
expect(await k.get('cloud.dify.ai', 'acct-1')).toBeUndefined()
})
it('delete is a no-op for missing entries', async () => {
const k = new KeyringBackend()
await expect(k.delete('cloud.dify.ai', 'gone')).resolves.toBeUndefined()
})
it('list returns empty array (keyring does not enumerate)', async () => {
const k = new KeyringBackend()
await k.put('cloud.dify.ai', 'acct-1', 'A')
expect(await k.list('cloud.dify.ai')).toEqual([])
})
it('swallows getPassword exceptions and returns undefined', async () => {
const k = new KeyringBackend()
getPassword.mockImplementationOnce(() => {
throw new Error('NoEntry')
})
expect(await k.get('cloud.dify.ai', 'acct-1')).toBeUndefined()
})
it('swallows delete exceptions', async () => {
const k = new KeyringBackend()
deletePassword.mockImplementationOnce(() => {
throw new Error('NoEntry')
})
await expect(k.delete('cloud.dify.ai', 'acct-1')).resolves.toBeUndefined()
})
it('lets put propagate exceptions (caller decides fallback)', async () => {
const k = new KeyringBackend()
setPassword.mockImplementationOnce(() => {
throw new Error('keyring locked')
})
await expect(k.put('cloud.dify.ai', 'acct-1', 'tok')).rejects.toThrow(/keyring locked/)
})
})
-35
View File
@@ -1,35 +0,0 @@
import type { TokenStore } from './store.js'
import { AsyncEntry } from '@napi-rs/keyring'
export const KEYRING_SERVICE = 'difyctl'
function username(host: string, accountId: string): string {
return `${host}::${accountId}`
}
export class KeyringBackend implements TokenStore {
async put(host: string, accountId: string, token: string): Promise<void> {
await new AsyncEntry(KEYRING_SERVICE, username(host, accountId)).setPassword(token)
}
async get(host: string, accountId: string): Promise<string | undefined> {
try {
const v = await new AsyncEntry(KEYRING_SERVICE, username(host, accountId)).getPassword()
return v ?? undefined
}
catch {
return undefined
}
}
async delete(host: string, accountId: string): Promise<void> {
try {
await new AsyncEntry(KEYRING_SERVICE, username(host, accountId)).deletePassword()
}
catch { /* missing entry is fine */ }
}
async list(_host: string): Promise<readonly string[]> {
return []
}
}
-75
View File
@@ -1,75 +0,0 @@
import type { TokenStore } from './store.js'
import { describe, expect, it, vi } from 'vitest'
import { selectStore } from './store.js'
function memBackend(label: string): TokenStore & { _label: string } {
const map = new Map<string, string>()
const k = (h: string, a: string) => `${h}::${a}`
return {
_label: label,
async put(h, a, t) { map.set(k(h, a), t) },
async get(h, a) { return map.get(k(h, a)) },
async delete(h, a) { map.delete(k(h, a)) },
async list() { return [] },
}
}
describe('selectStore', () => {
it('returns keychain when probe succeeds', async () => {
const k = memBackend('keyring')
const f = memBackend('file')
const result = await selectStore({
configDir: '/tmp/x',
factory: { keyring: () => k, file: () => f },
})
expect(result.mode).toBe('keychain')
expect(result.store).toBe(k)
})
it('falls back to file when keyring put throws', async () => {
const k = memBackend('keyring')
const f = memBackend('file')
k.put = vi.fn().mockRejectedValue(new Error('locked'))
const result = await selectStore({
configDir: '/tmp/x',
factory: { keyring: () => k, file: () => f },
})
expect(result.mode).toBe('file')
expect(result.store).toBe(f)
})
it('falls back to file when probe round-trip mismatches', async () => {
const k = memBackend('keyring')
const f = memBackend('file')
k.get = vi.fn().mockResolvedValue('something-else')
const result = await selectStore({
configDir: '/tmp/x',
factory: { keyring: () => k, file: () => f },
})
expect(result.mode).toBe('file')
expect(result.store).toBe(f)
})
it('falls back to file when keyring constructor throws', async () => {
const f = memBackend('file')
const result = await selectStore({
configDir: '/tmp/x',
factory: {
keyring: () => { throw new Error('no backend') },
file: () => f,
},
})
expect(result.mode).toBe('file')
expect(result.store).toBe(f)
})
it('cleans up probe entry after successful probe', async () => {
const k = memBackend('keyring')
const f = memBackend('file')
await selectStore({
configDir: '/tmp/x',
factory: { keyring: () => k, file: () => f },
})
expect(await k.get('__difyctl_probe__', '__probe__')).toBeUndefined()
})
})
-40
View File
@@ -1,40 +0,0 @@
import { FileBackend } from './file-backend.js'
import { KeyringBackend } from './keyring-backend.js'
export type TokenStore = {
put: (host: string, accountId: string, token: string) => Promise<void>
get: (host: string, accountId: string) => Promise<string | undefined>
delete: (host: string, accountId: string) => Promise<void>
list: (host: string) => Promise<readonly string[]>
}
export type StorageMode = 'keychain' | 'file'
export type SelectStoreOptions = {
readonly configDir: string
readonly factory?: {
readonly keyring?: () => TokenStore
readonly file?: (dir: string) => TokenStore
}
}
const PROBE_HOST = '__difyctl_probe__'
const PROBE_ACCOUNT = '__probe__'
const PROBE_VALUE = 'probe-v1'
export async function selectStore(opts: SelectStoreOptions): Promise<{ store: TokenStore, mode: StorageMode }> {
const fileFactory = opts.factory?.file ?? ((dir: string) => new FileBackend(dir))
const keyringFactory = opts.factory?.keyring ?? (() => new KeyringBackend())
try {
const k = keyringFactory()
await k.put(PROBE_HOST, PROBE_ACCOUNT, PROBE_VALUE)
const got = await k.get(PROBE_HOST, PROBE_ACCOUNT)
await k.delete(PROBE_HOST, PROBE_ACCOUNT)
if (got !== PROBE_VALUE)
throw new Error('keyring round-trip mismatch')
return { store: k, mode: 'keychain' }
}
catch {
return { store: fileFactory(opts.configDir), mode: 'file' }
}
}
+19 -12
View File
@@ -4,8 +4,8 @@ import { tmpdir } from 'node:os'
import { join } from 'node:path'
import yaml from 'js-yaml'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { CACHE_APP_INFO, cachePath } from '../store/manager.js'
import { YamlStore } from '../store/store.js'
import { ENV_CACHE_DIR } from '../store/dir.js'
import { CACHE_APP_INFO, cachePath, getCache } from '../store/manager.js'
import { platform } from '../sys/index.js'
import { FieldInfo, FieldParameters } from '../types/app-meta.js'
import { APP_INFO_TTL_MS, loadAppInfoCache } from './app-info.js'
@@ -35,18 +35,25 @@ function metaInfoOnly(): AppMeta {
describe('app-info disk cache', () => {
let dir: string
let prevCacheDir: string | undefined
beforeEach(async () => {
dir = await mkdtemp(join(tmpdir(), 'difyctl-cache-'))
prevCacheDir = process.env[ENV_CACHE_DIR]
process.env[ENV_CACHE_DIR] = dir
})
afterEach(async () => {
if (prevCacheDir === undefined)
delete process.env[ENV_CACHE_DIR]
else
process.env[ENV_CACHE_DIR] = prevCacheDir
await rm(dir, { recursive: true, force: true })
})
it('round-trips an entry across reloads', async () => {
const c1 = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const c1 = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
await c1.set('http://localhost:9999', 'app-1', metaInfoOnly())
const c2 = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const c2 = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
const got = c2.get('http://localhost:9999', 'app-1')
expect(got).toBeDefined()
expect(got?.meta.info?.id).toBe('app-1')
@@ -55,7 +62,7 @@ describe('app-info disk cache', () => {
it('isFresh respects TTL', async () => {
const now = new Date('2026-05-09T00:00:00Z')
const c = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)), now: () => now })
const c = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO), now: () => now })
await c.set('h', 'app-1', metaInfoOnly())
const r = c.get('h', 'app-1')
expect(r).toBeDefined()
@@ -66,23 +73,23 @@ describe('app-info disk cache', () => {
})
it('keys by (host, app_id) — different hosts isolate', async () => {
const c = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const c = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
await c.set('h1', 'app-1', metaInfoOnly())
expect(c.get('h2', 'app-1')).toBeUndefined()
expect(c.get('h1', 'app-1')).toBeDefined()
})
it('delete removes entry from disk', async () => {
const c1 = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const c1 = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
await c1.set('h', 'app-1', metaInfoOnly())
await c1.delete('h', 'app-1')
const c2 = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const c2 = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
expect(c2.get('h', 'app-1')).toBeUndefined()
})
it('writes file with 0600 permission', async () => {
const c = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const c = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
await c.set('h', 'app-1', metaInfoOnly())
const { stat } = await import('node:fs/promises')
const s = await stat(appInfoPath(dir))
@@ -91,19 +98,19 @@ describe('app-info disk cache', () => {
})
it('missing cache file is not an error', async () => {
const c = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const c = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
expect(c.get('h', 'app-1')).toBeUndefined()
})
it('corrupt cache file is treated as empty', async () => {
const { writeFile } = await import('node:fs/promises')
await writeFile(appInfoPath(dir), ': : not valid yaml', 'utf8')
const c = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const c = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
expect(c.get('h', 'app-1')).toBeUndefined()
})
it('updates same key in place (no growth)', async () => {
const c = await loadAppInfoCache({ store: new YamlStore(cachePath(dir, CACHE_APP_INFO)) })
const c = await loadAppInfoCache({ store: getCache(CACHE_APP_INFO) })
await c.set('h', 'app-1', metaInfoOnly())
const slim: AppMeta = {
...metaInfoOnly(),
+19 -12
View File
@@ -3,8 +3,8 @@ import { tmpdir } from 'node:os'
import { dirname, join } from 'node:path'
import yaml from 'js-yaml'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { CACHE_NUDGE, cachePath } from '../store/manager.js'
import { YamlStore } from '../store/store.js'
import { ENV_CACHE_DIR } from '../store/dir.js'
import { CACHE_NUDGE, cachePath, getCache } from '../store/manager.js'
import { loadNudgeStore, WARN_INTERVAL_MS } from './nudge-store.js'
function nudgeStorePath(dir: string): string {
@@ -15,21 +15,28 @@ const HOST = 'https://cloud.dify.ai'
describe('NudgeStore', () => {
let dir: string
let prevCacheDir: string | undefined
beforeEach(async () => {
dir = await mkdtemp(join(tmpdir(), 'difyctl-nudge-'))
prevCacheDir = process.env[ENV_CACHE_DIR]
process.env[ENV_CACHE_DIR] = dir
})
afterEach(async () => {
if (prevCacheDir === undefined)
delete process.env[ENV_CACHE_DIR]
else
process.env[ENV_CACHE_DIR] = prevCacheDir
await rm(dir, { recursive: true, force: true })
})
it('canWarn=true when no prior record exists', async () => {
const store = await loadNudgeStore({ store: new YamlStore(cachePath(dir, CACHE_NUDGE)) })
const store = await loadNudgeStore({ store: getCache(CACHE_NUDGE) })
expect(store.canWarn(HOST)).toBe(true)
})
it('canWarn=false within the silence window, true past it', async () => {
const t0 = new Date('2026-05-19T12:00:00.000Z')
const store = await loadNudgeStore({ store: new YamlStore(cachePath(dir, CACHE_NUDGE)), now: () => t0 })
const store = await loadNudgeStore({ store: getCache(CACHE_NUDGE), now: () => t0 })
await store.markWarned(HOST)
expect(store.canWarn(HOST, new Date('2026-05-19T18:00:00.000Z'))).toBe(false)
expect(store.canWarn(HOST, new Date('2026-05-20T12:00:00.000Z'))).toBe(true)
@@ -37,7 +44,7 @@ describe('NudgeStore', () => {
it('canWarn clamps negative elapsed under clock skew (treats as still in window)', async () => {
const t0 = new Date('2026-05-19T12:00:00.000Z')
const store = await loadNudgeStore({ store: new YamlStore(cachePath(dir, CACHE_NUDGE)), now: () => t0 })
const store = await loadNudgeStore({ store: getCache(CACHE_NUDGE), now: () => t0 })
await store.markWarned(HOST)
const pastClock = new Date('2026-05-19T11:00:00.000Z') // clock moved backwards 1h
expect(store.canWarn(HOST, pastClock)).toBe(false)
@@ -45,22 +52,22 @@ describe('NudgeStore', () => {
it('markWarned persists across store reloads', async () => {
const t0 = new Date('2026-05-19T12:00:00.000Z')
const s1 = await loadNudgeStore({ store: new YamlStore(cachePath(dir, CACHE_NUDGE)), now: () => t0 })
const s1 = await loadNudgeStore({ store: getCache(CACHE_NUDGE), now: () => t0 })
await s1.markWarned(HOST)
const s2 = await loadNudgeStore({ store: new YamlStore(cachePath(dir, CACHE_NUDGE)), now: () => t0 })
const s2 = await loadNudgeStore({ store: getCache(CACHE_NUDGE), now: () => t0 })
expect(s2.canWarn(HOST)).toBe(false)
})
it('treats a corrupt cache file as empty', async () => {
const path = nudgeStorePath(dir)
await writeCacheFile(path, '{ not valid json')
const store = await loadNudgeStore({ store: new YamlStore(cachePath(dir, CACHE_NUDGE)) })
const store = await loadNudgeStore({ store: getCache(CACHE_NUDGE) })
expect(store.canWarn(HOST)).toBe(true)
})
it('writes ISO timestamps under warned/<host> on disk', async () => {
const t = new Date('2026-05-19T12:00:00.000Z')
const store = await loadNudgeStore({ store: new YamlStore(cachePath(dir, CACHE_NUDGE)), now: () => t })
const store = await loadNudgeStore({ store: getCache(CACHE_NUDGE), now: () => t })
await store.markWarned(HOST)
const raw = await readFile(nudgeStorePath(dir), 'utf8')
const parsed = yaml.load(raw) as Record<string, unknown>
@@ -72,11 +79,11 @@ describe('NudgeStore', () => {
// warns about a different host. Without merge-on-write the second writer
// would clobber the first.
const t = new Date('2026-05-19T12:00:00.000Z')
const a = await loadNudgeStore({ store: new YamlStore(cachePath(dir, CACHE_NUDGE)), now: () => t })
const b = await loadNudgeStore({ store: new YamlStore(cachePath(dir, CACHE_NUDGE)), now: () => t })
const a = await loadNudgeStore({ store: getCache(CACHE_NUDGE), now: () => t })
const b = await loadNudgeStore({ store: getCache(CACHE_NUDGE), now: () => t })
await a.markWarned('https://a.example')
await b.markWarned('https://b.example')
const reread = await loadNudgeStore({ store: new YamlStore(cachePath(dir, CACHE_NUDGE)), now: () => t })
const reread = await loadNudgeStore({ store: getCache(CACHE_NUDGE), now: () => t })
expect(reread.canWarn('https://a.example')).toBe(false)
expect(reread.canWarn('https://b.example')).toBe(false)
})
+2 -5
View File
@@ -12,7 +12,6 @@ import { BaseError } from '../../errors/base.js'
import { ErrorCode } from '../../errors/codes.js'
import { formatErrorForCli } from '../../errors/format.js'
import { createClient } from '../../http/client.js'
import { resolveConfigDir } from '../../store/dir.js'
import { realStreams } from '../../sys/io/streams'
import { hostWithScheme } from '../../util/host.js'
import { versionInfo } from '../../version/info.js'
@@ -24,7 +23,6 @@ export type AuthedContext = {
readonly http: KyInstance
readonly host: string
readonly io: IOStreams
readonly configDir: string
readonly cache?: AppInfoCache
}
@@ -38,9 +36,8 @@ export async function buildAuthedContext(
cmd: Pick<Command, 'error'>,
opts: AuthedContextOptions,
): Promise<AuthedContext> {
const configDir = resolveConfigDir()
const io = realStreams(opts.format ?? '')
const bundle = await loadHosts(configDir)
const bundle = loadHosts()
if (bundle === undefined || bundle.tokens?.bearer === undefined || bundle.tokens.bearer === '') {
const err = new BaseError({
code: ErrorCode.NotLoggedIn,
@@ -61,7 +58,7 @@ export async function buildAuthedContext(
await runCompatNudge({ host, io })
return { bundle, http, host, io, configDir, cache }
return { bundle, http, host, io, cache }
}
// Best-effort nudge: never throws, never blocks. Lives here so every authed
@@ -2,7 +2,7 @@ import type { SessionListResponse, SessionRow } from '@dify/contracts/api/openap
import type { DifyMock } from '../../../../../test/fixtures/dify-mock/server.js'
import type { AccountSessionsClient } from '../../../../api/account-sessions.js'
import type { HostsBundle } from '../../../../auth/hosts.js'
import type { TokenStore } from '../../../../auth/store.js'
import type { Key, Store } from '../../../../store/store.js'
import { mkdtemp, readFile, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
@@ -10,26 +10,23 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { startMock } from '../../../../../test/fixtures/dify-mock/server.js'
import { saveHosts } from '../../../../auth/hosts.js'
import { createClient } from '../../../../http/client.js'
import { ENV_CONFIG_DIR, resolveConfigDir } from '../../../../store/dir.js'
import { tokenKey } from '../../../../store/manager.js'
import { bufferStreams } from '../../../../sys/io/streams'
import { listAllSessions, runDevicesList, runDevicesRevoke } from './devices.js'
class MemStore implements TokenStore {
readonly entries = new Map<string, string>()
async put(host: string, accountId: string, token: string): Promise<void> {
this.entries.set(`${host}::${accountId}`, token)
class MemStore implements Store {
readonly entries = new Map<string, unknown>()
get<T>(key: Key<T>): T {
return (this.entries.get(key.key) as T | undefined) ?? key.default
}
async get(host: string, accountId: string): Promise<string | undefined> {
return this.entries.get(`${host}::${accountId}`)
set<T>(key: Key<T>, value: T): void {
this.entries.set(key.key, value)
}
async delete(host: string, accountId: string): Promise<void> {
this.entries.delete(`${host}::${accountId}`)
}
async list(host: string): Promise<readonly string[]> {
const prefix = `${host}::`
return Array.from(this.entries.keys()).filter(k => k.startsWith(prefix))
unset<T>(key: Key<T>): void {
this.entries.delete(key.key)
}
}
@@ -93,11 +90,18 @@ describe('runDevicesList', () => {
describe('runDevicesRevoke', () => {
let mock: DifyMock
let configDir: string
let prevConfigDir: string | undefined
beforeEach(async () => {
mock = await startMock({ scenario: 'happy' })
configDir = await mkdtemp(join(tmpdir(), 'difyctl-devrevoke-'))
prevConfigDir = process.env[ENV_CONFIG_DIR]
process.env[ENV_CONFIG_DIR] = configDir
})
afterEach(async () => {
if (prevConfigDir === undefined)
delete process.env[ENV_CONFIG_DIR]
else
process.env[ENV_CONFIG_DIR] = prevConfigDir
await mock.stop()
await rm(configDir, { recursive: true, force: true })
})
@@ -106,11 +110,11 @@ describe('runDevicesRevoke', () => {
const io = bufferStreams()
const store = new MemStore()
const b = bundleFor(mock.url, 'tok-1')
await store.put(b.current_host, 'acct-1', 'dfoa_test')
await saveHosts(configDir, b)
store.set(tokenKey(b.current_host, 'acct-1'), 'dfoa_test')
saveHosts(b)
const http = createClient({ host: mock.url, bearer: 'dfoa_test' })
await runDevicesRevoke({ configDir, io, bundle: b, http, store, target: 'difyctl on desktop', all: false })
await runDevicesRevoke({ io, bundle: b, http, store, target: 'difyctl on desktop', all: false })
expect(io.outBuf()).toContain('Revoked 1 session(s)')
expect(store.entries.size).toBe(1)
})
@@ -121,7 +125,7 @@ describe('runDevicesRevoke', () => {
const b = bundleFor(mock.url, 'tok-1')
const http = createClient({ host: mock.url, bearer: 'dfoa_test' })
await runDevicesRevoke({ configDir, io, bundle: b, http, store, target: 'tok-2', all: false })
await runDevicesRevoke({ io, bundle: b, http, store, target: 'tok-2', all: false })
expect(io.outBuf()).toContain('Revoked 1 session(s)')
})
@@ -131,7 +135,7 @@ describe('runDevicesRevoke', () => {
const b = bundleFor(mock.url, 'tok-1')
const http = createClient({ host: mock.url, bearer: 'dfoa_test' })
await runDevicesRevoke({ configDir, io, bundle: b, http, store, target: 'web', all: false })
await runDevicesRevoke({ io, bundle: b, http, store, target: 'web', all: false })
expect(io.outBuf()).toContain('Revoked 1 session(s)')
})
@@ -141,7 +145,7 @@ describe('runDevicesRevoke', () => {
const b = bundleFor(mock.url, 'tok-1')
const http = createClient({ host: mock.url, bearer: 'dfoa_test' })
await expect(runDevicesRevoke({ configDir, io, bundle: b, http, store, target: 'difyctl', all: false }))
await expect(runDevicesRevoke({ io, bundle: b, http, store, target: 'difyctl', all: false }))
.rejects
.toThrow(/matches multiple/)
})
@@ -152,7 +156,7 @@ describe('runDevicesRevoke', () => {
const b = bundleFor(mock.url, 'tok-1')
const http = createClient({ host: mock.url, bearer: 'dfoa_test' })
await expect(runDevicesRevoke({ configDir, io, bundle: b, http, store, target: 'nonexistent', all: false }))
await expect(runDevicesRevoke({ io, bundle: b, http, store, target: 'nonexistent', all: false }))
.rejects
.toThrow(/no session matches/)
})
@@ -163,7 +167,7 @@ describe('runDevicesRevoke', () => {
const b = bundleFor(mock.url, 'tok-1')
const http = createClient({ host: mock.url, bearer: 'dfoa_test' })
await runDevicesRevoke({ configDir, io, bundle: b, http, store, all: true })
await runDevicesRevoke({ io, bundle: b, http, store, all: true })
expect(io.outBuf()).toContain('Revoked 2 session(s)')
})
@@ -171,20 +175,20 @@ describe('runDevicesRevoke', () => {
const io = bufferStreams()
const store = new MemStore()
const b = bundleFor(mock.url, 'tok-1')
await store.put(b.current_host, 'acct-1', 'dfoa_test')
await saveHosts(configDir, b)
store.set(tokenKey(b.current_host, 'acct-1'), 'dfoa_test')
saveHosts(b)
const http = createClient({ host: mock.url, bearer: 'dfoa_test' })
await runDevicesRevoke({ configDir, io, bundle: b, http, store, target: 'tok-1', all: false })
await runDevicesRevoke({ io, bundle: b, http, store, target: 'tok-1', all: false })
expect(store.entries.size).toBe(0)
await expect(readFile(join(configDir, 'hosts.yml'), 'utf8')).rejects.toThrow(/ENOENT/)
await expect(readFile(join(resolveConfigDir(), 'hosts.yml'), 'utf8')).rejects.toThrow(/ENOENT/)
})
it('no target + no --all: throws UsageMissingArg', async () => {
const io = bufferStreams()
const store = new MemStore()
const http = createClient({ host: mock.url, bearer: 'dfoa_test' })
await expect(runDevicesRevoke({ configDir, io, bundle: bundleFor(mock.url), http, store, all: false }))
await expect(runDevicesRevoke({ io, bundle: bundleFor(mock.url), http, store, all: false }))
.rejects
.toThrow(/specify a device label/)
})
@@ -1,15 +1,14 @@
import type { SessionRow } from '@dify/contracts/api/openapi/types.gen'
import type { KyInstance } from 'ky'
import type { HostsBundle } from '../../../../auth/hosts.js'
import type { TokenStore } from '../../../../auth/store.js'
import type { Store } from '../../../../store/store.js'
import type { IOStreams } from '../../../../sys/io/streams'
import { unlink } from 'node:fs/promises'
import { join } from 'node:path'
import { AccountSessionsClient } from '../../../../api/account-sessions.js'
import { HOSTS_FILE_NAME } from '../../../../auth/hosts.js'
import { clearLocal } from '../../../../auth/hosts.js'
import { BaseError } from '../../../../errors/base.js'
import { ErrorCode } from '../../../../errors/codes.js'
import { LIMIT_DEFAULT, LIMIT_MAX, parseLimit } from '../../../../limit/limit.js'
import { getTokenStore } from '../../../../store/manager.js'
import { colorEnabled, colorScheme } from '../../../../sys/io/color.js'
import { runWithSpinner } from '../../../../sys/io/spinner.js'
@@ -72,11 +71,11 @@ export async function listAllSessions(client: AccountSessionsClient): Promise<re
}
export type DevicesRevokeOptions = {
readonly configDir: string
readonly io: IOStreams
readonly bundle: HostsBundle | undefined
readonly http: KyInstance
readonly store: TokenStore
/** Optional override for tests; production code resolves via `getTokenStore`. */
readonly store?: Store
readonly target?: string
readonly all: boolean
readonly yes?: boolean
@@ -104,8 +103,10 @@ export async function runDevicesRevoke(opts: DevicesRevokeOptions): Promise<void
for (const id of ids)
await sessions.revoke(id)
if (selfHit)
await clearLocal(opts.configDir, b, opts.store)
if (selfHit) {
const tokens = opts.store ?? getTokenStore().store
clearLocal(b, tokens)
}
opts.io.out.write(`${cs.successIcon()} Revoked ${ids.length} session(s)\n`)
}
@@ -178,18 +179,3 @@ function renderTable(rows: readonly SessionRow[], currentId: string): string {
cells.map((c, i) => c.padEnd(widths[i] ?? 0)).join(' ').trimEnd()
return body.length === 0 ? `${fmt(header)}\n` : `${[fmt(header), ...body.map(fmt)].join('\n')}\n`
}
async function clearLocal(configDir: string, bundle: HostsBundle, store: TokenStore): Promise<void> {
const accountId = bundle.account?.id ?? bundle.external_subject?.email ?? 'default'
try {
await store.delete(bundle.current_host, accountId)
}
catch { /* best-effort */ }
try {
await unlink(join(configDir, HOSTS_FILE_NAME))
}
catch (err) {
if ((err as NodeJS.ErrnoException).code !== 'ENOENT')
throw err
}
}
@@ -1,4 +1,3 @@
import { selectStore } from '../../../../auth/store.js'
import { Args, Flags } from '../../../../framework/flags.js'
import { DifyCommand } from '../../../_shared/dify-command.js'
import { httpRetryFlag } from '../../../_shared/global-flags.js'
@@ -25,13 +24,10 @@ export default class DevicesRevoke extends DifyCommand {
async run(argv: string[]): Promise<void> {
const { args, flags } = this.parse(DevicesRevoke, argv)
const ctx = await this.authedCtx({ retryFlag: flags['http-retry'] })
const { store } = await selectStore({ configDir: ctx.configDir })
await runDevicesRevoke({
configDir: ctx.configDir,
io: ctx.io,
bundle: ctx.bundle,
http: ctx.http,
store,
target: args.target,
all: flags.all,
yes: flags.yes,
-2
View File
@@ -1,5 +1,4 @@
import { Flags } from '../../../framework/flags.js'
import { resolveConfigDir } from '../../../store/dir.js'
import { realStreams } from '../../../sys/io/streams'
import { DifyCommand } from '../../_shared/dify-command.js'
import { runLogin } from './login.js'
@@ -31,7 +30,6 @@ export default class Login extends DifyCommand {
async run(argv: string[]): Promise<void> {
const { flags } = this.parse(Login, argv)
await runLogin({
configDir: resolveConfigDir(),
io: realStreams(),
host: flags.host,
noBrowser: flags['no-browser'],
+21 -23
View File
@@ -1,5 +1,5 @@
import type { DifyMock } from '../../../../test/fixtures/dify-mock/server.js'
import type { TokenStore } from '../../../auth/store.js'
import type { Key, Store } from '../../../store/store.js'
import type { Clock } from './device-flow.js'
import { mkdtemp, readFile, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
@@ -8,6 +8,8 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { startMock } from '../../../../test/fixtures/dify-mock/server.js'
import { DeviceFlowApi } from '../../../api/oauth-device.js'
import { createClient } from '../../../http/client.js'
import { ENV_CONFIG_DIR } from '../../../store/dir.js'
import { tokenKey } from '../../../store/manager.js'
import { bufferStreams } from '../../../sys/io/streams'
import { runLogin } from './login.js'
@@ -18,38 +20,38 @@ const noopClock: Clock = {
const noopBrowser = async (): Promise<void> => { /* skip OS open */ }
class MemStore implements TokenStore {
readonly entries = new Map<string, string>()
async put(host: string, accountId: string, token: string): Promise<void> {
this.entries.set(`${host}::${accountId}`, token)
class MemStore implements Store {
readonly entries = new Map<string, unknown>()
get<T>(key: Key<T>): T {
return (this.entries.get(key.key) as T | undefined) ?? key.default
}
async get(host: string, accountId: string): Promise<string | undefined> {
return this.entries.get(`${host}::${accountId}`)
set<T>(key: Key<T>, value: T): void {
this.entries.set(key.key, value)
}
async delete(host: string, accountId: string): Promise<void> {
this.entries.delete(`${host}::${accountId}`)
}
async list(host: string): Promise<readonly string[]> {
const prefix = `${host}::`
return Array.from(this.entries.keys())
.filter(k => k.startsWith(prefix))
.map(k => k.slice(prefix.length))
unset<T>(key: Key<T>): void {
this.entries.delete(key.key)
}
}
describe('runLogin', () => {
let mock: DifyMock
let configDir: string
let prevConfigDir: string | undefined
beforeEach(async () => {
mock = await startMock({ scenario: 'happy' })
configDir = await mkdtemp(join(tmpdir(), 'difyctl-login-'))
prevConfigDir = process.env[ENV_CONFIG_DIR]
process.env[ENV_CONFIG_DIR] = configDir
})
afterEach(async () => {
if (prevConfigDir === undefined)
delete process.env[ENV_CONFIG_DIR]
else
process.env[ENV_CONFIG_DIR] = prevConfigDir
await mock.stop()
await rm(configDir, { recursive: true, force: true })
})
@@ -58,7 +60,6 @@ describe('runLogin', () => {
const io = bufferStreams()
const store = new MemStore()
const bundle = await runLogin({
configDir,
io,
host: mock.url,
noBrowser: true,
@@ -73,7 +74,7 @@ describe('runLogin', () => {
expect(bundle.account?.email).toBe('tester@dify.ai')
expect(bundle.workspace?.id).toBe('ws-1')
expect(bundle.available_workspaces).toHaveLength(2)
const stored = await store.get(bundle.current_host, 'acct-1')
const stored = store.get(tokenKey(bundle.current_host, 'acct-1'))
expect(stored).toBe('dfoa_test')
const hostsRaw = await readFile(join(configDir, 'hosts.yml'), 'utf8')
@@ -91,7 +92,6 @@ describe('runLogin', () => {
const io = bufferStreams()
const store = new MemStore()
const bundle = await runLogin({
configDir,
io,
host: mock.url,
noBrowser: true,
@@ -106,6 +106,8 @@ describe('runLogin', () => {
expect(bundle.account).toBeUndefined()
expect(bundle.external_subject?.email).toBe('sso@dify.ai')
expect(bundle.external_subject?.issuer).toBe('https://issuer.example')
const stored = await store.get(bundle.current_host, 'sso@dify.ai')
expect(stored).toBe('dfoe_test')
expect(io.outBuf()).toContain('external SSO')
expect(io.outBuf()).toContain('sso@dify.ai')
})
@@ -115,7 +117,6 @@ describe('runLogin', () => {
const io = bufferStreams()
const store = new MemStore()
await expect(runLogin({
configDir,
io,
host: mock.url,
noBrowser: true,
@@ -135,7 +136,6 @@ describe('runLogin', () => {
const io = bufferStreams()
const store = new MemStore()
await expect(runLogin({
configDir,
io,
host: mock.url,
noBrowser: true,
@@ -152,7 +152,6 @@ describe('runLogin', () => {
const io = bufferStreams()
const store = new MemStore()
await expect(runLogin({
configDir,
io,
host: mock.url,
noBrowser: true,
@@ -169,7 +168,6 @@ describe('runLogin', () => {
const io = bufferStreams()
const store = new MemStore()
await runLogin({
configDir,
io,
host: mock.url,
noBrowser: true,
+10 -11
View File
@@ -1,6 +1,6 @@
import type { CodeResponse, PollSuccess } from '../../../api/oauth-device.js'
import type { HostsBundle, StorageMode, Workspace } from '../../../auth/hosts.js'
import type { TokenStore } from '../../../auth/store.js'
import type { HostsBundle, Workspace } from '../../../auth/hosts.js'
import type { StorageMode, Store } from '../../../store/store.js'
import type { IOStreams } from '../../../sys/io/streams'
import type { BrowserEnv, BrowserOpener } from '../../../util/browser.js'
import type { Clock } from './device-flow.js'
@@ -8,21 +8,20 @@ import * as os from 'node:os'
import * as readline from 'node:readline'
import { DeviceFlowApi } from '../../../api/oauth-device.js'
import { saveHosts } from '../../../auth/hosts.js'
import { selectStore } from '../../../auth/store.js'
import { createClient } from '../../../http/client.js'
import { getTokenStore, tokenKey } from '../../../store/manager.js'
import { colorEnabled, colorScheme } from '../../../sys/io/color.js'
import { decideOpen, OpenDecision, openUrl, realEnv } from '../../../util/browser.js'
import { bareHost, DEFAULT_HOST, resolveHost, validateVerificationURI } from '../../../util/host.js'
import { awaitAuthorization, realClock } from './device-flow.js'
export type LoginOptions = {
readonly configDir: string
readonly io: IOStreams
readonly host?: string
readonly noBrowser?: boolean
readonly insecure?: boolean
readonly deviceLabel?: string
readonly store?: { readonly store: TokenStore, readonly mode: StorageMode }
readonly store?: { readonly store: Store, readonly mode: StorageMode }
readonly api?: DeviceFlowApi
readonly browserEnv?: BrowserEnv
readonly browserOpener?: BrowserOpener
@@ -59,11 +58,11 @@ export async function runLogin(opts: LoginOptions): Promise<HostsBundle> {
const success = await awaitAuthorization(api, code, { clock: opts.clock ?? realClock() })
const storeBundle = opts.store ?? await selectStore({ configDir: opts.configDir })
const storeBundle = opts.store ?? getTokenStore()
const bundle = bundleFromSuccess(host, success, storeBundle.mode)
await storeBundle.store.put(bundle.current_host, accountKey(bundle), success.token)
await saveHosts(opts.configDir, bundle)
storeBundle.store.set(tokenKey(bundle.current_host, accountKey(bundle)), success.token)
saveHosts(bundle)
renderLoggedIn(opts.io.out, cs, host, success)
return bundle
@@ -100,7 +99,7 @@ function renderCodePrompt(w: NodeJS.WritableStream, cs: ReturnType<typeof colorS
function renderLoggedIn(w: NodeJS.WritableStream, cs: ReturnType<typeof colorScheme>, host: string, s: PollSuccess): void {
const display = bareHost(host)
if (s.account !== undefined && s.account.email !== '') {
if (s.account && s.account.email !== '') {
w.write(`${cs.successIcon()} Logged in to ${display} as ${cs.bold(s.account.email)} (${s.account.name})\n`)
const ws = findDefaultWorkspace(s)
if (ws !== undefined)
@@ -140,11 +139,11 @@ function bundleFromSuccess(host: string, s: PollSuccess, mode: StorageMode): Hos
token_id: s.token_id,
tokens: { bearer: s.token },
}
if (s.account !== undefined) {
if (s.account) {
bundle.account = { id: s.account.id, email: s.account.email, name: s.account.name }
}
if (s.subject_email !== undefined && s.subject_email !== ''
&& (s.account === undefined || s.account.id === '')) {
&& (!s.account || s.account.id === '')) {
bundle.external_subject = {
email: s.subject_email,
issuer: s.subject_issuer ?? '',
+2 -6
View File
@@ -1,8 +1,6 @@
import type { KyInstance } from 'ky'
import { loadHosts } from '../../../auth/hosts.js'
import { selectStore } from '../../../auth/store.js'
import { createClient } from '../../../http/client.js'
import { resolveConfigDir } from '../../../store/dir.js'
import { runWithSpinner } from '../../../sys/io/spinner.js'
import { realStreams } from '../../../sys/io/streams'
import { hostWithScheme } from '../../../util/host.js'
@@ -18,9 +16,7 @@ export default class Logout extends DifyCommand {
async run(argv: string[]): Promise<void> {
this.parse(Logout, argv)
const configDir = resolveConfigDir()
const bundle = await loadHosts(configDir)
const { store } = await selectStore({ configDir })
const bundle = loadHosts()
let http: KyInstance | undefined
if (bundle !== undefined && bundle.current_host !== '' && bundle.tokens?.bearer !== undefined && bundle.tokens.bearer !== '') {
@@ -34,7 +30,7 @@ export default class Logout extends DifyCommand {
const io = realStreams()
await runWithSpinner(
{ io, label: 'Signing out', enabled: true, style: 'dify-dim' },
() => runLogout({ configDir, io, bundle, http, store }),
() => runLogout({ io, bundle, http }),
)
}
}
+31 -29
View File
@@ -1,6 +1,6 @@
import type { DifyMock } from '../../../../test/fixtures/dify-mock/server.js'
import type { HostsBundle } from '../../../auth/hosts.js'
import type { TokenStore } from '../../../auth/store.js'
import type { Key, Store } from '../../../store/store.js'
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
@@ -8,28 +8,23 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { startMock } from '../../../../test/fixtures/dify-mock/server.js'
import { saveHosts } from '../../../auth/hosts.js'
import { createClient } from '../../../http/client.js'
import { ENV_CONFIG_DIR } from '../../../store/dir.js'
import { tokenKey } from '../../../store/manager.js'
import { bufferStreams } from '../../../sys/io/streams'
import { runLogout } from './logout.js'
class MemStore implements TokenStore {
readonly entries = new Map<string, string>()
async put(host: string, accountId: string, token: string): Promise<void> {
this.entries.set(`${host}::${accountId}`, token)
class MemStore implements Store {
readonly entries = new Map<string, unknown>()
get<T>(key: Key<T>): T {
return (this.entries.get(key.key) as T | undefined) ?? key.default
}
async get(host: string, accountId: string): Promise<string | undefined> {
return this.entries.get(`${host}::${accountId}`)
set<T>(key: Key<T>, value: T): void {
this.entries.set(key.key, value)
}
async delete(host: string, accountId: string): Promise<void> {
this.entries.delete(`${host}::${accountId}`)
}
async list(host: string): Promise<readonly string[]> {
const prefix = `${host}::`
return Array.from(this.entries.keys())
.filter(k => k.startsWith(prefix))
.map(k => k.slice(prefix.length))
unset<T>(key: Key<T>): void {
this.entries.delete(key.key)
}
}
@@ -52,13 +47,20 @@ function fixtureBundle(host: string): HostsBundle {
describe('runLogout', () => {
let mock: DifyMock
let configDir: string
let prevConfigDir: string | undefined
beforeEach(async () => {
mock = await startMock({ scenario: 'happy' })
configDir = await mkdtemp(join(tmpdir(), 'difyctl-logout-'))
prevConfigDir = process.env[ENV_CONFIG_DIR]
process.env[ENV_CONFIG_DIR] = configDir
})
afterEach(async () => {
if (prevConfigDir === undefined)
delete process.env[ENV_CONFIG_DIR]
else
process.env[ENV_CONFIG_DIR] = prevConfigDir
await mock.stop()
await rm(configDir, { recursive: true, force: true })
})
@@ -67,11 +69,11 @@ describe('runLogout', () => {
const io = bufferStreams()
const store = new MemStore()
const bundle = fixtureBundle(mock.url)
await store.put(bundle.current_host, 'acct-1', 'dfoa_test')
await saveHosts(configDir, bundle)
store.set(tokenKey(bundle.current_host, 'acct-1'), 'dfoa_test')
saveHosts(bundle)
const http = createClient({ host: mock.url, bearer: 'dfoa_test' })
await runLogout({ configDir, io, bundle, http, store })
await runLogout({ io, bundle, http, store })
expect(store.entries.size).toBe(0)
await expect(readFile(join(configDir, 'hosts.yml'), 'utf8')).rejects.toThrow(/ENOENT/)
@@ -82,7 +84,7 @@ describe('runLogout', () => {
it('not-logged-in: throws BaseError', async () => {
const io = bufferStreams()
const store = new MemStore()
await expect(runLogout({ configDir, io, bundle: undefined, store })).rejects.toThrow(/not logged in/)
await expect(runLogout({ io, bundle: undefined, store })).rejects.toThrow(/not logged in/)
})
it('hosts.yml absent: still completes locally + emits success', async () => {
@@ -91,7 +93,7 @@ describe('runLogout', () => {
const bundle = fixtureBundle(mock.url)
const http = createClient({ host: mock.url, bearer: 'dfoa_test' })
await runLogout({ configDir, io, bundle, http, store })
await runLogout({ io, bundle, http, store })
expect(io.outBuf()).toContain('Logged out of')
})
@@ -100,12 +102,12 @@ describe('runLogout', () => {
const io = bufferStreams()
const store = new MemStore()
const bundle = fixtureBundle(mock.url)
await store.put(bundle.current_host, 'acct-1', 'dfoa_test')
await saveHosts(configDir, bundle)
store.set(tokenKey(bundle.current_host, 'acct-1'), 'dfoa_test')
saveHosts(bundle)
mock.setScenario('server-5xx')
const http = createClient({ host: mock.url, bearer: 'dfoa_test', retryAttempts: 0 })
await runLogout({ configDir, io, bundle, http, store })
await runLogout({ io, bundle, http, store })
expect(store.entries.size).toBe(0)
expect(io.errBuf()).toContain('server revoke failed')
@@ -117,11 +119,11 @@ describe('runLogout', () => {
const store = new MemStore()
const bundle = fixtureBundle(mock.url)
bundle.tokens = { bearer: 'dfp_personal_token' }
await store.put(bundle.current_host, 'acct-1', 'dfp_personal_token')
await saveHosts(configDir, bundle)
store.set(tokenKey(bundle.current_host, 'acct-1'), 'dfp_personal_token')
saveHosts(bundle)
const http = createClient({ host: mock.url, bearer: 'dfp_personal_token' })
await runLogout({ configDir, io, bundle, http, store })
await runLogout({ io, bundle, http, store })
expect(io.errBuf()).toBe('')
expect(store.entries.size).toBe(0)
@@ -131,11 +133,11 @@ describe('runLogout', () => {
const io = bufferStreams()
const store = new MemStore()
const bundle = fixtureBundle(mock.url)
await saveHosts(configDir, bundle)
saveHosts(bundle)
await writeFile(join(configDir, 'config.yml'), 'foo: bar\n', 'utf8')
const http = createClient({ host: mock.url, bearer: 'dfoa_test' })
await runLogout({ configDir, io, bundle, http, store })
await runLogout({ io, bundle, http, store })
const cfg = await readFile(join(configDir, 'config.yml'), 'utf8')
expect(cfg).toContain('foo: bar')
+7 -23
View File
@@ -1,21 +1,20 @@
import type { KyInstance } from 'ky'
import type { HostsBundle } from '../../../auth/hosts.js'
import type { TokenStore } from '../../../auth/store.js'
import type { Store } from '../../../store/store.js'
import type { IOStreams } from '../../../sys/io/streams'
import { unlink } from 'node:fs/promises'
import { join } from 'node:path'
import { AccountSessionsClient } from '../../../api/account-sessions.js'
import { HOSTS_FILE_NAME } from '../../../auth/hosts.js'
import { clearLocal } from '../../../auth/hosts.js'
import { BaseError } from '../../../errors/base.js'
import { ErrorCode } from '../../../errors/codes.js'
import { getTokenStore } from '../../../store/manager.js'
import { colorEnabled, colorScheme } from '../../../sys/io/color.js'
export type LogoutOptions = {
readonly configDir: string
readonly io: IOStreams
readonly bundle: HostsBundle | undefined
readonly http?: KyInstance
readonly store: TokenStore
/** Optional override for tests; production code resolves via `getTokenStore`. */
readonly store?: Store
}
export async function runLogout(opts: LogoutOptions): Promise<void> {
@@ -40,7 +39,8 @@ export async function runLogout(opts: LogoutOptions): Promise<void> {
}
}
await clearLocal(opts.configDir, bundle, opts.store)
const tokens = opts.store ?? getTokenStore().store
clearLocal(bundle, tokens)
if (revokeWarning !== '')
opts.io.err.write(revokeWarning)
@@ -52,19 +52,3 @@ const REVOCABLE_PREFIXES = ['dfoa_', 'dfoe_'] as const
function revokeAllowed(bearer: string): boolean {
return REVOCABLE_PREFIXES.some(p => bearer.startsWith(p))
}
async function clearLocal(configDir: string, bundle: HostsBundle, store: TokenStore): Promise<void> {
const accountId = bundle.account?.id ?? bundle.external_subject?.email ?? 'default'
try {
await store.delete(bundle.current_host, accountId)
}
catch { /* best-effort */ }
const hostsPath = join(configDir, HOSTS_FILE_NAME)
try {
await unlink(hostsPath)
}
catch (err) {
if ((err as NodeJS.ErrnoException).code !== 'ENOENT')
throw err
}
}
+1 -3
View File
@@ -1,6 +1,5 @@
import { loadHosts } from '../../../auth/hosts.js'
import { Flags } from '../../../framework/flags.js'
import { resolveConfigDir } from '../../../store/dir.js'
import { realStreams } from '../../../sys/io/streams'
import { DifyCommand } from '../../_shared/dify-command.js'
import { runStatus } from './status.js'
@@ -21,8 +20,7 @@ export default class Status extends DifyCommand {
async run(argv: string[]): Promise<void> {
const { flags } = this.parse(Status, argv)
const configDir = resolveConfigDir()
const bundle = await loadHosts(configDir)
const bundle = loadHosts()
await runStatus({ io: realStreams(), bundle, verbose: flags.verbose, json: flags.json })
}
}
+1 -3
View File
@@ -1,6 +1,5 @@
import { loadHosts } from '../../../auth/hosts.js'
import { Flags } from '../../../framework/flags.js'
import { resolveConfigDir } from '../../../store/dir.js'
import { realStreams } from '../../../sys/io/streams'
import { DifyCommand } from '../../_shared/dify-command.js'
import { runWhoami } from './whoami.js'
@@ -19,8 +18,7 @@ export default class Whoami extends DifyCommand {
async run(argv: string[]): Promise<void> {
const { flags } = this.parse(Whoami, argv)
const configDir = resolveConfigDir()
const bundle = await loadHosts(configDir)
const bundle = loadHosts()
await runWhoami({ io: realStreams(), bundle, json: flags.json })
}
}
+29 -24
View File
@@ -1,43 +1,49 @@
import { mkdtemp, writeFile } from 'node:fs/promises'
import { mkdtemp, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { beforeEach, describe, expect, it } from 'vitest'
import { FILE_NAME } from '../../../config/schema.js'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { isBaseError } from '../../../errors/base.js'
import { ErrorCode } from '../../../errors/codes.js'
import { YamlStore } from '../../../store/store.js'
import { ENV_CONFIG_DIR } from '../../../store/dir.js'
import { getConfigurationStore } from '../../../store/manager.js'
import { runConfigGet } from './run.js'
function makeStore(dir: string): YamlStore {
return new YamlStore(join(dir, FILE_NAME))
}
describe('runConfigGet', () => {
let dir: string
let prevConfigDir: string | undefined
beforeEach(async () => {
dir = await mkdtemp(join(tmpdir(), 'difyctl-get-'))
prevConfigDir = process.env[ENV_CONFIG_DIR]
process.env[ENV_CONFIG_DIR] = dir
})
it('returns set value with trailing newline', async () => {
await writeFile(
join(dir, FILE_NAME),
'schema_version: 1\ndefaults:\n format: yaml\n',
'utf8',
)
const out = runConfigGet({ store: makeStore(dir), key: 'defaults.format' })
afterEach(async () => {
if (prevConfigDir === undefined)
delete process.env[ENV_CONFIG_DIR]
else
process.env[ENV_CONFIG_DIR] = prevConfigDir
await rm(dir, { recursive: true, force: true })
})
it('returns set value with trailing newline', () => {
getConfigurationStore().setTyped({
schema_version: 1,
defaults: { format: 'yaml' },
})
const out = runConfigGet({ store: getConfigurationStore(), key: 'defaults.format' })
expect(out).toBe('yaml\n')
})
it('returns empty line when key is unset (matches Go fmt.Fprintln)', () => {
const out = runConfigGet({ store: makeStore(dir), key: 'defaults.format' })
const out = runConfigGet({ store: getConfigurationStore(), key: 'defaults.format' })
expect(out).toBe('\n')
})
it('throws BaseError(config_invalid_key) on unknown key', () => {
let caught: unknown
try {
runConfigGet({ store: makeStore(dir), key: 'bogus.key' })
runConfigGet({ store: getConfigurationStore(), key: 'bogus.key' })
}
catch (err) { caught = err }
expect(isBaseError(caught)).toBe(true)
@@ -45,13 +51,12 @@ describe('runConfigGet', () => {
expect(caught.code).toBe(ErrorCode.ConfigInvalidKey)
})
it('returns numeric limit as string', async () => {
await writeFile(
join(dir, FILE_NAME),
'schema_version: 1\ndefaults:\n limit: 75\n',
'utf8',
)
const out = runConfigGet({ store: makeStore(dir), key: 'defaults.limit' })
it('returns numeric limit as string', () => {
getConfigurationStore().setTyped({
schema_version: 1,
defaults: { limit: 75 },
})
const out = runConfigGet({ store: getConfigurationStore(), key: 'defaults.limit' })
expect(out).toBe('75\n')
})
})
+5 -2
View File
@@ -1,7 +1,8 @@
import { join } from 'node:path'
import { raw } from '../../../framework/output.js'
import { resolveConfigDir } from '../../../store/dir.js'
import { CONFIG_FILE_NAME } from '../../../store/manager.js'
import { DifyCommand } from '../../_shared/dify-command.js'
import { runConfigPath } from './run.js'
export default class ConfigPath extends DifyCommand {
static override description = 'Print the resolved config.yml path'
@@ -12,6 +13,8 @@ export default class ConfigPath extends DifyCommand {
async run(argv: string[]) {
this.parse(ConfigPath, argv)
return raw(runConfigPath({ dir: resolveConfigDir() }))
return raw(
join(resolveConfigDir(), CONFIG_FILE_NAME),
)
}
}
-14
View File
@@ -1,14 +0,0 @@
import { describe, expect, it } from 'vitest'
import { runConfigPath } from './run.js'
describe('runConfigPath', () => {
it('joins dir and config.yml with trailing newline', () => {
const out = runConfigPath({ dir: '/tmp/x' })
expect(out).toBe('/tmp/x/config.yml\n')
})
it('handles trailing slash on dir', () => {
const out = runConfigPath({ dir: '/tmp/x/' })
expect(out).toBe('/tmp/x/config.yml\n')
})
})
-10
View File
@@ -1,10 +0,0 @@
import { join } from 'node:path'
import { FILE_NAME } from '../../../config/schema.js'
export type RunConfigPathOptions = {
readonly dir: string
}
export function runConfigPath(opts: RunConfigPathOptions): string {
return `${join(opts.dir, FILE_NAME)}\n`
}
+39 -24
View File
@@ -1,35 +1,46 @@
import { mkdtemp, readFile } from 'node:fs/promises'
import { mkdtemp, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { beforeEach, describe, expect, it } from 'vitest'
import { FILE_NAME } from '../../../config/schema.js'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { loadConfig } from '../../../config/config-loader.js'
import { isBaseError } from '../../../errors/base.js'
import { ErrorCode, ExitCode } from '../../../errors/codes.js'
import { YamlStore } from '../../../store/store.js'
import { ENV_CONFIG_DIR } from '../../../store/dir.js'
import { getConfigurationStore } from '../../../store/manager.js'
import { runConfigSet } from './run.js'
function makeStore(dir: string): YamlStore {
return new YamlStore(join(dir, FILE_NAME))
}
describe('runConfigSet', () => {
let dir: string
let prevConfigDir: string | undefined
beforeEach(async () => {
dir = await mkdtemp(join(tmpdir(), 'difyctl-set-'))
prevConfigDir = process.env[ENV_CONFIG_DIR]
process.env[ENV_CONFIG_DIR] = dir
})
it('writes config.yml and returns "set k = v\\n"', async () => {
const out = runConfigSet({ store: makeStore(dir), key: 'defaults.format', value: 'json' })
afterEach(async () => {
if (prevConfigDir === undefined)
delete process.env[ENV_CONFIG_DIR]
else
process.env[ENV_CONFIG_DIR] = prevConfigDir
await rm(dir, { recursive: true, force: true })
})
it('persists the value and returns "set k = v\\n"', () => {
const out = runConfigSet({ store: getConfigurationStore(), key: 'defaults.format', value: 'json' })
expect(out).toBe('set defaults.format = json\n')
const raw = await readFile(join(dir, FILE_NAME), 'utf8')
expect(raw).toContain('format: json')
const r = loadConfig(getConfigurationStore())
expect(r.found).toBe(true)
if (r.found)
expect(r.config.defaults.format).toBe('json')
})
it('rejects invalid format value with config_invalid_value', async () => {
it('rejects invalid format value with config_invalid_value', () => {
let caught: unknown
try {
runConfigSet({ store: makeStore(dir), key: 'defaults.format', value: 'csv' })
runConfigSet({ store: getConfigurationStore(), key: 'defaults.format', value: 'csv' })
}
catch (err) { caught = err }
expect(isBaseError(caught)).toBe(true)
@@ -40,7 +51,7 @@ describe('runConfigSet', () => {
it('rejects unknown key with config_invalid_key', () => {
let caught: unknown
try {
runConfigSet({ store: makeStore(dir), key: 'bogus', value: 'x' })
runConfigSet({ store: getConfigurationStore(), key: 'bogus', value: 'x' })
}
catch (err) { caught = err }
expect(isBaseError(caught)).toBe(true)
@@ -48,18 +59,22 @@ describe('runConfigSet', () => {
expect(caught.code).toBe(ErrorCode.ConfigInvalidKey)
})
it('preserves prior keys when setting a new one', async () => {
runConfigSet({ store: makeStore(dir), key: 'defaults.format', value: 'yaml' })
runConfigSet({ store: makeStore(dir), key: 'defaults.limit', value: '40' })
const raw = await readFile(join(dir, FILE_NAME), 'utf8')
expect(raw).toContain('format: yaml')
expect(raw).toContain('limit: 40')
it('preserves prior keys when setting a new one', () => {
runConfigSet({ store: getConfigurationStore(), key: 'defaults.format', value: 'yaml' })
runConfigSet({ store: getConfigurationStore(), key: 'defaults.limit', value: '40' })
const r = loadConfig(getConfigurationStore())
expect(r.found).toBe(true)
if (r.found) {
expect(r.config.defaults.format).toBe('yaml')
expect(r.config.defaults.limit).toBe(40)
}
})
it('exit code for invalid value is Usage (2)', () => {
let caught: unknown
try {
runConfigSet({ store: makeStore(dir), key: 'defaults.format', value: 'csv' })
runConfigSet({ store: getConfigurationStore(), key: 'defaults.format', value: 'csv' })
}
catch (err) { caught = err }
expect(isBaseError(caught)).toBe(true)
@@ -70,7 +85,7 @@ describe('runConfigSet', () => {
it('exit code for unknown key is Usage (2)', () => {
let caught: unknown
try {
runConfigSet({ store: makeStore(dir), key: 'bogus', value: 'x' })
runConfigSet({ store: getConfigurationStore(), key: 'bogus', value: 'x' })
}
catch (err) { caught = err }
expect(isBaseError(caught)).toBe(true)
@@ -81,7 +96,7 @@ describe('runConfigSet', () => {
it('typed wrap chain: invalid defaults.limit surfaces ConfigInvalidValue (not UsageInvalidFlag)', () => {
let caught: unknown
try {
runConfigSet({ store: makeStore(dir), key: 'defaults.limit', value: 'abc' })
runConfigSet({ store: getConfigurationStore(), key: 'defaults.limit', value: 'abc' })
}
catch (err) { caught = err }
expect(isBaseError(caught)).toBe(true)
+37 -24
View File
@@ -1,48 +1,61 @@
import { mkdtemp, readFile, writeFile } from 'node:fs/promises'
import { mkdtemp, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { beforeEach, describe, expect, it } from 'vitest'
import { FILE_NAME } from '../../../config/schema.js'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { loadConfig } from '../../../config/config-loader.js'
import { isBaseError } from '../../../errors/base.js'
import { ErrorCode } from '../../../errors/codes.js'
import { YamlStore } from '../../../store/store.js'
import { ENV_CONFIG_DIR } from '../../../store/dir.js'
import { getConfigurationStore } from '../../../store/manager.js'
import { runConfigUnset } from './run.js'
function makeStore(dir: string): YamlStore {
return new YamlStore(join(dir, FILE_NAME))
}
describe('runConfigUnset', () => {
let dir: string
let prevConfigDir: string | undefined
beforeEach(async () => {
dir = await mkdtemp(join(tmpdir(), 'difyctl-unset-'))
prevConfigDir = process.env[ENV_CONFIG_DIR]
process.env[ENV_CONFIG_DIR] = dir
})
it('clears the requested key, leaves others intact', async () => {
await writeFile(
join(dir, FILE_NAME),
'schema_version: 1\ndefaults:\n format: json\n limit: 25\n',
'utf8',
)
const out = runConfigUnset({ store: makeStore(dir), key: 'defaults.format' })
expect(out).toBe('unset defaults.format\n')
const raw = await readFile(join(dir, FILE_NAME), 'utf8')
expect(raw).not.toContain('format:')
expect(raw).toContain('limit: 25')
afterEach(async () => {
if (prevConfigDir === undefined)
delete process.env[ENV_CONFIG_DIR]
else
process.env[ENV_CONFIG_DIR] = prevConfigDir
await rm(dir, { recursive: true, force: true })
})
it('is a no-op (writes empty config) when key was already unset', async () => {
const out = runConfigUnset({ store: makeStore(dir), key: 'defaults.format' })
it('clears the requested key, leaves others intact', () => {
getConfigurationStore().setTyped({
schema_version: 1,
defaults: { format: 'json', limit: 25 },
})
const out = runConfigUnset({ store: getConfigurationStore(), key: 'defaults.format' })
expect(out).toBe('unset defaults.format\n')
const raw = await readFile(join(dir, FILE_NAME), 'utf8')
expect(raw).toContain('schema_version: 1')
const r = loadConfig(getConfigurationStore())
expect(r.found).toBe(true)
if (r.found) {
expect(r.config.defaults.format).not.toBe('json')
expect(r.config.defaults.limit).toBe(25)
}
})
it('is a no-op (writes empty config) when key was already unset', () => {
const out = runConfigUnset({ store: getConfigurationStore(), key: 'defaults.format' })
expect(out).toBe('unset defaults.format\n')
const r = loadConfig(getConfigurationStore())
expect(r.found).toBe(true)
if (r.found)
expect(r.config.schema_version).toBe(1)
})
it('rejects unknown key', () => {
let caught: unknown
try {
runConfigUnset({ store: makeStore(dir), key: 'bogus' })
runConfigUnset({ store: getConfigurationStore(), key: 'bogus' })
}
catch (err) { caught = err }
expect(isBaseError(caught)).toBe(true)

Some files were not shown because too many files have changed in this diff Show More