Commit Graph
3311 Commits
Author SHA1 Message Date
Yansong Zhang 66212e3575 feat(api): implement zero-migration transparent upgrade (Phase 8)
Add two feature-flag-controlled upgrade paths that allow existing apps
and LLM nodes to transparently run through the Agent V2 engine without
any database migration:

1. AGENT_V2_TRANSPARENT_UPGRADE (default: off):
   When enabled, old apps (chat/completion/agent-chat) bypass legacy
   Easy-UI runners. VirtualWorkflowSynthesizer converts AppModelConfig
   to an in-memory Workflow (start -> agent-v2 -> answer) at runtime,
   then executes via AdvancedChatAppGenerator. Falls back to legacy
   path on any synthesis error.

   VirtualWorkflowSynthesizer maps:
   - model JSON -> ModelConfig
   - pre_prompt/chat_prompt_config -> prompt_template
   - agent_mode.tools -> ToolMetadata[]
   - agent_mode.strategy -> agent_strategy
   - dataset_configs -> context
   - file_upload -> vision

2. AGENT_V2_REPLACES_LLM (default: off):
   When enabled, DifyNodeFactory.create_node() transparently remaps
   nodes with type="llm" to type="agent-v2" before class resolution.
   Since AgentV2NodeData is a strict superset of LLMNodeData, the
   mapping is lossless. With tools=[], Agent V2 behaves identically
   to LLM Node.

Both flags default to False for safety. Turn off = instant rollback.
46 existing tests pass. Flask starts successfully.

Made-with: Cursor
2026-04-09 10:30:52 +08:00
Yansong Zhang 96374d7f6a refactor(api): replace legacy agent runners with StrategyFactory in AgentChatAppRunner (Phase 4)
Replace the hardcoded FunctionCallAgentRunner / CotChatAgentRunner /
CotCompletionAgentRunner selection in AgentChatAppRunner with the new
AgentAppRunner class that uses StrategyFactory from Phase 1.

Before: AgentChatAppRunner manually selects FC/CoT runner class based on
model features and LLM mode, then instantiates it directly.

After: AgentChatAppRunner instantiates AgentAppRunner (from sandbox branch),
which internally uses StrategyFactory.create_strategy() to auto-select
the right strategy, and uses ToolInvokeHook for proper agent_invoke
with file handling and thought persistence.

This unifies the agent execution engine: both the new Agent V2 workflow
node and the legacy agent-chat app now use the same StrategyFactory
and AgentPattern implementations.

Also fix: command and file_upload nodes use string node_type instead of
BuiltinNodeTypes.COMMAND/FILE_UPLOAD (not in current graphon version).

46 tests pass. Flask starts successfully.

Made-with: Cursor
2026-04-09 09:42:23 +08:00
Yansong Zhang 44491e427c feat(api): enable all sandbox/skill controller routes and resolve dependencies (P0)
Resolve the full dependency chain to enable all previously disabled controllers:

Enabled routes:
- sandbox_files: sandbox file browser API
- sandbox_providers: sandbox provider management API
- app_asset: app asset management API
- skills: skill extraction API
- CLI API blueprint: DifyCli callback endpoints (/cli/api/*)

Dependencies extracted (64 files, ~8000 lines):
- models/sandbox.py, models/app_asset.py: DB models
- core/zip_sandbox/: zip-based sandbox execution
- core/session/: CLI API session management
- core/memory/: base memory + node token buffer
- core/helper/creators.py: helper utilities
- core/llm_generator/: context models, output models, utils
- core/workflow/nodes/command/: command node type
- core/workflow/nodes/file_upload/: file upload node type
- core/app/entities/: app_asset_entities, app_bundle_entities, llm_generation_entities
- services/: asset_content, skill, workflow_collaboration, workflow_comment
- controllers/console/app/error.py: AppAsset error classes
- core/tools/utils/system_encryption.py

Import fixes:
- dify_graph.enums -> graphon.enums in skill_service.py
- get_signed_file_url_for_plugin -> get_signed_file_url in cli_api.py

All 5 controllers verified: import OK, Flask starts successfully.
46 existing tests still pass.

Made-with: Cursor
2026-04-09 09:36:16 +08:00
Yansong Zhang d3d9f21cdf feat(api): wire sandbox into Agent V2 node execution pipeline
Integrate the ported sandbox system with Agent V2 node:

- Add DIFY_SANDBOX_CONTEXT_KEY to app_invoke_entities for passing
  sandbox through run_context without modifying graphon
- DifyNodeFactory._resolve_sandbox() extracts sandbox from run_context
  and passes it to AgentV2Node constructor
- AgentV2Node accepts optional sandbox parameter
- AgentV2ToolManager supports dual execution paths:
  - _invoke_tool_directly(): standard ToolEngine.generic_invoke (no sandbox)
  - _invoke_tool_in_sandbox(): delegates to SandboxBashSession.run_tool()
    which uses DifyCli to call back to Dify API from inside the sandbox
- Graceful fallback: if sandbox execution fails, logs warning and returns
  error message (does not crash the agent loop)

To enable sandbox for an Agent workflow:
1. Create a Sandbox via SandboxBuilder
2. Add it to run_context under DIFY_SANDBOX_CONTEXT_KEY
3. Agent V2 nodes will automatically use sandbox for tool execution

46 existing tests still pass.

Made-with: Cursor
2026-04-08 17:46:34 +08:00
Yansong Zhang 0c7e7e0c4e feat(api): port Sandbox + VirtualEnvironment + Skill system from feat/support-agent-sandbox (Phase 5-6)
Port the complete infrastructure for agent sandbox execution and skill system:

Sandbox & Virtual Environment (core/sandbox/, core/virtual_environment/):
- Sandbox entity with lifecycle management (ready/failed/cancelled states)
- SandboxBuilder with fluent API for configuring providers
- 5 VM providers: Local, SSH, Docker, E2B, AWS CodeInterpreter
- VirtualEnvironment base with command execution, file transfer, transport layers
- Channel transport: pipe, queue, socket implementations
- Bash session management and DifyCli binary integration
- Storage: archive storage, file storage, noop storage, presign storage
- Initializers: DifyCli, AppAssets, DraftAppAssets, Skills
- Inspector: file browser, archive/runtime source, script utils
- Security: encryption utils, debug helpers

Skill & App Assets (core/skill/, core/app_assets/, core/app_bundle/):
- Skill entity and manager
- App asset accessor, builder pipeline (file, skill builders)
- App bundle source zip extractor
- Storage and converter utilities

API Endpoints:
- CLI API blueprint (controllers/cli_api/) for sandbox callback
- Sandbox provider management (workspace/sandbox_providers)
- Sandbox file browser (console/sandbox_files)
- App asset management (console/app/app_asset)
- Skill management (console/app/skills)
- Storage file endpoints (controllers/files/storage_files)

Services:
- Sandbox service, provider service, file service
- App asset service, app bundle service

Config:
- CliApiConfig, CreatorsPlatformConfig, CollaborationConfig
- FILES_API_URL for sandbox file access

Note: Controller route registration temporarily commented out (marked TODO)
pending resolution of deep dependency chains (socketio, workflow_comment,
command node, etc.). Core sandbox modules are fully ported and syntax-validated.
110 files changed, 10,549 insertions.

Made-with: Cursor
2026-04-08 17:39:02 +08:00
Yansong Zhang d9d1e9b63a fix(api): resolve Agent V2 node E2E runtime issues
Fixes discovered during end-to-end testing of Agent workflow execution:

1. ModelManager instantiation: use ModelManager.for_tenant() instead of
   ModelManager() which requires a ProviderManager argument
2. Variable template resolution: use VariableTemplateParser(template).format()
   instead of non-existent resolve_template() static method
3. invoke_llm() signature: remove unsupported 'user' keyword argument
4. Event dispatch: remove ModelInvokeCompletedEvent from _run() yield
   (graphon base Node._dispatch doesn't support it via singledispatch)
5. NodeRunResult metadata: use WorkflowNodeExecutionMetadataKey enum keys
   (TOTAL_TOKENS, TOTAL_PRICE, CURRENCY) instead of arbitrary string keys
6. SSE topic mismatch: use AppMode.AGENT (not ADVANCED_CHAT) in
   retrieve_events() so publisher and subscriber share the same channel
7. Celery task routing: add AppMode.AGENT to workflow_execute_task._run_app()
   alongside ADVANCED_CHAT

All issues verified fixed: Agent V2 node successfully invokes LLM and
returns "Hello there!" through the full SSE streaming pipeline.

Made-with: Cursor
2026-04-08 16:21:12 +08:00
Yansong Zhang 8f3a3ea03e feat(api): enable Agent mode in workflow/service APIs and add default config (Phase 7)
Ensure new Agent apps (AppMode.AGENT) can access all workflow-related
APIs and Service API chat endpoints:

- Add AppMode.AGENT to 13 workflow controller mode checks
- Add AppMode.AGENT to 4 workflow_run controller mode checks
- Add AppMode.AGENT to workflow_draft_variable controller
- Add AppMode.AGENT to Service API chat, conversation, message endpoints
- Add AgentV2Node.get_default_config() with prompt templates and strategy defaults
- 46 unit tests all passing (8 new Phase 7 tests)

Old agent/agent-chat paths remain completely unchanged.

Made-with: Cursor
2026-04-08 12:41:37 +08:00
Yansong Zhang 96641a93f6 feat(api): add Agent V2 node and new Agent app type (Phase 1-3)
Introduce a new unified Agent V2 workflow node that combines LLM capabilities
with agent tool-calling loops, along with a new AppMode.AGENT for standalone
agent apps backed by single-node workflows.

Phase 1 — Agent Patterns:
- Add core/agent/patterns/ module (AgentPattern, FunctionCallStrategy,
  ReActStrategy, StrategyFactory) ported from feat/support-agent-sandbox
- Add ExecutionContext, AgentLog, AgentResult entities
- Add Tool.to_prompt_message_tool() for LLM-consumable tool conversion

Phase 2 — Agent V2 Workflow Node:
- Add core/workflow/nodes/agent_v2/ (AgentV2Node, AgentV2NodeData,
  AgentV2ToolManager, AgentV2EventAdapter)
- Register agent-v2 node type in DifyNodeFactory
- No-tools path: single LLM call (LLM Node equivalent)
- Tools path: FC/ReAct loop via StrategyFactory

Phase 3 — Agent App Type:
- Add AppMode.AGENT to model enum
- Add WorkflowGraphFactory for auto-generating start->agent_v2->answer graphs
- AppService.create_app() creates workflow draft for AGENT mode
- AppGenerateService.generate() routes AGENT to AdvancedChatAppGenerator
- Console API and DSL import/export support AGENT mode
- Default app template for AGENT mode

Old agent/agent-chat/LLM node paths are fully preserved.
38 unit tests all passing.

Made-with: Cursor
2026-04-08 12:31:23 +08:00
corevibe555andGitHub b1adb5652e refactor(api): deduplicate I18nObject in datasource entities (#34701) 2026-04-08 01:36:56 +00:00
corevibe555andGitHub c825d5dcf6 refactor(api): tighten types for Tenant.custom_config_dict and MCPToolProvider.headers (#34698) 2026-04-08 01:36:42 +00:00
corevibe555andGitHub 624db69f12 refactor(api): remove duplicated RAG entities from services layer (#34689) 2026-04-07 23:36:59 +00:00
corevibe555andGitHub 80a7843f45 refactor(api): migrate consumers to shared RAG domain entities from core/rag/entities/ (#34692) 2026-04-07 23:22:56 +00:00
StatxcandGitHub 5aa2524d33 refactor(api): type I18nObject.to_dict with I18nObjectDict TypedDict (#34680) 2026-04-07 22:57:32 +00:00
PulakeshandGitHub 2575a3a3ab refactor(api): clean up AssistantPromptMessage typing in CotChatAgentRunner (#34681) 2026-04-07 22:53:14 +00:00
corevibe555andGitHub d2ee486900 refactor(api): extract shared RAG domain entities into core/rag/entity (#34685) 2026-04-07 22:43:37 +00:00
StatxcandGitHub c44ddd9831 refactor(api): type Chroma and AnalyticDB config params dicts with TypedDicts (#34678) 2026-04-07 13:27:12 +00:00
StatxcandGitHub e645cbd8f8 refactor(api): type VDB config params dicts with TypedDicts (#34677) 2026-04-07 13:23:42 +00:00
YBoyandGitHub f09be969bb refactor(api): type single-node graph structure with TypedDicts in workflow_entry (#34671) 2026-04-07 13:18:00 +00:00
StatxcandGitHub 597a0b4d9f refactor(api): type indexing result with IndexingResultDict TypedDict (#34672) 2026-04-07 13:17:39 +00:00
StatxcandGitHub 779cce3c61 refactor(api): type gen_index_struct_dict with VectorIndexStructDict TypedDict (#34675) 2026-04-07 13:17:20 +00:00
StatxcandGitHub b5d9a71cf9 refactor(api): type VDB to_index_struct with VectorIndexStructDict TypedDict (#34674) 2026-04-07 13:17:04 +00:00
DreamandGitHub 89ce61cfea refactor(api): replace json.loads with Pydantic validation in security and tools layers (#34380) 2026-04-07 12:11:51 +00:00
StatxcandGitHub 19c80f0f0e refactor(api): type error stream response with TypedDict (#34641) 2026-04-07 05:57:42 +00:00
YBoyandGitHub c5a0bde3ec refactor(api): type aliyun trace utils with TypedDict and tighten return types (#34642) 2026-04-07 05:57:22 +00:00
YBoyandGitHub 84d8940dbf refactor(api): type app parameter feature toggles with FeatureToggleD… (#34651) 2026-04-07 05:53:50 +00:00
RenzoandGitHub 72adb5468c refactor: migrate session.query to select API in retrieval_service (#34638) 2026-04-07 04:46:30 +00:00
RenzoandGitHub b55bef4438 refactor: migrate session.query to select API in core misc modules (#34608) 2026-04-07 04:08:34 +00:00
StatxcandGitHub 0bce6b35b4 refactor(api): type LLM generator results with TypedDict (#34621) 2026-04-07 01:06:08 +00:00
YBoyandGitHub 12e93d374f refactor(api): type MCP tool schema and arguments with TypedDict (#34612) 2026-04-07 01:02:06 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>Asuka Minato
5b862a43e0 chore(deps-dev): bump the dev group in /api with 6 updates (#34579)
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Asuka Minato <[email protected]>
2026-04-06 11:49:54 +00:00
YBoyandGitHub 01ba0e050f refactor(api): reuse IdentityDict TypedDict in logging filters (#34593) 2026-04-06 11:30:21 +00:00
YBoyandGitHub e178451d04 refactor(api): type log identity dict with IdentityDict TypedDict (#34485) 2026-04-03 02:25:02 +00:00
lifandGitHub 2e29ac2829 fix: remove redundant cast in MCP base session (#34461)
Signed-off-by: majiayu000 <[email protected]>
2026-04-02 12:36:21 +00:00
8f9dbf269e chore(api): align Python support with 3.12 (#34419)
Co-authored-by: Asuka Minato <[email protected]>
2026-04-02 05:07:32 +00:00
YBoyandGitHub 2d29345f26 refactor(api): type OpsTraceProviderConfigMap with TracingProviderCon… (#34424) 2026-04-02 01:47:08 +00:00
jimmyzhuuandGitHub b23ea0397a fix: apply Baidu Vector DB connection timeout when initializing Mochow client (#34328) 2026-04-01 06:16:09 +00:00
DreamGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
c51cd42cb4 refactor(api): replace json.loads with Pydantic validation in controllers and infra layers (#34277)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-01 05:41:44 +00:00
Full Stack EngineerGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>Crazywoola
09ee8ea1f5 fix: support qa_preview shape in IndexProcessor preview formatting (#34151)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Crazywoola <[email protected]>
2026-04-01 04:22:23 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
4bd388669a refactor: core/app pipeline, core/datasource, and core/indexing_runner (#34359)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-01 02:20:56 +00:00
wangxiaoleiGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
57f358a96b perf: use global httpx client instead of per request create new one (#34311)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-01 01:19:32 +00:00
99andGitHub f27d669f87 chore: normalize frozenset literals and myscale typing (#34327) 2026-03-31 08:21:22 +00:00
tmimmanuelGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
1344c3b280 refactor: use EnumText for model_type in provider models (#34300)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-03-31 00:31:33 +00:00
tmimmanuelandGitHub 5897b28355 refactor: use EnumText for Provider.quota_type and consolidate ProviderQuotaType (#34299) 2026-03-31 00:29:57 +00:00
bc14ad6a8f fix: map checkbox and json_object types in MCP schema publishing (#34226)
Signed-off-by: majiayu000 <[email protected]>
Co-authored-by: Asuka Minato <[email protected]>
2026-03-30 15:05:57 +00:00
lifandGitHub ae9a16a397 fix: upgrade langfuse SDK to v3+ for LLM-as-judge support (#34265)
Signed-off-by: majiayu000 <[email protected]>
2026-03-30 13:06:55 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>Asuka Minato
456684dfc3 refactor: core/rag docstore, datasource, embedding, rerank, retrieval (#34203)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Asuka Minato <[email protected]>
2026-03-30 08:09:49 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>Asuka Minato
364d7ebc40 refactor: core/tools, agent, callback_handler, encrypter, llm_generator, plugin, inner_api (#34205)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Asuka Minato <[email protected]>
2026-03-28 10:14:43 +00:00
99GitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
40591a7c50 refactor(api): use standalone graphon package (#34209)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-03-27 21:05:32 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
32d394d65b refactor: select in core/ops trace manager and trace providers (#34197)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-03-27 14:00:26 +00:00
tmimmanuelGitHubAsuka Minatoautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
66fab8722c refactor: use EnumText for credential_type in TriggerSubscription (#34174)
Co-authored-by: Asuka Minato <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-03-27 10:53:51 +00:00