Commit Graph
975 Commits
Author SHA1 Message Date
Yansong Zhang 90cce7693f revert: remove all sandbox and skill related code
Remove ~12,900 lines of sandbox/skill code that was ported from
feat/support-agent-sandbox. This reverts to direct tool execution
(the original behavior before sandbox integration).

Removed:
- core/sandbox/ (SandboxBuilder, bash tools, providers, initializers)
- core/skill/ (SkillManager, assembler, entities)
- core/virtual_environment/ (5 provider implementations)
- core/zip_sandbox/ (archive operations)
- core/app_assets/ (asset management)
- core/app_bundle/ (bundle management)
- controllers/cli_api/ (DifyCli callback endpoints)
- services/sandbox/ (provider service)
- services/skill_service, app_asset_service, app_bundle_service
- models/sandbox.py, app_asset.py
- bin/dify-cli-* (3 platform binaries)
- web sandbox-provider-page and service
- SandboxLayer, _resolve_sandbox_context, _invoke_tool_in_sandbox
- CliApiConfig, DIFY_SANDBOX_CONTEXT_KEY
- sandbox-related migrations

Preserved: All Agent V2 core functionality (agent-v2 node, strategy
engine, transparent upgrade, LLM remapping, memory, context, tools
via direct execution).

Made-with: Cursor
2026-04-13 10:42:36 +08:00
Yansong Zhang 77c182f738 feat(api): propagate all app features in transparent upgrade
VirtualWorkflowSynthesizer._build_features() now extracts ALL legacy
app features from AppModelConfig into the synthesized workflow.features:

- opening_statement + suggested_questions
- sensitive_word_avoidance (keywords/API moderation)
- more_like_this
- speech_to_text / text_to_speech
- retriever_resource

Previously workflow.features was hardcoded to "{}", losing all these
features during transparent upgrade. Now AdvancedChatAppRunner's
moderation, opening text, and other feature layers work correctly
for transparently upgraded old apps.

Made-with: Cursor
2026-04-10 18:47:18 +08:00
Yansong Zhang e50c36526e fix(api): fix transparent upgrade SSE channel mismatch and chat mode routing
- workflow_execute_task: add AppMode.CHAT/AGENT_CHAT/COMPLETION to the
  AdvancedChatAppGenerator routing branch so transparently upgraded old
  apps can execute through the workflow engine.
- app_generate_service: use app_model.mode (not hardcoded AppMode.AGENT)
  for SSE event subscription channel, ensuring the subscriber and
  Celery publisher use the same Redis channel key.

Made-with: Cursor
2026-04-09 17:27:41 +08:00
Yansong Zhang e2e16772a1 fix(api): fix DSL import, memory loading, and remaining test coverage
1. DSL Import fix: change self._session.commit() to self._session.flush()
   in app_dsl_service.py _create_or_update_app() to avoid "closed transaction"
   error. DSL import now works: export agent app -> import -> new app created.

2. Memory loading attempt: added _load_memory_messages() to AgentV2Node
   that loads TokenBufferMemory from conversation history. However, chatflow
   engine manages conversations differently from easy-UI (conversation may
   not be in DB at query time, or uses ConversationVariablePersistenceLayer
   instead of Message table). Memory needs further investigation.

Test results:
- Multi-turn memory: Turn 1 OK, Turn 2 LLM doesn't see history (needs deeper fix)
- Service API with API Key: PASSED (answer="Sixteen" for 8+8)
- DSL Import: PASSED (status=completed, new app created)
- Token aggregation: PASSED (node=49, workflow=49)

Known: memory in multi-turn chatflow needs to use graphon's built-in
memory mechanism (MemoryConfig on node + ConversationVariablePersistenceLayer)
rather than direct DB query.

Made-with: Cursor
2026-04-09 14:47:55 +08:00
Yansong Zhang 482a004efe fix(api): fix duplicate answer and completion app upgrade issues
1. Remove StreamChunkEvent from AgentV2Node._run_without_tools():
   The agent-v2 node was yielding StreamChunkEvent during LLM streaming,
   AND the downstream answer node was outputting the same text via
   {{#agent.text#}} variable reference, causing "FourFour" duplication.
   Now text only flows through outputs.text -> answer node (single path).

2. Map inputs to query for completion app transparent upgrade:
   Completion apps send {inputs: {query: "..."}} not {query: "..."}.
   VirtualWorkflowSynthesizer route now extracts query from inputs
   when the top-level query is missing.

Verified:
- Old chat app: "What is 2+2?" -> "Four" (was "FourFour")
- Old completion app: {inputs: {query: "What is 3+3?"}} -> "3 + 3 = 6" (was failing)
- Old agent-chat app: still works

Made-with: Cursor
2026-04-09 12:02:43 +08:00
Yansong Zhang 7052257c8d fix(api): use lazy workflow persistence for transparent upgrade of old apps
VirtualWorkflowSynthesizer.ensure_workflow() creates a real draft
workflow on first call for a legacy app, persisting it to the database.
On subsequent calls, returns the existing draft.

This is needed because AdvancedChatAppGenerator's worker thread looks
up workflows from the database by ID. Instead of hacking the generator
to skip DB lookups, we treat this as a lazy one-time upgrade: the old
app gets a real workflow that can also be edited in the workflow editor.

Verified: old chat app created on main branch ("What is 2+2?" -> "Four")
and old agent-chat app ("Say hello" -> "Hello!") both successfully
execute through the Agent V2 engine with AGENT_V2_TRANSPARENT_UPGRADE=true.

Made-with: Cursor
2026-04-09 11:28:16 +08:00
Yansong Zhang edfcab6455 fix(api): add AGENT mode to app list filtering
Add AppMode.AGENT branch in get_paginate_apps() so that
filtering apps by mode=agent works correctly.
Discovered during comprehensive E2E testing.

14/14 E2E tests pass covering:
- A: New Agent app full lifecycle (create, draft, configs, publish, run)
- B: Old app creation compat (chat, completion, agent-chat, advanced-chat, workflow)
- C: App listing and filtering (all modes, agent filter)
- D: Workflow editor compat (block configs)
- E: DSL export

Made-with: Cursor
2026-04-09 10:54:05 +08:00
Yansong Zhang 66212e3575 feat(api): implement zero-migration transparent upgrade (Phase 8)
Add two feature-flag-controlled upgrade paths that allow existing apps
and LLM nodes to transparently run through the Agent V2 engine without
any database migration:

1. AGENT_V2_TRANSPARENT_UPGRADE (default: off):
   When enabled, old apps (chat/completion/agent-chat) bypass legacy
   Easy-UI runners. VirtualWorkflowSynthesizer converts AppModelConfig
   to an in-memory Workflow (start -> agent-v2 -> answer) at runtime,
   then executes via AdvancedChatAppGenerator. Falls back to legacy
   path on any synthesis error.

   VirtualWorkflowSynthesizer maps:
   - model JSON -> ModelConfig
   - pre_prompt/chat_prompt_config -> prompt_template
   - agent_mode.tools -> ToolMetadata[]
   - agent_mode.strategy -> agent_strategy
   - dataset_configs -> context
   - file_upload -> vision

2. AGENT_V2_REPLACES_LLM (default: off):
   When enabled, DifyNodeFactory.create_node() transparently remaps
   nodes with type="llm" to type="agent-v2" before class resolution.
   Since AgentV2NodeData is a strict superset of LLMNodeData, the
   mapping is lossless. With tools=[], Agent V2 behaves identically
   to LLM Node.

Both flags default to False for safety. Turn off = instant rollback.
46 existing tests pass. Flask starts successfully.

Made-with: Cursor
2026-04-09 10:30:52 +08:00
Yansong Zhang 44491e427c feat(api): enable all sandbox/skill controller routes and resolve dependencies (P0)
Resolve the full dependency chain to enable all previously disabled controllers:

Enabled routes:
- sandbox_files: sandbox file browser API
- sandbox_providers: sandbox provider management API
- app_asset: app asset management API
- skills: skill extraction API
- CLI API blueprint: DifyCli callback endpoints (/cli/api/*)

Dependencies extracted (64 files, ~8000 lines):
- models/sandbox.py, models/app_asset.py: DB models
- core/zip_sandbox/: zip-based sandbox execution
- core/session/: CLI API session management
- core/memory/: base memory + node token buffer
- core/helper/creators.py: helper utilities
- core/llm_generator/: context models, output models, utils
- core/workflow/nodes/command/: command node type
- core/workflow/nodes/file_upload/: file upload node type
- core/app/entities/: app_asset_entities, app_bundle_entities, llm_generation_entities
- services/: asset_content, skill, workflow_collaboration, workflow_comment
- controllers/console/app/error.py: AppAsset error classes
- core/tools/utils/system_encryption.py

Import fixes:
- dify_graph.enums -> graphon.enums in skill_service.py
- get_signed_file_url_for_plugin -> get_signed_file_url in cli_api.py

All 5 controllers verified: import OK, Flask starts successfully.
46 existing tests still pass.

Made-with: Cursor
2026-04-09 09:36:16 +08:00
Yansong Zhang 0c7e7e0c4e feat(api): port Sandbox + VirtualEnvironment + Skill system from feat/support-agent-sandbox (Phase 5-6)
Port the complete infrastructure for agent sandbox execution and skill system:

Sandbox & Virtual Environment (core/sandbox/, core/virtual_environment/):
- Sandbox entity with lifecycle management (ready/failed/cancelled states)
- SandboxBuilder with fluent API for configuring providers
- 5 VM providers: Local, SSH, Docker, E2B, AWS CodeInterpreter
- VirtualEnvironment base with command execution, file transfer, transport layers
- Channel transport: pipe, queue, socket implementations
- Bash session management and DifyCli binary integration
- Storage: archive storage, file storage, noop storage, presign storage
- Initializers: DifyCli, AppAssets, DraftAppAssets, Skills
- Inspector: file browser, archive/runtime source, script utils
- Security: encryption utils, debug helpers

Skill & App Assets (core/skill/, core/app_assets/, core/app_bundle/):
- Skill entity and manager
- App asset accessor, builder pipeline (file, skill builders)
- App bundle source zip extractor
- Storage and converter utilities

API Endpoints:
- CLI API blueprint (controllers/cli_api/) for sandbox callback
- Sandbox provider management (workspace/sandbox_providers)
- Sandbox file browser (console/sandbox_files)
- App asset management (console/app/app_asset)
- Skill management (console/app/skills)
- Storage file endpoints (controllers/files/storage_files)

Services:
- Sandbox service, provider service, file service
- App asset service, app bundle service

Config:
- CliApiConfig, CreatorsPlatformConfig, CollaborationConfig
- FILES_API_URL for sandbox file access

Note: Controller route registration temporarily commented out (marked TODO)
pending resolution of deep dependency chains (socketio, workflow_comment,
command node, etc.). Core sandbox modules are fully ported and syntax-validated.
110 files changed, 10,549 insertions.

Made-with: Cursor
2026-04-08 17:39:02 +08:00
Yansong Zhang d9d1e9b63a fix(api): resolve Agent V2 node E2E runtime issues
Fixes discovered during end-to-end testing of Agent workflow execution:

1. ModelManager instantiation: use ModelManager.for_tenant() instead of
   ModelManager() which requires a ProviderManager argument
2. Variable template resolution: use VariableTemplateParser(template).format()
   instead of non-existent resolve_template() static method
3. invoke_llm() signature: remove unsupported 'user' keyword argument
4. Event dispatch: remove ModelInvokeCompletedEvent from _run() yield
   (graphon base Node._dispatch doesn't support it via singledispatch)
5. NodeRunResult metadata: use WorkflowNodeExecutionMetadataKey enum keys
   (TOTAL_TOKENS, TOTAL_PRICE, CURRENCY) instead of arbitrary string keys
6. SSE topic mismatch: use AppMode.AGENT (not ADVANCED_CHAT) in
   retrieve_events() so publisher and subscriber share the same channel
7. Celery task routing: add AppMode.AGENT to workflow_execute_task._run_app()
   alongside ADVANCED_CHAT

All issues verified fixed: Agent V2 node successfully invokes LLM and
returns "Hello there!" through the full SSE streaming pipeline.

Made-with: Cursor
2026-04-08 16:21:12 +08:00
Yansong Zhang 1835a1dc5d fix(api): allow AGENT mode in workflow features validation
Add AppMode.AGENT to validate_features_structure() match case
alongside ADVANCED_CHAT, fixing 'Invalid app mode: agent' error
when creating Agent apps (which auto-generate a workflow draft).

Discovered during E2E testing of the full create -> draft -> publish flow.

Made-with: Cursor
2026-04-08 13:19:59 +08:00
Yansong Zhang 96641a93f6 feat(api): add Agent V2 node and new Agent app type (Phase 1-3)
Introduce a new unified Agent V2 workflow node that combines LLM capabilities
with agent tool-calling loops, along with a new AppMode.AGENT for standalone
agent apps backed by single-node workflows.

Phase 1 — Agent Patterns:
- Add core/agent/patterns/ module (AgentPattern, FunctionCallStrategy,
  ReActStrategy, StrategyFactory) ported from feat/support-agent-sandbox
- Add ExecutionContext, AgentLog, AgentResult entities
- Add Tool.to_prompt_message_tool() for LLM-consumable tool conversion

Phase 2 — Agent V2 Workflow Node:
- Add core/workflow/nodes/agent_v2/ (AgentV2Node, AgentV2NodeData,
  AgentV2ToolManager, AgentV2EventAdapter)
- Register agent-v2 node type in DifyNodeFactory
- No-tools path: single LLM call (LLM Node equivalent)
- Tools path: FC/ReAct loop via StrategyFactory

Phase 3 — Agent App Type:
- Add AppMode.AGENT to model enum
- Add WorkflowGraphFactory for auto-generating start->agent_v2->answer graphs
- AppService.create_app() creates workflow draft for AGENT mode
- AppGenerateService.generate() routes AGENT to AdvancedChatAppGenerator
- Console API and DSL import/export support AGENT mode
- Default app template for AGENT mode

Old agent/agent-chat/LLM node paths are fully preserved.
38 unit tests all passing.

Made-with: Cursor
2026-04-08 12:31:23 +08:00
corevibe555andGitHub c825d5dcf6 refactor(api): tighten types for Tenant.custom_config_dict and MCPToolProvider.headers (#34698) 2026-04-08 01:36:42 +00:00
corevibe555andGitHub 624db69f12 refactor(api): remove duplicated RAG entities from services layer (#34689) 2026-04-07 23:36:59 +00:00
corevibe555andGitHub 80a7843f45 refactor(api): migrate consumers to shared RAG domain entities from core/rag/entities/ (#34692) 2026-04-07 23:22:56 +00:00
corevibe555andGitHub f8f7b0ec1a refactor(api): deduplicate shared auth request payloads into auth_entities.py (#34694) 2026-04-07 22:51:11 +00:00
DreamandGitHub 89ce61cfea refactor(api): replace json.loads with Pydantic validation in security and tools layers (#34380) 2026-04-07 12:11:51 +00:00
StatxcandGitHub 63db9a7a2f refactor(api): type load balancing config dicts with TypedDict (#34639) 2026-04-07 05:58:10 +00:00
YBoyandGitHub 1261e5e5e8 refactor(api): type webhook validation result and workflow inputs with TypedDict (#34645) 2026-04-07 05:57:02 +00:00
PulakeshandGitHub bceb0eee9b refactor(api): migrate dict returns to TypedDicts in billing service (#34649) 2026-04-07 05:56:02 +00:00
RenzoandGitHub 1194957fde refactor: migrate session.query to select API in end_user_service and small tasks (#34620) 2026-04-07 04:25:55 +00:00
RenzoandGitHub 68bd29eda2 refactor: migrate session.query to select API in sync task and services (#34619) 2026-04-07 04:23:14 +00:00
YBoyGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
f67a811f7f refactor: replace dict params with BaseModel payloads in TagService (#34422)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-07 04:20:02 +00:00
StatxcandGitHub a7b6307d32 refactor(api): type dataset service dicts with TypedDict (#34625) 2026-04-07 02:10:52 +00:00
StatxcandGitHub 2883ad6764 refactor(api): type plugin migration results with TypedDict (#34627) 2026-04-07 02:10:23 +00:00
PulakeshandGitHub 0feff5b048 refactor(api): enforce strict typing on retrieval_model to resolve FIXME (#34614) 2026-04-07 01:10:53 +00:00
YBoyGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
89e23456f0 refactor(api): type invitation detail with InvitationDetailDict TypedDict (#34613)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-07 01:03:31 +00:00
YBoyandGitHub 922f9242e4 refactor(api): type crawl status dicts with CrawlStatusDict TypedDict (#34611) 2026-04-07 01:01:04 +00:00
YBoyandGitHub 7fc0a791a2 refactor(api): type document summary status detail with TypedDict (#34610) 2026-04-07 01:00:39 +00:00
YBoyandGitHub 5ad906ea6a refactor(api): type workflow run related counts with RelatedCountsDict TypedDict (#34530) 2026-04-06 13:17:01 +00:00
YBoyandGitHub 1e5cd69205 refactor(api): type archive manifest with ArchiveManifestDict TypedDict (#34594) 2026-04-06 11:35:11 +00:00
YBoyGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
b71b9f80b9 refactor(api): type workflow run delete/count results with RunsWithRelatedCountsDict TypedDict (#34531)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-05 16:11:41 +00:00
agenthaulkGitHubagenthaulkautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
ee87289917 refactor: convert AppMode if/elif to match/case in app_generate_service (#30001) (#34563)
Co-authored-by: agenthaulk <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-05 10:23:51 +00:00
agenthaulkGitHubagenthaulkautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
5ad8c3e249 refactor: convert AppMode if/elif to match/case in service files (#30001) (#34562)
Co-authored-by: agenthaulk <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-05 10:22:11 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
eca0cdc7a9 refactor: select in dataset_service (SegmentService and remaining cla… (#34547)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-05 00:13:06 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
779e6b8e0b refactor: select in datasource_provider_service (#34548)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-05 00:12:15 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
c2428361c4 refactor: select in dataset_service (DocumentService class) (#34528)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 22:52:01 +00:00
RenzoGitHubAsuka Minatoautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
68e4d13f36 refactor: select in annotation_service (#34503)
Co-authored-by: Asuka Minato <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 22:47:22 +00:00
YBoyandGitHub 8a398f3105 refactor(api): type messages cleanup stats with MessagesCleanStatsDict TypedDict (#34527) 2026-04-03 12:29:41 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
e85d9a0d72 refactor: select in dataset_service (DatasetService class) (#34525)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 12:01:31 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>Asuka Minato
06dde4f503 refactor: select in account_service (TenantService class) (#34499)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Asuka Minato <[email protected]>
2026-04-03 11:03:45 +00:00
64ddec0d67 refactor(api): type annotation service dicts with TypedDict (#34482)
Co-authored-by: Asuka Minato <[email protected]>
2026-04-03 06:25:52 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
da3b0caf5e refactor: select in account_service (RegisterService class) (#34500)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 06:21:26 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
608958de1c refactor: select in external_knowledge_service (#34493)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 03:42:16 +00:00
RenzoGitHubautofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
7eb632eb34 refactor: select in rag_pipeline (#34495)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 03:42:01 +00:00
RenzoandGitHub 33d4fd357c refactor: select in account_service (AccountService class) (#34496) 2026-04-03 03:41:46 +00:00
YBoyandGitHub 71d299d0d3 refactor(api): type hit testing retrieve responses with TypedDict (#34484) 2026-04-03 02:25:30 +00:00
YBoyandGitHub 9a6222f245 refactor(api): type webhook data extraction with RawWebhookDataDict TypedDict (#34486) 2026-04-03 02:24:17 +00:00
YBoyandGitHub affe5ed30b refactor(api): type get_knowledge_rate_limit with KnowledgeRateLimitD… (#34483) 2026-04-03 02:23:32 +00:00