chore(api): restore summary index session callsites

Co-authored-by: WH-2099 <wh2099@pm.me>

.agents/skills/e2e-cucumber-playwright/SKILL.md

@@ -5,7 +5,7 @@ description: Write, update, or review Dify end-to-end tests under `e2e/` that us
 
 # Dify E2E Cucumber + Playwright
 
-Use this skill for Dify's repository-level E2E suite in `e2e/`. Use [`e2e/AGENTS.md`](../../../e2e/AGENTS.md) as the canonical guide for local architecture and conventions, then apply Playwright/Cucumber best practices only where they fit the current suite.
+Use this skill for Dify's repository-level E2E suite in `e2e/`. Use [`e2e/AGENTS.md`](../../../e2e/AGENTS.md) as the canonical package guide for local architecture and conventions, then read any feature-scoped `AGENTS.md` that owns the target area. Apply Playwright/Cucumber best practices only where they fit the current suite.
 
 ## Scope
 
@@ -25,6 +25,8 @@ Use this skill for Dify's repository-level E2E suite in `e2e/`. Use [`e2e/AGENTS
 4. Read [`references/cucumber-best-practices.md`](references/cucumber-best-practices.md) only when scenario wording, step granularity, tags, or expression design are involved.
 5. Re-check official Playwright or Cucumber docs with the available documentation tools before introducing a new framework pattern.
 
+Keep this skill focused on Cucumber, Playwright, and package-level E2E guidance. Put feature-specific conventions in the owning feature's `AGENTS.md` instead of adding them here.
+
 ## Local Rules
 
 - `e2e/` uses Cucumber for scenarios and Playwright as the browser layer.
@@ -43,6 +45,7 @@ Use this skill for Dify's repository-level E2E suite in `e2e/`. Use [`e2e/AGENTS
    - Inspect the target feature area.
    - Reuse an existing step when wording and behavior already match.
    - Add a new step only for a genuinely new user action or assertion.
+   - Before adding several similar steps, scan the target capability for an existing domain noun that can be parameterized without hiding behavior.
    - Keep edits close to the current capability folder unless the step is broadly reusable.
 2. Write behavior-first scenarios.
    - Describe user-observable behavior, not DOM mechanics.
@@ -51,15 +54,19 @@ Use this skill for Dify's repository-level E2E suite in `e2e/`. Use [`e2e/AGENTS
 3. Write step definitions in the local style.
    - Keep one step to one user-visible action or one assertion.
    - Prefer Cucumber Expressions such as `{string}` and `{int}`.
+   - Use a bounded regex only when the accepted values are a small explicit domain set and Cucumber Expressions would make the Gherkin less natural.
+   - Do not create one-off steps for each case variant when the same domain action or outcome applies to named surfaces, modes, or resources.
    - Scope locators to stable containers when the page has repeated elements.
    - Avoid page-object layers or extra helper abstractions unless repeated complexity clearly justifies them.
 4. Use Playwright in the local style.
    - Prefer user-facing locators: `getByRole`, `getByLabel`, `getByPlaceholder`, `getByText`, then `getByTestId` for explicit contracts.
    - Use web-first `expect(...)` assertions.
    - Do not use `waitForTimeout`, manual polling, or raw visibility checks when a locator action or retrying assertion already expresses the behavior.
+   - Use `expect.poll` for API persistence, backend eventual consistency, captured browser events, or other non-DOM state; prefer locator assertions for DOM readiness and visible UI state.
+   - If a product element has real user-facing semantics but no accessible name, prefer fixing that accessible contract over adding a test id.
 5. Validate narrowly.
    - Run the narrowest tagged scenario or flow that exercises the change.
-   - Run `pnpm -C e2e check`.
+   - Run `vpr lint --fix --quiet` from the repository root and `pnpm -C e2e type-check`.
    - Broaden verification only when the change affects hooks, tags, setup, or shared step semantics.
 
 ## Review Checklist

.agents/skills/e2e-cucumber-playwright/references/cucumber-best-practices.md

@@ -39,6 +39,7 @@ Prefer reuse when:
 - the user action is genuinely the same
 - the expected outcome is genuinely the same
 - the wording stays natural across features
+- the parameter is a real product domain value such as a named surface, mode, resource, or status
 
 Write a new step when:
 
@@ -46,6 +47,8 @@ Write a new step when:
 - reusing the old wording would make the scenario misleading
 - a supposedly generic step would become an implementation-detail wrapper
 
+Do not optimize for a low step count by making vague steps. Optimize for a small set of truthful, domain-owned steps.
+
 ### 4. Prefer Cucumber Expressions
 
 Use Cucumber Expressions for parameters unless regex is clearly necessary.
@@ -59,6 +62,8 @@ Common examples:
 
 Keep expressions readable. If a step needs complicated parsing logic, first ask whether the scenario wording should be simpler.
 
+Use regex for a bounded natural-language alternative only when it keeps Gherkin readable, for example `/(Web app|Backend service API)/`. Avoid broad regexes that accept unowned language.
+
 ### 5. Keep step definitions thin and meaningful
 
 Step definitions are glue between Gherkin and automation, not a second abstraction language.

.agents/skills/e2e-cucumber-playwright/references/playwright-best-practices.md

@@ -41,6 +41,7 @@ Also remember:
 - repeated content usually needs scoping to a stable container
 - exact text matching is often too brittle when role/name or label already exists
 - `getByTestId` is acceptable when semantics are weak but the contract is intentional
+- when a real UI region, card, status, or icon lacks an accessible name, prefer adding that semantic contract in product code before falling back to `getByTestId`
 
 ### 3. Use web-first assertions
 
@@ -62,6 +63,8 @@ Avoid:
 
 If a condition genuinely needs custom retry logic, use Playwright's polling/assertion tools deliberately and keep that choice local and explicit.
 
+Use `expect.poll` for non-DOM truth such as API state, backend eventual consistency, generated resources, or captured browser events. For DOM state, use locator assertions so Playwright can apply actionability and web-first retry semantics.
+
 ### 4. Let actions wait for actionability
 
 Locator actions already wait for the element to be actionable. Do not preface every click/fill with extra timing logic unless the action needs a specific visible/ready assertion for clarity.

.agents/skills/frontend-code-review/references/dify-ui.md

@@ -40,9 +40,13 @@ Flag:
 - JavaScript conditional class logic for visual states that the Dify UI/Base UI primitive already exposes through `data-*` attributes or CSS variables.
 - Controlled props added when uncontrolled DOM state or CSS variables would be enough.
 - Thin wrappers that rename Base UI parts without adding semantics.
+- Generic Base UI selection primitives wrapped without preserving their value generics, such as `Select.Root<Value, Multiple>`, `RadioGroup<Value>`, or `Radio.Root<Value>`.
+- Shared select/radio option components that type selected values as `string` while callers pass enums, unions, booleans, numbers, objects, or nullable placeholder values.
 
 Prefer Base UI/Dify UI data attributes and CSS variables for visual state: `data-open`, `data-checked`, `data-disabled`, `data-highlighted`, `data-popup-open`, `group-data-*`, `peer-data-*`, `has-[:focus-visible]`, and primitive CSS variables such as anchor width or transform origin. Use JS conditional classes for product/business state that the primitive does not expose.
 
+For non-string `Select` and `RadioGroup` values, prefer explicit domain generics at the root and at child value carriers. JSX children do not inherit the parent generic, so `RadioGroup<PromptMode>` should compose with `Radio<PromptMode>`, `RadioRoot<PromptMode>`, or option values from a typed collection. For `Select`, prefer the Base UI `items` collection pattern for typed value-to-label rendering, and flag string coercion helpers used only to recover display labels.
+
 ## Forms
 
 Flag:

.agents/skills/frontend-testing/SKILL.md

@@ -102,11 +102,11 @@ describe('ComponentName', () => {
     })
   })
 
-  // Props tests (REQUIRED)
+  // Props tests (REQUIRED when props change observable behavior)
   describe('Props', () => {
-    it('should apply custom className', () => {
-      render(<Component className="custom" />)
-      expect(screen.getByRole('button')).toHaveClass('custom')
+    it('should disable the action when disabled', () => {
+      render(<Component disabled />)
+      expect(screen.getByRole('button')).toBeDisabled()
     })
   })
 
@@ -220,6 +220,7 @@ Every test should clearly separate:
 ### 2. Black-Box Testing
 
 - Test observable behavior, not implementation details
+- Test product contracts, not cosmetic implementation. Do not add or expand unit tests only to lock pure style classes, spacing, colors, backgrounds, or layout micro-adjustments. Cover visual-only fixes with browser/manual verification, screenshots, or E2E/visual checks when risk justifies it. Add unit tests only when the change affects user-observable behavior, accessibility semantics, state, data flow, routing, or a stable component API contract.
 - Use semantic queries (`getByRole` with accessible `name`, `getByLabelText`, `getByPlaceholderText`, `getByText`, and scoped `within(...)`)
 - Treat `getByTestId` as a last resort. If a control cannot be found by role/name, label, landmark, or dialog scope, fix the component accessibility first instead of adding or relying on `data-testid`.
 - Remove production `data-testid` attributes when semantic selectors can cover the behavior. Keep them only for non-visual mocked boundaries, editor/browser shims such as Monaco, canvas/chart output, or third-party widgets with no accessible DOM in the test environment.
@@ -273,7 +274,7 @@ it('should disable input when isReadOnly is true')
 ### Always Required (All Components)
 
 1. **Rendering**: Component renders without crashing
-1. **Props**: Required props, optional props, default values
+1. **Props**: Required props, optional props, default values that change observable behavior. Do not test pass-through styling props such as `className` unless they are an explicit, stable component API whose absence would break a real integration contract.
 1. **Edge Cases**: null, undefined, empty values, boundary conditions
 
 ### Conditional (When Present)

.agents/skills/how-to-write-component/SKILL.md

@@ -1,128 +1,128 @@
 ---
 name: how-to-write-component
-description: React/TypeScript component style guide. Use when writing, refactoring, or reviewing React components, especially around abstraction choices, props typing, state boundaries, shared local state with Jotai atoms, API types, query/mutation contracts, navigation, memoization, wrappers, and empty-state handling.
+description: Use when writing, refactoring, or reviewing React/TypeScript components in Dify web, especially decisions about component ownership, props/types, URL/query state, Jotai state, async state, generated API contracts, queries/mutations, overlays, effects, navigation, performance, and empty states.
 ---
 
 # How To Write A Component
 
-Use this as the decision guide for React/TypeScript component structure. Existing code is reference material, not automatic precedent; when it conflicts with these rules, adapt the approach instead of reproducing the violation.
+Use this as the component decision guide for Dify web. Existing code is reference material, not automatic precedent; if touched code violates these rules, adapt it and fix equivalent patterns in the same feature branch.
+
+## First Decisions
+
+| Question | Default | Promote or extract only when |
+| --- | --- | --- |
+| Where should code live? | Keep it local to the feature workflow, route, or owner. | Multiple verticals need the same stable primitive. |
+| How should route/tab folders be named? | Match the current route segment, tab name, or user-visible surface. | Keep a historical or broader parent only when it still owns multiple surfaces. |
+| Who owns state, data, and handlers? | The lowest component that uses them. | A parent coordinates shared loading, errors, empty UI, selection, submission, navigation, or one consistent snapshot. |
+| Should this become Jotai state? | Keep synchronous UI/form state in component or DOM state. | Siblings need one source of truth, the value drives atoms, or scoped workflow state must survive hidden/unmounted steps. |
+| Should URL state enter Jotai? | Let Next.js route params and `nuqs` own URL state and updates. | Query atoms or shared derived atoms need a read-only bridge hydrated at the route/surface boundary. |
+| Should this query/mutation become an atom? | Use TanStack Query hooks at the lowest owner. | It reads atom state, feeds derived atoms, or participates in shared Jotai workflow orchestration. |
+| Should this be a helper/wrapper? | Prefer direct readable code at the use site. | The name captures a stable domain rule or the wrapper owns real behavior, validation, state, error handling, or semantics. |
+| Is an Effect needed? | No. Derive during render or handle the user action in the event handler. | It synchronizes with an external system such as browser APIs, subscriptions, timers, analytics, or imperative DOM/non-React widgets. |
 
 ## Core Defaults
 
-- Search before adding UI, hooks, helpers, or styling patterns. Reuse existing base components, feature components, hooks, utilities, and design styles when they fit.
-- Group code by feature workflow, route, or ownership area: components, hooks, local types, query helpers, atoms, constants, and small utilities should live near the code that changes with them.
-- Promote code to shared only when multiple verticals need the same stable primitive. Otherwise keep it local and compose shared primitives inside the owning feature.
-- Prefer local code and purpose-named helpers over catch-all utility modules; do not group workflow-specific defaults, validation, payload shaping, or metadata merging in a generic utils file just because they share a DTO.
-- Keep source/default selection, validation, and payload shaping close to the workflow that owns the behavior. Do not extract a shared helper just because two flows read the same DTO when their priority order, fallback behavior, or submit semantics differ.
-- Prefer direct, readable conditionals at the use site for small branch-specific decisions, especially form source selection and request payload assembly. Extract only when the helper name captures a stable domain rule and removes repeated complexity without hiding flow-specific behavior.
-- When fixing an invalid pattern, scan the touched feature or branch for equivalent patterns and fix them together.
-- Follow Dify's CSS-first Tailwind v4 contract from `packages/dify-ui/README.md` and `packages/dify-ui/AGENTS.md`. Prefer design-system tokens, utilities, and radius mappings over generic Tailwind guidance.
+- Search before adding UI, hooks, helpers, query utilities, or styling patterns. Reuse existing base components, feature components, hooks, utilities, and design styles when they fit.
+- Follow Dify's CSS-first Tailwind v4 contract from `packages/dify-ui/README.md` and `packages/dify-ui/AGENTS.md`. Prefer design-system tokens, utilities, and radius mappings over generic Tailwind choices.
+- Group feature code by workflow, route, or ownership area with route-aligned names: components, hooks, local types, query helpers, atoms, constants, tests, and small utilities should live near the code that changes with them.
+- For each feature module, keep a module-local `README.md` as a boundary note. Start with the module name, a brief one-sentence description, then split dependencies into `Internal Modules` and `External Modules` sections; keep both sections and write `None.` when one category is empty. `Internal Modules` lists modules inside the same overall feature using paths from that feature root, such as `shared/domain/runtime-status`; `External Modules` lists project modules outside the feature using paths from the web root without a `web/` prefix, such as `app/components/base/skeleton`. Omit npm packages, workspace package dependencies, and whitelisted plumbing modules. Do not copy caller-relative import paths into the README.
+- Module README whitelist: `@/service/client`, `@/next/*`.
+- Keep source/default selection, validation, dirty checks, and payload shaping close to the workflow that owns submit behavior. Do not hide flow-specific priority order, fallback behavior, or submit semantics in generic utilities.
+- Prefer direct conditionals for small branch-specific decisions, especially form source selection and request payload assembly.
+- Loading states for page sections, cards, lists, tables, forms, and drawers should be skeletons scoped to the content being loaded. Use spinners only for small inline busy indicators.
 
-## Feature Workflow Layout
+## Layout And Ownership
 
-- State-heavy wizards, drawers, modals, and secondary workflows work best as a small feature surface with route/entry files, a single feature-local state file, and feature-local UI.
-- Keep `ui/` shallow with owner files that map to the workflow's real composition boundaries and major visual regions.
-- Owner files contain the section components, field components, skeletons, and one-off helper components that belong to their visual region.
-- Folders represent groups of related files with a shared owner and a stable reason to change together.
-- The entry file handles route integration, provider wiring, close behavior, and feature surface mounting. The composition owner handles high-level workflow branching, and the closest visual owner handles section branching.
+- State-heavy wizards, drawers, modals, and secondary workflows can be a small feature surface: an entry file, one feature-local state file when Jotai is actually needed, and shallow `ui/` owners that match real visual regions.
+- The entry file handles route integration, provider wiring, close behavior, and surface mounting. The composition owner handles high-level workflow branching. The closest visual owner handles section branching.
+- When a page or tab maps to a route segment, name its feature folder after that route/tab surface instead of a stale parent grouping. Remove misleading intermediate folders when only one surface remains.
+- When a tab folder grows into several independent sections or action areas, split the first level by product/visual owners. Keep the root for the entry component and cross-owner state, colocate tests with the owner folder, and put truly shared local UI under a specifically named `components/` file.
+- Repeated TanStack query calls in sibling components are acceptable when each component independently consumes the data; TanStack Query deduplicates and shares cache.
+- Pass stable domain identity across boundaries. Do not forward derived presentation state when the receiver can derive it from its own data source.
+- A component that owns a visual surface should also own data access, loading, empty, and error states for content rendered inside it unless a parent truly coordinates that state.
+- Avoid prop drilling. One pass-through layer is acceptable; repeated forwarding means ownership should move down or into feature-scoped Jotai UI state. Keep server/cache state in Query and API flow.
+- Do not replace prop drilling with one large view-model hook threaded through section props. Move each hook, query, derived value, and handler to the concrete section that consumes it.
+- Keep callbacks in a parent only for workflow coordination such as form submission, shared selection, batch behavior, or navigation. Otherwise let the child, menu, or row own the action.
 
-## Ownership
+## Feature-Scoped Jotai
 
-- Put local state, queries, mutations, handlers, and derived UI data in the lowest component that uses them. Extract a purpose-built owner component only when the logic has no natural home.
-- Repeated TanStack query calls in sibling components are acceptable when each component independently consumes the data. Do not hoist a query only because it is duplicated; TanStack Query handles deduplication and cache sharing.
-- Hoist state, queries, or callbacks to a parent only when the parent consumes the data, coordinates shared loading/error/empty UI, needs one consistent snapshot, or owns a workflow spanning children.
-- Pass stable domain identity across boundaries; avoid forwarding derived presentation state when the receiver can derive it from its own data source. A component that owns a visual surface should also own the data access, loading, empty, and error states for content rendered inside it unless a parent truly coordinates that state.
-- Loading states for visual surfaces should use skeleton placeholders scoped to the content that is actually loading, with shape, density, and dimensions close to the final UI. Avoid generic loading text or centered spinners for page sections, cards, lists, tables, forms, and drawers; reserve spinners for small inline busy indicators such as an in-progress status icon.
-- Avoid prop drilling. One pass-through layer is acceptable; repeated forwarding means ownership should move down or into feature-scoped Jotai UI state. Keep server/cache state in query and API data flow.
-- Do not replace prop drilling with one top-level hook that returns a large view model and then thread that object through section props. Move each hook, query, derived value, and handler to the concrete section that consumes it, or use feature-scoped Jotai atoms for simple shared form/UI state when siblings need the same source of truth.
-- When using feature-scoped Jotai state for a form, drawer, or other secondary surface, scope the store to that surface instance when stale cross-instance state is possible. Initialize stable config at the owning boundary, then let descendants read only the atoms or purpose-named hooks they actually need.
-- Keep callbacks in a parent only for workflow coordination such as form submission, shared selection, batch behavior, or navigation. Otherwise let the child or row own its action.
-- Prefer uncontrolled DOM state and CSS variables before adding controlled props.
-
-## Feature-Scoped Jotai State
-
-- A module's feature-local state lives in one state file for Jotai-backed features: primitive atoms, query atoms, derived atoms, write-only action atoms, mutation atoms, submission orchestration, provider exports, and optional scope configuration.
-- Atom order in the state file follows the dependency graph: types/constants, editable primitives, query atoms, query-data derived atoms, readiness/business derived atoms, write actions, mutation atoms, submission orchestration, provider exports.
-- Derived atom names read as business facts. Write atom names read as user or workflow commands.
-- UI components read and write the exact atom they use with `useAtomValue` or `useSetAtom`. Repeated workflow semantics live in named derived atoms or write atoms.
-- Non-query derived atoms return a narrow value with a clear domain name. Query atoms expose the TanStack Query result object so loading, error, fetch, and pagination state stay attached to the query contract.
-- Write-only atoms own state transitions that update multiple primitives, reset dependent state, guard stale async work, or advance the workflow.
-- `jotai-tanstack-query` atoms use the same QueryClient as the React Query provider. Query atoms belong in feature state when atoms are the feature's local state surface.
-- Jotai scope is an optional instance-isolation tool for secondary surfaces with independent local state. Query atoms keep shared cache behavior through the shared QueryClient.
+- A Jotai-backed feature has one feature-local state file for shared primitive atoms, query atoms, derived atoms, write-only actions, mutation atoms, submission orchestration, provider exports, and optional scope configuration.
+- Keep component-owned synchronous UI state local even inside Jotai features: dialog open flags, menus/popovers, confirmations, field drafts, and selected local options usually belong in component state.
+- Use uncontrolled `@langgenius/dify-ui/form` and `@langgenius/dify-ui/field` controls for edit/create forms whose fields are read only at submit time. Initialize query-backed defaults with `defaultValue` and keyed remounts.
+- Promote form state to atoms only when another component must react to in-progress values, a draft must survive unmount/remount in the scoped workflow, or multiple steps share the same editable draft before submit.
+- Treat `useParams`, route args, and `nuqs` query state as framework-owned state. When atom logic needs those values, hydrate primitive atoms at the route or surface boundary, such as with `useHydrateAtoms(..., { dangerouslyForceHydrate: true })`; keep URL updates in the route/query-state APIs instead of write atoms.
+- Within a route-owned feature, choose one source for route identity. If route params are bridged into feature atoms, use that bridge consistently for route-derived queries and actions instead of also threading the same route id through page, tab, and section props.
+- For async work tied to atom state, use `atomWithQuery` or `atomWithMutation`; write atoms should update only the inputs that drive those atoms. This applies to pure frontend async work as well as network requests, so do not hand-roll loading/error/in-flight state with `useState` or `useRef` for atom-orchestrated async behavior. For component-owned remote work, use `useQuery` or `useMutation` directly.
+- Row-local async state belongs to the row owner unless it participates in a shared Jotai workflow or needs atom-scoped reset semantics.
+- Leave query and mutation atoms unscoped so they keep shared QueryClient cache and invalidation behavior. Scope resettable primitives and explicit hydration tuples; scope a derived atom only when every dependency should be private to that surface.
+- For scoped primitives that are always hydrated by `ScopeProvider`, prefer `atomWithLazy<T>(() => { throw new Error(...) })` when consumers should see a non-null type.
+- Order state files by dependency graph: types/constants, primitives, query atoms, query-data derived atoms, business/readiness derived atoms, write actions, mutation atoms, submission orchestration, provider exports.
+- Name derived atoms as business facts and write atoms as user or workflow commands. Components should read or write the exact atom they need with `useAtomValue` or `useSetAtom`.
+- Menu/dialog `open` state usually stays local, but a scoped atom is acceptable when a composed menu plus secondary surface would otherwise pass confusing `open`/`onClose` props through unrelated layers. Scope that primitive with the surface instance so reset behavior stays local.
+- Keep independent dialog lifecycles separate. Avoid one discriminated "current action dialog" atom when dialogs have separate open state, loading guards, or reset behavior.
 
 ## Components, Props, And Types
 
 - Type component signatures directly; do not use `FC` or `React.FC`.
 - Prefer `function` for top-level components and module helpers. Use arrow functions for local callbacks, handlers, and lambda-style APIs.
 - Prefer named exports. Use default exports only where the framework requires them, such as Next.js route files.
+- Avoid barrel files that only re-export secondary owners. `index.tsx` is acceptable for a route/tab entry component; import header controls, switches, sections, and row owners from their concrete owner files.
 - Type simple one-off props inline. Use a named `Props` type only when reused, exported, complex, or clearer.
 - Use API-generated or API-returned types at component boundaries. Keep small UI conversion helpers and one-off UI extensions beside the component that needs them.
-- Do not create type aliases that only rename another type. Use an alias only when it encodes a real UI concept, refinement, or reusable local contract.
-- Name values by their domain role and backend API contract, and keep that name stable across the call chain, especially persistent IDs and route params. Normalize framework or route params at the boundary.
-- Keep fallback and invariant checks at the lowest component that already handles that state; avoid defensive fallbacks that mask impossible states.
-- Do not extract fallback helpers whose only behavior is hiding missing display data. The component that renders the surface owns the empty, disabled, hidden, or placeholder state.
+- Preserve domain value types for selection components. Do not widen enum, union, boolean, numeric, object, or nullable select/radio values to `string`; keep wrappers and option value carriers typed from their feature option collection.
+- Avoid `common.tsx` buckets for shared UI. Use a feature-local `components/` folder with concrete filenames that describe the shared role.
+- Do not create type aliases that only rename another type. Use aliases only for real UI concepts, refinements, or reusable local contracts.
+- Name values by their domain role and backend API contract, especially persistent IDs and route params. Normalize framework or route params at the boundary.
+- Put fallback and invariant checks in the lowest component that already handles that state. Do not extract helpers whose only behavior is hiding missing display data.
 
-## Generated API Contracts
+## Generated API And Nullable Data
 
 - Treat generated contracts as authoritative at API, query, mutation, cache, and service boundaries. For enterprise APIs, use `packages/contracts/generated/enterprise/*`.
-- Do not hand-write local request/response/reply/page/cache-data types that mirror generated DTOs. Import or infer the generated type.
-- Do not widen generated fields or enums for compatibility. Normalize legacy input at the boundary, then return the generated field type.
-- Do not repair generated or API-returned contract fields in components unless the API contract or product requirement says they need normalization. Treat enums, statuses, and presence flags as exact contract values.
-- Use generated enum objects and union types directly in props, comparisons, status logic, and i18n keys. Do not add local enum constants or parallel frontend enum/status layers unless they model real product state not represented by the API. Presentation-only tone maps should be keyed by the generated enum.
-- Normalize or coerce only at a real boundary, such as user-entered forms, search, URL/query params, file names, DOM IDs, or legacy adapters. Preserve user-entered values when whitespace or formatting can be meaningful.
-- Do not coerce nullable or optional API strings to `''` in query, derived model, or payload-building code. Keep `undefined` or `null` until the final boundary that requires a string.
-- Local UI models are fine for presentation, form state, select options, or guarded required-field refinements. Name them as UI concepts, not generated DTO mirrors.
-- Required-value refinements are allowed only after same-branch filtering or early return. Prefer nullable-tolerant props for render-only data.
-- When a component needs a stricter shape than a generated DTO, refine once at the API/query-to-UI boundary into a purpose-named UI type instead of hiding missing fields with generic fallback or coercion helpers.
-
-## Nullable API Data
-
-- Prefer nullable-tolerant call boundaries. Pass API-returned types through for render-only rows, and let the component render fallback, disabled state, or nothing.
-- Narrow only where a real value is required, such as mutation params, route hrefs, select values, or query input. Build that target model with `flatMap`, a local loop, or an early return so the required value is captured in the same branch.
-- If design says a field is the display value, use that field. Only the final component should decide whether a nullable display value renders a placeholder, hides content, or disables an action.
-- Do not wrap required arrays or fields in null-fallback helpers. Use empty collection fallbacks only for not-yet-loaded query data or genuinely nullable collections at the owning render boundary.
-- Do not drop rows only to satisfy props or React keys; use a stable fallback key when possible.
-- Use conditional spreads or explicit pushes for conditional array items instead of `undefined` placeholders followed by a narrowing filter.
-- Avoid truthiness type guards, `filter(Boolean)`, `filter(item => item.id)`, and `!` after those filters.
-- Use type guards only for meaningful domain or runtime validation, such as enum membership, object shape, or a reusable business invariant.
+- Do not hand-write DTO mirrors, widen generated fields/enums, or add parallel frontend enum/status layers unless they model product state not represented by the API.
+- Use generated enum objects and union types directly in props, comparisons, status logic, and i18n keys. Presentation-only tone maps should be keyed by generated enums.
+- Normalize or coerce only at real boundaries: user-entered forms, search, URL/query params, file names, DOM IDs, or legacy adapters.
+- Do not coerce nullable or optional API strings to `''` in query, derived model, or payload-building code. Keep `null` or `undefined` until the final boundary requiring a string.
+- Do not use `value || undefined` for mutation fields where `''` means "clear this value". Trim or normalize at the form boundary, then preserve intentional empty strings.
+- Prefer nullable-tolerant render props for API-returned rows. Narrow only where a real value is required, such as mutation params, route hrefs, select values, query input, or required React keys.
+- Build required values in the same branch that proves them, using `flatMap`, a local loop, or an early return. Avoid truthiness guards, `filter(Boolean)`, `filter(item => item.id)`, and `!` after filters.
+- Use conditional spreads or explicit pushes for conditional array items instead of `undefined` placeholders followed by narrowing filters.
+- Empty collection fallbacks are for not-yet-loaded query data or genuinely nullable collections at the owning render boundary, not for hiding required API fields.
 
 ## Queries And Mutations
 
-- Keep `web/contract/*` as the single source of truth for API shape; follow existing domain/router patterns and the `{ params, query?, body? }` input shape.
-- Consume queries directly with `useQuery(consoleQuery.xxx.queryOptions(...))` or `useQuery(marketplaceQuery.xxx.queryOptions(...))`.
-- Avoid pass-through hooks and thin `web/service/use-*` wrappers that only rename `queryOptions()` or `mutationOptions()`. Extract a small `queryOptions` helper only when repeated call-site options justify it.
-- Keep feature hooks for real orchestration, workflow state, or shared domain behavior.
-- For TanStack cache data, use generated or query-derived types; do not create local wrappers for `getQueryData` or `getQueriesData`.
-- For generated oRPC `queryOptions()` / `infiniteOptions()`, do not pass `skipToken` as `input`; keep a valid placeholder input shape and use `enabled` to gate missing required params because the OpenAPI codec encodes input eagerly.
-- Consume mutations directly with `useMutation(consoleQuery.xxx.mutationOptions(...))` or `useMutation(marketplaceQuery.xxx.mutationOptions(...))`; use oRPC clients as `mutationFn` only for custom flows.
-- Put shared cache behavior in `createTanstackQueryUtils(...experimental_defaults...)`; components may add UI feedback callbacks, but should not own shared invalidation rules.
+- Keep `web/contract/*` as the API shape source of truth and follow the `{ params, query?, body? }` input shape.
+- Consume generated queries with `useQuery(consoleQuery.xxx.queryOptions(...))` or `useQuery(marketplaceQuery.xxx.queryOptions(...))`.
+- If a generated query input comes from an atom, including a route-identity bridge atom, keep the query in `atomWithQuery`; do not unwrap the atom in a component just to call `useQuery`.
+- Consume owner-local mutations with `useMutation(consoleQuery.xxx.mutationOptions(...))` or `useMutation(marketplaceQuery.xxx.mutationOptions(...))` when pending/error state is not consumed by feature atoms.
+- In `atomWithQuery`, `atomWithInfiniteQuery`, and `atomWithMutation`, return generated `queryOptions()`, `infiniteOptions()`, or `mutationOptions()` directly. Pass `enabled`, `retry`, `placeholderData`, `select`, and pagination options into the generated call instead of spreading options into a hand-built object.
+- For generated oRPC options with missing required input, branch the whole input with `input: condition ? validInput : skipToken` and `enabled: Boolean(condition)`. Never place `skipToken` inside a nested placeholder payload or coerce required IDs to `''`.
+- When prefetch and render use the same request, extract local query options or a query-options atom so `prefetchQuery` and `useQuery`/`atomWithQuery` share the exact options.
+- For custom query or mutation functions, wrap options with TanStack `queryOptions(...)` or `mutationOptions(...)`.
+- Do not extract generated `queryOptions(...)` into a helper solely to share input construction; extract only when prefetch/render must share exact options or the helper owns real domain behavior.
+- Avoid pass-through hooks and thin `web/service/use-*` wrappers that only rename generated options. Keep feature hooks for real orchestration, workflow state, or shared domain behavior.
+- Put shared cache behavior in `createTanstackQueryUtils(...experimental_defaults...)`. Component or atom callbacks may handle local toasts, closing dialogs, and navigation, but should not replace shared invalidation or patch shared server state locally.
+- For overlays that may open heavier secondary content, prefetch from the trigger/menu open event with `queryClient.prefetchQuery(queryOptions)` when `onOpenChange` is available. Do not mount hidden subscribers just to warm cache.
 - Do not use deprecated `useInvalid` or `useReset`.
 - Prefer `mutate(...)`; use `mutateAsync(...)` only when Promise semantics are required, and wrap awaited calls in `try/catch`.
 
-## Component Boundaries
+## Boundaries And Overlays
 
-- Use the first level below a page or tab to organize independent page sections when it adds real structure. This layer is layout/semantic first, not automatically the data owner.
-- Treat component names, semantic roles, and user- or design-marked visual regions as boundary constraints. Do not expand a child component's responsibility just because its data is useful nearby; keep adjacent UI as a sibling owner or introduce a correctly named broader owner.
-- Split deeper components by the data and state each layer actually needs. Each component should access only necessary data, and ownership should stay at the lowest consumer.
+- Use the first level below a page or tab to organize independent page sections when it adds structure or the root folder becomes noisy. This layer is layout/semantic first, not automatically the data owner.
+- Treat component names, semantic roles, and user- or design-marked visual regions as boundary constraints. Keep adjacent UI as a sibling owner or introduce a correctly named broader owner.
 - Keep cohesive forms, menu bodies, and one-off helpers local unless they need their own state, reuse, or semantic boundary.
-- Separate hidden secondary surfaces from the trigger's main flow. For dialogs, dropdowns, popovers, and similar branches, extract a small local component that owns the trigger, open state, and hidden content when it would obscure the parent flow.
-- Preserve composability by separating behavior ownership from layout ownership. A dropdown action may own its trigger, open state, and menu content; the caller owns placement such as slots, offsets, and alignment.
-- Avoid unnecessary DOM hierarchy. Do not add wrapper elements unless they provide layout, semantics, accessibility, state ownership, or integration with a library API; prefer fragments or styling an existing element when possible.
-- Avoid shallow wrappers, hook-to-props adapter components, layout-only render-prop wrappers, and prop renaming unless the wrapper adds validation, orchestration, error handling, state ownership, or a real semantic boundary. If a component only calls a hook and forwards every returned field to one child, move the hook into that child or make the wrapper own a real surface.
+- Separate hidden secondary surfaces from the trigger's main flow. For dialogs, dropdowns, popovers, and similar branches, extract a small local component when hidden content would obscure the parent.
+- Preserve composability by separating behavior ownership from placement ownership: an action can own trigger/open/menu content while the caller owns slots, offsets, and alignment.
+- When a dialog, dropdown, or popover accepts controlled `open`, mount it unconditionally unless unmounting is required for performance or reset semantics. Use keyed scope or local state reset instead of `{open && <Surface />}` wrappers.
+- When opening a dialog from a menu item, keep the menu and dialog as sibling surfaces. Let the menu command open the dialog, and mount the dialog outside menu popup content.
+- For dialogs and alert dialogs, keep the root responsible for `open` wiring and put query/mutation hooks inside the content component when work should mount only after the overlay opens.
+- Prefer uncontrolled overlay roots when the library can own open state. Use `onOpenChange` for side effects and CSS/data selectors for open-state styling.
+- Avoid wrapper DOM unless it provides layout, semantics, accessibility, state ownership, or library integration. Avoid shallow wrappers, hook-to-props adapters, layout-only render props, children pass-through wrappers, and prop renaming unless they add real behavior or a real boundary.
 
-## You Might Not Need An Effect
-
-- Use Effects only to synchronize with external systems such as browser APIs, non-React widgets, subscriptions, timers, analytics that must run because the component was shown, or imperative DOM integration.
-- Do not use Effects to transform props or state for rendering. Calculate derived values during render, and use `useMemo` only when the calculation is actually expensive.
-- Do not use Effects to handle user actions. Put action-specific logic in the event handler where the cause is known.
-- Do not use Effects to copy one state value into another state value representing the same concept. Pick one source of truth and derive the rest during render.
-- Do not reset or adjust state from props with an Effect. Prefer a `key` reset, storing a stable ID and deriving the selected object, or guarded same-component render-time adjustment when truly necessary.
-- Prefer framework data APIs or TanStack Query for data fetching instead of writing request Effects in components.
-- If an Effect still seems necessary, first name the external system it synchronizes with. If there is no external system, remove the Effect and restructure the state or event flow.
-
-## Navigation And Performance
+## Effects, Navigation, And Performance
 
+- Use Effects only to synchronize with external systems. Do not use Effects to transform props/state for rendering, handle user actions, copy state, reset state from props, or fetch data.
+- For forms initialized from query data, prefer keyed remounts or surface-entry atom hydration over Effects that copy query data into form state.
+- Prefer framework data APIs or TanStack Query for data fetching.
 - Prefer `Link` for normal navigation. Use router APIs only for command-flow side effects such as mutation success, guarded redirects, or form submission.
-- Before reaching for `memo`, first try moving changing state down to the smallest component that actually uses it so unrelated sibling trees stay untouched.
-- If changing state must wrap other content, lift the unchanged content up and pass it as `children` so the stateful wrapper can update without React visiting that subtree.
+- Before using `memo`, move changing state down to the smallest component that uses it. If state must wrap stable content, lift the stable content up and pass it as `children`.
 - Avoid `memo`, `useMemo`, and `useCallback` unless there is a clear performance reason.

.github/CODEOWNERS

@@ -26,6 +26,9 @@
 /cli/ @GareArc
 /.github/workflows/cli-tests.yml @GareArc
 
+# E2E
+/e2e/ @lyzno1
+
 # Backend (default owner, more specific rules below will override)
 /api/ @QuantumGhost
 

.github/labeler.yml

@@ -7,3 +7,9 @@ web:
           - 'pnpm-lock.yaml'
           - 'pnpm-workspace.yaml'
           - '.nvmrc'
+
+e2e:
+  - changed-files:
+      - any-glob-to-any-file:
+          - 'e2e/**'
+          - '.github/workflows/web-e2e.yml'

.github/scripts/check-hotfix-cherry-picks.sh

@@ -45,7 +45,7 @@ while IFS= read -r commit_sha; do
   )
 
   if [[ -z "$source_sha" ]]; then
-    error "Commit $commit_sha ($subject) is missing cherry-pick provenance. $REMEDIATION_HINT"
+    error "Commit $commit_sha ($subject) is missing cherry-pick provenance. $REMEDIATION_HINT If version differences prevent using git cherry-pick -x, manually add '(cherry picked from commit <sha>)' to the commit message."
     failed=1
     continue
   fi

.github/workflows/api-tests.yml

@@ -29,13 +29,13 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           fetch-depth: 0
           persist-credentials: false
 
       - name: Setup UV and Python
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: true
           python-version: ${{ matrix.python-version }}
@@ -91,13 +91,13 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           fetch-depth: 0
           persist-credentials: false
 
       - name: Setup UV and Python
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: true
           python-version: ${{ matrix.python-version }}
@@ -142,13 +142,13 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           fetch-depth: 0
           persist-credentials: false
 
       - name: Setup UV and Python
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: true
           python-version: "3.12"

.github/workflows/autofix.yml

@@ -20,7 +20,7 @@ jobs:
         run: echo "autofix.ci updates pull request branches, not merge group refs."
 
       - if: github.event_name != 'merge_group'
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
 
       - name: Check Docker Compose inputs
         if: github.event_name != 'merge_group'
@@ -51,6 +51,18 @@ jobs:
         with:
           files: |
             api/**
+      - name: Check frontend contract inputs
+        if: github.event_name != 'merge_group'
+        id: frontend-contract-changes
+        uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6
+        with:
+          files: |
+            api/**
+            packages/contracts/openapi-ts.api.config.ts
+            packages/contracts/package.json
+            packages/contracts/openapi/**
+            pnpm-lock.yaml
+            pnpm-workspace.yaml
       - name: Check dify-agent inputs
         if: github.event_name != 'merge_group'
         id: dify-agent-changes
@@ -61,12 +73,12 @@ jobs:
             dify-agent/pyproject.toml
             dify-agent/uv.lock
       - if: github.event_name != 'merge_group'
-        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
         with:
           python-version: "3.11"
 
       - if: github.event_name != 'merge_group'
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
 
       - name: Generate Docker Compose
         if: github.event_name != 'merge_group' && steps.docker-compose-changes.outputs.any_changed == 'true'
@@ -143,8 +155,8 @@ jobs:
           uv run dev/generate_swagger_markdown_docs.py --swagger-dir ../packages/contracts/openapi --markdown-dir openapi/markdown --keep-swagger-json
 
       - name: Generate frontend contracts
-        if: github.event_name != 'merge_group' && steps.api-changes.outputs.any_changed == 'true'
-        run: pnpm --dir packages/contracts gen-api-contract-from-openapi
+        if: github.event_name != 'merge_group' && steps.frontend-contract-changes.outputs.any_changed == 'true'
+        run: pnpm --dir packages/contracts gen-api-contract
 
       - name: ESLint autofix
         if: github.event_name != 'merge_group' && steps.web-changes.outputs.any_changed == 'true'

.github/workflows/build-push.yml

@@ -8,8 +8,6 @@ on:
       - "build/**"
       - "release/e-*"
       - "hotfix/**"
-      - "feat/hitl-backend"
-      - "feat/rbac"
     tags:
       - "*"
 
@@ -23,6 +21,7 @@ env:
   DIFY_WEB_IMAGE_NAME: ${{ vars.DIFY_WEB_IMAGE_NAME || 'langgenius/dify-web' }}
   DIFY_API_IMAGE_NAME: ${{ vars.DIFY_API_IMAGE_NAME || 'langgenius/dify-api' }}
   DIFY_AGENT_IMAGE_NAME: ${{ vars.DIFY_AGENT_IMAGE_NAME || 'langgenius/dify-agent-backend' }}
+  DIFY_AGENT_LOCAL_SANDBOX_IMAGE_NAME: ${{ vars.DIFY_AGENT_LOCAL_SANDBOX_IMAGE_NAME || 'langgenius/dify-agent-local-sandbox' }}
 
 jobs:
   build:
@@ -76,6 +75,20 @@ jobs:
             file: "dify-agent/Dockerfile"
             platform: linux/arm64
             runs_on: depot-ubuntu-24.04-4
+          - service_name: "build-agent-local-sandbox-amd64"
+            image_name_env: "DIFY_AGENT_LOCAL_SANDBOX_IMAGE_NAME"
+            artifact_context: "local-sandbox"
+            build_context: "{{defaultContext}}:dify-agent"
+            file: "docker/local-sandbox/Dockerfile"
+            platform: linux/amd64
+            runs_on: depot-ubuntu-24.04-4
+          - service_name: "build-agent-local-sandbox-arm64"
+            image_name_env: "DIFY_AGENT_LOCAL_SANDBOX_IMAGE_NAME"
+            artifact_context: "local-sandbox"
+            build_context: "{{defaultContext}}:dify-agent"
+            file: "docker/local-sandbox/Dockerfile"
+            platform: linux/arm64
+            runs_on: depot-ubuntu-24.04-4
 
     steps:
       - name: Prepare
@@ -84,7 +97,7 @@ jobs:
           echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
 
       - name: Login to Docker Hub
-        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
+        uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
         with:
           username: ${{ env.DOCKERHUB_USER }}
           password: ${{ env.DOCKERHUB_TOKEN }}
@@ -94,7 +107,7 @@ jobs:
 
       - name: Extract metadata for Docker
         id: meta
-        uses: docker/metadata-action@80c7e94dd9b9319bd5eb7a0e0fe9291e23a2a2e9 # v6.1.0
+        uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
         with:
           images: ${{ env[matrix.image_name_env] }}
 
@@ -141,12 +154,15 @@ jobs:
           - service_name: "validate-agent-amd64"
             build_context: "{{defaultContext}}"
             file: "dify-agent/Dockerfile"
+          - service_name: "validate-agent-local-sandbox-amd64"
+            build_context: "{{defaultContext}}:dify-agent"
+            file: "docker/local-sandbox/Dockerfile"
     steps:
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
+        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
 
       - name: Validate Docker image
-        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
         with:
           push: false
           context: ${{ matrix.build_context }}
@@ -169,6 +185,9 @@ jobs:
           - service_name: "merge-agent-images"
             image_name_env: "DIFY_AGENT_IMAGE_NAME"
             context: "agent"
+          - service_name: "merge-agent-local-sandbox-images"
+            image_name_env: "DIFY_AGENT_LOCAL_SANDBOX_IMAGE_NAME"
+            context: "local-sandbox"
     steps:
       - name: Download digests
         uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
@@ -178,14 +197,14 @@ jobs:
           merge-multiple: true
 
       - name: Login to Docker Hub
-        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
+        uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
         with:
           username: ${{ env.DOCKERHUB_USER }}
           password: ${{ env.DOCKERHUB_TOKEN }}
 
       - name: Extract metadata for Docker
         id: meta
-        uses: docker/metadata-action@80c7e94dd9b9319bd5eb7a0e0fe9291e23a2a2e9 # v6.1.0
+        uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
         with:
           images: ${{ env[matrix.image_name_env] }}
           tags: |

.github/workflows/cli-e2e.yml

@@ -79,7 +79,7 @@ jobs:
       ws2_app_id:                 ${{ steps.out.outputs.DIFY_E2E_WS2_APP_ID }}
 
     steps:
-      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v4
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v4
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false
@@ -123,7 +123,7 @@ jobs:
         shell: bash
 
     steps:
-      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v4
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v4
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false
@@ -170,7 +170,7 @@ jobs:
         shell: bash
 
     steps:
-      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v4
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v4
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false
@@ -233,7 +233,7 @@ jobs:
         shell: bash
 
     steps:
-      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v4
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v4
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false
@@ -295,7 +295,7 @@ jobs:
         shell: bash
 
     steps:
-      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v4
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v4
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false
@@ -351,7 +351,7 @@ jobs:
         shell: bash
 
     steps:
-      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v4
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v4
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false

.github/workflows/cli-edge.yml

@@ -23,7 +23,7 @@ jobs:
         working-directory: ./cli
     steps:
       - name: Checkout
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
           fetch-depth: 0

.github/workflows/cli-release.yml

@@ -35,7 +35,7 @@ jobs:
       dify_tag: ${{ steps.resolve.outputs.dify_tag }}
     steps:
       - name: Checkout
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 
@@ -98,7 +98,7 @@ jobs:
       DIFY_TAG: ${{ needs.validate.outputs.dify_tag }}
     steps:
       - name: Checkout
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
           fetch-depth: 1

.github/workflows/cli-smoke.yml

@@ -24,7 +24,7 @@ jobs:
         shell: bash
     steps:
       - name: Checkout cli ref
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           ref: ${{ inputs.cli_ref || github.ref }}
           persist-credentials: false

.github/workflows/cli-tests.yml

@@ -30,7 +30,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 

.github/workflows/db-migration-test.yml

@@ -13,13 +13,13 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           fetch-depth: 0
           persist-credentials: false
 
       - name: Setup UV and Python
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: true
           python-version: "3.12"
@@ -63,13 +63,13 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           fetch-depth: 0
           persist-credentials: false
 
       - name: Setup UV and Python
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: true
           python-version: "3.12"

.github/workflows/deploy-hitl.yml

@@ -1,25 +0,0 @@
-name: Deploy HITL
-
-on:
-  workflow_run:
-    workflows: ["Build and Push API & Web"]
-    branches:
-      - "build/feat/hitl"
-    types:
-      - completed
-
-jobs:
-  deploy:
-    runs-on: depot-ubuntu-24.04
-    if: |
-      github.event.workflow_run.conclusion == 'success' &&
-      github.event.workflow_run.head_branch == 'build/feat/hitl'
-    steps:
-      - name: Deploy to server
-        uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2 # v1.2.5
-        with:
-          host: ${{ secrets.HITL_SSH_HOST }}
-          username: ${{ secrets.SSH_USER }}
-          key: ${{ secrets.SSH_PRIVATE_KEY }}
-          script: |
-            ${{ vars.SSH_SCRIPT || secrets.SSH_SCRIPT }}

.github/workflows/docker-build.yml

@@ -77,10 +77,10 @@ jobs:
             file: "web/Dockerfile"
     steps:
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
+        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
 
       - name: Build Docker Image
-        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
         with:
           push: false
           context: ${{ matrix.context }}

.github/workflows/hotfix-cherry-pick.yml

@@ -24,7 +24,7 @@ jobs:
     name: Require cherry-pick provenance
     runs-on: depot-ubuntu-24.04
     steps:
-      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           fetch-depth: 0
 

.github/workflows/main-ci.yml

@@ -6,8 +6,6 @@ on:
   merge_group:
     branches: ["main"]
     types: [checks_requested]
-  push:
-    branches: ["main"]
 
 permissions:
   actions: write
@@ -48,8 +46,8 @@ jobs:
       vdb-changed: ${{ steps.changes.outputs.vdb }}
       migration-changed: ${{ steps.changes.outputs.migration }}
     steps:
-      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
-      - uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      - uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v4.0.2
         id: changes
         with:
           filters: |
@@ -324,6 +322,7 @@ jobs:
       - check-changes
     if: needs.pre_job.outputs.should_skip != 'true' && needs.check-changes.outputs.e2e-changed == 'true'
     uses: ./.github/workflows/web-e2e.yml
+    secrets: inherit
 
   web-e2e-skip:
     name: Skip Web Full-Stack E2E

.github/workflows/post-merge.yml

@@ -0,0 +1,54 @@
+name: Post-Merge Checks
+
+on:
+  push:
+    branches: ["main"]
+
+permissions:
+  contents: read
+
+concurrency:
+  group: post-merge-${{ github.sha }}
+  cancel-in-progress: false
+
+jobs:
+  check-changes:
+    name: Check Changed Files
+    runs-on: depot-ubuntu-24.04
+    outputs:
+      external-e2e-changed: ${{ steps.changes.outputs.external_e2e }}
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      - uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v4.0.2
+        id: changes
+        with:
+          filters: |
+            external_e2e:
+              - 'e2e/features/agent-v2/**'
+              - 'e2e/scripts/**'
+              - 'e2e/support/**'
+              - '.github/workflows/post-merge.yml'
+              - '.github/workflows/web-e2e.yml'
+              - '.github/actions/setup-web/**'
+              - 'dify-agent/**'
+              - 'api/clients/agent_backend/**'
+              - 'api/core/app/apps/agent_app/**'
+              - 'api/core/workflow/nodes/agent_v2/**'
+              - 'api/controllers/console/agent/**'
+              - 'api/services/agent/**'
+              - 'api/core/plugin/**'
+              - 'api/services/plugin/**'
+              - 'api/core/tools/**'
+              - 'api/services/tools/**'
+              - 'web/features/agent-v2/**'
+              - 'web/app/(commonLayout)/roster/**'
+              - 'web/app/components/workflow/nodes/agent-v2/**'
+
+  external-e2e:
+    name: External Runtime E2E
+    needs: check-changes
+    if: needs.check-changes.outputs.external-e2e-changed == 'true'
+    uses: ./.github/workflows/web-e2e.yml
+    with:
+      run-external-runtime: true
+    secrets: inherit

.github/workflows/pyrefly-diff.yml

@@ -17,12 +17,12 @@ jobs:
       pull-requests: write
     steps:
       - name: Checkout PR branch
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           fetch-depth: 0
 
       - name: Setup Python & UV
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: true
 

.github/workflows/pyrefly-type-coverage-comment.yml

@@ -21,10 +21,10 @@ jobs:
     if: ${{ github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.pull_requests[0].head.repo.full_name != github.repository }}
     steps:
       - name: Checkout default branch (trusted code)
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
 
       - name: Setup Python & UV
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: true
 

.github/workflows/pyrefly-type-coverage.yml

@@ -17,12 +17,12 @@ jobs:
       pull-requests: write
     steps:
       - name: Checkout PR branch
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           fetch-depth: 0
 
       - name: Setup Python & UV
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: true
 

.github/workflows/style.yml

@@ -19,9 +19,10 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
+          fetch-depth: 0
 
       - name: Check changed files
         id: changed-files
@@ -29,11 +30,13 @@ jobs:
         with:
           files: |
             api/**
+            scripts/check_no_new_getattr.py
+            scripts/ast_grep_rules/no_new_getattr.yml
             .github/workflows/style.yml
 
       - name: Setup UV and Python
         if: steps.changed-files.outputs.any_changed == 'true'
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: false
           python-version: "3.12"
@@ -51,8 +54,22 @@ jobs:
         if: steps.changed-files.outputs.any_changed == 'true'
         run: uv run --project api --dev python api/dev/lint_response_contracts.py --fail-on-mismatch
 
+      - name: Fetch merge target ref for getattr guard
+        if: steps.changed-files.outputs.any_changed == 'true'
+        run: git fetch --no-tags --depth=1 origin +refs/heads/main:refs/remotes/origin/main
+
+      - name: Bind merge target branch for getattr guard
+        if: steps.changed-files.outputs.any_changed == 'true'
+        run: git show-ref --verify --quiet refs/heads/main || git branch main origin/main
+
+      - name: Run No New Getattr Guard
+        if: steps.changed-files.outputs.any_changed == 'true'
+        run: uv run --project api python scripts/check_no_new_getattr.py --mode ci --merge-target main
+
       - name: Run Type Checks
         if: steps.changed-files.outputs.any_changed == 'true'
+        env:
+          PYREFLY_OUTPUT_FORMAT: github
         run: make type-check-core
 
       - name: Dotenv check
@@ -71,7 +88,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 
@@ -105,6 +122,14 @@ jobs:
         if: steps.changed-files.outputs.any_changed == 'true'
         run: vp run knip
 
+      - name: Web dead code check production
+        if: steps.changed-files.outputs.any_changed == 'true'
+        run: vp run knip:production
+
+      - name: Web production unused declarations check
+        if: steps.changed-files.outputs.any_changed == 'true'
+        run: vp run knip:production-unused-check
+
   ts-common-style:
     name: TS Common
     runs-on: depot-ubuntu-24.04
@@ -114,7 +139,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 
@@ -143,7 +168,7 @@ jobs:
       - name: Restore ESLint cache
         if: steps.changed-files.outputs.any_changed == 'true'
         id: eslint-cache-restore
-        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: .eslintcache
           key: ${{ runner.os }}-eslint-${{ hashFiles('pnpm-lock.yaml', 'eslint.config.mjs', 'web/eslint.config.mjs', 'web/eslint.constants.mjs', 'web/plugins/eslint/**') }}-${{ github.sha }}
@@ -160,7 +185,7 @@ jobs:
 
       - name: Save ESLint cache
         if: steps.changed-files.outputs.any_changed == 'true' && success() && steps.eslint-cache-restore.outputs.cache-hit != 'true'
-        uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: .eslintcache
           key: ${{ steps.eslint-cache-restore.outputs.cache-primary-key }}
@@ -171,7 +196,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           fetch-depth: 0
           persist-credentials: false
@@ -189,7 +214,7 @@ jobs:
             .editorconfig
 
       - name: Super-linter
-        uses: super-linter/super-linter/slim@9e863354e3ff62e0727d37183162c4a88873df41 # v8.6.0
+        uses: super-linter/super-linter/slim@4ce20838b8ab83717e78138c5b3a1407148e0918 # v8.7.0
         if: steps.changed-files.outputs.any_changed == 'true'
         env:
           BASH_SEVERITY: warning

.github/workflows/tool-test-sdks.yaml

@@ -24,7 +24,7 @@ jobs:
         working-directory: sdks/nodejs-client
 
     steps:
-      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 

.github/workflows/translate-i18n-claude.yml

@@ -40,7 +40,7 @@ jobs:
 
     steps:
       - name: Checkout repository
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           fetch-depth: 0
           token: ${{ secrets.GITHUB_TOKEN }}
@@ -158,7 +158,7 @@ jobs:
 
       - name: Run Claude Code for Translation Sync
         if: steps.context.outputs.CHANGED_FILES != ''
-        uses: anthropics/claude-code-action@806af32823ef69c8ef357086c573a902af641307 # v1.0.151
+        uses: anthropics/claude-code-action@558b1d6cab4085c7753fe402c10bef0fbb92ac7a # v1.0.165
         with:
           anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
           github_token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/trigger-i18n-sync.yml

@@ -21,7 +21,7 @@ jobs:
 
     steps:
       - name: Checkout repository
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           fetch-depth: 0
 

.github/workflows/vdb-tests-full.yml

@@ -24,7 +24,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 
@@ -36,7 +36,7 @@ jobs:
           remove_tool_cache: true
 
       - name: Setup UV and Python
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: true
           python-version: ${{ matrix.python-version }}

.github/workflows/vdb-tests.yml

@@ -21,7 +21,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 
@@ -33,7 +33,7 @@ jobs:
           remove_tool_cache: true
 
       - name: Setup UV and Python
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: true
           python-version: ${{ matrix.python-version }}

.github/workflows/web-e2e.yml

@@ -2,6 +2,11 @@ name: Web Full-Stack E2E
 
 on:
   workflow_call:
+    inputs:
+      run-external-runtime:
+        required: false
+        type: boolean
+        default: false
 
 permissions:
   contents: read
@@ -20,7 +25,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 
@@ -28,11 +33,13 @@ jobs:
         uses: ./.github/actions/setup-web
 
       - name: Setup UV and Python
-        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
         with:
           enable-cache: true
           python-version: "3.12"
-          cache-dependency-glob: api/uv.lock
+          cache-dependency-glob: |
+            api/uv.lock
+            dify-agent/uv.lock
 
       - name: Install API dependencies
         run: uv sync --project api --dev
@@ -51,12 +58,52 @@ jobs:
           E2E_INIT_PASSWORD: E2eInit12345
         run: vp run e2e:full
 
+      - name: Run external runtime E2E tests
+        if: ${{ inputs.run-external-runtime }}
+        working-directory: ./e2e
+        env:
+          E2E_ADMIN_EMAIL: e2e-admin@example.com
+          E2E_ADMIN_NAME: E2E Admin
+          E2E_ADMIN_PASSWORD: E2eAdmin12345
+          E2E_AGENT_DECISION_MODEL_NAME: ${{ vars.E2E_AGENT_DECISION_MODEL_NAME || 'gpt-5.5' }}
+          E2E_AGENT_DECISION_MODEL_PROVIDER: ${{ vars.E2E_AGENT_DECISION_MODEL_PROVIDER || 'openai' }}
+          E2E_AGENT_DECISION_MODEL_TYPE: ${{ vars.E2E_AGENT_DECISION_MODEL_TYPE || 'llm' }}
+          E2E_EXTERNAL_RUNTIME_SEED_SPECS: ${{ vars.E2E_EXTERNAL_RUNTIME_SEED_SPECS }}
+          E2E_EXTERNAL_RUNTIME_TAGS: ${{ vars.E2E_EXTERNAL_RUNTIME_TAGS }}
+          E2E_FORCE_WEB_BUILD: "1"
+          E2E_INIT_PASSWORD: E2eInit12345
+          E2E_MARKETPLACE_API_URL: ${{ vars.E2E_MARKETPLACE_API_URL }}
+          E2E_MARKETPLACE_PLUGIN_IDS: ${{ vars.E2E_MARKETPLACE_PLUGIN_IDS }}
+          E2E_MARKETPLACE_PLUGIN_UNIQUE_IDENTIFIERS: ${{ vars.E2E_MARKETPLACE_PLUGIN_UNIQUE_IDENTIFIERS }}
+          E2E_MODEL_PROVIDER_CREDENTIALS_JSON: ${{ secrets.E2E_MODEL_PROVIDER_CREDENTIALS_JSON }}
+          E2E_START_AGENT_BACKEND: ${{ vars.E2E_START_AGENT_BACKEND || '1' }}
+          E2E_STABLE_MODEL_NAME: ${{ vars.E2E_STABLE_MODEL_NAME || 'gpt-5-nano' }}
+          E2E_STABLE_MODEL_PROVIDER: ${{ vars.E2E_STABLE_MODEL_PROVIDER || 'openai' }}
+          E2E_STABLE_MODEL_TYPE: ${{ vars.E2E_STABLE_MODEL_TYPE || 'llm' }}
+        run: |
+          if [[ -z "${E2E_MODEL_PROVIDER_CREDENTIALS_JSON}" ]]; then
+            echo "E2E_MODEL_PROVIDER_CREDENTIALS_JSON is required for external runtime E2E." >&2
+            exit 1
+          fi
+
+          if [[ -d cucumber-report ]]; then
+            rm -rf cucumber-report-non-external
+            mv cucumber-report cucumber-report-non-external
+          fi
+
+          trap 'vp run e2e:middleware:down' EXIT
+          vp run e2e:middleware:up
+          vp run e2e:external:prepare
+          vp run e2e:external
+
       - name: Upload Cucumber report
         if: ${{ !cancelled() }}
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: cucumber-report
-          path: e2e/cucumber-report
+          path: |
+            e2e/cucumber-report
+            e2e/cucumber-report-non-external
           retention-days: 7
 
       - name: Upload E2E logs

.github/workflows/web-tests.yml

@@ -31,7 +31,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 
@@ -64,7 +64,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 
@@ -102,7 +102,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 
@@ -134,7 +134,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
 

.superpowers/sdd/hitl-timeout-semantics-impl-report.md

@@ -0,0 +1,30 @@
+# HITL timeout semantics implementation report
+
+## What changed
+
+- Updated `api/core/workflow/nodes/human_input/callback.py` so `DifyHITLCallback` now preserves Dify's timeout split at the boundary:
+  - `HumanInputFormStatus.TIMEOUT` returns the graphon timeout branch via `Expired(selected_handle="__timeout__", ...)`.
+  - `HumanInputFormStatus.EXPIRED` is treated as an invalid resume state and raises `AssertionError`.
+  - `HumanInputFormStatus.WAITING` with a past global deadline is treated as an invalid resume state and raises `AssertionError`.
+  - `HumanInputFormStatus.WAITING` with only the node-level deadline expired still returns the timeout branch.
+- Added `created_at` to `HumanInputFormEntity` and `_HumanInputFormEntityImpl` so the callback can compute the global deadline using Dify's shared `HUMAN_INPUT_GLOBAL_TIMEOUT_SECONDS` invariant.
+- Kept the submitted and pause flows unchanged.
+- Added focused unit coverage in `api/tests/unit_tests/core/workflow/test_human_input_callback.py` for:
+  - node timeout branch
+  - global expiration rejection
+  - waiting-form past node deadline timeout
+  - waiting-form past global deadline rejection
+
+## Verification
+
+- `uv run --project api pytest -o addopts='' api/tests/unit_tests/core/workflow/test_human_input_callback.py api/tests/unit_tests/core/workflow/nodes/human_input/test_human_input_form_filled_event.py -q`
+- `git diff --check`
+
+## Result
+
+- The focused test set is expected to pass with the new `created_at` boundary in place.
+- No unrelated files were modified.
+
+## Concerns
+
+- The callback now fails fast on invalid resume states by design. That is intentional, but any caller that previously relied on `EXPIRED` being mapped to the timeout branch will now see an assertion failure instead.

README.md

@@ -116,7 +116,7 @@ All of Dify's offerings come with corresponding APIs, so you could effortlessly
 ## Using Dify
 
 - **Cloud <br/>**
-  We host a [Dify Cloud](https://dify.ai) service for anyone to try with zero setup. It provides all the capabilities of the self-deployed version, and includes 200 free GPT-4 calls in the sandbox plan.
+  We host a [Dify Cloud](https://dify.ai) service for anyone to try with zero setup. It provides all the capabilities of the self-deployed version, and includes 200 free GPT-4 calls in the sandbox plan. If you run into issues with Dify Cloud, [contact our Cloud support team](mailto:cloud@dify.ai?subject=%5BGitHub%5DDify%20Cloud%20Support).
 
 - **Self-hosting Dify Community Edition<br/>**
   Quickly get Dify running in your environment with this [starter guide](#quick-start).

api/.env.example

@@ -36,6 +36,9 @@ FILES_ACCESS_TIMEOUT=300
 # Collaboration mode toggle
 ENABLE_COLLABORATION_MODE=true
 
+# Learn app feature toggle
+ENABLE_LEARN_APP=true
+
 # Access token expiration time in minutes
 ACCESS_TOKEN_EXPIRE_MINUTES=60
 
@@ -768,7 +771,6 @@ EVENT_BUS_REDIS_CHANNEL_TYPE=pubsub
 # Whether to use Redis cluster mode while use redis as event bus.
 #  It's highly recommended to enable this for large deployments.
 EVENT_BUS_REDIS_USE_CLUSTERS=false
-EVENT_BUS_LISTENER_JOIN_TIMEOUT_MS=2000
 
 # Whether to Enable human input timeout check task
 ENABLE_HUMAN_INPUT_TIMEOUT_TASK=true

api/clients/agent_backend/__init__.py

@@ -32,6 +32,8 @@ from clients.agent_backend.factory import create_agent_backend_run_client
 from clients.agent_backend.fake_client import FakeAgentBackendRunClient, FakeAgentBackendScenario
 from clients.agent_backend.request_builder import (
     AGENT_SOUL_PROMPT_LAYER_ID,
+    DIFY_CONFIG_LAYER_ID,
+    DIFY_CORE_TOOLS_LAYER_ID,
     DIFY_EXECUTION_CONTEXT_LAYER_ID,
     DIFY_KNOWLEDGE_BASE_LAYER_ID,
     DIFY_PLUGIN_TOOLS_LAYER_ID,
@@ -47,6 +49,8 @@ from clients.agent_backend.request_builder import (
 
 __all__ = [
     "AGENT_SOUL_PROMPT_LAYER_ID",
+    "DIFY_CONFIG_LAYER_ID",
+    "DIFY_CORE_TOOLS_LAYER_ID",
     "DIFY_EXECUTION_CONTEXT_LAYER_ID",
     "DIFY_KNOWLEDGE_BASE_LAYER_ID",
     "DIFY_PLUGIN_TOOLS_LAYER_ID",

api/clients/agent_backend/event_adapter.py

@@ -67,6 +67,7 @@ class AgentBackendRunSucceededInternalEvent(AgentBackendInternalEventBase):
     type: Literal[AgentBackendInternalEventType.RUN_SUCCEEDED] = AgentBackendInternalEventType.RUN_SUCCEEDED
     output: JsonValue
     session_snapshot: CompositorSessionSnapshot
+    usage: dict[str, JsonValue] | None = None
 
 
 class AgentBackendDeferredToolCallInternalEvent(AgentBackendInternalEventBase):
@@ -76,6 +77,7 @@ class AgentBackendDeferredToolCallInternalEvent(AgentBackendInternalEventBase):
     deferred_tool_call: DeferredToolCallPayload
     message: str | None = None
     session_snapshot: CompositorSessionSnapshot
+    usage: dict[str, JsonValue] | None = None
 
 
 class AgentBackendRunFailedInternalEvent(AgentBackendInternalEventBase):
@@ -140,6 +142,7 @@ class AgentBackendRunEventAdapter:
                             deferred_tool_call=event.data.deferred_tool_call,
                             message=_deferred_tool_call_message(event.data.deferred_tool_call),
                             session_snapshot=event.data.session_snapshot,
+                            usage=_agent_run_usage(event.data.usage),
                         )
                     ]
                 return [
@@ -148,6 +151,7 @@ class AgentBackendRunEventAdapter:
                         source_event_id=event.id,
                         output=event.data.output,
                         session_snapshot=event.data.session_snapshot,
+                        usage=_agent_run_usage(event.data.usage),
                     )
                 ]
             case RunFailedEvent():
@@ -184,3 +188,13 @@ def _deferred_tool_call_message(payload: DeferredToolCallPayload) -> str:
             return title
 
     return f"Agent backend requested external input via deferred tool '{payload.tool_name}'."
+
+
+def _agent_run_usage(usage: object | None) -> dict[str, JsonValue] | None:
+    """Return JSON-safe usage metadata from optional Agent backend usage."""
+    if usage is None:
+        return None
+    dumped = _EVENT_DATA_ADAPTER.dump_python(usage, mode="json")
+    if not isinstance(dumped, dict):
+        return None
+    return cast(dict[str, JsonValue], dumped)

api/clients/agent_backend/request_builder.py

@@ -20,6 +20,8 @@ from agenton.layers import ExitIntent
 from agenton_collections.layers.plain import PLAIN_PROMPT_LAYER_TYPE_ID, PromptLayerConfig
 from agenton_collections.layers.pydantic_ai import PYDANTIC_AI_HISTORY_LAYER_TYPE_ID
 from dify_agent.layers.ask_human import DIFY_ASK_HUMAN_LAYER_TYPE_ID, DifyAskHumanLayerConfig
+from dify_agent.layers.config import DIFY_CONFIG_LAYER_TYPE_ID, DifyConfigLayerConfig
+from dify_agent.layers.dify_core_tools import DIFY_CORE_TOOLS_LAYER_TYPE_ID, DifyCoreToolsLayerConfig
 from dify_agent.layers.dify_plugin import (
     DIFY_PLUGIN_LLM_LAYER_TYPE_ID,
     DIFY_PLUGIN_TOOLS_LAYER_TYPE_ID,
@@ -54,8 +56,10 @@ WORKFLOW_NODE_JOB_PROMPT_LAYER_ID = "workflow_node_job_prompt"
 WORKFLOW_USER_PROMPT_LAYER_ID = "workflow_user_prompt"
 AGENT_APP_USER_PROMPT_LAYER_ID = "agent_app_user_prompt"
 DIFY_EXECUTION_CONTEXT_LAYER_ID = "execution_context"
+DIFY_CONFIG_LAYER_ID = "config"
 DIFY_DRIVE_LAYER_ID = "drive"
 DIFY_PLUGIN_TOOLS_LAYER_ID = "tools"
+DIFY_CORE_TOOLS_LAYER_ID = "core_tools"
 DIFY_KNOWLEDGE_BASE_LAYER_ID = "knowledge"
 DIFY_ASK_HUMAN_LAYER_ID = "ask_human"
 DIFY_SHELL_LAYER_ID = "shell"
@@ -78,6 +82,28 @@ def _filter_snapshot_to_specs(
     return CompositorSessionSnapshot(schema_version=snapshot.schema_version, layers=filtered_layers)
 
 
+def _shell_layer_deps() -> dict[str, str]:
+    return {"execution_context": DIFY_EXECUTION_CONTEXT_LAYER_ID}
+
+
+def _drive_layer_deps() -> dict[str, str]:
+    return {"shell": DIFY_SHELL_LAYER_ID}
+
+
+def _config_layer_deps() -> dict[str, str]:
+    return {"shell": DIFY_SHELL_LAYER_ID}
+
+
+def _shell_config_with_drive_ref(
+    shell_config: DifyShellLayerConfig | None,
+    drive_config: DifyDriveLayerConfig | None,
+) -> DifyShellLayerConfig:
+    config = shell_config or DifyShellLayerConfig()
+    if drive_config is None:
+        return config
+    return config.model_copy(update={"agent_stub_drive_ref": drive_config.drive_ref})
+
+
 class AgentBackendModelConfig(BaseModel):
     """API-side model/plugin selection before it is converted to Dify Agent layers."""
 
@@ -141,9 +167,11 @@ class AgentBackendWorkflowNodeRunInput(BaseModel):
     idempotency_key: str | None = None
     output: AgentBackendOutputConfig | None = None
     tools: DifyPluginToolsLayerConfig | None = None
+    core_tools: DifyCoreToolsLayerConfig | None = None
     knowledge: DifyKnowledgeBaseLayerConfig | None = None
+    config_layer_config: DifyConfigLayerConfig | None = None
     # Drive Skills & Files declaration (dify.drive) — an index the agent pulls
-    # through the back proxy, never inline content; see AGENT_DRIVE_MANIFEST_ENABLED.
+    # through the back proxy, never inline content.
     drive_config: DifyDriveLayerConfig | None = None
     # Human-in-the-loop ask_human deferred tool (dify.ask_human). Present only when
     # the Agent Soul configures human involvement; a deferred call ends the run and
@@ -188,9 +216,11 @@ class AgentBackendAgentAppRunInput(BaseModel):
     idempotency_key: str | None = None
     output: AgentBackendOutputConfig | None = None
     tools: DifyPluginToolsLayerConfig | None = None
+    core_tools: DifyCoreToolsLayerConfig | None = None
     knowledge: DifyKnowledgeBaseLayerConfig | None = None
+    config_layer_config: DifyConfigLayerConfig | None = None
     # Drive Skills & Files declaration (dify.drive) — an index the agent pulls
-    # through the back proxy, never inline content; see AGENT_DRIVE_MANIFEST_ENABLED.
+    # through the back proxy, never inline content.
     drive_config: DifyDriveLayerConfig | None = None
     # Human-in-the-loop ask_human deferred tool (dify.ask_human). Present only when
     # the Agent Soul configures human involvement (ENG-635).
@@ -224,9 +254,11 @@ class AgentBackendRunRequestBuilder:
         """Build an Agent App conversation-turn run request.
 
         Layer graph: optional Agent Soul system prompt → user prompt →
-        execution context → optional history (multi-turn) → LLM → optional
-        plugin tools / knowledge search → optional structured output. Mirrors the workflow-node
-        layer ordering minus the workflow-job / previous-node prompt.
+        execution context → optional shell / config / drive / history
+        (multi-turn) → LLM → optional plugin-direct tools / core-routed tools /
+        knowledge search / ask_human / structured output. Mirrors the
+        workflow-node layer ordering minus the workflow-job / previous-node
+        prompt.
         """
         layers: list[RunLayerSpec] = []
         if run_input.agent_soul_prompt:
@@ -256,13 +288,42 @@ class AgentBackendRunRequestBuilder:
             ]
         )
 
+        include_shell = (
+            run_input.include_shell or run_input.config_layer_config is not None or run_input.drive_config is not None
+        )
+        if include_shell:
+            # Sandboxed bash workspace (dify.shell). It enters before config/drive
+            # so eager pulls materialize content in the same filesystem used by
+            # model commands.
+            layers.append(
+                RunLayerSpec(
+                    name=DIFY_SHELL_LAYER_ID,
+                    type=DIFY_SHELL_LAYER_TYPE_ID,
+                    deps=_shell_layer_deps(),
+                    metadata=run_input.metadata,
+                    config=_shell_config_with_drive_ref(run_input.shell_config, run_input.drive_config),
+                )
+            )
+
+        if run_input.config_layer_config is not None:
+            layers.append(
+                RunLayerSpec(
+                    name=DIFY_CONFIG_LAYER_ID,
+                    type=DIFY_CONFIG_LAYER_TYPE_ID,
+                    deps=_config_layer_deps(),
+                    metadata=run_input.metadata,
+                    config=run_input.config_layer_config,
+                )
+            )
+
         if run_input.drive_config is not None:
-            # Drive Skills & Files declaration (dify.drive): a config-only index;
-            # the agent pulls listed entries through the back proxy by drive_ref.
+            # Drive Skills & Files declaration (dify.drive): the catalog plus
+            # prompt-mentioned entries eagerly pulled through the shell layer.
             layers.append(
                 RunLayerSpec(
                     name=DIFY_DRIVE_LAYER_ID,
                     type=DIFY_DRIVE_LAYER_TYPE_ID,
+                    deps=_drive_layer_deps(),
                     metadata=run_input.metadata,
                     config=run_input.drive_config,
                 )
@@ -294,17 +355,31 @@ class AgentBackendRunRequestBuilder:
         )
 
         if run_input.tools is not None and run_input.tools.tools:
+            plugin_tool_deps = {"execution_context": DIFY_EXECUTION_CONTEXT_LAYER_ID}
+            if include_shell:
+                plugin_tool_deps["shell"] = DIFY_SHELL_LAYER_ID
             layers.append(
                 RunLayerSpec(
                     name=DIFY_PLUGIN_TOOLS_LAYER_ID,
                     type=DIFY_PLUGIN_TOOLS_LAYER_TYPE_ID,
-                    deps={"execution_context": DIFY_EXECUTION_CONTEXT_LAYER_ID},
+                    deps=plugin_tool_deps,
                     metadata=run_input.metadata,
                     config=run_input.tools,
                 )
             )
 
-        if run_input.knowledge is not None and run_input.knowledge.dataset_ids:
+        if run_input.core_tools is not None and run_input.core_tools.tools:
+            layers.append(
+                RunLayerSpec(
+                    name=DIFY_CORE_TOOLS_LAYER_ID,
+                    type=DIFY_CORE_TOOLS_LAYER_TYPE_ID,
+                    deps={"execution_context": DIFY_EXECUTION_CONTEXT_LAYER_ID},
+                    metadata=run_input.metadata,
+                    config=run_input.core_tools,
+                )
+            )
+
+        if run_input.knowledge is not None and run_input.knowledge.sets:
             layers.append(
                 RunLayerSpec(
                     name=DIFY_KNOWLEDGE_BASE_LAYER_ID,
@@ -328,20 +403,6 @@ class AgentBackendRunRequestBuilder:
                 )
             )
 
-        if run_input.include_shell:
-            # Sandboxed bash workspace (dify.shell). Depends on execution_context so
-            # the agent server can mint per-command Agent Stub env (back proxy);
-            # shellctl connection itself is server-injected.
-            layers.append(
-                RunLayerSpec(
-                    name=DIFY_SHELL_LAYER_ID,
-                    type=DIFY_SHELL_LAYER_TYPE_ID,
-                    deps={"execution_context": DIFY_EXECUTION_CONTEXT_LAYER_ID},
-                    metadata=run_input.metadata,
-                    config=run_input.shell_config or DifyShellLayerConfig(),
-                )
-            )
-
         if run_input.output is not None:
             layers.append(
                 RunLayerSpec(
@@ -383,7 +444,8 @@ class AgentBackendRunRequestBuilder:
         non-plugin layer graph that produced the snapshot. Plugin layers
         (``dify.plugin.llm``, ``dify.plugin.tools``) are excluded from both the
         composition and the snapshot before submission because their configs
-        require credentials that are not persisted between runs.
+        may carry credentials or runtime-only declarations that are not
+        persisted between runs.
         """
         if not runtime_layer_specs:
             raise ValueError(
@@ -416,8 +478,9 @@ class AgentBackendRunRequestBuilder:
         """Build a workflow Agent Node run request without defining another wire schema.
 
         Layer graph mirrors the workflow surface: prompts → execution context →
-        optional drive/history → LLM → optional plugin tools / knowledge search
-        → optional auxiliary layers such as ask_human, shell, and structured output.
+        optional shell / config / drive / history → LLM → optional
+        plugin-direct tools / core-routed tools / knowledge search /
+        ask_human / structured output.
         """
         layers: list[RunLayerSpec] = []
         if run_input.agent_soul_prompt:
@@ -436,7 +499,7 @@ class AgentBackendRunRequestBuilder:
                     name=WORKFLOW_NODE_JOB_PROMPT_LAYER_ID,
                     type=PLAIN_PROMPT_LAYER_TYPE_ID,
                     metadata={**run_input.metadata, "origin": "workflow_node_job"},
-                    config=PromptLayerConfig(prefix=run_input.workflow_node_job_prompt),
+                    config=PromptLayerConfig(user=run_input.workflow_node_job_prompt),
                 ),
                 RunLayerSpec(
                     name=WORKFLOW_USER_PROMPT_LAYER_ID,
@@ -453,13 +516,42 @@ class AgentBackendRunRequestBuilder:
             ]
         )
 
+        include_shell = (
+            run_input.include_shell or run_input.config_layer_config is not None or run_input.drive_config is not None
+        )
+        if include_shell:
+            # Sandboxed bash workspace (dify.shell). It enters before drive so
+            # drive can materialize mentioned targets with `dify-agent drive pull`
+            # in the same shell-visible filesystem used by model commands.
+            layers.append(
+                RunLayerSpec(
+                    name=DIFY_SHELL_LAYER_ID,
+                    type=DIFY_SHELL_LAYER_TYPE_ID,
+                    deps=_shell_layer_deps(),
+                    metadata=run_input.metadata,
+                    config=_shell_config_with_drive_ref(run_input.shell_config, run_input.drive_config),
+                )
+            )
+
+        if run_input.config_layer_config is not None:
+            layers.append(
+                RunLayerSpec(
+                    name=DIFY_CONFIG_LAYER_ID,
+                    type=DIFY_CONFIG_LAYER_TYPE_ID,
+                    deps=_config_layer_deps(),
+                    metadata=run_input.metadata,
+                    config=run_input.config_layer_config,
+                )
+            )
+
         if run_input.drive_config is not None:
-            # Drive Skills & Files declaration (dify.drive): a config-only index;
-            # the agent pulls listed entries through the back proxy by drive_ref.
+            # Drive Skills & Files declaration (dify.drive): the catalog plus
+            # prompt-mentioned entries eagerly pulled through the shell layer.
             layers.append(
                 RunLayerSpec(
                     name=DIFY_DRIVE_LAYER_ID,
                     type=DIFY_DRIVE_LAYER_TYPE_ID,
+                    deps=_drive_layer_deps(),
                     metadata=run_input.metadata,
                     config=run_input.drive_config,
                 )
@@ -493,17 +585,31 @@ class AgentBackendRunRequestBuilder:
         )
 
         if run_input.tools is not None and run_input.tools.tools:
+            plugin_tool_deps = {"execution_context": DIFY_EXECUTION_CONTEXT_LAYER_ID}
+            if include_shell:
+                plugin_tool_deps["shell"] = DIFY_SHELL_LAYER_ID
             layers.append(
                 RunLayerSpec(
                     name=DIFY_PLUGIN_TOOLS_LAYER_ID,
                     type=DIFY_PLUGIN_TOOLS_LAYER_TYPE_ID,
-                    deps={"execution_context": DIFY_EXECUTION_CONTEXT_LAYER_ID},
+                    deps=plugin_tool_deps,
                     metadata=run_input.metadata,
                     config=run_input.tools,
                 )
             )
 
-        if run_input.knowledge is not None and run_input.knowledge.dataset_ids:
+        if run_input.core_tools is not None and run_input.core_tools.tools:
+            layers.append(
+                RunLayerSpec(
+                    name=DIFY_CORE_TOOLS_LAYER_ID,
+                    type=DIFY_CORE_TOOLS_LAYER_TYPE_ID,
+                    deps={"execution_context": DIFY_EXECUTION_CONTEXT_LAYER_ID},
+                    metadata=run_input.metadata,
+                    config=run_input.core_tools,
+                )
+            )
+
+        if run_input.knowledge is not None and run_input.knowledge.sets:
             layers.append(
                 RunLayerSpec(
                     name=DIFY_KNOWLEDGE_BASE_LAYER_ID,
@@ -527,20 +633,6 @@ class AgentBackendRunRequestBuilder:
                 )
             )
 
-        if run_input.include_shell:
-            # Sandboxed bash workspace (dify.shell). Depends on execution_context so
-            # the agent server can mint per-command Agent Stub env (back proxy);
-            # shellctl connection itself is server-injected.
-            layers.append(
-                RunLayerSpec(
-                    name=DIFY_SHELL_LAYER_ID,
-                    type=DIFY_SHELL_LAYER_TYPE_ID,
-                    deps={"execution_context": DIFY_EXECUTION_CONTEXT_LAYER_ID},
-                    metadata=run_input.metadata,
-                    config=run_input.shell_config or DifyShellLayerConfig(),
-                )
-            )
-
         if run_input.output is not None:
             layers.append(
                 RunLayerSpec(

api/commands/__init__.py

@@ -22,9 +22,10 @@ from .plugin import (
     setup_system_trigger_oauth_client,
     transform_datasource_credentials,
 )
-from .rbac import migrate_member_roles_to_rbac
+from .rbac import migrate_dataset_permissions_to_rbac, migrate_member_roles_to_rbac
 from .retention import (
     archive_workflow_runs,
+    archive_workflow_runs_plan,
     clean_expired_messages,
     clean_workflow_runs,
     cleanup_orphaned_draft_variables,
@@ -51,6 +52,7 @@ from .vector import (
 __all__ = [
     "add_qdrant_index",
     "archive_workflow_runs",
+    "archive_workflow_runs_plan",
     "backfill_plugin_auto_upgrade",
     "clean_expired_messages",
     "clean_workflow_runs",
@@ -74,6 +76,7 @@ __all__ = [
     "legacy_model_types",
     "migrate_annotation_vector_database",
     "migrate_data_for_plugin",
+    "migrate_dataset_permissions_to_rbac",
     "migrate_knowledge_vector_database",
     "migrate_member_roles_to_rbac",
     "migrate_oss",

api/commands/account.py

@@ -25,7 +25,7 @@ def reset_password(email, new_password, password_confirm):
         return
     normalized_email = email.strip().lower()
 
-    account = AccountService.get_account_by_email_with_case_fallback(email.strip())
+    account = AccountService.get_account_by_email_with_case_fallback(db.session(), email.strip())
 
     if not account:
         click.echo(click.style(f"Account not found for email: {email}", fg="red"))
@@ -67,7 +67,7 @@ def reset_email(email, new_email, email_confirm):
         return
     normalized_new_email = new_email.strip().lower()
 
-    account = AccountService.get_account_by_email_with_case_fallback(email.strip())
+    account = AccountService.get_account_by_email_with_case_fallback(db.session(), email.strip())
 
     if not account:
         click.echo(click.style(f"Account not found for email: {email}", fg="red"))
@@ -133,8 +133,9 @@ def create_tenant(email: str, language: str | None = None, name: str | None = No
         password=new_password,
         language=language,
         create_workspace_required=False,
+        session=db.session(),
     )
-    TenantService.create_owner_tenant_if_not_exist(account, name)
+    TenantService.create_owner_tenant_if_not_exist(account, name, session=db.session())
 
     click.echo(
         click.style(

api/commands/data_migrate.py

@@ -7,6 +7,7 @@ from typing import cast
 
 import click
 
+from commands.rbac import migrate_dataset_permissions_to_rbac
 from extensions.ext_database import db
 from graphon.model_runtime.entities.model_entities import ModelType
 from services.legacy_model_type_migration import (
@@ -177,3 +178,4 @@ def legacy_model_types(
 
 
 data_migrate.add_command(legacy_model_types)
+data_migrate.add_command(migrate_dataset_permissions_to_rbac)

api/commands/data_migration.py

@@ -9,7 +9,9 @@ from uuid import UUID
 import click
 import sqlalchemy as sa
 import yaml
+from sqlalchemy.orm import Session
 
+from core.db.session_factory import session_factory
 from extensions.ext_database import db
 from models import Tenant
 from models.model import App
@@ -106,7 +108,8 @@ def export_migration_data(input_file: str | None, output_file: str | None, overw
         assert output_file is not None
         raw_config = _load_json_object(input_file, "Export config")
         selection = ExportConfigParser().parse(raw_config)
-        result = MigrationExportService().export(selection)
+        with session_factory.create_session() as session:
+            result = MigrationExportService().export(selection, session=session)
         MigrationPackageService().save_package(result.package, output_file, overwrite=overwrite)
         click.echo(click.style(f"Output written to {output_file}", fg="green"))
         _render_report(result.report_items, context=_with_output_path(result.report_context, output_file))
@@ -153,19 +156,21 @@ def import_migration_data(
         _require_options(("--input", input_file))
         assert input_file is not None
         package = MigrationPackageService().load_package(input_file)
-        result = MigrationImportService().import_package(
-            ImportRequest(
-                package=package,
-                cli_target_tenant=target_tenant,
-                operator_email=operator_email,
-                options_override=_build_options_override(
-                    package.metadata.import_options,
-                    id_strategy=id_strategy,
-                    conflict_strategy=conflict_strategy,
-                    create_app_api_token_on_import=create_app_api_token_on_import,
+        with session_factory.create_session() as session:
+            result = MigrationImportService().import_package(
+                ImportRequest(
+                    package=package,
+                    cli_target_tenant=target_tenant,
+                    operator_email=operator_email,
+                    options_override=_build_options_override(
+                        package.metadata.import_options,
+                        id_strategy=id_strategy,
+                        conflict_strategy=conflict_strategy,
+                        create_app_api_token_on_import=create_app_api_token_on_import,
+                    ),
                 ),
+                session=session,
             )
-        )
         _render_report(result.report_items, context=result.report_context)
     except MigrationDataError as exc:
         raise click.ClickException(str(exc)) from exc
@@ -213,7 +218,9 @@ def migration_data_wizard() -> None:
             default=True,
             show_default=False,
         )
-        auto_tools = _discover_auto_tools([app for app in apps if app.id in set(app_ids)], include_referenced_tools)
+        auto_tools = _discover_auto_tools(
+            [app for app in apps if app.id in set(app_ids)], include_referenced_tools, session=db.session()
+        )
         auto_tools = _resolve_auto_tool_names(tenant.id, auto_tools)
         _print_auto_tools(auto_tools)
         additional_tools = _prompt_additional_tools(tenant.id, auto_tools)
@@ -248,7 +255,8 @@ def migration_data_wizard() -> None:
             conflict_strategy=conflict_strategy,
             output_file=output_file,
         )
-        result = MigrationExportService().export(selection)
+        with session_factory.create_session() as session:
+            result = MigrationExportService().export(selection, session=session)
         MigrationPackageService().save_package(result.package, output_file, overwrite=overwrite)
         click.echo(click.style(f"Output written to {output_file}", fg="green"))
         _print_wizard_step("Report")
@@ -389,13 +397,13 @@ def _prompt_import_options() -> tuple[bool, bool, str, str]:
     return include_secrets, create_tokens, id_strategy, conflict_strategy
 
 
-def _discover_auto_tools(apps: list[App], include_referenced_tools: bool) -> WizardToolMap:
+def _discover_auto_tools(apps: list[App], include_referenced_tools: bool, *, session: Session) -> WizardToolMap:
     auto_tools: WizardToolMap = {"api_tools": {}, "workflow_tools": {}, "mcp_tools": {}}
     if not include_referenced_tools:
         return auto_tools
     discovery_service = DependencyDiscoveryService()
     for app in apps:
-        dsl_content = AppDslService.export_dsl(app_model=app, include_secret=False)
+        dsl_content = AppDslService.export_dsl(app_model=app, session=session, include_secret=False)
         raw_dsl = yaml.safe_load(dsl_content) if dsl_content else {}
         dsl = raw_dsl if isinstance(raw_dsl, dict) else {}
         for dependency in discovery_service.discover_from_dsl(dsl):

api/commands/plugin.py

@@ -16,7 +16,7 @@ from core.plugin.plugin_service import PluginService
 from core.tools.utils.system_encryption import encrypt_system_params
 from extensions.ext_database import db
 from models import Tenant
-from models.account import TenantPluginAutoUpgradeStrategy
+from models.account import TenantPluginAutoUpgradeCategory, TenantPluginAutoUpgradeStrategy
 from models.oauth import DatasourceOauthParamConfig, DatasourceProvider
 from models.provider_ids import DatasourceProviderID, ToolProviderID
 from models.source import DataSourceApiKeyAuthBinding, DataSourceOauthBinding
@@ -188,13 +188,13 @@ def transform_datasource_credentials(environment: str):
         firecrawl_plugin_id = "langgenius/firecrawl_datasource"
         jina_plugin_id = "langgenius/jina_datasource"
         if environment == "online":
-            notion_plugin_unique_identifier = plugin_migration._fetch_plugin_unique_identifier(notion_plugin_id)
-            firecrawl_plugin_unique_identifier = plugin_migration._fetch_plugin_unique_identifier(firecrawl_plugin_id)
-            jina_plugin_unique_identifier = plugin_migration._fetch_plugin_unique_identifier(jina_plugin_id)
+            notion_package_identifier = plugin_migration._fetch_latest_package_identifier(notion_plugin_id)
+            firecrawl_package_identifier = plugin_migration._fetch_latest_package_identifier(firecrawl_plugin_id)
+            jina_package_identifier = plugin_migration._fetch_latest_package_identifier(jina_plugin_id)
         else:
-            notion_plugin_unique_identifier = None
-            firecrawl_plugin_unique_identifier = None
-            jina_plugin_unique_identifier = None
+            notion_package_identifier = None
+            firecrawl_package_identifier = None
+            jina_package_identifier = None
         oauth_credential_type = CredentialType.OAUTH2
         api_key_credential_type = CredentialType.API_KEY
 
@@ -219,9 +219,9 @@ def transform_datasource_credentials(environment: str):
                     installed_plugins = installer_manager.list_plugins(tenant_id)
                     installed_plugins_ids = [plugin.plugin_id for plugin in installed_plugins]
                     if notion_plugin_id not in installed_plugins_ids:
-                        if notion_plugin_unique_identifier:
+                        if notion_package_identifier:
                             # install notion plugin
-                            PluginService.install_from_marketplace_pkg(tenant_id, [notion_plugin_unique_identifier])
+                            PluginService.install_from_marketplace_pkg(tenant_id, [notion_package_identifier])
                     auth_count = 0
                     for notion_tenant_credential in notion_tenant_credentials:
                         auth_count += 1
@@ -279,9 +279,9 @@ def transform_datasource_credentials(environment: str):
                     installed_plugins = installer_manager.list_plugins(tenant_id)
                     installed_plugins_ids = [plugin.plugin_id for plugin in installed_plugins]
                     if firecrawl_plugin_id not in installed_plugins_ids:
-                        if firecrawl_plugin_unique_identifier:
+                        if firecrawl_package_identifier:
                             # install firecrawl plugin
-                            PluginService.install_from_marketplace_pkg(tenant_id, [firecrawl_plugin_unique_identifier])
+                            PluginService.install_from_marketplace_pkg(tenant_id, [firecrawl_package_identifier])
 
                     auth_count = 0
                     for firecrawl_tenant_credential in firecrawl_tenant_credentials:
@@ -343,10 +343,10 @@ def transform_datasource_credentials(environment: str):
                     installed_plugins = installer_manager.list_plugins(tenant_id)
                     installed_plugins_ids = [plugin.plugin_id for plugin in installed_plugins]
                     if jina_plugin_id not in installed_plugins_ids:
-                        if jina_plugin_unique_identifier:
+                        if jina_package_identifier:
                             # install jina plugin
-                            logger.debug("Installing Jina plugin %s", jina_plugin_unique_identifier)
-                            PluginService.install_from_marketplace_pkg(tenant_id, [jina_plugin_unique_identifier])
+                            logger.debug("Installing Jina plugin %s", jina_package_identifier)
+                            PluginService.install_from_marketplace_pkg(tenant_id, [jina_package_identifier])
 
                     auth_count = 0
                     for jina_tenant_credential in jina_tenant_credentials:
@@ -406,7 +406,7 @@ def migrate_data_for_plugin():
 
 
 def _candidate_auto_upgrade_strategy_tenant_ids_stmt(limit: int | None = None):
-    category_count = len(TenantPluginAutoUpgradeStrategy.PluginCategory)
+    category_count = len(TenantPluginAutoUpgradeCategory)
     stmt = (
         select(TenantPluginAutoUpgradeStrategy.tenant_id)
         .group_by(TenantPluginAutoUpgradeStrategy.tenant_id)
@@ -472,6 +472,7 @@ def backfill_plugin_auto_upgrade(
         try:
             result = PluginAutoUpgradeService.backfill_strategy_categories(
                 current_tenant_id,
+                session=db.session(),
             )
         except Exception as e:
             failed_count += 1

api/commands/rbac.py

@@ -1,11 +1,56 @@
 from __future__ import annotations
 
+import json
+from collections.abc import Iterator
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
 import click
 from sqlalchemy import select
+from sqlalchemy.orm import Session
 
+from configs import dify_config
 from core.db.session_factory import session_factory
-from models import TenantAccountJoin, TenantAccountRole
-from services.enterprise.rbac_service import ListOption, RBACService
+from core.rbac import RBACResourceWhitelistScope
+from models import Dataset, DatasetPermission, DatasetPermissionEnum, TenantAccountJoin, TenantAccountRole
+from services.enterprise.rbac_service import ListOption, RBACService, ReplaceMemberBindings, ReplaceUserAccessPolicies
+
+_RBAC_DEFAULT_ACCESS_POLICY_ID = "default"
+
+_LEGACY_ROLE_TO_BUILTIN_TAG = {
+    TenantAccountRole.OWNER.value: "owner",
+    TenantAccountRole.ADMIN.value: "admin",
+    TenantAccountRole.EDITOR.value: "editor",
+    TenantAccountRole.NORMAL.value: "normal",
+    TenantAccountRole.DATASET_OPERATOR.value: "dataset_operator",
+}
+
+
+def _resolve_builtin_role_ids(tenant_id: str, operator_account_id: str) -> dict[str, str]:
+    """Resolve every legacy workspace role to the current tenant's builtin RBAC role id.
+
+    The migration replays the old `TenantAccountJoin.role` values onto the
+    RBAC member-role binding API. Builtin RBAC roles are tenant-scoped and
+    identified by runtime ids, so the command must look them up per tenant.
+    """
+    roles = RBACService.Roles.list(
+        tenant_id=tenant_id,
+        account_id=operator_account_id,
+        options=ListOption(page_number=1, results_per_page=100),
+    ).data
+    role_id_by_tag = {
+        role.role_tag: role.id
+        for role in roles
+        if role.is_builtin and role.category == "global_system_default" and role.role_tag
+    }
+    resolved: dict[str, str] = {}
+    for legacy_role, expected_builtin_tag in _LEGACY_ROLE_TO_BUILTIN_TAG.items():
+        role_id = role_id_by_tag.get(expected_builtin_tag)
+        if expected_builtin_tag == "dataset_operator" and not dify_config.DATASET_OPERATOR_ENABLED:
+            continue
+        if not role_id:
+            raise ValueError(f"Builtin RBAC role not found for tenant={tenant_id}, legacy_role={legacy_role}")
+        resolved[legacy_role] = role_id
+    return resolved
 
 
 def _resolve_builtin_role_id(tenant_id: str, operator_account_id: str, legacy_role: str) -> str:
@@ -15,26 +60,105 @@ def _resolve_builtin_role_id(tenant_id: str, operator_account_id: str, legacy_ro
     RBAC member-role binding API. Builtin RBAC roles are tenant-scoped and
     identified by runtime ids, so the command must look them up per tenant.
     """
-    expected_builtin_tag = {
-        TenantAccountRole.OWNER.value: "owner",
-        TenantAccountRole.ADMIN.value: "admin",
-        TenantAccountRole.EDITOR.value: "editor",
-        TenantAccountRole.NORMAL.value: "normal",
-        TenantAccountRole.DATASET_OPERATOR.value: "dataset_operator",
-    }.get(legacy_role)
-    if not expected_builtin_tag:
+    if legacy_role not in _LEGACY_ROLE_TO_BUILTIN_TAG:
         raise ValueError(f"Unsupported legacy workspace role: {legacy_role}")
 
-    roles = RBACService.Roles.list(
+    return _resolve_builtin_role_ids(tenant_id, operator_account_id)[legacy_role]
+
+
+def _iter_tenant_member_batches(
+    tenant_id: str | None,
+    *,
+    db_batch_size: int,
+    api_batch_size: int,
+) -> Iterator[tuple[str, str, list[tuple[str, str]]]]:
+    """Yield legacy member roles in tenant-scoped API-sized batches.
+
+    Rows are projected to primitive values and streamed from the database, so
+    the command never materializes every TenantAccountJoin ORM object. The
+    iterator only keeps one tenant's API-sized batches in memory while it
+    finds that tenant's owner account.
+    """
+    with session_factory.create_session() as session:
+        stmt = (
+            select(TenantAccountJoin.tenant_id, TenantAccountJoin.account_id, TenantAccountJoin.role)
+            .order_by(TenantAccountJoin.tenant_id.asc(), TenantAccountJoin.id.asc())
+            .execution_options(yield_per=db_batch_size)
+        )
+        if tenant_id:
+            stmt = stmt.where(TenantAccountJoin.tenant_id == tenant_id)
+
+        current_tenant_id: str | None = None
+        owner_account_id: str | None = None
+        batches: list[list[tuple[str, str]]] = []
+        batch: list[tuple[str, str]] = []
+
+        def flush_current_tenant() -> Iterator[tuple[str, str, list[tuple[str, str]]]]:
+            if current_tenant_id is None:
+                return
+            if batch:
+                batches.append(batch.copy())
+            if not owner_account_id:
+                raise ValueError(f"Workspace owner not found for tenant={current_tenant_id}")
+            for item in batches:
+                yield current_tenant_id, owner_account_id, item
+
+        for row in session.execute(stmt):
+            workspace_id = str(row.tenant_id)
+            if current_tenant_id is not None and workspace_id != current_tenant_id:
+                yield from flush_current_tenant()
+                owner_account_id = None
+                batches = []
+                batch = []
+            current_tenant_id = workspace_id
+            account_id = str(row.account_id)
+            role = str(row.role)
+            if role == TenantAccountRole.OWNER.value:
+                owner_account_id = account_id
+            batch.append((account_id, role))
+            if len(batch) >= api_batch_size:
+                batches.append(batch)
+                batch = []
+
+        yield from flush_current_tenant()
+
+
+def _member_already_has_role(current_roles_by_account_id: dict[str, set[str]], account_id: str, role_id: str) -> bool:
+    return current_roles_by_account_id.get(account_id) == {role_id}
+
+
+def _replace_member_role(
+    tenant_id: str,
+    operator_account_id: str,
+    member_account_id: str,
+    role_id: str,
+    *,
+    session: Session,
+) -> str:
+    RBACService.MemberRoles.replace(
         tenant_id=tenant_id,
         account_id=operator_account_id,
-        options=ListOption(page_number=1, results_per_page=100),
-    ).data
-    for role in roles:
-        if role.is_builtin and role.category == "global_system_default" and role.role_tag == expected_builtin_tag:
-            return str(role.id)
+        member_account_id=member_account_id,
+        role_ids=[role_id],
+        session=session,
+    )
+    return member_account_id
 
-    raise ValueError(f"Builtin RBAC role not found for tenant={tenant_id}, legacy_role={legacy_role}")
+
+def _replace_member_role_with_new_session(
+    tenant_id: str,
+    operator_account_id: str,
+    member_account_id: str,
+    role_id: str,
+) -> str:
+    with session_factory.create_session() as session:
+        return _replace_member_role(
+            tenant_id=tenant_id,
+            operator_account_id=operator_account_id,
+            member_account_id=member_account_id,
+            role_id=role_id,
+            session=session,
+        )
 
 
 @click.command(
@@ -42,7 +166,16 @@ def _resolve_builtin_role_id(tenant_id: str, operator_account_id: str, legacy_ro
 )
 @click.option("--tenant-id", help="Only migrate a single workspace.")
 @click.option("--dry-run", is_flag=True, default=False, help="Preview the migration without writing RBAC bindings.")
-def migrate_member_roles_to_rbac(tenant_id: str | None, dry_run: bool) -> None:
+@click.option("--db-batch-size", default=5000, show_default=True, help="Rows fetched per database batch.")
+@click.option("--api-batch-size", default=200, show_default=True, help="Members checked per RBAC batch_get call.")
+@click.option("--workers", default=1, show_default=True, help="Concurrent member role replace calls per tenant batch.")
+def migrate_member_roles_to_rbac(
+    tenant_id: str | None,
+    dry_run: bool,
+    db_batch_size: int,
+    api_batch_size: int,
+    workers: int,
+) -> None:
     """Backfill RBAC member-role bindings from legacy `TenantAccountJoin.role` data.
 
     This is an offline migration command for workspaces that already have
@@ -50,63 +183,329 @@ def migrate_member_roles_to_rbac(tenant_id: str | None, dry_run: bool) -> None:
     member-role binding store.
     """
     click.echo(click.style("Starting RBAC member-role migration.", fg="green"))
+    if workers < 1:
+        raise click.BadParameter("workers must be >= 1", param_hint="--workers")
 
-    with session_factory.create_session() as session:
-        stmt = select(TenantAccountJoin).order_by(TenantAccountJoin.tenant_id.asc(), TenantAccountJoin.id.asc())
-        if tenant_id:
-            stmt = stmt.where(TenantAccountJoin.tenant_id == tenant_id)
+    tenant_count = 0
+    scanned_count = 0
+    skipped_count = 0
+    migrated_count = 0
+    current_tenant_id: str | None = None
+    role_ids_by_legacy_role: dict[str, str] = {}
 
-        joins = list(session.scalars(stmt).all())
+    for workspace_id, owner_account_id, batch in _iter_tenant_member_batches(
+        tenant_id,
+        db_batch_size=db_batch_size,
+        api_batch_size=api_batch_size,
+    ):
+        scanned_count += len(batch)
+        if workspace_id != current_tenant_id:
+            tenant_count += 1
+            current_tenant_id = workspace_id
+            role_ids_by_legacy_role = _resolve_builtin_role_ids(workspace_id, owner_account_id)
+            click.echo(f"tenant={workspace_id}")
 
-    if not joins:
+        current_roles_by_account_id: dict[str, set[str]] = {}
+        if not dry_run:
+            current_roles = RBACService.MemberRoles.batch_get(
+                tenant_id=workspace_id,
+                account_id=owner_account_id,
+                member_account_ids=[account_id for account_id, _ in batch],
+            )
+            current_roles_by_account_id = {
+                item.account_id: {str(role.id) for role in item.roles} for item in current_roles
+            }
+
+        replace_jobs: list[tuple[str, str]] = []
+        for member_account_id, legacy_role in batch:
+            resolved_role_id = role_ids_by_legacy_role.get(legacy_role)
+            if not resolved_role_id:
+                raise ValueError(f"Unsupported legacy workspace role: {legacy_role}")
+
+            if dry_run:
+                click.echo(
+                    f"tenant={workspace_id} member={member_account_id} "
+                    f"legacy_role={legacy_role} -> rbac_role_id={resolved_role_id}"
+                )
+                continue
+
+            if _member_already_has_role(current_roles_by_account_id, member_account_id, resolved_role_id):
+                skipped_count += 1
+                continue
+
+            replace_jobs.append((member_account_id, resolved_role_id))
+
+        if replace_jobs:
+            if workers == 1:
+                with session_factory.create_session() as session:
+                    for member_account_id, resolved_role_id in replace_jobs:
+                        _replace_member_role(
+                            workspace_id,
+                            owner_account_id,
+                            member_account_id,
+                            resolved_role_id,
+                            session=session,
+                        )
+                        migrated_count += 1
+            else:
+                with ThreadPoolExecutor(max_workers=workers) as executor:
+                    futures = [
+                        executor.submit(
+                            _replace_member_role_with_new_session,
+                            workspace_id,
+                            owner_account_id,
+                            member_account_id,
+                            resolved_role_id,
+                        )
+                        for member_account_id, resolved_role_id in replace_jobs
+                    ]
+                    for future in as_completed(futures):
+                        future.result()
+                        migrated_count += 1
+
+        if scanned_count % 10000 == 0:
+            click.echo(
+                f"progress scanned={scanned_count} migrated={migrated_count} skipped={skipped_count}",
+                err=True,
+            )
+
+    if scanned_count == 0:
         click.echo(click.style("No workspace members found for migration.", fg="yellow"))
         return
 
-    owner_account_by_tenant: dict[str, str] = {}
-    resolved_role_ids: dict[tuple[str, str], str] = {}
-    migrated_count = 0
-
-    for join in joins:
-        workspace_id = str(join.tenant_id)
-        member_account_id = str(join.account_id)
-        legacy_role = str(join.role)
-
-        if workspace_id not in owner_account_by_tenant:
-            owner_join = next(
-                (
-                    item
-                    for item in joins
-                    if str(item.tenant_id) == workspace_id and str(item.role) == TenantAccountRole.OWNER.value
-                ),
-                None,
-            )
-            if not owner_join:
-                raise ValueError(f"Workspace owner not found for tenant={workspace_id}")
-            owner_account_by_tenant[workspace_id] = str(owner_join.account_id)
-
-        operator_account_id = owner_account_by_tenant[workspace_id]
-        cache_key = (workspace_id, legacy_role)
-        if cache_key not in resolved_role_ids:
-            resolved_role_ids[cache_key] = _resolve_builtin_role_id(workspace_id, operator_account_id, legacy_role)
-
-        resolved_role_id = resolved_role_ids[cache_key]
+    if dry_run:
         click.echo(
-            f"tenant={workspace_id} member={member_account_id} "
-            f"legacy_role={legacy_role} -> rbac_role_id={resolved_role_id}"
+            click.style(
+                f"Dry run completed. Scanned {scanned_count} members across {tenant_count} tenants. "
+                "No RBAC bindings were written.",
+                fg="yellow",
+            )
+        )
+    else:
+        click.echo(
+            click.style(
+                f"RBAC member-role migration completed. Scanned {scanned_count} members across {tenant_count} tenants, "
+                f"migrated {migrated_count}, skipped {skipped_count} already up-to-date.",
+                fg="green",
+            )
         )
 
-        if dry_run:
-            continue
 
-        RBACService.MemberRoles.replace(
-            tenant_id=workspace_id,
-            account_id=operator_account_id,
-            member_account_id=member_account_id,
-            role_ids=[resolved_role_id],
-        )
-        migrated_count += 1
+def _dataset_permission_enum(permission: DatasetPermissionEnum | str | None) -> DatasetPermissionEnum:
+    if permission is None:
+        return DatasetPermissionEnum.ONLY_ME
+    try:
+        return DatasetPermissionEnum(permission)
+    except ValueError as exc:
+        raise ValueError(f"Unsupported legacy dataset permission: {permission}") from exc
+
+
+def _rbac_dataset_scope_for_legacy_permission(permission: DatasetPermissionEnum) -> RBACResourceWhitelistScope:
+    if permission is DatasetPermissionEnum.ALL_TEAM:
+        return RBACResourceWhitelistScope.ALL
+    if permission in {DatasetPermissionEnum.ONLY_ME, DatasetPermissionEnum.PARTIAL_TEAM}:
+        return RBACResourceWhitelistScope.SPECIFIC
+    raise ValueError(f"Unsupported legacy dataset permission: {permission}")
+
+
+def _emit_dataset_permission_migration_event(payload: dict[str, object]) -> None:
+    click.echo(json.dumps(payload, sort_keys=True))
+
+
+@click.command(
+    "rbac-migrate-dataset-permissions",
+    help=(
+        "Migrate legacy dataset permission scopes and partial members into RBAC dataset access bindings. "
+        "Side effect: replacing each dataset whitelist clears existing per-user policy bindings; "
+        "the command then recreates legacy partial-member default bindings."
+    ),
+)
+@click.option("--tenant-id", help="Only migrate datasets in a single workspace.")
+@click.option("--dataset-id", help="Only migrate a single dataset.")
+@click.option("--batch-size", default=500, show_default=True, type=click.IntRange(min=1))
+@click.option(
+    "--dry-run/--apply",
+    default=True,
+    show_default=True,
+    help="Preview the migration without writing RBAC bindings. Use --apply to write changes.",
+)
+def migrate_dataset_permissions_to_rbac(
+    tenant_id: str | None,
+    dataset_id: str | None,
+    batch_size: int,
+    dry_run: bool,
+) -> None:
+    """Backfill RBAC dataset access config from legacy `Dataset.permission`.
+
+    Legacy mapping:
+    - all_team_members -> RBAC dataset whitelist scope "all"
+    - partial_members  -> RBAC dataset whitelist scope "specific" plus each partial member gets the
+      virtual default policy
+    - only_me          -> RBAC dataset whitelist scope "specific" with no member policy bindings
+
+    The command replaces each dataset's RBAC whitelist scope first. RBAC clears
+    existing per-user policy bindings during that replace, then this command
+    recreates the legacy partial-member default bindings. Re-running it is
+    therefore idempotent for a dataset's current legacy configuration.
+    """
+    click.echo(click.style("Starting RBAC dataset permission migration.", fg="green"))
+
+    scanned_count = 0
+    scope_migrated_count = 0
+    user_policy_migrated_count = 0
+    partial_dataset_count = 0
+
+    last_dataset_id: str | None = None
+    while True:
+        with session_factory.create_session() as session:
+            stmt = (
+                select(Dataset.id, Dataset.tenant_id, Dataset.permission, Dataset.created_by)
+                .order_by(Dataset.id.asc())
+                .limit(batch_size)
+            )
+            if tenant_id:
+                stmt = stmt.where(Dataset.tenant_id == tenant_id)
+            if dataset_id:
+                stmt = stmt.where(Dataset.id == dataset_id)
+            if last_dataset_id:
+                stmt = stmt.where(Dataset.id > last_dataset_id)
+
+            dataset_rows = list(session.execute(stmt).all())
+            if not dataset_rows:
+                break
+
+            dataset_ids = [str(row.id) for row in dataset_rows]
+            partial_members_by_dataset_id: dict[str, list[str]] = {item: [] for item in dataset_ids}
+            permission_rows = session.execute(
+                select(DatasetPermission.dataset_id, DatasetPermission.account_id).where(
+                    DatasetPermission.dataset_id.in_(dataset_ids)
+                )
+            ).all()
+            for row in permission_rows:
+                partial_members_by_dataset_id[str(row.dataset_id)].append(str(row.account_id))
+
+        for dataset in dataset_rows:
+            workspace_id = str(dataset.tenant_id)
+            current_dataset_id = str(dataset.id)
+            operator_account_id = str(dataset.created_by)
+            permission_value = _dataset_permission_enum(dataset.permission)
+            scope = _rbac_dataset_scope_for_legacy_permission(permission_value)
+            partial_member_ids = sorted(set(partial_members_by_dataset_id[current_dataset_id]))
+            should_bind_partial_members = permission_value is DatasetPermissionEnum.PARTIAL_TEAM
+
+            click.echo(
+                f"tenant={workspace_id} dataset={current_dataset_id} "
+                f"operator={operator_account_id} "
+                f"legacy_permission={permission_value} -> rbac_scope={scope} "
+                f"partial_members={len(partial_member_ids) if should_bind_partial_members else 0}"
+            )
+
+            scanned_count += 1
+            replace_whitelist_payload = ReplaceMemberBindings(scope=scope)
+            if dry_run:
+                _emit_dataset_permission_migration_event(
+                    {
+                        "event": "dataset_permission_migration_proposed_change",
+                        "action": "replace_whitelist",
+                        "dry_run": True,
+                        "tenant_id": workspace_id,
+                        "dataset_id": current_dataset_id,
+                        "operator_account_id": operator_account_id,
+                        "before": {
+                            "legacy_dataset_permission": permission_value.value,
+                            "legacy_partial_member_ids": partial_member_ids if should_bind_partial_members else [],
+                        },
+                        "after": {
+                            "rbac_whitelist_scope": scope.value,
+                        },
+                        "call": {
+                            "method": "RBACService.DatasetAccess.replace_whitelist",
+                            "kwargs": {
+                                "tenant_id": workspace_id,
+                                "account_id": operator_account_id,
+                                "dataset_id": current_dataset_id,
+                                "payload": replace_whitelist_payload.model_dump(mode="json"),
+                            },
+                        },
+                    }
+                )
+            if not dry_run:
+                RBACService.DatasetAccess.replace_whitelist(
+                    tenant_id=workspace_id,
+                    account_id=operator_account_id,
+                    dataset_id=current_dataset_id,
+                    payload=replace_whitelist_payload,
+                )
+                scope_migrated_count += 1
+
+            if should_bind_partial_members:
+                partial_dataset_count += 1
+                for member_account_id in partial_member_ids:
+                    replace_user_access_policies_payload = ReplaceUserAccessPolicies(
+                        access_policy_ids=[_RBAC_DEFAULT_ACCESS_POLICY_ID],
+                    )
+                    if dry_run:
+                        _emit_dataset_permission_migration_event(
+                            {
+                                "event": "dataset_permission_migration_proposed_change",
+                                "action": "replace_user_access_policies",
+                                "dry_run": True,
+                                "tenant_id": workspace_id,
+                                "dataset_id": current_dataset_id,
+                                "operator_account_id": operator_account_id,
+                                "target_account_id": member_account_id,
+                                "before": {
+                                    "legacy_dataset_permission": permission_value.value,
+                                    "legacy_partial_member_id": member_account_id,
+                                },
+                                "after": {
+                                    "rbac_user_access_policy_ids": [_RBAC_DEFAULT_ACCESS_POLICY_ID],
+                                },
+                                "call": {
+                                    "method": "RBACService.DatasetAccess.replace_user_access_policies",
+                                    "kwargs": {
+                                        "tenant_id": workspace_id,
+                                        "account_id": operator_account_id,
+                                        "dataset_id": current_dataset_id,
+                                        "target_account_id": member_account_id,
+                                        "payload": replace_user_access_policies_payload.model_dump(mode="json"),
+                                    },
+                                },
+                            }
+                        )
+                        continue
+                    RBACService.DatasetAccess.replace_user_access_policies(
+                        tenant_id=workspace_id,
+                        account_id=operator_account_id,
+                        dataset_id=current_dataset_id,
+                        target_account_id=member_account_id,
+                        payload=replace_user_access_policies_payload,
+                    )
+                    user_policy_migrated_count += 1
+
+        last_dataset_id = dataset_ids[-1]
+
+        if dataset_id:
+            break
+
+    if scanned_count == 0:
+        click.echo(click.style("No datasets found for migration.", fg="yellow"))
+        return
 
     if dry_run:
-        click.echo(click.style("Dry run completed. No RBAC bindings were written.", fg="yellow"))
+        click.echo(
+            click.style(
+                f"Dry run completed. Scanned {scanned_count} datasets; "
+                f"{partial_dataset_count} partial-member datasets would be migrated.",
+                fg="yellow",
+            )
+        )
     else:
-        click.echo(click.style(f"RBAC member-role migration completed. Migrated {migrated_count} members.", fg="green"))
+        click.echo(
+            click.style(
+                "RBAC dataset permission migration completed. "
+                f"Scanned {scanned_count} datasets, migrated {scope_migrated_count} scopes, "
+                f"wrote {user_policy_migrated_count} user default-policy bindings.",
+                fg="green",
+            )
+        )

api/commands/retention.py

@@ -12,10 +12,171 @@ from services.clear_free_plan_tenant_expired_logs import ClearFreePlanTenantExpi
 from services.retention.conversation.messages_clean_policy import create_message_clean_policy
 from services.retention.conversation.messages_clean_service import MessagesCleanService
 from services.retention.workflow_run.clear_free_plan_expired_workflow_run_logs import WorkflowRunCleanup
+from services.retention.workflow_run.tenant_prefix import tenant_prefix_condition
 from tasks.remove_app_and_related_data_task import delete_draft_variables_batch
 
 logger = logging.getLogger(__name__)
 
+_HEX_PREFIXES = tuple("0123456789abcdef")
+
+
+class WorkflowRunArchivePlanRow(TypedDict):
+    tenant_prefix: str
+    total_tenants: int
+    workflow_runs: int
+    workflow_node_executions: int
+    paid_tenants: int
+    unpaid_tenants: int
+
+
+class WorkflowRunArchiveTenantPlan(TypedDict):
+    archive_tenant_ids: list[str] | None
+    paid_tenant_ids: list[str]
+    unpaid_tenant_ids: list[str]
+
+
+def _normalize_utc_datetime(value: datetime.datetime) -> datetime.datetime:
+    if value.tzinfo is None:
+        return value.replace(tzinfo=datetime.UTC)
+    return value.astimezone(datetime.UTC)
+
+
+def _parse_tenant_prefixes(prefixes: str | None) -> list[str]:
+    if not prefixes:
+        return []
+
+    parsed = []
+    for raw_prefix in prefixes.split(","):
+        prefix = raw_prefix.strip().lower()
+        if not prefix:
+            continue
+        if len(prefix) != 1 or prefix not in _HEX_PREFIXES:
+            raise click.UsageError("--tenant-prefixes must be a comma-separated list of hex digits, e.g. 0,1,a,f.")
+        parsed.append(prefix)
+    return sorted(set(parsed))
+
+
+def _get_archive_candidate_tenant_ids_by_prefix(
+    prefix: str,
+    *,
+    start_from: datetime.datetime | None,
+    end_before: datetime.datetime,
+) -> list[str]:
+    from graphon.enums import WorkflowExecutionStatus
+    from models.workflow import WorkflowRun
+    from services.retention.workflow_run.archive_paid_plan_workflow_run import WorkflowRunArchiver
+
+    conditions = [
+        WorkflowRun.created_at < end_before,
+        WorkflowRun.status.in_(WorkflowExecutionStatus.ended_values()),
+        WorkflowRun.type.in_(WorkflowRunArchiver.ARCHIVED_TYPE),
+        tenant_prefix_condition(WorkflowRun.tenant_id, prefix),
+    ]
+    if start_from is not None:
+        conditions.append(WorkflowRun.created_at >= start_from)
+
+    tenant_ids = db.session.scalars(
+        sa.select(WorkflowRun.tenant_id).where(*conditions).distinct().order_by(WorkflowRun.tenant_id)
+    ).all()
+    return list(tenant_ids)
+
+
+def _filter_paid_workflow_archive_tenant_ids(tenant_ids: list[str]) -> tuple[list[str], list[str]]:
+    from configs import dify_config
+    from enums.cloud_plan import CloudPlan
+    from services.billing_service import BillingService
+
+    tenant_ids = sorted(set(tenant_ids))
+    if not tenant_ids:
+        return [], []
+    if not dify_config.BILLING_ENABLED:
+        return tenant_ids, []
+
+    plans = BillingService.get_plan_bulk_with_cache(tenant_ids)
+    paid_tenant_ids = [
+        tenant_id
+        for tenant_id in tenant_ids
+        if plans.get(tenant_id) and plans[tenant_id].get("plan") in (CloudPlan.PROFESSIONAL, CloudPlan.TEAM)
+    ]
+    unpaid_tenant_ids = sorted(set(tenant_ids) - set(paid_tenant_ids))
+    return paid_tenant_ids, unpaid_tenant_ids
+
+
+def _resolve_archive_tenant_ids_from_plan(
+    *,
+    tenant_ids: str | None,
+    tenant_prefixes: list[str],
+    start_from: datetime.datetime | None,
+    end_before: datetime.datetime,
+) -> WorkflowRunArchiveTenantPlan:
+    """
+    Resolve the archive tenant scope once before scanning workflow_runs.
+
+    Prefix rollout should use the tenant list collected by the same planning path, then archive by
+    tenant_id IN (...). Scanning workflow_runs with a tenant prefix range in every archive run is too expensive on
+    the large production table this command is meant to shrink.
+    """
+    if tenant_ids:
+        requested_tenant_ids = [tid.strip() for tid in tenant_ids.split(",") if tid.strip()]
+    elif tenant_prefixes:
+        requested_tenant_ids = []
+        for prefix in tenant_prefixes:
+            requested_tenant_ids.extend(
+                _get_archive_candidate_tenant_ids_by_prefix(
+                    prefix,
+                    start_from=start_from,
+                    end_before=end_before,
+                )
+            )
+    else:
+        return WorkflowRunArchiveTenantPlan(
+            archive_tenant_ids=None,
+            paid_tenant_ids=[],
+            unpaid_tenant_ids=[],
+        )
+
+    paid_tenant_ids, unpaid_tenant_ids = _filter_paid_workflow_archive_tenant_ids(requested_tenant_ids)
+    return WorkflowRunArchiveTenantPlan(
+        archive_tenant_ids=paid_tenant_ids,
+        paid_tenant_ids=paid_tenant_ids,
+        unpaid_tenant_ids=unpaid_tenant_ids,
+    )
+
+
+def _resolve_archive_time_range(
+    *,
+    before_days: int,
+    from_days_ago: int | None,
+    to_days_ago: int | None,
+    start_from: datetime.datetime | None,
+    end_before: datetime.datetime | None,
+) -> tuple[int, datetime.datetime | None, datetime.datetime | None]:
+    if (start_from is None) ^ (end_before is None):
+        raise click.UsageError("--start-from and --end-before must be provided together.")
+
+    if (from_days_ago is None) ^ (to_days_ago is None):
+        raise click.UsageError("--from-days-ago and --to-days-ago must be provided together.")
+
+    if from_days_ago is not None and to_days_ago is not None:
+        if start_from or end_before:
+            raise click.UsageError("Choose either day offsets or explicit dates, not both.")
+        if from_days_ago <= to_days_ago:
+            raise click.UsageError("--from-days-ago must be greater than --to-days-ago.")
+        now = datetime.datetime.now(datetime.UTC)
+        start_from = now - datetime.timedelta(days=from_days_ago)
+        end_before = now - datetime.timedelta(days=to_days_ago)
+        before_days = 0
+
+    if start_from is not None:
+        start_from = _normalize_utc_datetime(start_from)
+    if end_before is not None:
+        end_before = _normalize_utc_datetime(end_before)
+
+    if start_from and end_before and start_from >= end_before:
+        raise click.UsageError("--start-from must be earlier than --end-before.")
+
+    return before_days, start_from, end_before
+
 
 @click.command("clear-free-plan-tenant-expired-logs", help="Clear free plan tenant expired logs.")
 @click.option("--days", prompt=True, help="The days to clear free plan tenant expired logs.", default=30)
@@ -139,11 +300,150 @@ def clean_workflow_runs(
     )
 
 
+@click.command(
+    "archive-workflow-runs-plan",
+    help="Plan workflow run archive rollout by tenant ID first hex digit.",
+)
+@click.option("--before-days", default=90, show_default=True, help="Plan runs older than N days.")
+@click.option(
+    "--from-days-ago",
+    default=None,
+    type=click.IntRange(min=0),
+    help="Lower bound in days ago (older). Must be paired with --to-days-ago.",
+)
+@click.option(
+    "--to-days-ago",
+    default=None,
+    type=click.IntRange(min=0),
+    help="Upper bound in days ago (newer). Must be paired with --from-days-ago.",
+)
+@click.option(
+    "--start-from",
+    type=click.DateTime(formats=["%Y-%m-%d", "%Y-%m-%dT%H:%M:%S"]),
+    default=None,
+    help="Plan runs created at or after this timestamp (UTC if no timezone).",
+)
+@click.option(
+    "--end-before",
+    type=click.DateTime(formats=["%Y-%m-%d", "%Y-%m-%dT%H:%M:%S"]),
+    default=None,
+    help="Plan runs created before this timestamp (UTC if no timezone).",
+)
+@click.option(
+    "--include-archived",
+    is_flag=True,
+    help="Compatibility no-op for V2 bundle archive; plan counts source rows in the requested window.",
+)
+def archive_workflow_runs_plan(
+    before_days: int,
+    from_days_ago: int | None,
+    to_days_ago: int | None,
+    start_from: datetime.datetime | None,
+    end_before: datetime.datetime | None,
+    include_archived: bool,
+):
+    """
+    Print the 16 tenant-prefix rollout rows used to choose archive execution order.
+
+    Counts use the same workflow run eligibility as archive-workflow-runs: ended runs,
+    supported workflow types, and the requested created_at window. V2 bundle archive
+    does not maintain per-run archive logs, so this plan reports source-table volume.
+    """
+    from graphon.enums import WorkflowExecutionStatus
+    from models.workflow import WorkflowNodeExecutionModel, WorkflowRun
+    from services.retention.workflow_run.archive_paid_plan_workflow_run import WorkflowRunArchiver
+
+    before_days, start_from, end_before = _resolve_archive_time_range(
+        before_days=before_days,
+        from_days_ago=from_days_ago,
+        to_days_ago=to_days_ago,
+        start_from=start_from,
+        end_before=end_before,
+    )
+    plan_end_before = end_before or datetime.datetime.now(datetime.UTC) - datetime.timedelta(days=before_days)
+    if include_archived:
+        click.echo(click.style("--include-archived is a no-op for V2 bundle archive plans.", fg="yellow"))
+
+    rows: list[WorkflowRunArchivePlanRow] = []
+    for prefix in _HEX_PREFIXES:
+        tenant_ids = _get_archive_candidate_tenant_ids_by_prefix(
+            prefix,
+            start_from=start_from,
+            end_before=plan_end_before,
+        )
+        total_tenants = len(tenant_ids)
+        paid_tenant_ids, unpaid_tenant_ids = _filter_paid_workflow_archive_tenant_ids(tenant_ids)
+
+        run_conditions = [
+            WorkflowRun.created_at < plan_end_before,
+            WorkflowRun.status.in_(WorkflowExecutionStatus.ended_values()),
+            WorkflowRun.type.in_(WorkflowRunArchiver.ARCHIVED_TYPE),
+            tenant_prefix_condition(WorkflowRun.tenant_id, prefix),
+        ]
+        if start_from is not None:
+            run_conditions.append(WorkflowRun.created_at >= start_from)
+        workflow_runs = (
+            db.session.scalar(sa.select(sa.func.count()).select_from(WorkflowRun).where(*run_conditions)) or 0
+        )
+        candidate_runs = sa.select(WorkflowRun.id).where(*run_conditions).subquery()
+        workflow_node_executions = (
+            db.session.scalar(
+                sa.select(sa.func.count())
+                .select_from(WorkflowNodeExecutionModel)
+                .join(candidate_runs, WorkflowNodeExecutionModel.workflow_run_id == candidate_runs.c.id)
+            )
+            or 0
+        )
+
+        rows.append(
+            WorkflowRunArchivePlanRow(
+                tenant_prefix=prefix,
+                total_tenants=total_tenants,
+                workflow_runs=workflow_runs,
+                workflow_node_executions=workflow_node_executions,
+                paid_tenants=len(paid_tenant_ids),
+                unpaid_tenants=len(unpaid_tenant_ids),
+            )
+        )
+
+    click.echo(
+        click.style(
+            f"Workflow archive plan for runs before {plan_end_before.isoformat()}"
+            f"{f' and at/after {start_from.isoformat()}' if start_from else ''}.",
+            fg="white",
+        )
+    )
+    click.echo(
+        click.style(
+            "fixed_archive_window="
+            f"{start_from.isoformat() if start_from else 'unbounded'},{plan_end_before.isoformat()}",
+            fg="white",
+        )
+    )
+    click.echo("tenant_prefix,total_tenants,workflow_runs,workflow_node_executions,paid_tenants,unpaid_tenants")
+    for row in rows:
+        click.echo(
+            f"{row['tenant_prefix']},{row['total_tenants']},{row['workflow_runs']},"
+            f"{row['workflow_node_executions']},{row['paid_tenants']},{row['unpaid_tenants']}"
+        )
+
+    ordered_rows = sorted(
+        rows,
+        key=lambda row: (row["workflow_runs"] + row["workflow_node_executions"], row["tenant_prefix"]),
+    )
+    click.echo("suggested_execution_order=" + ",".join(row["tenant_prefix"] for row in ordered_rows))
+
+
 @click.command(
     "archive-workflow-runs",
     help="Archive workflow runs for paid plan tenants to S3-compatible storage.",
 )
 @click.option("--tenant-ids", default=None, help="Optional comma-separated tenant IDs for grayscale rollout.")
+@click.option(
+    "--tenant-prefixes",
+    default=None,
+    help="Optional comma-separated tenant ID first hex digits for rollout waves, e.g. 0,1,a,f.",
+)
 @click.option("--before-days", default=90, show_default=True, help="Archive runs older than N days.")
 @click.option(
     "--from-days-ago",
@@ -169,13 +469,36 @@ def clean_workflow_runs(
     default=None,
     help="Archive runs created before this timestamp (UTC if no timezone).",
 )
-@click.option("--batch-size", default=100, show_default=True, help="Batch size for processing.")
-@click.option("--workers", default=1, show_default=True, type=int, help="Concurrent workflow runs to archive.")
+@click.option("--batch-size", default=10000, show_default=True, help="Maximum workflow runs per archive bundle.")
+@click.option(
+    "--workers",
+    default=1,
+    show_default=True,
+    type=int,
+    help="Reserved; bundle archive currently runs serially.",
+)
+@click.option(
+    "--run-shard-index",
+    default=None,
+    type=click.IntRange(min=0),
+    help="Zero-based workflow run shard index for parallel cron jobs. Must be paired with --run-shard-total.",
+)
+@click.option(
+    "--run-shard-total",
+    default=None,
+    type=click.IntRange(min=1, max=16),
+    help="Total workflow run shard count for parallel cron jobs. Must be paired with --run-shard-index.",
+)
 @click.option("--limit", default=None, type=int, help="Maximum number of runs to archive.")
 @click.option("--dry-run", is_flag=True, help="Preview without archiving.")
-@click.option("--delete-after-archive", is_flag=True, help="Delete runs and related data after archiving.")
+@click.option(
+    "--delete-after-archive",
+    is_flag=True,
+    help="Not supported by bundle archive; use a separate bundle delete workflow after validation.",
+)
 def archive_workflow_runs(
     tenant_ids: str | None,
+    tenant_prefixes: str | None,
     before_days: int,
     from_days_ago: int | None,
     to_days_ago: int | None,
@@ -183,6 +506,8 @@ def archive_workflow_runs(
     end_before: datetime.datetime | None,
     batch_size: int,
     workers: int,
+    run_shard_index: int | None,
+    run_shard_total: int | None,
     limit: int | None,
     dry_run: bool,
     delete_after_archive: bool,
@@ -190,14 +515,19 @@ def archive_workflow_runs(
     """
     Archive workflow runs for paid plan tenants older than the specified days.
 
-    This command archives the following tables to storage:
+    This command writes V2 tenant/month/shard archive bundles. Each bundle contains Parquet snapshots from:
+    - workflow_runs
+    - workflow_app_logs
     - workflow_node_executions
     - workflow_node_execution_offload
     - workflow_pauses
     - workflow_pause_reasons
     - workflow_trigger_logs
 
-    The workflow_runs and workflow_app_logs tables are preserved for UI listing.
+    Source database rows are always preserved by archive. Deletion must be handled by
+    a separate bundle-level delete workflow after manifest, checksum, row-count, and
+    restore-sampling validation. In --dry-run mode, no storage or database writes
+    happen; the command estimates per-table Parquet bytes and object size instead.
     """
     from services.retention.workflow_run.archive_paid_plan_workflow_run import WorkflowRunArchiver
 
@@ -209,32 +539,67 @@ def archive_workflow_runs(
         )
     )
 
-    if (start_from is None) ^ (end_before is None):
-        click.echo(click.style("start-from and end-before must be provided together.", fg="red"))
-        return
-
-    if (from_days_ago is None) ^ (to_days_ago is None):
-        click.echo(click.style("from-days-ago and to-days-ago must be provided together.", fg="red"))
-        return
-
-    if from_days_ago is not None and to_days_ago is not None:
-        if start_from or end_before:
-            click.echo(click.style("Choose either day offsets or explicit dates, not both.", fg="red"))
-            return
-        if from_days_ago <= to_days_ago:
-            click.echo(click.style("from-days-ago must be greater than to-days-ago.", fg="red"))
-            return
-        now = datetime.datetime.now()
-        start_from = now - datetime.timedelta(days=from_days_ago)
-        end_before = now - datetime.timedelta(days=to_days_ago)
-        before_days = 0
-
-    if start_from and end_before and start_from >= end_before:
-        click.echo(click.style("start-from must be earlier than end-before.", fg="red"))
+    uses_relative_window = start_from is None and end_before is None
+    try:
+        before_days, start_from, end_before = _resolve_archive_time_range(
+            before_days=before_days,
+            from_days_ago=from_days_ago,
+            to_days_ago=to_days_ago,
+            start_from=start_from,
+            end_before=end_before,
+        )
+        parsed_tenant_prefixes = _parse_tenant_prefixes(tenant_prefixes)
+    except click.UsageError as e:
+        click.echo(click.style(e.message, fg="red"))
         return
+    plan_end_before = end_before or datetime.datetime.now(datetime.UTC) - datetime.timedelta(days=before_days)
     if workers < 1:
         click.echo(click.style("workers must be at least 1.", fg="red"))
         return
+    if (run_shard_index is None) ^ (run_shard_total is None):
+        click.echo(click.style("run-shard-index and run-shard-total must be provided together.", fg="red"))
+        return
+    if run_shard_index is not None and run_shard_total is not None and run_shard_index >= run_shard_total:
+        click.echo(click.style("run-shard-index must be less than run-shard-total.", fg="red"))
+        return
+    if delete_after_archive:
+        click.echo(click.style("delete-after-archive is not supported by bundle archive.", fg="red"))
+        return
+    if uses_relative_window:
+        click.echo(
+            click.style(
+                "Relative archive windows are evaluated at command start. For multi-day prefix/shard rollout, "
+                "reuse absolute --start-from/--end-before values from archive-workflow-runs-plan.",
+                fg="yellow",
+            )
+        )
+
+    try:
+        tenant_plan = _resolve_archive_tenant_ids_from_plan(
+            tenant_ids=tenant_ids,
+            tenant_prefixes=parsed_tenant_prefixes,
+            start_from=start_from,
+            end_before=plan_end_before,
+        )
+    except Exception:
+        logger.exception("Failed to resolve workflow archive tenant plan")
+        click.echo(click.style("Failed to resolve workflow archive tenant plan.", fg="red"))
+        return
+
+    planned_tenant_ids = tenant_plan["archive_tenant_ids"]
+    planned_paid_tenant_ids = tenant_plan["paid_tenant_ids"] if planned_tenant_ids is not None else None
+    paid_tenants = len(tenant_plan["paid_tenant_ids"])
+    unpaid_tenants = len(tenant_plan["unpaid_tenant_ids"])
+    if planned_tenant_ids is not None:
+        click.echo(
+            click.style(
+                f"Resolved archive tenant plan: paid_tenants={paid_tenants}, unpaid_tenants={unpaid_tenants}.",
+                fg="white",
+            )
+        )
+        if not planned_tenant_ids:
+            click.echo(click.style("No paid tenants matched the archive plan; nothing to archive.", fg="yellow"))
+            return
 
     archiver = WorkflowRunArchiver(
         days=before_days,
@@ -242,7 +607,11 @@ def archive_workflow_runs(
         start_from=start_from,
         end_before=end_before,
         workers=workers,
-        tenant_ids=[tid.strip() for tid in tenant_ids.split(",")] if tenant_ids else None,
+        tenant_ids=planned_tenant_ids,
+        tenant_prefixes=parsed_tenant_prefixes,
+        paid_tenant_ids=planned_paid_tenant_ids,
+        run_shard_index=run_shard_index,
+        run_shard_total=run_shard_total,
         limit=limit,
         dry_run=dry_run,
         delete_after_archive=delete_after_archive,
@@ -252,7 +621,9 @@ def archive_workflow_runs(
         click.style(
             f"Summary: processed={summary.total_runs_processed}, archived={summary.runs_archived}, "
             f"skipped={summary.runs_skipped}, failed={summary.runs_failed}, "
-            f"time={summary.total_elapsed_time:.2f}s",
+            f"bundles_archived={summary.bundles_archived}, bundles_skipped={summary.bundles_skipped}, "
+            f"bundles_failed={summary.bundles_failed}, "
+            f"object_size_bytes={summary.total_object_size_bytes}, time={summary.total_elapsed_time:.2f}s",
             fg="cyan",
         )
     )
@@ -268,6 +639,52 @@ def archive_workflow_runs(
     )
 
 
+def _echo_bundle_archive_operation_summary(summary) -> None:
+    status = "completed successfully" if summary.bundles_failed == 0 else "completed with failures"
+    fg = "green" if summary.bundles_failed == 0 else "red"
+    click.echo(
+        click.style(
+            f"{summary.operation} {status}. "
+            f"bundles_success={summary.bundles_succeeded} bundles_failed={summary.bundles_failed} "
+            f"runs={summary.runs_processed} rows={summary.rows_processed} "
+            f"archive_bytes={summary.archive_bytes} duration={summary.elapsed_time:.2f}s "
+            f"validation_time={summary.validation_time:.2f}s "
+            f"runs_per_second={summary.runs_per_second:.2f} rows_per_second={summary.rows_per_second:.2f} "
+            f"bytes_per_second={summary.bytes_per_second:.2f}",
+            fg=fg,
+        )
+    )
+    click.echo(click.style("table,row_count", fg="white"))
+    for table_name in [
+        "workflow_runs",
+        "workflow_app_logs",
+        "workflow_node_executions",
+        "workflow_node_execution_offload",
+        "workflow_pauses",
+        "workflow_pause_reasons",
+        "workflow_trigger_logs",
+    ]:
+        click.echo(f"{table_name},{summary.table_counts.get(table_name, 0)}")
+    for result in summary.results:
+        if result.success:
+            click.echo(
+                click.style(
+                    f"  bundle={result.bundle_id} tenant={result.tenant_id} runs={result.run_count} "
+                    f"rows={result.row_count} archive_bytes={result.archive_bytes} "
+                    f"time={result.elapsed_time:.2f}s validation={result.validation_time:.2f}s",
+                    fg="white",
+                )
+            )
+        else:
+            click.echo(
+                click.style(
+                    f"  failed bundle={result.bundle_id} tenant={result.tenant_id} "
+                    f"object_prefix={result.object_prefix} error={result.error}",
+                    fg="red",
+                )
+            )
+
+
 @click.command(
     "restore-workflow-runs",
     help="Restore archived workflow runs from S3-compatible storage.",
@@ -290,8 +707,8 @@ def archive_workflow_runs(
     default=None,
     help="Optional upper bound (exclusive) for created_at; must be paired with --start-from.",
 )
-@click.option("--workers", default=1, show_default=True, type=int, help="Concurrent workflow runs to restore.")
-@click.option("--limit", type=int, default=100, show_default=True, help="Maximum number of runs to restore.")
+@click.option("--workers", default=1, show_default=True, type=int, help="V1 --run-id compatibility only.")
+@click.option("--limit", type=int, default=100, show_default=True, help="Maximum number of V2 bundles to restore.")
 @click.option("--dry-run", is_flag=True, help="Preview without restoring.")
 def restore_workflow_runs(
     tenant_ids: str | None,
@@ -303,15 +720,18 @@ def restore_workflow_runs(
     dry_run: bool,
 ):
     """
-    Restore an archived workflow run from storage to the database.
+    Restore archived workflow runs from storage to the database.
 
-    This restores the following tables:
+    Batch restore uses V2 bundle metadata and validates archive objects before writing source rows. This restores:
+    - workflow_runs
+    - workflow_app_logs
     - workflow_node_executions
     - workflow_node_execution_offload
     - workflow_pauses
     - workflow_pause_reasons
     - workflow_trigger_logs
     """
+    from services.retention.workflow_run.bundle_archive_maintenance import WorkflowRunBundleArchiveMaintenance
     from services.retention.workflow_run.restore_archived_workflow_run import WorkflowRunRestore
 
     parsed_tenant_ids = None
@@ -335,39 +755,46 @@ def restore_workflow_runs(
         )
     )
 
-    restorer = WorkflowRunRestore(dry_run=dry_run, workers=workers)
     if run_id:
+        restorer = WorkflowRunRestore(dry_run=dry_run, workers=workers)
         results = [restorer.restore_by_run_id(run_id)]
-    else:
-        assert start_from is not None
-        assert end_before is not None
-        results = restorer.restore_batch(
-            parsed_tenant_ids,
-            start_date=start_from,
-            end_date=end_before,
-            limit=limit,
-        )
+        end_time = datetime.datetime.now(datetime.UTC)
+        elapsed = end_time - start_time
 
-    end_time = datetime.datetime.now(datetime.UTC)
-    elapsed = end_time - start_time
+        successes = sum(1 for result in results if result.success)
+        failures = len(results) - successes
 
-    successes = sum(1 for result in results if result.success)
-    failures = len(results) - successes
-
-    if failures == 0:
-        click.echo(
-            click.style(
-                f"Restore completed successfully. success={successes} duration={elapsed}",
-                fg="green",
+        if failures == 0:
+            click.echo(
+                click.style(
+                    f"Restore completed successfully. success={successes} duration={elapsed}",
+                    fg="green",
+                )
             )
-        )
-    else:
-        click.echo(
-            click.style(
-                f"Restore completed with failures. success={successes} failed={failures} duration={elapsed}",
-                fg="red",
+        else:
+            click.echo(
+                click.style(
+                    f"Restore completed with failures. success={successes} failed={failures} duration={elapsed}",
+                    fg="red",
+                )
             )
+        return
+
+    if workers != 1:
+        click.echo(
+            click.style("--workers is ignored for V2 bundle restore; bundles are processed serially.", fg="yellow")
         )
+    assert start_from is not None
+    assert end_before is not None
+    bundle_restorer = WorkflowRunBundleArchiveMaintenance(dry_run=dry_run, strict_content_validation=True)
+    summary = bundle_restorer.restore_batch(
+        tenant_ids=parsed_tenant_ids,
+        start_date=start_from,
+        end_date=end_before,
+        limit=limit,
+    )
+    _echo_bundle_archive_operation_summary(summary)
+    return
 
 
 @click.command(
@@ -392,8 +819,20 @@ def restore_workflow_runs(
     default=None,
     help="Optional upper bound (exclusive) for created_at; must be paired with --start-from.",
 )
-@click.option("--limit", type=int, default=100, show_default=True, help="Maximum number of runs to delete.")
+@click.option("--limit", type=int, default=100, show_default=True, help="Maximum number of V2 bundles to delete.")
 @click.option("--dry-run", is_flag=True, help="Preview without deleting.")
+@click.option(
+    "--skip-bad-archives",
+    is_flag=True,
+    help="Continue batch deletion when one archive object fails validation.",
+)
+@click.option(
+    "--restore-sample-interval",
+    type=int,
+    default=0,
+    show_default=True,
+    help="Run restore dry-run after every N successful deletes; 0 disables restore sampling.",
+)
 def delete_archived_workflow_runs(
     tenant_ids: str | None,
     run_id: str | None,
@@ -401,10 +840,16 @@ def delete_archived_workflow_runs(
     end_before: datetime.datetime | None,
     limit: int,
     dry_run: bool,
+    skip_bad_archives: bool,
+    restore_sample_interval: int,
 ):
     """
     Delete archived workflow runs from the database.
+
+    Batch delete uses V2 bundle metadata and validates object existence, manifest schema, object size, checksum, row
+    counts, and source/archive content checksums before deleting source rows. `--run-id` keeps the V1 per-run path.
     """
+    from services.retention.workflow_run.bundle_archive_maintenance import WorkflowRunBundleArchiveMaintenance
     from services.retention.workflow_run.delete_archived_workflow_run import ArchivedWorkflowRunDeletion
 
     parsed_tenant_ids = None
@@ -417,6 +862,8 @@ def delete_archived_workflow_runs(
         raise click.UsageError("--start-from and --end-before must be provided together.")
     if run_id is None and (start_from is None or end_before is None):
         raise click.UsageError("--start-from and --end-before are required for batch delete.")
+    if restore_sample_interval < 0:
+        raise click.BadParameter("restore-sample-interval must be >= 0")
 
     start_time = datetime.datetime.now(datetime.UTC)
     target_desc = f"workflow run {run_id}" if run_id else "workflow runs"
@@ -427,56 +874,85 @@ def delete_archived_workflow_runs(
         )
     )
 
-    deleter = ArchivedWorkflowRunDeletion(dry_run=dry_run)
     if run_id:
-        results = [deleter.delete_by_run_id(run_id)]
-    else:
-        assert start_from is not None
-        assert end_before is not None
-        results = deleter.delete_batch(
-            parsed_tenant_ids,
-            start_date=start_from,
-            end_date=end_before,
-            limit=limit,
+        deleter = ArchivedWorkflowRunDeletion(
+            dry_run=dry_run,
+            skip_bad_archives=skip_bad_archives,
+            restore_sample_interval=restore_sample_interval,
         )
+        results = [deleter.delete_by_run_id(run_id)]
+        for result in results:
+            if result.success:
+                click.echo(
+                    click.style(
+                        f"{'[DRY RUN] Would delete' if dry_run else 'Deleted'} "
+                        f"workflow run {result.run_id} (tenant={result.tenant_id}, "
+                        f"archive_key={result.archive_key}, counts={result.validated_counts})",
+                        fg="green",
+                    )
+                )
+                if result.restore_sampled:
+                    sample_status = "passed" if result.restore_sample_success else "failed"
+                    click.echo(
+                        click.style(
+                            f"  restore dry-run sample {sample_status} for workflow run {result.run_id}",
+                            fg="green" if result.restore_sample_success else "red",
+                        )
+                    )
+            else:
+                click.echo(
+                    click.style(
+                        f"Failed to delete workflow run {result.run_id}: {result.error}",
+                        fg="red",
+                    )
+                )
+                click.echo(
+                    click.style(
+                        "  runbook: pause this delete window, verify archive storage object and manifest/checksum, "
+                        "retry the same run after fixing storage or DB drift, or rerun with --skip-bad-archives "
+                        "to quarantine this run and continue the batch.",
+                        fg="yellow",
+                    )
+                )
 
-    for result in results:
-        if result.success:
+        end_time = datetime.datetime.now(datetime.UTC)
+        elapsed = end_time - start_time
+
+        successes = sum(1 for result in results if result.success)
+        failures = len(results) - successes
+
+        if failures == 0:
             click.echo(
                 click.style(
-                    f"{'[DRY RUN] Would delete' if dry_run else 'Deleted'} "
-                    f"workflow run {result.run_id} (tenant={result.tenant_id})",
+                    f"Delete completed successfully. success={successes} duration={elapsed}",
                     fg="green",
                 )
             )
         else:
             click.echo(
                 click.style(
-                    f"Failed to delete workflow run {result.run_id}: {result.error}",
+                    f"Delete completed with failures. success={successes} failed={failures} duration={elapsed}",
                     fg="red",
                 )
             )
+        return
 
-    end_time = datetime.datetime.now(datetime.UTC)
-    elapsed = end_time - start_time
-
-    successes = sum(1 for result in results if result.success)
-    failures = len(results) - successes
-
-    if failures == 0:
-        click.echo(
-            click.style(
-                f"Delete completed successfully. success={successes} duration={elapsed}",
-                fg="green",
-            )
-        )
-    else:
-        click.echo(
-            click.style(
-                f"Delete completed with failures. success={successes} failed={failures} duration={elapsed}",
-                fg="red",
-            )
-        )
+    if restore_sample_interval:
+        click.echo(click.style("--restore-sample-interval is ignored for V2 bundle delete.", fg="yellow"))
+    assert start_from is not None
+    assert end_before is not None
+    bundle_deleter = WorkflowRunBundleArchiveMaintenance(
+        dry_run=dry_run,
+        strict_content_validation=True,
+        stop_on_error=not skip_bad_archives,
+    )
+    summary = bundle_deleter.delete_batch(
+        tenant_ids=parsed_tenant_ids,
+        start_date=start_from,
+        end_date=end_before,
+        limit=limit,
+    )
+    _echo_bundle_archive_operation_summary(summary)
 
 
 def _find_orphaned_draft_variables(batch_size: int = 1000) -> list[str]:

api/commands/vector.py

@@ -13,6 +13,7 @@ from core.rag.index_processor.constant.built_in_field import BuiltInField
 from core.rag.index_processor.constant.index_type import IndexStructureType, IndexTechniqueType
 from core.rag.models.document import ChildDocument, Document
 from extensions.ext_database import db
+from libs.pagination import paginate_query
 from models.dataset import Dataset, DatasetCollectionBinding, DatasetMetadata, DatasetMetadataBinding, DocumentSegment
 from models.dataset import Document as DatasetDocument
 from models.enums import DatasetMetadataType, IndexingStatus, SegmentStatus
@@ -183,7 +184,7 @@ def migrate_knowledge_vector_database():
                 .order_by(Dataset.created_at.desc())
             )
 
-            datasets = db.paginate(select=stmt, page=page, per_page=50, max_per_page=50, error_out=False)
+            datasets = paginate_query(stmt, page=page, per_page=50, max_per_page=50)
             if not datasets.items:
                 break
         except SQLAlchemyError:
@@ -409,7 +410,7 @@ def old_metadata_migration():
                 .where(DatasetDocument.doc_metadata.is_not(None))
                 .order_by(DatasetDocument.created_at.desc())
             )
-            documents = db.paginate(select=stmt, page=page, per_page=50, max_per_page=50, error_out=False)
+            documents = paginate_query(stmt, page=page, per_page=50, max_per_page=50)
         except SQLAlchemyError:
             raise
         if not documents:

api/configs/enterprise/__init__.py

@@ -16,7 +16,7 @@ class EnterpriseFeatureConfig(BaseSettings):
 
     CAN_REPLACE_LOGO: bool = Field(
         description="Allow customization of the enterprise logo.",
-        default=True,
+        default=False,
     )
 
     ENTERPRISE_REQUEST_TIMEOUT: int = Field(
@@ -34,6 +34,12 @@ class EnterpriseFeatureConfig(BaseSettings):
         default=False,
     )
 
+    ENTERPRISE_RBAC_REQUEST_TIMEOUT: int = Field(
+        ge=1,
+        description="Maximum timeout in seconds for inner RBAC requests.",
+        default=30,
+    )
+
 
 class EnterpriseTelemetryConfig(BaseSettings):
     """

api/configs/extra/agent_backend_config.py

@@ -1,4 +1,4 @@
-from pydantic import Field
+from pydantic import Field, NonNegativeFloat
 from pydantic_settings import BaseSettings
 
 
@@ -32,12 +32,10 @@ class AgentBackendConfig(BaseSettings):
         default=False,
     )
 
-    AGENT_DRIVE_MANIFEST_ENABLED: bool = Field(
+    AGENT_APP_TEXT_DELTA_DEBOUNCE_SECONDS: NonNegativeFloat = Field(
         description=(
-            "Inject the dify.drive layer (Skills & Files drive manifest declaration) "
-            "into Agent runs. The declaration is an index only — the agent backend "
-            "pulls the actual SKILL.md / files through the back proxy. Keep it off "
-            "until the agent backend registers the dify.drive layer type."
+            "Buffer Agent App assistant text deltas for up to this many seconds before "
+            "publishing SSE chunks. Set to 0 to publish each delta immediately."
         ),
-        default=False,
+        default=0.5,
     )

api/configs/feature/__init__.py

@@ -1073,6 +1073,12 @@ class MailConfig(BaseSettings):
         default=None,
     )
 
+
+class HomepageConfig(BaseSettings):
+    """
+    Configuration for homepage feature toggles exposed through system features.
+    """
+
     ENABLE_TRIAL_APP: bool = Field(
         description="Enable trial app",
         default=False,
@@ -1083,6 +1089,11 @@ class MailConfig(BaseSettings):
         default=False,
     )
 
+    ENABLE_LEARN_APP: bool = Field(
+        description="Enable Learn App",
+        default=True,
+    )
+
 
 class RagEtlConfig(BaseSettings):
     """
@@ -1489,6 +1500,7 @@ class FeatureConfig(
     EndpointConfig,
     FileAccessConfig,
     FileUploadConfig,
+    HomepageConfig,
     HttpConfig,
     InnerAPIConfig,
     IndexingConfig,

api/configs/middleware/cache/redis_pubsub_config.py

@@ -2,7 +2,6 @@ from typing import Literal, Protocol, cast
 from urllib.parse import quote_plus, urlunparse
 
 from pydantic import AliasChoices, Field
-from pydantic.types import NonNegativeInt
 from pydantic_settings import BaseSettings
 
 
@@ -71,24 +70,6 @@ class RedisPubSubConfig(BaseSettings):
         default=600,
     )
 
-    PUBSUB_LISTENER_JOIN_TIMEOUT_MS: NonNegativeInt = Field(
-        validation_alias=AliasChoices("EVENT_BUS_LISTENER_JOIN_TIMEOUT_MS", "PUBSUB_LISTENER_JOIN_TIMEOUT_MS"),
-        description=(
-            "Maximum time (milliseconds) that ``Subscription.close()`` waits for its listener thread to "
-            "finish before returning. Bounds the tail latency between a terminal event being delivered to "
-            "an SSE client and the response stream actually closing.\n\n"
-            "The listener thread blocks on a polling read (XREAD BLOCK for streams, get_message timeout "
-            "for pubsub/sharded) with a fixed 1s window, so close() naturally has to wait up to ~1s for "
-            "the thread to notice the subscription was closed. Setting this lower (e.g. 100) lets close() "
-            "return promptly while the daemon listener thread cleans itself up on the next poll "
-            "boundary - safe because the listener holds no critical state and exits within one poll "
-            "window. Setting it higher (e.g. 5000) gives the listener more grace before close() gives up "
-            "and logs a warning. Default 2000ms preserves the pre-change behaviour.\n\n"
-            "Also accepts ENV: EVENT_BUS_LISTENER_JOIN_TIMEOUT_MS."
-        ),
-        default=2000,
-    )
-
     def _build_default_pubsub_url(self) -> str:
         defaults = _redis_defaults(self)
         if not defaults.REDIS_HOST or not defaults.REDIS_PORT:

api/controllers/common/agent_app_parameters.py

@@ -0,0 +1,54 @@
+from typing import Any
+
+from sqlalchemy import select
+
+from core.app.apps.agent_app.app_feature_projection import merge_agent_app_features
+from core.app.apps.agent_app.app_variable_projection import agent_app_variables_to_user_input_form
+from core.app.apps.agent_app.errors import AgentAppGeneratorError, AgentAppNotPublishedError
+from extensions.ext_database import db
+from models.agent import Agent, AgentConfigSnapshot, AgentStatus
+from models.agent_config_entities import AgentSoulConfig
+from models.model import App
+
+
+def get_published_agent_app_feature_dict_and_user_input_form(
+    app_model: App,
+) -> tuple[dict[str, Any], list[dict[str, Any]]]:
+    """Return public Agent App parameters backed by the published Agent Soul."""
+    app_model_config = app_model.app_model_config
+
+    agent_id = app_model.bound_agent_id
+    if not agent_id:
+        raise AgentAppGeneratorError("Agent App has no bound Agent")
+
+    agent = db.session.scalar(
+        select(Agent)
+        .where(
+            Agent.tenant_id == app_model.tenant_id,
+            Agent.id == agent_id,
+            Agent.status == AgentStatus.ACTIVE,
+        )
+        .limit(1)
+    )
+    if agent is None:
+        raise AgentAppGeneratorError("Agent App has no bound Agent")
+    # active_config_is_published means the draft has no unpublished edits; the public app
+    # can still read parameters from the active snapshot while a newer draft is pending.
+    if not agent.active_config_snapshot_id:
+        raise AgentAppNotPublishedError("Agent has not been published")
+
+    snapshot = db.session.scalar(
+        select(AgentConfigSnapshot)
+        .where(
+            AgentConfigSnapshot.tenant_id == app_model.tenant_id,
+            AgentConfigSnapshot.agent_id == agent.id,
+            AgentConfigSnapshot.id == agent.active_config_snapshot_id,
+        )
+        .limit(1)
+    )
+    if snapshot is None:
+        raise AgentAppGeneratorError("Agent published version not found")
+
+    agent_soul = AgentSoulConfig.model_validate(snapshot.config_snapshot_dict)
+    features_dict = merge_agent_app_features(agent_soul=agent_soul, app_model_config=app_model_config)
+    return features_dict, agent_app_variables_to_user_input_form(agent_soul.app_variables)

api/controllers/common/app_access.py

@@ -0,0 +1,108 @@
+from __future__ import annotations
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+from typing import TYPE_CHECKING
+
+from extensions.ext_database import db
+from services.enterprise import rbac_service as enterprise_rbac_service
+
+if TYPE_CHECKING:
+    from services.app_service import AppListBaseParams
+    from services.enterprise.rbac_service import MyPermissionsResponse
+
+# Permission keys (dot-notation, from MyPermissionsResponse) that grant
+# list/preview access to an app. Keep this the single source of truth for both
+# the console and OpenAPI app-list endpoints.
+APP_LIST_PERMISSION_KEYS: frozenset[str] = frozenset({"app.preview", "app.acl.preview", "app.full_access"})
+
+# Workspace permission key that lets a caller see apps they maintain even when
+# those apps are not in their preview whitelist.
+_MANAGE_OWN_APPS_PERMISSION_KEY = "app.create_and_management"
+
+
+def has_app_list_permission(permission_keys: Sequence[str]) -> bool:
+    """Return True if any of ``permission_keys`` grants app list/preview access."""
+    return any(permission_key in APP_LIST_PERMISSION_KEYS for permission_key in permission_keys)
+
+
+@dataclass(frozen=True)
+class AppAccessFilter:
+    """Resolved RBAC visibility for app list/read endpoints.
+
+    ``accessible_app_ids`` of ``None`` means the caller can see every app in the
+    workspace (unrestricted). Otherwise it is the exact set of app ids the
+    caller may preview; combined with ``can_manage_own_apps`` it also covers
+    apps the caller maintains.
+    """
+
+    accessible_app_ids: set[str] | None
+    can_manage_own_apps: bool
+
+    @classmethod
+    def unrestricted(cls) -> AppAccessFilter:
+        """Filter that imposes no restriction (RBAC disabled / not applicable)."""
+        return cls(accessible_app_ids=None, can_manage_own_apps=False)
+
+    def is_app_accessible(self, app_id: str, maintainer: str | None, account_id: str) -> bool:
+        """Whether a single app is visible to the caller under this filter.
+
+        Mirrors the service-layer query gate: an app is visible when the filter
+        is unrestricted, the app id is whitelisted, or the caller maintains it
+        and holds ``app.create_and_management``.
+        """
+        if self.accessible_app_ids is None:
+            return True
+        if app_id in self.accessible_app_ids:
+            return True
+        return self.can_manage_own_apps and maintainer is not None and maintainer == account_id
+
+    def apply_to_params(self, params: AppListBaseParams) -> None:
+        if self.accessible_app_ids is None:
+            return
+        params.accessible_app_ids = sorted(self.accessible_app_ids)
+        params.include_own_apps = self.can_manage_own_apps
+
+
+def resolve_app_access_filter(
+    tenant_id: str,
+    account_id: str,
+    *,
+    permissions: MyPermissionsResponse | None = None,
+) -> AppAccessFilter:
+    """Compute the RBAC app-access filter for ``account_id`` in ``tenant_id``.
+
+    Pass ``permissions`` when the caller has already fetched the snapshot (the
+    console controller reuses it for per-app permission keys) to avoid a second
+    inner-API round trip; otherwise it is fetched here.
+    """
+    if permissions is None:
+        permissions = enterprise_rbac_service.RBACService.MyPermissions.get(tenant_id, account_id, session=db.session())
+    whitelist_scope = enterprise_rbac_service.RBACService.AppAccess.whitelist_resources(tenant_id, account_id)
+
+    can_manage_own_apps = _MANAGE_OWN_APPS_PERMISSION_KEY in permissions.workspace.permission_keys
+    has_default_preview = has_app_list_permission(permissions.app.default_permission_keys) or has_app_list_permission(
+        permissions.workspace.permission_keys
+    )
+
+    permission_app_ids: set[str] | None = None
+    if not has_default_preview:
+        # Collect apps the caller can preview via per-app permission overrides.
+        permission_app_ids = {
+            override.resource_id
+            for override in permissions.app.overrides
+            if has_app_list_permission(override.permission_keys)
+        }
+
+    accessible_app_ids: set[str] | None
+    if getattr(whitelist_scope, "unrestricted", False):
+        accessible_app_ids = permission_app_ids
+    else:
+        accessible_app_ids = set(whitelist_scope.resource_ids)
+        if permission_app_ids is not None:
+            accessible_app_ids |= permission_app_ids
+        elif has_default_preview:
+            # Default preview overrides the whitelist restriction.
+            accessible_app_ids = None
+
+    return AppAccessFilter(accessible_app_ids=accessible_app_ids, can_manage_own_apps=can_manage_own_apps)

api/controllers/common/fields.py

@@ -183,6 +183,7 @@ class Site(BaseModel):
     description: str | None = None
     copyright: str | None = None
     privacy_policy: str | None = None
+    input_placeholder: str | None = None
     custom_disclaimer: str | None = None
     default_language: str
     show_workflow_steps: bool

api/controllers/common/session.py

@@ -0,0 +1,57 @@
+"""Controller session decorators.
+
+`with_session` is an HTTP controller helper: it opens one SQLAlchemy session
+for a Resource handler and injects it as the first argument after `self`.
+Handlers use a transaction by default so migrated write paths keep
+commit/rollback handling; pure read handlers may opt out with `write=False`.
+"""
+
+from collections.abc import Callable
+from functools import wraps
+from typing import Concatenate, overload
+
+from sqlalchemy.orm import Session
+
+from core.db.session_factory import session_factory
+
+
+@overload
+def with_session[T, **P, R](
+    view: Callable[Concatenate[T, Session, P], R],
+    *,
+    write: bool = True,
+) -> Callable[Concatenate[T, P], R]: ...
+
+
+@overload
+def with_session[T, **P, R](
+    view: None = None,
+    *,
+    write: bool = True,
+) -> Callable[[Callable[Concatenate[T, Session, P], R]], Callable[Concatenate[T, P], R]]: ...
+
+
+def with_session[T, **P, R](
+    view: Callable[Concatenate[T, Session, P], R] | None = None,
+    *,
+    write: bool = True,
+) -> (
+    Callable[Concatenate[T, P], R] | Callable[[Callable[Concatenate[T, Session, P], R]], Callable[Concatenate[T, P], R]]
+):
+    """Inject a request-scoped session, using a transaction only for write handlers."""
+
+    def decorator(view: Callable[Concatenate[T, Session, P], R]) -> Callable[Concatenate[T, P], R]:
+        @wraps(view)
+        def wrapper(self: T, *args: P.args, **kwargs: P.kwargs) -> R:
+            if write:
+                with session_factory.get_session_maker().begin() as session:
+                    return view(self, session, *args, **kwargs)
+
+            with session_factory.create_session() as session:
+                return view(self, session, *args, **kwargs)
+
+        return wrapper
+
+    if view is None:
+        return decorator
+    return decorator(view)

api/controllers/common/wraps.py

@@ -1,23 +1,3 @@
-"""Shared decorator utilities for Dify controller layers.
-
-This module provides decorators that are not tied to any single API group (e.g.
-console, inner, service).  Currently it exposes the RBAC permission gate, which
-can be applied to any blueprint.
-
-Key exports
------------
-``rbac_permission_required`` – decorator that enforces enterprise RBAC access
-    control.  When ``RBAC_ENABLED`` is ``False`` it is a no-op.
-
-``RBACPermission``, ``RBACResourceScope`` – re-exported from ``core.rbac`` so
-    callers only need a single import site.
-
-Private helpers
----------------
-``_extract_resource_id``, ``_is_resource_owned_by_current_user`` – kept module-
-    private but accessible via the module namespace for unit-test patching.
-"""
-
 from collections.abc import Callable
 from functools import wraps
 
@@ -32,7 +12,57 @@ from models.dataset import Dataset
 from models.model import App
 from services.enterprise.rbac_service import RBACService
 
-__all__ = ["RBACPermission", "RBACResourceScope", "rbac_permission_required"]
+__all__ = ["RBACPermission", "RBACResourceScope", "enforce_rbac_access", "rbac_permission_required"]
+
+
+def enforce_rbac_access(
+    *,
+    tenant_id: str,
+    account_id: str,
+    resource_type: RBACResourceScope,
+    scene: RBACPermission,
+    resource_required: bool = True,
+    path_args: dict[str, object] | None = None,
+) -> None:
+    """Enforce enterprise RBAC for an explicit account/tenant pair.
+
+    This is the flask-login-independent core of the RBAC gate so it can run
+    inside request-handling layers that resolve the caller themselves (e.g. the
+    openapi auth pipeline, which has the account on ``AuthData`` before
+    flask-login is mounted).
+
+    No-op when ``RBAC_ENABLED`` is ``False``. For resource-scoped checks the
+    resource ID is taken from ``path_args`` merged with ``request.view_args``;
+    resource ownership short-circuits the check. Raises ``Forbidden`` when
+    access is denied. For workspace-level checks pass ``resource_required=False``
+    so the RBAC request omits ``resource_id``.
+
+    Args:
+        tenant_id: The tenant the access is evaluated against.
+        account_id: The account requesting access.
+        resource_type: The :class:`RBACResourceScope` member (app/dataset/workspace).
+        scene: The :class:`RBACPermission` permission point, e.g. ``RBACPermission.APP_DELETE``.
+        resource_required: Whether a concrete resource ID is required.
+        path_args: Extra path arguments to merge with ``request.view_args``.
+    """
+    if not dify_config.RBAC_ENABLED:
+        return
+
+    check_resource_type = None if resource_type == RBACResourceScope.WORKSPACE else resource_type
+    resource_id = None
+    if resource_required and check_resource_type:
+        resource_id = _extract_resource_id(resource_type, path_args)
+        if _is_resource_owned_by_current_user(tenant_id, account_id, resource_type, resource_id):
+            return
+    allowed = RBACService.CheckAccess.check(
+        tenant_id,
+        account_id,
+        scene=scene,
+        resource_type=check_resource_type,
+        resource_id=resource_id,
+    )
+    if not allowed:
+        raise Forbidden()
 
 
 def rbac_permission_required[**P, R](
@@ -41,14 +71,12 @@ def rbac_permission_required[**P, R](
     *,
     resource_required: bool = True,
 ) -> Callable[[Callable[P, R]], Callable[P, R]]:
-    """Check enterprise RBAC permissions for the current user.
+    """Check enterprise RBAC permissions for the current flask-login user.
 
     When ``RBAC_ENABLED`` is ``False`` the decorator is a no-op and the
-    request passes through unchanged. When enabled it extracts the resource ID
-    from ``request.view_args`` for resource-scoped checks, calls the RBAC
-    service ``check-access`` endpoint, and raises ``Forbidden`` if the access
-    is denied. For workspace-level checks, set ``resource_required=False`` so
-    the RBAC request omits ``resource_id``.
+    request passes through unchanged. When enabled it resolves the current
+    account/tenant and delegates to :func:`enforce_rbac_access`, raising
+    ``Forbidden`` if access is denied.
 
     Args:
         resource_type: The :class:`RBACResourceScope` member (app/dataset/workspace).
@@ -63,23 +91,14 @@ def rbac_permission_required[**P, R](
                 return view(*args, **kwargs)
 
             current_user, current_tenant_id = current_account_with_tenant()
-            check_resource_type = None if resource_type == RBACResourceScope.WORKSPACE else resource_type
-            resource_id = None
-            if resource_required and check_resource_type:
-                resource_id = _extract_resource_id(resource_type, kwargs)
-                if _is_resource_owned_by_current_user(current_tenant_id, current_user.id, resource_type, resource_id):
-                    return view(*args, **kwargs)
-            allowed = RBACService.CheckAccess.check(
-                current_tenant_id,
-                current_user.id,
+            enforce_rbac_access(
+                tenant_id=current_tenant_id,
+                account_id=current_user.id,
+                resource_type=resource_type,
                 scene=scene,
-                resource_type=check_resource_type,
-                resource_id=resource_id,
+                resource_required=resource_required,
+                path_args=kwargs,
             )
-
-            if not allowed:
-                raise Forbidden()
-
             return view(*args, **kwargs)
 
         return decorated

api/controllers/console/__init__.py

@@ -54,6 +54,7 @@ from .app import (
     agent_app_access,
     agent_app_feature,
     agent_app_sandbox,
+    agent_config_inspector,
     agent_drive_inspector,
     annotation,
     app,
@@ -157,6 +158,7 @@ __all__ = [
     "agent_app_feature",
     "agent_app_sandbox",
     "agent_composer",
+    "agent_config_inspector",
     "agent_drive_inspector",
     "agent_providers",
     "agent_roster",

api/controllers/console/agent/app_helpers.py

@@ -6,5 +6,15 @@ from services.agent.roster_service import AgentRosterService
 
 
 def resolve_agent_app_model(*, tenant_id: str, agent_id: UUID) -> App:
-    """Resolve the hidden Agent App backing an Agent Console resource."""
+    """Resolve a roster Agent's public Agent App."""
     return AgentRosterService(db.session).get_agent_app_model(tenant_id=tenant_id, agent_id=str(agent_id))
+
+
+def resolve_agent_runtime_app_model(*, tenant_id: str, agent_id: UUID) -> App:
+    """Resolve the App that backs an Agent runtime surface.
+
+    This accepts both roster Agent Apps and workflow-only inline Agents with a
+    hidden backing App.
+    """
+
+    return AgentRosterService(db.session).get_agent_runtime_app_model(tenant_id=tenant_id, agent_id=str(agent_id))

api/controllers/console/agent/composer.py

@@ -1,10 +1,10 @@
 from uuid import UUID
 
+from flask import request
 from flask_restx import Resource
 
-from controllers.common.schema import register_response_schema_models, register_schema_models
+from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models
 from controllers.console import console_ns
-from controllers.console.agent.app_helpers import resolve_agent_app_model
 from controllers.console.app.wraps import get_app_model
 from controllers.console.wraps import (
     RBACPermission,
@@ -16,6 +16,7 @@ from controllers.console.wraps import (
     with_current_tenant_id,
     with_current_user_id,
 )
+from extensions.ext_database import db
 from fields.agent_fields import (
     AgentAppComposerResponse,
     AgentComposerCandidatesResponse,
@@ -28,9 +29,15 @@ from libs.login import login_required
 from models.model import App, AppMode
 from services.agent.composer_service import AgentComposerService
 from services.agent.composer_validator import ComposerConfigValidator
-from services.entities.agent_entities import ComposerSavePayload
+from services.entities.agent_entities import (
+    ComposerSavePayload,
+    WorkflowAgentComposerQuery,
+    WorkflowComposerCopyFromRosterPayload,
+)
 
-register_schema_models(console_ns, ComposerSavePayload)
+register_schema_models(
+    console_ns, ComposerSavePayload, WorkflowAgentComposerQuery, WorkflowComposerCopyFromRosterPayload
+)
 register_response_schema_models(
     console_ns,
     AgentAppComposerResponse,
@@ -41,27 +48,29 @@ register_response_schema_models(
 )
 
 
-def _resolve_agent_app_id(*, tenant_id: str, agent_id: UUID) -> str:
-    return resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id).id
-
-
 @console_ns.route("/apps/<uuid:app_id>/workflows/draft/nodes/<string:node_id>/agent-composer")
 class WorkflowAgentComposerApi(Resource):
     @console_ns.response(
         200, "Workflow agent composer state", console_ns.models[WorkflowAgentComposerResponse.__name__]
     )
+    @console_ns.doc(params=query_params_from_model(WorkflowAgentComposerQuery))
     @setup_required
     @login_required
     @account_initialization_required
     @get_app_model(mode=[AppMode.WORKFLOW, AppMode.ADVANCED_CHAT])
+    @with_current_user_id
     @with_current_tenant_id
-    def get(self, tenant_id: str, app_model: App, node_id: str):
+    def get(self, tenant_id: str, account_id: str, app_model: App, node_id: str):
+        query = WorkflowAgentComposerQuery.model_validate(request.args.to_dict(flat=True))
         return dump_response(
             WorkflowAgentComposerResponse,
             AgentComposerService.load_workflow_composer(
                 tenant_id=tenant_id,
                 app_id=app_model.id,
                 node_id=node_id,
+                account_id=account_id,
+                snapshot_id=query.snapshot_id,
+                session=db.session(),
             ),
         )
 
@@ -87,6 +96,40 @@ class WorkflowAgentComposerApi(Resource):
                 node_id=node_id,
                 account_id=account_id,
                 payload=payload,
+                session=db.session(),
+            ),
+        )
+
+
+@console_ns.route("/apps/<uuid:app_id>/workflows/draft/nodes/<string:node_id>/agent-composer/copy-from-roster")
+class WorkflowAgentComposerCopyFromRosterApi(Resource):
+    @console_ns.expect(console_ns.models[WorkflowComposerCopyFromRosterPayload.__name__])
+    @console_ns.response(
+        200,
+        "Workflow roster agent copied to inline agent",
+        console_ns.models[WorkflowAgentComposerResponse.__name__],
+    )
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @edit_permission_required
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_EDIT)
+    @get_app_model(mode=[AppMode.WORKFLOW, AppMode.ADVANCED_CHAT])
+    @with_current_user_id
+    @with_current_tenant_id
+    def post(self, tenant_id: str, account_id: str, app_model: App, node_id: str):
+        payload = WorkflowComposerCopyFromRosterPayload.model_validate(console_ns.payload or {})
+        return dump_response(
+            WorkflowAgentComposerResponse,
+            AgentComposerService.copy_workflow_composer_from_roster(
+                tenant_id=tenant_id,
+                app_id=app_model.id,
+                node_id=node_id,
+                account_id=account_id,
+                source_agent_id=payload.source_agent_id,
+                source_snapshot_id=payload.source_snapshot_id,
+                idempotency_key=payload.idempotency_key,
+                session=db.session(),
             ),
         )
 
@@ -104,13 +147,15 @@ class WorkflowAgentComposerValidateApi(Resource):
     @with_current_tenant_id
     def post(self, tenant_id: str, app_model: App, node_id: str):
         payload = ComposerSavePayload.model_validate(console_ns.payload or {})
-        ComposerConfigValidator.validate_save_payload(payload)
+        ComposerConfigValidator.validate_publish_payload(payload)
+        AgentComposerService.validate_knowledge_datasets(tenant_id=tenant_id, agent_soul=payload.agent_soul)
         findings = AgentComposerService.collect_validation_findings(
             tenant_id=tenant_id,
             payload=payload,
             agent_id=AgentComposerService.resolve_workflow_node_agent_id(
-                tenant_id=tenant_id, app_id=app_model.id, node_id=node_id
+                tenant_id=tenant_id, app_id=app_model.id, node_id=node_id, session=db.session()
             ),
+            session=db.session(),
         )
         return dump_response(AgentComposerValidateResponse, {"result": "success", "errors": [], **findings})
 
@@ -134,6 +179,7 @@ class WorkflowAgentComposerCandidatesApi(Resource):
                 app_id=app_model.id,
                 node_id=node_id,
                 user_id=current_user_id,
+                session=db.session(),
             ),
         )
 
@@ -156,7 +202,9 @@ class WorkflowAgentComposerImpactApi(Resource):
             )
         return dump_response(
             AgentComposerImpactResponse,
-            AgentComposerService.calculate_impact(tenant_id=tenant_id, current_snapshot_id=current_snapshot_id),
+            AgentComposerService.calculate_impact(
+                tenant_id=tenant_id, current_snapshot_id=current_snapshot_id, session=db.session()
+            ),
         )
 
 
@@ -184,6 +232,7 @@ class WorkflowAgentComposerSaveToRosterApi(Resource):
                 node_id=node_id,
                 account_id=account_id,
                 payload=payload,
+                session=db.session(),
             ),
         )
 
@@ -196,10 +245,9 @@ class AgentComposerApi(Resource):
     @account_initialization_required
     @with_current_tenant_id
     def get(self, tenant_id: str, agent_id: UUID):
-        app_id = _resolve_agent_app_id(tenant_id=tenant_id, agent_id=agent_id)
         return dump_response(
             AgentAppComposerResponse,
-            AgentComposerService.load_agent_app_composer(tenant_id=tenant_id, app_id=app_id),
+            AgentComposerService.load_agent_composer(tenant_id=tenant_id, agent_id=str(agent_id), session=db.session()),
         )
 
     @console_ns.expect(console_ns.models[ComposerSavePayload.__name__])
@@ -212,15 +260,15 @@ class AgentComposerApi(Resource):
     @with_current_user_id
     @with_current_tenant_id
     def put(self, tenant_id: str, account_id: str, agent_id: UUID):
-        app_id = _resolve_agent_app_id(tenant_id=tenant_id, agent_id=agent_id)
         payload = ComposerSavePayload.model_validate(console_ns.payload or {})
         return dump_response(
             AgentAppComposerResponse,
-            AgentComposerService.save_agent_app_composer(
+            AgentComposerService.save_agent_composer(
                 tenant_id=tenant_id,
-                app_id=app_id,
+                agent_id=str(agent_id),
                 account_id=account_id,
                 payload=payload,
+                session=db.session(),
             ),
         )
 
@@ -236,13 +284,15 @@ class AgentComposerValidateApi(Resource):
     @account_initialization_required
     @with_current_tenant_id
     def post(self, tenant_id: str, agent_id: UUID):
-        _resolve_agent_app_id(tenant_id=tenant_id, agent_id=agent_id)
+        AgentComposerService.load_agent_composer(tenant_id=tenant_id, agent_id=str(agent_id), session=db.session())
         payload = ComposerSavePayload.model_validate(console_ns.payload or {})
-        ComposerConfigValidator.validate_save_payload(payload)
+        ComposerConfigValidator.validate_publish_payload(payload)
+        AgentComposerService.validate_knowledge_datasets(tenant_id=tenant_id, agent_soul=payload.agent_soul)
         findings = AgentComposerService.collect_validation_findings(
             tenant_id=tenant_id,
             payload=payload,
             agent_id=str(agent_id),
+            session=db.session(),
         )
         return dump_response(AgentComposerValidateResponse, {"result": "success", "errors": [], **findings})
 
@@ -258,12 +308,12 @@ class AgentComposerCandidatesApi(Resource):
     @with_current_user_id
     @with_current_tenant_id
     def get(self, tenant_id: str, current_user_id: str, agent_id: UUID):
-        app_id = _resolve_agent_app_id(tenant_id=tenant_id, agent_id=agent_id)
         return dump_response(
             AgentComposerCandidatesResponse,
             AgentComposerService.get_agent_app_candidates(
                 tenant_id=tenant_id,
-                app_id=app_id,
+                agent_id=str(agent_id),
                 user_id=current_user_id,
+                session=db.session(),
             ),
         )

api/controllers/console/agent/roster.py

@@ -3,17 +3,23 @@ from uuid import UUID
 from flask import abort, request
 from flask_restx import Resource
 from pydantic import AliasChoices, BaseModel, Field, field_validator
+from sqlalchemy import func, select
 
-from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models
+from controllers.common.schema import (
+    query_params_from_model,
+    query_params_from_request,
+    register_response_schema_models,
+    register_schema_models,
+)
 from controllers.console import console_ns
-from controllers.console.agent.app_helpers import resolve_agent_app_model
+from controllers.console.agent.app_helpers import resolve_agent_app_model, resolve_agent_runtime_app_model
+from controllers.console.apikey import ApiKeyItem, ApiKeyList, BaseApiKeyListResource, BaseApiKeyResource
 from controllers.console.app.app import (
-    AppDetailWithSite as GenericAppDetailWithSite,
+    APP_LIST_QUERY_ARRAY_FIELDS,
+    AppListQuery,
 )
 from controllers.console.app.app import (
-    AppListQuery,
-    CopyAppPayload,
-    _normalize_app_list_query_args,
+    AppDetailWithSite as GenericAppDetailWithSite,
 )
 from controllers.console.app.app import (
     AppPagination as GenericAppPagination,
@@ -25,17 +31,24 @@ from controllers.console.app.app import (
     UpdateAppPayload as GenericUpdateAppPayload,
 )
 from controllers.console.wraps import (
+    RBACPermission,
+    RBACResourceScope,
     account_initialization_required,
     edit_permission_required,
     enterprise_license_required,
+    is_admin_or_owner_required,
+    rbac_permission_required,
     setup_required,
     with_current_tenant_id,
     with_current_user,
 )
 from extensions.ext_database import db
 from fields.agent_fields import (
+    AgentConfigDraftSummaryResponse,
     AgentConfigSnapshotDetailResponse,
     AgentConfigSnapshotListResponse,
+    AgentConfigSnapshotRestoreResponse,
+    AgentConfigSnapshotSummaryResponse,
     AgentInviteOptionsResponse,
     AgentLogListResponse,
     AgentLogMessageListResponse,
@@ -44,11 +57,16 @@ from fields.agent_fields import (
     AgentRosterListResponse,
     AgentStatisticSummaryEnvelopeResponse,
 )
+from fields.base import ResponseModel
 from libs.datetime_utils import parse_time_range
 from libs.helper import dump_response
 from libs.login import login_required
 from models import Account
-from models.model import IconType
+from models.agent import Agent, AgentStatus
+from models.agent_config_entities import AgentSoulConfig
+from models.enums import ApiTokenType
+from models.model import ApiToken, App, IconType
+from services.agent.composer_service import AgentComposerService
 from services.agent.errors import AgentNotFoundError
 from services.agent.observability_service import (
     AgentLogQueryParams,
@@ -58,7 +76,7 @@ from services.agent.observability_service import (
 from services.agent.roster_service import AgentRosterService
 from services.app_service import AppListParams, AppService, CreateAppParams
 from services.enterprise.enterprise_service import EnterpriseService
-from services.entities.agent_entities import RosterListQuery
+from services.entities.agent_entities import ComposerSavePayload, RosterListQuery
 from services.feature_service import FeatureService
 
 
@@ -73,33 +91,68 @@ class AgentIdPath(BaseModel):
 class AgentAppCreatePayload(BaseModel):
     name: str = Field(..., min_length=1, description="Agent name")
     description: str | None = Field(default=None, description="Agent description (max 400 chars)", max_length=400)
-    role: str = Field(..., min_length=1, description="Agent role", max_length=255)
+    role: str | None = Field(default=None, description="Agent role", max_length=255)
     icon_type: IconType | None = Field(default=None, description="Icon type")
     icon: str | None = Field(default=None, description="Icon")
     icon_background: str | None = Field(default=None, description="Icon background color")
 
     @field_validator("role")
     @classmethod
-    def validate_role(cls, value: str) -> str:
-        role = value.strip()
-        if not role:
-            raise ValueError("Agent role is required.")
-        return role
+    def validate_role(cls, value: str | None) -> str | None:
+        if value is None:
+            return None
+        return value.strip()
 
 
 # Keep agent-app roster DTOs agent-specific instead of reusing the shared
 # /apps response/request models. The roster surface needs Agent-only fields such
 # as `role`, while the generic console/apps contracts must stay unchanged.
 class AgentAppUpdatePayload(GenericUpdateAppPayload):
-    role: str = Field(..., min_length=1, description="Agent role", max_length=255)
+    role: str | None = Field(default=None, description="Agent role", max_length=255)
 
     @field_validator("role")
     @classmethod
-    def validate_role(cls, value: str) -> str:
-        role = value.strip()
-        if not role:
-            raise ValueError("Agent role is required.")
-        return role
+    def validate_role(cls, value: str | None) -> str | None:
+        if value is None:
+            return None
+        return value.strip()
+
+
+class AgentAppCopyPayload(BaseModel):
+    name: str | None = Field(default=None, description="Name for the copied agent")
+    description: str | None = Field(default=None, description="Description for the copied agent", max_length=400)
+    role: str | None = Field(default=None, description="Role for the copied agent", max_length=255)
+    icon_type: IconType | None = Field(default=None, description="Icon type")
+    icon: str | None = Field(default=None, description="Icon")
+    icon_background: str | None = Field(default=None, description="Icon background color")
+
+    @field_validator("role")
+    @classmethod
+    def validate_role(cls, value: str | None) -> str | None:
+        if value is None:
+            return None
+        return value.strip()
+
+
+class AgentApiStatusPayload(BaseModel):
+    enable_api: bool = Field(..., description="Enable or disable Agent service API")
+
+
+class AgentApiAccessResponse(BaseModel):
+    enabled: bool
+    service_api_base_url: str
+    streaming_only: bool = True
+    chat_endpoint: str
+    stop_endpoint: str
+    conversations_endpoint: str
+    messages_endpoint: str
+    files_upload_endpoint: str
+    parameters_endpoint: str
+    info_endpoint: str
+    meta_endpoint: str
+    api_rpm: int
+    api_rph: int
+    api_key_count: int
 
 
 class AgentAppPublishedReferenceResponse(BaseModel):
@@ -144,11 +197,9 @@ class AgentLogsQuery(BaseModel):
     def empty_list_values_to_list(cls, value: object) -> list[str]:
         if value in (None, ""):
             return []
-        if isinstance(value, str):
-            return [value]
         if isinstance(value, list):
-            return [item for item in value if item]
-        return []
+            return [str(item).strip() for item in value if str(item).strip()]
+        raise ValueError("Unsupported query list type.")
 
     @field_validator("sort_by")
     @classmethod
@@ -185,6 +236,9 @@ class AgentStatisticsQuery(BaseModel):
 
 class AgentAppPartial(GenericAppPartial):
     app_id: str | None = None
+    backing_app_id: str | None = None
+    hidden_app_backed: bool = False
+    debug_conversation_id: str | None = None
     role: str | None = None
     active_config_is_published: bool = False
     published_reference_count: int = 0
@@ -193,10 +247,51 @@ class AgentAppPartial(GenericAppPartial):
 
 class AgentAppDetailWithSite(GenericAppDetailWithSite):
     app_id: str | None = None
+    backing_app_id: str | None = None
+    hidden_app_backed: bool = False
+    debug_conversation_id: str | None = None
+    debug_conversation_has_messages: bool = False
+    debug_conversation_message_count: int = 0
     role: str | None = None
     active_config_is_published: bool = False
 
 
+class AgentDebugConversationRefreshResponse(BaseModel):
+    debug_conversation_id: str
+    debug_conversation_has_messages: bool = False
+    debug_conversation_message_count: int = 0
+
+
+class AgentPublishPayload(BaseModel):
+    version_note: str | None = Field(default=None, description="Optional note for this published Agent version")
+
+
+class AgentPublishResponse(ResponseModel):
+    result: str
+    active_config_snapshot_id: str
+    active_config_snapshot: AgentConfigSnapshotSummaryResponse | None = None
+    draft: AgentConfigDraftSummaryResponse | None = None
+
+
+class AgentBuildDraftCheckoutPayload(BaseModel):
+    force: bool = Field(default=False, description="Overwrite the existing current-user build draft")
+
+
+class AgentBuildDraftResponse(ResponseModel):
+    variant: str
+    draft: AgentConfigDraftSummaryResponse
+    agent_soul: AgentSoulConfig
+
+
+class AgentBuildDraftApplyResponse(BaseModel):
+    result: str
+    draft: dict[str, object]
+
+
+class AgentSimpleResultResponse(BaseModel):
+    result: str
+
+
 class AgentAppPagination(GenericAppPagination):
     data: list[AgentAppPartial] = Field(  # type: ignore[assignment]  # pyrefly: ignore[bad-override-mutable-attribute]
         validation_alias=AliasChoices("items", "data")
@@ -207,7 +302,11 @@ register_schema_models(
     console_ns,
     AgentAppCreatePayload,
     AgentAppUpdatePayload,
-    CopyAppPayload,
+    AgentAppCopyPayload,
+    AgentPublishPayload,
+    AgentBuildDraftCheckoutPayload,
+    ComposerSavePayload,
+    AgentApiStatusPayload,
     AgentInviteOptionsQuery,
     AgentLogsQuery,
     AgentStatisticsQuery,
@@ -218,11 +317,18 @@ register_schema_models(
 register_response_schema_models(
     console_ns,
     AgentAppPagination,
+    AgentApiAccessResponse,
     AgentAppPublishedReferenceResponse,
     AgentAppDetailWithSite,
     AgentAppPartial,
+    AgentDebugConversationRefreshResponse,
+    AgentPublishResponse,
+    AgentBuildDraftResponse,
+    AgentBuildDraftApplyResponse,
+    AgentSimpleResultResponse,
     AgentConfigSnapshotDetailResponse,
     AgentConfigSnapshotListResponse,
+    AgentConfigSnapshotRestoreResponse,
     AgentInviteOptionsResponse,
     AgentLogListResponse,
     AgentLogMessageListResponse,
@@ -237,7 +343,7 @@ def _agent_roster_service() -> AgentRosterService:
     return AgentRosterService(db.session)
 
 
-def _serialize_agent_app_detail(app_model) -> dict:
+def _serialize_agent_app_detail(app_model, *, current_user: Account, agent_id: str | None = None) -> dict:
     """Serialize an Agent App detail using roster-only DTOs.
 
     `/agent` responses are roster-shaped rather than raw app-shaped: `id`
@@ -254,12 +360,35 @@ def _serialize_agent_app_detail(app_model) -> dict:
 
     roster_service = _agent_roster_service()
     payload = AgentAppDetailWithSite.model_validate(app_model, from_attributes=True).model_dump(mode="json")
-    agent = roster_service.get_app_backing_agent(tenant_id=app_model.tenant_id, app_id=str(app_model.id))
+    agent = (
+        db.session.scalar(
+            select(Agent).where(
+                Agent.tenant_id == app_model.tenant_id,
+                Agent.id == agent_id,
+                Agent.status == AgentStatus.ACTIVE,
+            )
+        )
+        if agent_id
+        else roster_service.get_app_backing_agent(tenant_id=app_model.tenant_id, app_id=str(app_model.id))
+    )
     if not agent:
         raise AgentNotFoundError()
     payload.pop("bound_agent_id", None)
-    payload["app_id"] = str(app_model.id)
+    payload["app_id"] = agent.app_id
+    payload["backing_app_id"] = roster_service.runtime_backing_app_id(agent)
+    payload["hidden_app_backed"] = bool(agent.backing_app_id and agent.backing_app_id != agent.app_id)
     payload["id"] = agent.id
+    debug_conversation_id = roster_service.get_or_create_agent_app_debug_conversation_id(
+        tenant_id=app_model.tenant_id,
+        agent_id=agent.id,
+        account_id=current_user.id,
+    )
+    message_count = roster_service.count_agent_app_debug_conversation_messages(
+        conversation_id=debug_conversation_id,
+    )
+    payload["debug_conversation_id"] = debug_conversation_id
+    payload["debug_conversation_has_messages"] = message_count > 0
+    payload["debug_conversation_message_count"] = message_count
     payload["role"] = agent.role or ""
     payload["active_config_is_published"] = roster_service.active_config_is_published(
         tenant_id=app_model.tenant_id,
@@ -268,7 +397,7 @@ def _serialize_agent_app_detail(app_model) -> dict:
     return payload
 
 
-def _serialize_agent_app_pagination(app_pagination, *, tenant_id: str) -> dict:
+def _serialize_agent_app_pagination(app_pagination, *, tenant_id: str, current_user: Account) -> dict:
     """Serialize Agent App lists with roster-shaped items.
 
     Each item starts from the shared App list shape, then drops
@@ -291,6 +420,11 @@ def _serialize_agent_app_pagination(app_pagination, *, tenant_id: str) -> dict:
         tenant_id=tenant_id,
         agent_ids=[agent.id for agent in agents_by_app_id.values()],
     )
+    debug_conversation_ids_by_agent_id = roster_service.load_or_create_agent_app_debug_conversation_ids_by_agent_id(
+        tenant_id=tenant_id,
+        agents=list(agents_by_app_id.values()),
+        account_id=current_user.id,
+    )
     payload = AgentAppPagination.model_validate(app_pagination, from_attributes=True).model_dump(mode="json")
     for item in payload["data"]:
         app_id = item["id"]
@@ -298,7 +432,10 @@ def _serialize_agent_app_pagination(app_pagination, *, tenant_id: str) -> dict:
         agent = agents_by_app_id.get(app_id)
         if agent:
             item["app_id"] = app_id
+            item["backing_app_id"] = agent.backing_app_id or app_id
+            item["hidden_app_backed"] = False
             item["id"] = agent.id
+            item["debug_conversation_id"] = debug_conversation_ids_by_agent_id.get(agent.id)
             item["role"] = agent.role or ""
             item["active_config_is_published"] = active_config_is_published_by_agent_id.get(agent.id, False)
             published_references = published_references_by_agent_id.get(agent.id, [])
@@ -323,6 +460,38 @@ def _resolve_agent_app_model(*, tenant_id: str, agent_id: UUID):
     return resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
 
 
+def _agent_api_key_count(app_id: str) -> int:
+    return (
+        db.session.scalar(
+            select(func.count(ApiToken.id)).where(
+                ApiToken.type == ApiTokenType.APP,
+                ApiToken.app_id == app_id,
+            )
+        )
+        or 0
+    )
+
+
+def _serialize_agent_api_access(app_model: App) -> dict:
+    base_url = app_model.api_base_url
+    response = AgentApiAccessResponse(
+        enabled=bool(app_model.enable_api),
+        service_api_base_url=base_url,
+        chat_endpoint=f"{base_url}/chat-messages",
+        stop_endpoint=f"{base_url}/chat-messages/{{task_id}}/stop",
+        conversations_endpoint=f"{base_url}/conversations",
+        messages_endpoint=f"{base_url}/messages",
+        files_upload_endpoint=f"{base_url}/files/upload",
+        parameters_endpoint=f"{base_url}/parameters",
+        info_endpoint=f"{base_url}/info",
+        meta_endpoint=f"{base_url}/meta",
+        api_rpm=app_model.api_rpm or 0,
+        api_rph=app_model.api_rph or 0,
+        api_key_count=_agent_api_key_count(str(app_model.id)),
+    )
+    return response.model_dump(mode="json")
+
+
 def _agent_observability_service() -> AgentObservabilityService:
     return AgentObservabilityService(db.session)
 
@@ -335,16 +504,11 @@ def _parse_observability_time_range(start: str | None, end: str | None, account:
         abort(400, description=str(exc))
 
 
-def _multi_query_values(name: str, legacy_name: str | None = None) -> list[str]:
-    values: list[str] = []
-    for query_name in (name, f"{name}[]"):
-        values.extend(request.args.getlist(query_name))
-    if legacy_name:
-        values.extend(request.args.getlist(legacy_name))
-    parsed: list[str] = []
-    for value in values:
-        parsed.extend(item.strip() for item in value.split(",") if item.strip())
-    return parsed
+def _query_values(name: str, alias_name: str | None = None) -> list[str]:
+    values = request.args.getlist(name)
+    if alias_name:
+        values.extend(request.args.getlist(alias_name))
+    return [value.strip() for value in values if value.strip()]
 
 
 @console_ns.route("/agent")
@@ -357,11 +521,12 @@ class AgentAppListApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def get(self, current_tenant_id: str, current_user: Account):
-        args = AppListQuery.model_validate(_normalize_app_list_query_args(request.args))
+        args = query_params_from_request(AppListQuery, list_fields=APP_LIST_QUERY_ARRAY_FIELDS)
         params = AppListParams(
             page=args.page,
             limit=args.limit,
             mode="agent",
+            sort_by=args.sort_by,
             name=args.name,
             tag_ids=args.tag_ids,
             creator_ids=args.creator_ids,
@@ -369,12 +534,16 @@ class AgentAppListApi(Resource):
             status="normal",
         )
 
-        app_pagination = AppService().get_paginate_apps(current_user.id, current_tenant_id, params, db.session)
+        app_pagination = AppService().get_paginate_apps(current_user.id, current_tenant_id, params, db.session())
         if app_pagination is None:
             empty = AgentAppPagination(page=args.page, limit=args.limit, total=0, has_more=False, data=[])
             return empty.model_dump(mode="json")
 
-        return _serialize_agent_app_pagination(app_pagination, tenant_id=current_tenant_id)
+        return _serialize_agent_app_pagination(
+            app_pagination,
+            tenant_id=current_tenant_id,
+            current_user=current_user,
+        )
 
     @console_ns.expect(console_ns.models[AgentAppCreatePayload.__name__])
     @console_ns.response(201, "Agent app created successfully", console_ns.models[AgentAppDetailWithSite.__name__])
@@ -392,14 +561,14 @@ class AgentAppListApi(Resource):
             name=args.name,
             description=args.description,
             mode="agent",
-            agent_role=args.role,
+            agent_role=args.role or "",
             icon_type=args.icon_type,
             icon=args.icon,
             icon_background=args.icon_background,
         )
 
-        app = AppService().create_app(current_tenant_id, params, current_user)
-        return _serialize_agent_app_detail(app), 201
+        app = AppService().create_app(current_tenant_id, params, current_user, session=db.session())
+        return _serialize_agent_app_detail(app, current_user=current_user), 201
 
 
 @console_ns.route("/agent/<uuid:agent_id>")
@@ -409,10 +578,11 @@ class AgentAppApi(Resource):
     @login_required
     @account_initialization_required
     @enterprise_license_required
+    @with_current_user
     @with_current_tenant_id
-    def get(self, tenant_id: str, agent_id: UUID):
-        app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
-        return _serialize_agent_app_detail(app_model)
+    def get(self, tenant_id: str, current_user: Account, agent_id: UUID):
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        return _serialize_agent_app_detail(app_model, current_user=current_user, agent_id=str(agent_id))
 
     @console_ns.expect(console_ns.models[AgentAppUpdatePayload.__name__])
     @console_ns.response(200, "Agent app updated successfully", console_ns.models[AgentAppDetailWithSite.__name__])
@@ -422,8 +592,9 @@ class AgentAppApi(Resource):
     @login_required
     @account_initialization_required
     @edit_permission_required
+    @with_current_user
     @with_current_tenant_id
-    def put(self, tenant_id: str, agent_id: UUID):
+    def put(self, tenant_id: str, current_user: Account, agent_id: UUID):
         app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
         args = AgentAppUpdatePayload.model_validate(console_ns.payload)
         args_dict: AppService.ArgsDict = {
@@ -436,8 +607,8 @@ class AgentAppApi(Resource):
             "max_active_requests": args.max_active_requests or 0,
             "role": args.role,
         }
-        updated = AppService().update_app(app_model, args_dict)
-        return _serialize_agent_app_detail(updated)
+        updated = AppService().update_app(app_model, args_dict, session=db.session())
+        return _serialize_agent_app_detail(updated, current_user=current_user)
 
     @console_ns.response(204, "Agent app deleted successfully")
     @console_ns.response(403, "Insufficient permissions")
@@ -448,13 +619,152 @@ class AgentAppApi(Resource):
     @with_current_tenant_id
     def delete(self, tenant_id: str, agent_id: UUID):
         app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
-        AppService().delete_app(app_model)
+        AppService().delete_app(app_model, session=db.session())
         return "", 204
 
 
+@console_ns.route("/agent/<uuid:agent_id>/debug-conversation/refresh")
+class AgentDebugConversationRefreshApi(Resource):
+    @console_ns.response(
+        200,
+        "Agent debug conversation refreshed",
+        console_ns.models[AgentDebugConversationRefreshResponse.__name__],
+    )
+    @console_ns.response(403, "Insufficient permissions")
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @edit_permission_required
+    @with_current_user
+    @with_current_tenant_id
+    def post(self, tenant_id: str, current_user: Account, agent_id: UUID):
+        debug_conversation_id = _agent_roster_service().refresh_agent_app_debug_conversation_id(
+            tenant_id=tenant_id,
+            agent_id=str(agent_id),
+            account_id=current_user.id,
+        )
+        return AgentDebugConversationRefreshResponse(
+            debug_conversation_id=debug_conversation_id,
+            debug_conversation_has_messages=False,
+            debug_conversation_message_count=0,
+        ).model_dump(mode="json")
+
+
+@console_ns.route("/agent/<uuid:agent_id>/publish")
+class AgentPublishApi(Resource):
+    @console_ns.expect(console_ns.models[AgentPublishPayload.__name__])
+    @console_ns.response(200, "Agent draft published", console_ns.models[AgentPublishResponse.__name__])
+    @console_ns.response(403, "Insufficient permissions")
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @edit_permission_required
+    @with_current_user
+    @with_current_tenant_id
+    def post(self, tenant_id: str, current_user: Account, agent_id: UUID):
+        args = AgentPublishPayload.model_validate(console_ns.payload or {})
+        return AgentComposerService.publish_agent_app_draft(
+            tenant_id=tenant_id,
+            agent_id=str(agent_id),
+            account_id=current_user.id,
+            version_note=args.version_note,
+            session=db.session(),
+        )
+
+
+@console_ns.route("/agent/<uuid:agent_id>/build-draft/checkout")
+class AgentBuildDraftCheckoutApi(Resource):
+    @console_ns.expect(console_ns.models[AgentBuildDraftCheckoutPayload.__name__])
+    @console_ns.response(200, "Agent build draft checked out", console_ns.models[AgentBuildDraftResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @edit_permission_required
+    @with_current_user
+    @with_current_tenant_id
+    def post(self, tenant_id: str, current_user: Account, agent_id: UUID):
+        args = AgentBuildDraftCheckoutPayload.model_validate(console_ns.payload or {})
+        return AgentComposerService.checkout_agent_app_build_draft(
+            tenant_id=tenant_id,
+            agent_id=str(agent_id),
+            account_id=current_user.id,
+            force=args.force,
+            session=db.session(),
+        )
+
+
+@console_ns.route("/agent/<uuid:agent_id>/build-draft")
+class AgentBuildDraftApi(Resource):
+    @console_ns.response(200, "Agent build draft", console_ns.models[AgentBuildDraftResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @edit_permission_required
+    @with_current_user
+    @with_current_tenant_id
+    def get(self, tenant_id: str, current_user: Account, agent_id: UUID):
+        return AgentComposerService.load_agent_app_build_draft(
+            tenant_id=tenant_id,
+            agent_id=str(agent_id),
+            account_id=current_user.id,
+            session=db.session(),
+        )
+
+    @console_ns.expect(console_ns.models[ComposerSavePayload.__name__])
+    @console_ns.response(200, "Agent build draft saved", console_ns.models[AgentBuildDraftResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @edit_permission_required
+    @with_current_user
+    @with_current_tenant_id
+    def put(self, tenant_id: str, current_user: Account, agent_id: UUID):
+        payload = ComposerSavePayload.model_validate(console_ns.payload or {})
+        return AgentComposerService.save_agent_app_build_draft(
+            tenant_id=tenant_id,
+            agent_id=str(agent_id),
+            account_id=current_user.id,
+            payload=payload,
+            session=db.session(),
+        )
+
+    @console_ns.response(200, "Agent build draft discarded", console_ns.models[AgentSimpleResultResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @edit_permission_required
+    @with_current_user
+    @with_current_tenant_id
+    def delete(self, tenant_id: str, current_user: Account, agent_id: UUID):
+        return AgentComposerService.discard_agent_app_build_draft(
+            tenant_id=tenant_id,
+            agent_id=str(agent_id),
+            account_id=current_user.id,
+            session=db.session(),
+        )
+
+
+@console_ns.route("/agent/<uuid:agent_id>/build-draft/apply")
+class AgentBuildDraftApplyApi(Resource):
+    @console_ns.response(200, "Agent build draft applied", console_ns.models[AgentBuildDraftApplyResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @edit_permission_required
+    @with_current_user
+    @with_current_tenant_id
+    def post(self, tenant_id: str, current_user: Account, agent_id: UUID):
+        return AgentComposerService.apply_agent_app_build_draft(
+            tenant_id=tenant_id,
+            agent_id=str(agent_id),
+            account_id=current_user.id,
+            session=db.session(),
+        )
+
+
 @console_ns.route("/agent/<uuid:agent_id>/copy")
 class AgentAppCopyApi(Resource):
-    @console_ns.expect(console_ns.models[CopyAppPayload.__name__])
+    @console_ns.expect(console_ns.models[AgentAppCopyPayload.__name__])
     @console_ns.response(201, "Agent app copied successfully", console_ns.models[AgentAppDetailWithSite.__name__])
     @console_ns.response(403, "Insufficient permissions")
     @console_ns.response(400, "Invalid request parameters")
@@ -465,18 +775,88 @@ class AgentAppCopyApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def post(self, tenant_id: str, current_user: Account, agent_id: UUID):
-        args = CopyAppPayload.model_validate(console_ns.payload or {})
+        args = AgentAppCopyPayload.model_validate(console_ns.payload or {})
         copied_app = _agent_roster_service().duplicate_agent_app(
             tenant_id=tenant_id,
             agent_id=str(agent_id),
             account=current_user,
             name=args.name,
             description=args.description,
+            role=args.role,
             icon_type=args.icon_type,
             icon=args.icon,
             icon_background=args.icon_background,
         )
-        return _serialize_agent_app_detail(copied_app), 201
+        return _serialize_agent_app_detail(copied_app, current_user=current_user), 201
+
+
+@console_ns.route("/agent/<uuid:agent_id>/api-access")
+class AgentApiAccessApi(Resource):
+    @console_ns.response(200, "Agent service API access", console_ns.models[AgentApiAccessResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @with_current_tenant_id
+    def get(self, tenant_id: str, agent_id: UUID):
+        app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        return _serialize_agent_api_access(app_model)
+
+
+@console_ns.route("/agent/<uuid:agent_id>/api-enable")
+class AgentApiStatusApi(Resource):
+    @console_ns.expect(console_ns.models[AgentApiStatusPayload.__name__])
+    @console_ns.response(200, "Agent service API status updated", console_ns.models[AgentApiAccessResponse.__name__])
+    @console_ns.response(403, "Insufficient permissions")
+    @setup_required
+    @login_required
+    @is_admin_or_owner_required
+    @account_initialization_required
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_RELEASE_AND_VERSION)
+    @with_current_tenant_id
+    def post(self, tenant_id: str, agent_id: UUID):
+        app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        args = AgentApiStatusPayload.model_validate(console_ns.payload)
+        app_model = AppService().update_app_api_status(app_model, args.enable_api, session=db.session())
+        return _serialize_agent_api_access(app_model)
+
+
+@console_ns.route("/agent/<uuid:agent_id>/api-keys")
+class AgentApiKeyListApi(BaseApiKeyListResource):
+    resource_type = ApiTokenType.APP
+    resource_model = App
+    resource_id_field = "app_id"
+    token_prefix = "app-"
+
+    @console_ns.response(200, "Agent service API keys", console_ns.models[ApiKeyList.__name__])
+    @with_current_tenant_id
+    def get(self, tenant_id: str, agent_id: UUID) -> dict[str, object]:
+        app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        return dump_response(ApiKeyList, self._get_api_key_list(str(app_model.id), tenant_id))
+
+    @console_ns.response(201, "Agent service API key created", console_ns.models[ApiKeyItem.__name__])
+    @console_ns.response(400, "Maximum keys exceeded")
+    @with_current_tenant_id
+    @edit_permission_required
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_RELEASE_AND_VERSION)
+    def post(self, tenant_id: str, agent_id: UUID) -> tuple[dict[str, object], int]:
+        app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        return dump_response(ApiKeyItem, self._create_api_key(str(app_model.id), tenant_id)), 201
+
+
+@console_ns.route("/agent/<uuid:agent_id>/api-keys/<uuid:api_key_id>")
+class AgentApiKeyApi(BaseApiKeyResource):
+    resource_type = ApiTokenType.APP
+    resource_model = App
+    resource_id_field = "app_id"
+
+    @console_ns.response(204, "Agent service API key deleted")
+    @with_current_user
+    @with_current_tenant_id
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_RELEASE_AND_VERSION)
+    def delete(self, tenant_id: str, current_user: Account, agent_id: UUID, api_key_id: UUID) -> tuple[str, int]:
+        app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        self._delete_api_key(str(app_model.id), str(api_key_id), tenant_id, current_user)
+        return "", 204
 
 
 @console_ns.route("/agent/invite-options")
@@ -511,10 +891,10 @@ class AgentLogsApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def get(self, tenant_id: str, current_user: Account, agent_id: UUID):
-        app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         query_data: dict[str, object] = dict(request.args.to_dict(flat=True))
-        query_data["sources"] = _multi_query_values("sources", "source")
-        query_data["statuses"] = _multi_query_values("statuses", "status")
+        query_data["sources"] = _query_values("sources", "source")
+        query_data["statuses"] = _query_values("statuses", "status")
         query = AgentLogsQuery.model_validate(query_data)
         start, end = _parse_observability_time_range(query.start, query.end, current_user)
         try:
@@ -548,10 +928,10 @@ class AgentLogMessagesApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def get(self, tenant_id: str, current_user: Account, agent_id: UUID, conversation_id: UUID):
-        app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         query_data: dict[str, object] = dict(request.args.to_dict(flat=True))
-        query_data["sources"] = _multi_query_values("sources", "source")
-        query_data["statuses"] = _multi_query_values("statuses", "status")
+        query_data["sources"] = _query_values("sources", "source")
+        query_data["statuses"] = _query_values("statuses", "status")
         query = AgentLogsQuery.model_validate(query_data)
         start, end = _parse_observability_time_range(query.start, query.end, current_user)
         try:
@@ -585,7 +965,7 @@ class AgentLogSourcesApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def get(self, tenant_id: str, current_user: Account, agent_id: UUID):
-        app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         payload = _agent_observability_service().list_log_sources(app=app_model, agent_id=str(agent_id))
         return dump_response(AgentLogSourceListResponse, payload)
 
@@ -604,7 +984,7 @@ class AgentStatisticsSummaryApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def get(self, tenant_id: str, current_user: Account, agent_id: UUID):
-        app_model = _resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         query = AgentStatisticsQuery.model_validate(request.args.to_dict(flat=True))
         timezone = current_user.timezone or "UTC"
         start, end = _parse_observability_time_range(query.start, query.end, current_user)
@@ -649,3 +1029,24 @@ class AgentRosterVersionDetailApi(Resource):
                 version_id=str(version_id),
             ),
         )
+
+
+@console_ns.route("/agent/<uuid:agent_id>/versions/<uuid:version_id>/restore")
+class AgentRosterVersionRestoreApi(Resource):
+    @console_ns.response(200, "Agent version restored", console_ns.models[AgentConfigSnapshotRestoreResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @edit_permission_required
+    @with_current_user
+    @with_current_tenant_id
+    def post(self, tenant_id: str, current_user: Account, agent_id: UUID, version_id: UUID):
+        return dump_response(
+            AgentConfigSnapshotRestoreResponse,
+            _agent_roster_service().restore_agent_version(
+                tenant_id=tenant_id,
+                agent_id=str(agent_id),
+                version_id=str(version_id),
+                account_id=current_user.id,
+            ),
+        )

api/controllers/console/app/agent.py

@@ -1,4 +1,3 @@
-import logging
 from typing import Any
 from uuid import UUID
 
@@ -14,7 +13,7 @@ from controllers.common.schema import (
     register_schema_models,
 )
 from controllers.console import console_ns
-from controllers.console.agent.app_helpers import resolve_agent_app_model
+from controllers.console.agent.app_helpers import resolve_agent_runtime_app_model
 from controllers.console.app.wraps import get_app_model
 from controllers.console.wraps import (
     RBACPermission,
@@ -30,7 +29,6 @@ from fields.base import ResponseModel
 from libs.helper import uuid_value
 from libs.login import login_required
 from models import Account
-from models.agent_config_entities import AgentFileRefConfig, AgentSkillRefConfig
 from models.model import App, AppMode, UploadFile
 from services.agent.composer_service import AgentComposerService
 from services.agent.skill_package_service import SkillManifest, SkillPackageError
@@ -49,8 +47,6 @@ from services.agent_drive_service import (
 )
 from services.agent_service import AgentService
 
-logger = logging.getLogger(__name__)
-
 _WORKFLOW_AGENT_DRIVE_APP_MODES = [AppMode.WORKFLOW, AppMode.ADVANCED_CHAT]
 _AGENT_SKILL_UPLOAD_PARAMS = {
     "file": {
@@ -130,8 +126,16 @@ class AgentLogResponse(ResponseModel):
     files: list[Any] = Field(default_factory=list)
 
 
+class AgentUploadedSkillResponse(ResponseModel):
+    name: str
+    description: str
+    path: str
+    skill_md_key: str
+    archive_key: str | None = None
+
+
 class AgentSkillUploadResponse(ResponseModel):
-    skill: AgentSkillRefConfig
+    skill: AgentUploadedSkillResponse
     manifest: SkillManifest
 
 
@@ -145,13 +149,11 @@ class AgentDriveFileResponse(ResponseModel):
 
 class AgentDriveFileCommitResponse(ResponseModel):
     file: AgentDriveFileResponse
-    config_version_id: str | None = None
 
 
 class AgentDriveDeleteResponse(ResponseModel):
     result: str
     removed_keys: list[str] = Field(default_factory=list)
-    config_version_id: str | None = None
 
 
 register_schema_models(console_ns, AgentLogQuery, AgentDriveFilePayload, AgentDriveDeleteFileByAgentQuery)
@@ -161,6 +163,7 @@ register_response_schema_models(
     AgentDriveFileCommitResponse,
     AgentDriveFileResponse,
     AgentLogResponse,
+    AgentUploadedSkillResponse,
     AgentSkillUploadResponse,
     SkillToolInferenceResult,
 )
@@ -169,7 +172,7 @@ register_response_schema_models(
 def _resolve_agent_id(app_model: App, node_id: str | None) -> str | None:
     if node_id and app_model.mode != AppMode.AGENT:
         return AgentComposerService.resolve_workflow_node_agent_id(
-            tenant_id=app_model.tenant_id, app_id=app_model.id, node_id=node_id
+            tenant_id=app_model.tenant_id, app_id=app_model.id, node_id=node_id, session=db.session()
         )
     return app_model.bound_agent_id
 
@@ -199,6 +202,7 @@ def _upload_skill_for_app(*, current_user: Account, app_model: App):
             tenant_id=app_model.tenant_id,
             user_id=current_user.id,
             agent_id=agent_id,
+            session=db.session(),
         )
     except (SkillPackageError, AgentDriveError) as exc:
         return {"code": exc.code, "message": exc.message}, exc.status_code
@@ -237,29 +241,12 @@ def _commit_drive_file_for_app(*, current_user: Account, app_model: App, allow_n
                     value_owned_by_drive=True,
                 )
             ],
+            session=db.session(),
         )
     except AgentDriveError as exc:
         return {"code": exc.code, "message": exc.message}, exc.status_code
 
     row = committed[0]
-    file_ref = AgentFileRefConfig.model_validate(
-        {
-            "id": row["key"],
-            "name": upload_file.name,
-            "file_id": upload_file.id,
-            "drive_key": row["key"],
-            "type": row.get("mime_type"),
-            "size": row.get("size"),
-        }
-    )
-    config_version_id = AgentComposerService.add_drive_file_ref(
-        tenant_id=app_model.tenant_id,
-        agent_id=agent_id,
-        account_id=current_user.id,
-        file_ref=file_ref,
-        app_id=app_model.id,
-        node_id=node_id,
-    )
     return {
         "file": {
             "name": upload_file.name,
@@ -268,7 +255,6 @@ def _commit_drive_file_for_app(*, current_user: Account, app_model: App, allow_n
             "size": row.get("size"),
             "mime_type": row.get("mime_type"),
         },
-        "config_version_id": config_version_id,
     }, 201
 
 
@@ -283,24 +269,18 @@ def _delete_drive_file_for_app(*, current_user: Account, app_model: App, allow_n
     except AgentDriveError as exc:
         return {"code": exc.code, "message": exc.message}, exc.status_code
 
-    config_version_id = AgentComposerService.remove_drive_refs(
-        tenant_id=app_model.tenant_id,
-        agent_id=agent_id,
-        account_id=current_user.id,
-        file_key=key,
-        app_id=app_model.id,
-        node_id=node_id,
-    )
-    removed_keys: list[str] = []
     try:
-        removed_keys = AgentDriveService().delete(tenant_id=app_model.tenant_id, agent_id=agent_id, key=key)
+        result = AgentDriveService().commit(
+            tenant_id=app_model.tenant_id,
+            user_id=current_user.id,
+            agent_id=agent_id,
+            items=[DriveCommitItem(key=key, file_ref=None)],
+            session=db.session(),
+        )
     except AgentDriveError as exc:
         return {"code": exc.code, "message": exc.message}, exc.status_code
-    except Exception:
-        # Soul-first ordering: the ref is already gone; orphan KV rows are
-        # harmless and an idempotent DELETE retry cleans them.
-        logger.exception("agent drive delete failed for key %s (soul already updated)", key)
-    return {"result": "success", "removed_keys": removed_keys, "config_version_id": config_version_id}
+    removed_keys = [item["key"] for item in result if item.get("removed")]
+    return {"result": "success", "removed_keys": removed_keys}
 
 
 def _delete_skill_for_app(*, current_user: Account, app_model: App, slug: str, allow_node_id: bool = True):
@@ -312,22 +292,21 @@ def _delete_skill_for_app(*, current_user: Account, app_model: App, slug: str, a
     if "/" in slug or not slug.strip():
         return {"code": "drive_key_invalid", "message": "skill slug must be a single path segment"}, 400
 
-    config_version_id = AgentComposerService.remove_drive_refs(
-        tenant_id=app_model.tenant_id,
-        agent_id=agent_id,
-        account_id=current_user.id,
-        skill_slug=slug,
-        app_id=app_model.id,
-        node_id=node_id,
-    )
-    removed_keys: list[str] = []
     try:
-        removed_keys = AgentDriveService().delete(tenant_id=app_model.tenant_id, agent_id=agent_id, prefix=f"{slug}/")
+        result = AgentDriveService().commit(
+            tenant_id=app_model.tenant_id,
+            user_id=current_user.id,
+            agent_id=agent_id,
+            items=[
+                DriveCommitItem(key=f"{slug}/SKILL.md", file_ref=None),
+                DriveCommitItem(key=f"{slug}/.DIFY-SKILL-FULL.zip", file_ref=None),
+            ],
+            session=db.session(),
+        )
     except AgentDriveError as exc:
         return {"code": exc.code, "message": exc.message}, exc.status_code
-    except Exception:
-        logger.exception("agent drive delete failed for skill %s (soul already updated)", slug)
-    return {"result": "success", "removed_keys": removed_keys, "config_version_id": config_version_id}
+    removed_keys = [item["key"] for item in result if item.get("removed")]
+    return {"result": "success", "removed_keys": removed_keys}
 
 
 def _infer_skill_tools_for_app(*, app_model: App, slug: str):
@@ -338,7 +317,9 @@ def _infer_skill_tools_for_app(*, app_model: App, slug: str):
     if "/" in slug or not slug.strip():
         return {"code": "drive_key_invalid", "message": "skill slug must be a single path segment"}, 400
     try:
-        return SkillToolInferenceService().infer(tenant_id=app_model.tenant_id, agent_id=agent_id, slug=slug)
+        return SkillToolInferenceService().infer(
+            tenant_id=app_model.tenant_id, agent_id=agent_id, slug=slug, session=db.session()
+        )
     except SkillToolInferenceError as exc:
         return {"code": exc.code, "message": exc.message}, exc.status_code
 
@@ -360,7 +341,7 @@ class AgentLogApi(Resource):
         """Get agent logs"""
         args = AgentLogQuery.model_validate(request.args.to_dict(flat=True))
 
-        return AgentService.get_agent_logs(app_model, args.conversation_id, args.message_id)
+        return AgentService.get_agent_logs(app_model, args.conversation_id, args.message_id, db.session())
 
 
 @console_ns.route("/agent/<uuid:agent_id>/skills/upload")
@@ -376,7 +357,7 @@ class AgentSkillUploadByAgentApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def post(self, tenant_id: str, current_user: Account, agent_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         return _upload_skill_for_app(current_user=current_user, app_model=app_model)
 
 
@@ -419,7 +400,7 @@ class AgentDriveFilesByAgentApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def post(self, tenant_id: str, current_user: Account, agent_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         return _commit_drive_file_for_app(current_user=current_user, app_model=app_model, allow_node_id=False)
 
     @console_ns.doc("delete_agent_drive_file_by_agent")
@@ -432,7 +413,7 @@ class AgentDriveFilesByAgentApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def delete(self, tenant_id: str, current_user: Account, agent_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         return _delete_drive_file_for_app(current_user=current_user, app_model=app_model, allow_node_id=False)
 
 
@@ -455,7 +436,7 @@ class AgentDriveFilesApi(Resource):
         return _commit_drive_file_for_app(current_user=current_user, app_model=app_model)
 
     @console_ns.doc("delete_agent_drive_file")
-    @console_ns.doc(description="Delete one drive file by key; soul ref first, then the KV row (ENG-625 D5)")
+    @console_ns.doc(description="Delete one drive file by key via drive commit-null semantics")
     @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(AgentDriveDeleteFileQuery)})
     @console_ns.response(200, "File removed", console_ns.models[AgentDriveDeleteResponse.__name__])
     @setup_required
@@ -479,16 +460,14 @@ class AgentSkillByAgentApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def delete(self, tenant_id: str, current_user: Account, agent_id: UUID, slug: str):
-        app_model = resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         return _delete_skill_for_app(current_user=current_user, app_model=app_model, slug=slug, allow_node_id=False)
 
 
 @console_ns.route("/apps/<uuid:app_id>/agent/skills/<string:slug>")
 class AgentSkillApi(Resource):
     @console_ns.doc("delete_agent_skill")
-    @console_ns.doc(
-        description="Delete a standardized skill: soul ref first, then the <slug>/ drive prefix (ENG-625 D5)"
-    )
+    @console_ns.doc(description="Delete a standardized skill by removing its known drive keys via commit-null")
     @console_ns.doc(
         params={
             "app_id": "Application ID",
@@ -521,7 +500,7 @@ class AgentSkillInferToolsByAgentApi(Resource):
     @account_initialization_required
     @with_current_tenant_id
     def post(self, tenant_id: str, agent_id: UUID, slug: str):
-        app_model = resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         return _infer_skill_tools_for_app(app_model=app_model, slug=slug)
 
 

api/controllers/console/app/agent_app_feature.py

@@ -17,7 +17,7 @@ from pydantic import BaseModel, Field
 from controllers.common.fields import SimpleResultResponse
 from controllers.common.schema import register_response_schema_models, register_schema_models
 from controllers.console import console_ns
-from controllers.console.agent.app_helpers import resolve_agent_app_model
+from controllers.console.agent.app_helpers import resolve_agent_runtime_app_model
 from controllers.console.wraps import (
     RBACPermission,
     RBACResourceScope,
@@ -30,7 +30,6 @@ from controllers.console.wraps import (
 )
 from events.app_event import app_model_config_was_updated
 from extensions.ext_database import db
-from libs.helper import dump_response
 from libs.login import login_required
 from models import Account
 from models.agent_config_entities import (
@@ -87,13 +86,16 @@ class AgentAppFeatureConfigResource(Resource):
     @with_current_user
     @with_current_tenant_id
     def post(self, tenant_id: str, current_user: Account, agent_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         args = AgentAppFeaturesPayload.model_validate(console_ns.payload or {})
 
         new_app_model_config = AgentAppFeatureConfigService.update_features(
-            app_model=app_model, account=current_user, config=args.model_dump(exclude_none=True), session=db.session
+            app_model=app_model,
+            account=current_user,
+            config=args.model_dump(exclude_none=True),
+            session=db.session(),
         )
 
         app_model_config_was_updated.send(app_model, app_model_config=new_app_model_config)
 
-        return dump_response(SimpleResultResponse, {"result": "success"})
+        return SimpleResultResponse(result="success").model_dump(mode="json")

api/controllers/console/app/agent_app_sandbox.py

@@ -22,9 +22,10 @@ from controllers.common.schema import (
     register_schema_models,
 )
 from controllers.console import console_ns
-from controllers.console.agent.app_helpers import resolve_agent_app_model
+from controllers.console.agent.app_helpers import resolve_agent_runtime_app_model
 from controllers.console.app.wraps import get_app_model
 from controllers.console.wraps import account_initialization_required, setup_required, with_current_tenant_id
+from extensions.ext_database import db
 from fields.base import ResponseModel
 from libs.login import login_required
 from models.model import App, AppMode
@@ -44,6 +45,10 @@ class AgentSandboxListQuery(BaseModel):
     path: str = Field(default=".", description="Directory path relative to the sandbox workspace")
 
 
+class AgentSandboxInfoQuery(BaseModel):
+    conversation_id: str = Field(min_length=1, description="Agent App conversation ID")
+
+
 class AgentSandboxFileQuery(BaseModel):
     conversation_id: str = Field(min_length=1, description="Agent App conversation ID")
     path: str = Field(min_length=1, description="File path relative to the sandbox workspace")
@@ -91,6 +96,11 @@ class SandboxListResponse(ResponseModel):
     truncated: bool = False
 
 
+class SandboxInfoResponse(ResponseModel):
+    session_id: str
+    workspace_cwd: str
+
+
 class SandboxReadResponse(ResponseModel):
     path: str
     size: int | None = None
@@ -99,14 +109,8 @@ class SandboxReadResponse(ResponseModel):
     text: str | None = None
 
 
-class SandboxToolFileResponse(ResponseModel):
-    transfer_method: Literal["tool_file"] = "tool_file"
-    reference: str
-
-
 class SandboxUploadResponse(ResponseModel):
-    path: str
-    file: SandboxToolFileResponse
+    url: str
 
 
 register_schema_models(
@@ -114,7 +118,13 @@ register_schema_models(
     AgentSandboxUploadPayload,
     WorkflowAgentSandboxUploadPayload,
 )
-register_response_schema_models(console_ns, SandboxListResponse, SandboxReadResponse, SandboxUploadResponse)
+register_response_schema_models(
+    console_ns,
+    SandboxInfoResponse,
+    SandboxListResponse,
+    SandboxReadResponse,
+    SandboxUploadResponse,
+)
 
 
 def _handle(exc: Exception) -> tuple[dict[str, object], int]:
@@ -133,6 +143,30 @@ def _handle(exc: Exception) -> tuple[dict[str, object], int]:
     raise exc
 
 
+@console_ns.route("/agent/<uuid:agent_id>/sandbox")
+class AgentAppSandboxInfoResource(Resource):
+    @console_ns.doc("get_agent_app_sandbox_info")
+    @console_ns.doc(description="Get basic information for an Agent App conversation sandbox")
+    @console_ns.doc(params={"agent_id": "Agent ID", **query_params_from_model(AgentSandboxInfoQuery)})
+    @console_ns.response(200, "Sandbox information returned", console_ns.models[SandboxInfoResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @with_current_tenant_id
+    def get(self, tenant_id: str, agent_id: UUID):
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        query = query_params_from_request(AgentSandboxInfoQuery)
+        try:
+            result = AgentAppSandboxService().get_info(
+                tenant_id=tenant_id,
+                app_id=app_model.id,
+                conversation_id=query.conversation_id,
+            )
+        except Exception as exc:
+            return _handle(exc)
+        return result.model_dump()
+
+
 @console_ns.route("/agent/<uuid:agent_id>/sandbox/files")
 class AgentAppSandboxListResource(Resource):
     @console_ns.doc("list_agent_app_sandbox_files")
@@ -144,7 +178,7 @@ class AgentAppSandboxListResource(Resource):
     @account_initialization_required
     @with_current_tenant_id
     def get(self, tenant_id: str, agent_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         query = query_params_from_request(AgentSandboxListQuery)
         try:
             result = AgentAppSandboxService().list_files(
@@ -169,7 +203,7 @@ class AgentAppSandboxReadResource(Resource):
     @account_initialization_required
     @with_current_tenant_id
     def get(self, tenant_id: str, agent_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         query = query_params_from_request(AgentSandboxFileQuery)
         try:
             result = AgentAppSandboxService().read_file(
@@ -186,7 +220,7 @@ class AgentAppSandboxReadResource(Resource):
 @console_ns.route("/agent/<uuid:agent_id>/sandbox/files/upload")
 class AgentAppSandboxUploadResource(Resource):
     @console_ns.doc("upload_agent_app_sandbox_file")
-    @console_ns.doc(description="Upload one Agent App sandbox file as a Dify ToolFile mapping")
+    @console_ns.doc(description="Upload one Agent App sandbox file and return a signed download URL")
     @console_ns.expect(console_ns.models[AgentSandboxUploadPayload.__name__])
     @console_ns.response(200, "Uploaded", console_ns.models[SandboxUploadResponse.__name__])
     @setup_required
@@ -194,7 +228,7 @@ class AgentAppSandboxUploadResource(Resource):
     @account_initialization_required
     @with_current_tenant_id
     def post(self, tenant_id: str, agent_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         payload = AgentSandboxUploadPayload.model_validate(request.get_json(silent=True) or {})
         try:
             result = AgentAppSandboxService().upload_file(
@@ -236,6 +270,7 @@ class WorkflowAgentSandboxListResource(Resource):
                 node_id=node_id,
                 node_execution_id=query.node_execution_id,
                 path=query.path,
+                session=db.session(),
             )
         except Exception as exc:
             return _handle(exc)
@@ -272,6 +307,7 @@ class WorkflowAgentSandboxReadResource(Resource):
                 node_id=node_id,
                 node_execution_id=query.node_execution_id,
                 path=query.path,
+                session=db.session(),
             )
         except Exception as exc:
             return _handle(exc)
@@ -283,7 +319,7 @@ class WorkflowAgentSandboxReadResource(Resource):
 )
 class WorkflowAgentSandboxUploadResource(Resource):
     @console_ns.doc("upload_workflow_agent_sandbox_file")
-    @console_ns.doc(description="Upload one workflow Agent sandbox file as a Dify ToolFile mapping")
+    @console_ns.doc(description="Upload one workflow Agent sandbox file and return a signed download URL")
     @console_ns.expect(console_ns.models[WorkflowAgentSandboxUploadPayload.__name__])
     @console_ns.response(200, "Uploaded", console_ns.models[SandboxUploadResponse.__name__])
     @setup_required
@@ -301,6 +337,7 @@ class WorkflowAgentSandboxUploadResource(Resource):
                 node_id=node_id,
                 node_execution_id=payload.node_execution_id,
                 path=payload.path,
+                session=db.session(),
             )
         except Exception as exc:
             return _handle(exc)

api/controllers/console/app/agent_drive_inspector.py

@@ -10,8 +10,12 @@ backend — drive data lives in the API's own DB/storage, served straight from
 
 from __future__ import annotations
 
+import json
+from collections.abc import Mapping
+from typing import Any
 from uuid import UUID
 
+from flask import Response
 from flask_restx import Resource
 from pydantic import BaseModel, Field
 
@@ -21,9 +25,10 @@ from controllers.common.schema import (
     register_response_schema_models,
 )
 from controllers.console import console_ns
-from controllers.console.agent.app_helpers import resolve_agent_app_model
+from controllers.console.agent.app_helpers import resolve_agent_runtime_app_model
 from controllers.console.app.wraps import get_app_model
 from controllers.console.wraps import account_initialization_required, setup_required, with_current_tenant_id
+from extensions.ext_database import db
 from fields.base import ResponseModel
 from libs.login import login_required
 from models.model import App, AppMode
@@ -49,6 +54,10 @@ class AgentDriveFileByAgentQuery(BaseModel):
     key: str = Field(min_length=1, description="Drive key, e.g. tender-analyzer/SKILL.md")
 
 
+class AgentDriveSkillInspectQuery(BaseModel):
+    node_id: str | None = Field(default=None, description="Workflow node ID (workflow composer variant)")
+
+
 class AgentDriveItemResponse(ResponseModel):
     key: str
     size: int | None = None
@@ -56,12 +65,63 @@ class AgentDriveItemResponse(ResponseModel):
     hash: str | None = None
     file_kind: str
     created_at: int | None = None
+    is_skill: bool | None = None
+    skill_metadata: str | None = None
 
 
 class AgentDriveListResponse(ResponseModel):
     items: list[AgentDriveItemResponse] = Field(default_factory=list)
 
 
+class AgentDriveSkillItemResponse(ResponseModel):
+    path: str
+    skill_md_key: str
+    archive_key: str | None = None
+    name: str
+    description: str
+    size: int | None = None
+    mime_type: str | None = None
+    hash: str | None = None
+    created_at: int | None = None
+
+
+class AgentDriveSkillListResponse(ResponseModel):
+    items: list[AgentDriveSkillItemResponse] = Field(default_factory=list)
+
+
+class AgentDriveSkillFileResponse(ResponseModel):
+    path: str
+    name: str
+    type: str
+    drive_key: str | None = None
+    available_in_drive: bool
+
+
+class AgentDriveSkillMarkdownResponse(ResponseModel):
+    key: str
+    size: int | None = None
+    truncated: bool
+    binary: bool
+    text: str | None = None
+
+
+class AgentDriveSkillInspectResponse(ResponseModel):
+    path: str
+    skill_md_key: str
+    archive_key: str | None = None
+    name: str
+    description: str
+    size: int | None = None
+    mime_type: str | None = None
+    hash: str | None = None
+    created_at: int | None = None
+    source: str
+    files: list[AgentDriveSkillFileResponse] = Field(default_factory=list)
+    file_tree: list[dict[str, Any]] = Field(default_factory=list)
+    skill_md: AgentDriveSkillMarkdownResponse
+    warnings: list[str] = Field(default_factory=list)
+
+
 class AgentDrivePreviewResponse(ResponseModel):
     key: str
     size: int | None = None
@@ -75,7 +135,12 @@ class AgentDriveDownloadResponse(ResponseModel):
 
 
 register_response_schema_models(
-    console_ns, AgentDriveListResponse, AgentDrivePreviewResponse, AgentDriveDownloadResponse
+    console_ns,
+    AgentDriveDownloadResponse,
+    AgentDriveListResponse,
+    AgentDrivePreviewResponse,
+    AgentDriveSkillInspectResponse,
+    AgentDriveSkillListResponse,
 )
 
 
@@ -83,7 +148,7 @@ def _resolve_agent_id(app_model: App, node_id: str | None) -> str | None:
     """Agent identity for the drive: app-bound agent, or the workflow node binding."""
     if node_id:
         return AgentComposerService.resolve_workflow_node_agent_id(
-            tenant_id=app_model.tenant_id, app_id=app_model.id, node_id=node_id
+            tenant_id=app_model.tenant_id, app_id=app_model.id, node_id=node_id, session=db.session()
         )
     return app_model.bound_agent_id
 
@@ -96,6 +161,13 @@ def _handle(exc: AgentDriveError) -> tuple[dict[str, object], int]:
     return {"code": exc.code, "message": exc.message}, exc.status_code
 
 
+def _json_response(data: Mapping[str, Any]):
+    return Response(
+        response=json.dumps(data, ensure_ascii=False, separators=(",", ":")),
+        content_type="application/json; charset=utf-8",
+    )
+
+
 _WORKFLOW_APP_MODES = [AppMode.WORKFLOW, AppMode.ADVANCED_CHAT]
 
 
@@ -111,14 +183,60 @@ class AgentDriveListByAgentApi(Resource):
     @with_current_tenant_id
     def get(self, tenant_id: str, agent_id: UUID):
         query = query_params_from_request(AgentDriveListByAgentQuery)
-        resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         try:
-            items = AgentDriveService().manifest(tenant_id=tenant_id, agent_id=str(agent_id), prefix=query.prefix)
+            items = AgentDriveService().manifest(
+                tenant_id=tenant_id, agent_id=str(agent_id), prefix=query.prefix, session=db.session()
+            )
         except AgentDriveError as exc:
             return _handle(exc)
         return {"items": [{k: v for k, v in item.items() if k != "file_id"} for item in items]}
 
 
+@console_ns.route("/agent/<uuid:agent_id>/drive/skills")
+class AgentDriveSkillListByAgentApi(Resource):
+    @console_ns.doc("list_agent_drive_skills_by_agent")
+    @console_ns.doc(description="List drive-backed skills for an Agent App")
+    @console_ns.doc(params={"agent_id": "Agent ID"})
+    @console_ns.response(200, "Drive skills", console_ns.models[AgentDriveSkillListResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @with_current_tenant_id
+    def get(self, tenant_id: str, agent_id: UUID):
+        resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        try:
+            items = AgentDriveService().list_skills(tenant_id=tenant_id, agent_id=str(agent_id), session=db.session())
+        except AgentDriveError as exc:
+            return _handle(exc)
+        return {"items": items}
+
+
+@console_ns.route("/agent/<uuid:agent_id>/drive/skills/<path:skill_path>/inspect")
+class AgentDriveSkillInspectByAgentApi(Resource):
+    @console_ns.doc("inspect_agent_drive_skill_by_agent")
+    @console_ns.doc(description="Inspect one drive-backed skill for slash-menu hover/detail UI")
+    @console_ns.doc(params={"agent_id": "Agent ID", "skill_path": "Skill path/slug, e.g. tender-analyzer"})
+    @console_ns.response(200, "Drive skill inspect view", console_ns.models[AgentDriveSkillInspectResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @with_current_tenant_id
+    def get(self, tenant_id: str, agent_id: UUID, skill_path: str):
+        resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        try:
+            return _json_response(
+                AgentDriveService().inspect_skill(
+                    tenant_id=tenant_id,
+                    agent_id=str(agent_id),
+                    skill_path=skill_path,
+                    session=db.session(),
+                )
+            )
+        except AgentDriveError as exc:
+            return _handle(exc)
+
+
 @console_ns.route("/agent/<uuid:agent_id>/drive/files/preview")
 class AgentDrivePreviewByAgentApi(Resource):
     @console_ns.doc("preview_agent_drive_file_by_agent")
@@ -131,9 +249,11 @@ class AgentDrivePreviewByAgentApi(Resource):
     @with_current_tenant_id
     def get(self, tenant_id: str, agent_id: UUID):
         query = query_params_from_request(AgentDriveFileByAgentQuery)
-        resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         try:
-            return AgentDriveService().preview(tenant_id=tenant_id, agent_id=str(agent_id), key=query.key)
+            return AgentDriveService().preview(
+                tenant_id=tenant_id, agent_id=str(agent_id), key=query.key, session=db.session()
+            )
         except AgentDriveError as exc:
             return _handle(exc)
 
@@ -150,9 +270,11 @@ class AgentDriveDownloadByAgentApi(Resource):
     @with_current_tenant_id
     def get(self, tenant_id: str, agent_id: UUID):
         query = query_params_from_request(AgentDriveFileByAgentQuery)
-        resolve_agent_app_model(tenant_id=tenant_id, agent_id=agent_id)
+        resolve_agent_runtime_app_model(tenant_id=tenant_id, agent_id=agent_id)
         try:
-            url = AgentDriveService().download_url(tenant_id=tenant_id, agent_id=str(agent_id), key=query.key)
+            url = AgentDriveService().download_url(
+                tenant_id=tenant_id, agent_id=str(agent_id), key=query.key, session=db.session()
+            )
         except AgentDriveError as exc:
             return _handle(exc)
         return {"url": url}
@@ -174,7 +296,9 @@ class AgentDriveListApi(Resource):
         if not agent_id:
             return _agent_not_bound()
         try:
-            items = AgentDriveService().manifest(tenant_id=app_model.tenant_id, agent_id=agent_id, prefix=query.prefix)
+            items = AgentDriveService().manifest(
+                tenant_id=app_model.tenant_id, agent_id=agent_id, prefix=query.prefix, session=db.session()
+            )
         except AgentDriveError as exc:
             return _handle(exc)
         # the inner manifest exposes file_id for agent-side pulls; the console
@@ -182,6 +306,64 @@ class AgentDriveListApi(Resource):
         return {"items": [{k: v for k, v in item.items() if k != "file_id"} for item in items]}
 
 
+@console_ns.route("/apps/<uuid:app_id>/agent/drive/skills")
+class AgentDriveSkillListApi(Resource):
+    @console_ns.doc("list_agent_drive_skills")
+    @console_ns.doc(description="List drive-backed skills for the bound agent")
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(AgentDriveListQuery)})
+    @console_ns.response(200, "Drive skills", console_ns.models[AgentDriveSkillListResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @get_app_model(mode=_WORKFLOW_APP_MODES)
+    def get(self, app_model: App):
+        query = query_params_from_request(AgentDriveListQuery)
+        agent_id = _resolve_agent_id(app_model, query.node_id)
+        if not agent_id:
+            return _agent_not_bound()
+        try:
+            items = AgentDriveService().list_skills(
+                tenant_id=app_model.tenant_id, agent_id=agent_id, session=db.session()
+            )
+        except AgentDriveError as exc:
+            return _handle(exc)
+        return {"items": items}
+
+
+@console_ns.route("/apps/<uuid:app_id>/agent/drive/skills/<path:skill_path>/inspect")
+class AgentDriveSkillInspectApi(Resource):
+    @console_ns.doc("inspect_agent_drive_skill")
+    @console_ns.doc(description="Inspect one drive-backed skill for slash-menu hover/detail UI")
+    @console_ns.doc(
+        params={
+            "app_id": "Application ID",
+            "skill_path": "Skill path/slug, e.g. tender-analyzer",
+            **query_params_from_model(AgentDriveSkillInspectQuery),
+        }
+    )
+    @console_ns.response(200, "Drive skill inspect view", console_ns.models[AgentDriveSkillInspectResponse.__name__])
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @get_app_model(mode=_WORKFLOW_APP_MODES)
+    def get(self, app_model: App, skill_path: str):
+        query = query_params_from_request(AgentDriveSkillInspectQuery)
+        agent_id = _resolve_agent_id(app_model, query.node_id)
+        if not agent_id:
+            return _agent_not_bound()
+        try:
+            return _json_response(
+                AgentDriveService().inspect_skill(
+                    tenant_id=app_model.tenant_id,
+                    agent_id=agent_id,
+                    skill_path=skill_path,
+                    session=db.session(),
+                )
+            )
+        except AgentDriveError as exc:
+            return _handle(exc)
+
+
 @console_ns.route("/apps/<uuid:app_id>/agent/drive/files/preview")
 class AgentDrivePreviewApi(Resource):
     @console_ns.doc("preview_agent_drive_file")
@@ -198,7 +380,9 @@ class AgentDrivePreviewApi(Resource):
         if not agent_id:
             return _agent_not_bound()
         try:
-            return AgentDriveService().preview(tenant_id=app_model.tenant_id, agent_id=agent_id, key=query.key)
+            return AgentDriveService().preview(
+                tenant_id=app_model.tenant_id, agent_id=agent_id, key=query.key, session=db.session()
+            )
         except AgentDriveError as exc:
             return _handle(exc)
 
@@ -219,7 +403,9 @@ class AgentDriveDownloadApi(Resource):
         if not agent_id:
             return _agent_not_bound()
         try:
-            url = AgentDriveService().download_url(tenant_id=app_model.tenant_id, agent_id=agent_id, key=query.key)
+            url = AgentDriveService().download_url(
+                tenant_id=app_model.tenant_id, agent_id=agent_id, key=query.key, session=db.session()
+            )
         except AgentDriveError as exc:
             return _handle(exc)
         return {"url": url}
@@ -232,4 +418,8 @@ __all__ = [
     "AgentDriveListByAgentApi",
     "AgentDrivePreviewApi",
     "AgentDrivePreviewByAgentApi",
+    "AgentDriveSkillInspectApi",
+    "AgentDriveSkillInspectByAgentApi",
+    "AgentDriveSkillListApi",
+    "AgentDriveSkillListByAgentApi",
 ]

api/controllers/console/app/annotation.py

@@ -1,9 +1,11 @@
 from typing import Any, Literal
 from uuid import UUID
 
-from flask import abort, make_response, request
+from flask import abort, request
 from flask_restx import Resource
 from pydantic import BaseModel, Field, TypeAdapter, field_validator
+from sqlalchemy import select
+from werkzeug.exceptions import NotFound
 
 from controllers.common.errors import NoFileUploadedError, TooManyFilesError
 from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models
@@ -19,17 +21,21 @@ from controllers.console.wraps import (
     rbac_permission_required,
     setup_required,
 )
+from extensions.ext_database import db
 from extensions.ext_redis import redis_client
 from fields.annotation_fields import (
     Annotation,
     AnnotationExportList,
     AnnotationHitHistory,
     AnnotationHitHistoryList,
+    AnnotationJobStatusDetailResponse,
+    AnnotationJobStatusResponse,
     AnnotationList,
 )
 from fields.base import ResponseModel
-from libs.helper import uuid_value
-from libs.login import login_required
+from libs.helper import dump_response, uuid_value
+from libs.login import current_account_with_tenant, login_required
+from models.model import App
 from services.annotation_service import (
     AppAnnotationService,
     EnableAnnotationArgs,
@@ -37,6 +43,17 @@ from services.annotation_service import (
     UpdateAnnotationSettingArgs,
     UpsertAnnotationArgs,
 )
+from services.app_ref_service import AppRef, AppRefService
+
+
+def _get_app_ref(app_id: str) -> AppRef:
+    _, current_tenant_id = current_account_with_tenant()
+    app = db.session.scalar(
+        select(App).where(App.id == app_id, App.tenant_id == current_tenant_id, App.status == "normal").limit(1)
+    )
+    if app is None:
+        raise NotFound("App not found")
+    return AppRefService.create_app_ref(app)
 
 
 class AnnotationReplyPayload(BaseModel):
@@ -98,23 +115,23 @@ class AnnotationFilePayload(BaseModel):
         return uuid_value(value)
 
 
-class AnnotationJobStatusResponse(ResponseModel):
-    job_id: str | None = None
-    job_status: str | None = None
-    error_msg: str | None = None
-    record_count: int | None = None
-
-
-class AnnotationEmbeddingModelResponse(ResponseModel):
+class AnnotationSettingEmbeddingModelResponse(ResponseModel):
     embedding_provider_name: str | None = None
     embedding_model_name: str | None = None
 
 
 class AnnotationSettingResponse(ResponseModel):
-    id: str | None = None
     enabled: bool
+    id: str | None = None
     score_threshold: float | None = None
-    embedding_model: AnnotationEmbeddingModelResponse | None = None
+    embedding_model: AnnotationSettingEmbeddingModelResponse | None = None
+
+
+class AnnotationBatchImportResponse(ResponseModel):
+    job_id: str | None = None
+    job_status: str | None = None
+    record_count: int | None = None
+    error_msg: str | None = None
 
 
 register_schema_models(
@@ -141,7 +158,10 @@ register_response_schema_models(
     AnnotationHitHistory,
     AnnotationHitHistoryList,
     AnnotationJobStatusResponse,
+    AnnotationJobStatusDetailResponse,
+    AnnotationSettingEmbeddingModelResponse,
     AnnotationSettingResponse,
+    AnnotationBatchImportResponse,
 )
 
 
@@ -171,7 +191,7 @@ class AnnotationReplyActionApi(Resource):
                 result = AppAnnotationService.enable_app_annotation(enable_args, str(app_id))
             case "disable":
                 result = AppAnnotationService.disable_app_annotation(str(app_id))
-        return result, 200
+        return dump_response(AnnotationJobStatusResponse, result), 200
 
 
 @console_ns.route("/apps/<uuid:app_id>/annotation-setting")
@@ -191,8 +211,8 @@ class AppAnnotationSettingDetailApi(Resource):
     @edit_permission_required
     @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_VIEW_LAYOUT)
     def get(self, app_id: UUID):
-        result = AppAnnotationService.get_app_annotation_setting_by_app_id(str(app_id))
-        return result, 200
+        result = AppAnnotationService.get_app_annotation_setting_by_app_id(str(app_id), session=db.session())
+        return dump_response(AnnotationSettingResponse, result), 200
 
 
 @console_ns.route("/apps/<uuid:app_id>/annotation-settings/<uuid:annotation_setting_id>")
@@ -215,9 +235,9 @@ class AppAnnotationSettingUpdateApi(Resource):
 
         setting_args: UpdateAnnotationSettingArgs = {"score_threshold": args.score_threshold}
         result = AppAnnotationService.update_app_annotation_setting(
-            str(app_id), annotation_setting_id_str, setting_args
+            str(app_id), annotation_setting_id_str, setting_args, session=db.session()
         )
-        return result, 200
+        return dump_response(AnnotationSettingResponse, result), 200
 
 
 @console_ns.route("/apps/<uuid:app_id>/annotation-reply/<string:action>/status/<uuid:job_id>")
@@ -226,9 +246,7 @@ class AnnotationReplyActionStatusApi(Resource):
     @console_ns.doc(description="Get status of annotation reply action job")
     @console_ns.doc(params={"app_id": "Application ID", "job_id": "Job ID", "action": "Action type"})
     @console_ns.response(
-        200,
-        "Job status retrieved successfully",
-        console_ns.models[AnnotationJobStatusResponse.__name__],
+        200, "Job status retrieved successfully", console_ns.models[AnnotationJobStatusDetailResponse.__name__]
     )
     @console_ns.response(403, "Insufficient permissions")
     @setup_required
@@ -250,7 +268,9 @@ class AnnotationReplyActionStatusApi(Resource):
             app_annotation_error_key = f"{action}_app_annotation_error_{job_id_str}"
             error_msg = redis_client.get(app_annotation_error_key).decode()
 
-        return {"job_id": job_id_str, "job_status": job_status, "error_msg": error_msg}, 200
+        return AnnotationJobStatusDetailResponse(
+            job_id=job_id_str, job_status=job_status, error_msg=error_msg
+        ).model_dump(mode="json"), 200
 
 
 @console_ns.route("/apps/<uuid:app_id>/annotations")
@@ -272,16 +292,13 @@ class AnnotationApi(Resource):
         limit = args.limit
         keyword = args.keyword
 
-        annotation_list, total = AppAnnotationService.get_annotation_list_by_app_id(str(app_id), page, limit, keyword)
-        annotation_models = TypeAdapter(list[Annotation]).validate_python(annotation_list, from_attributes=True)
-        response = AnnotationList(
-            data=annotation_models,
-            has_more=len(annotation_list) == limit,
-            limit=limit,
-            total=total,
-            page=page,
+        annotation_list, total = AppAnnotationService.get_annotation_list_by_app_id(
+            str(app_id), page, limit, keyword, session=db.session()
         )
-        return response.model_dump(mode="json"), 200
+        annotation_models = TypeAdapter(list[Annotation]).validate_python(annotation_list, from_attributes=True)
+        return AnnotationList(
+            data=annotation_models, has_more=len(annotation_list) == limit, limit=limit, total=total, page=page
+        ).model_dump(mode="json"), 200
 
     @console_ns.doc("create_annotation")
     @console_ns.doc(description="Create a new annotation for an app")
@@ -306,8 +323,10 @@ class AnnotationApi(Resource):
             upsert_args["message_id"] = args.message_id
         if args.question is not None:
             upsert_args["question"] = args.question
-        annotation = AppAnnotationService.up_insert_app_annotation_from_message(upsert_args, str(app_id))
-        return Annotation.model_validate(annotation, from_attributes=True).model_dump(mode="json")
+        annotation = AppAnnotationService.up_insert_app_annotation_from_message(
+            upsert_args, str(app_id), session=db.session()
+        )
+        return dump_response(Annotation, annotation), 201
 
     @setup_required
     @login_required
@@ -329,11 +348,12 @@ class AnnotationApi(Resource):
                     "message": "annotation_ids are required if the parameter is provided.",
                 }, 400
 
-            AppAnnotationService.delete_app_annotations_in_batch(str(app_id), annotation_ids)
+            app_ref = _get_app_ref(str(app_id))
+            AppAnnotationService.delete_app_annotations_in_batch(app_ref, annotation_ids, session=db.session())
             return "", 204
         # If no annotation_ids are provided, handle clearing all annotations
         else:
-            AppAnnotationService.clear_all_annotations(str(app_id))
+            AppAnnotationService.clear_all_annotations(str(app_id), session=db.session())
             return "", 204
 
 
@@ -354,16 +374,16 @@ class AnnotationExportApi(Resource):
     @edit_permission_required
     @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_VIEW_LAYOUT)
     def get(self, app_id: UUID):
-        annotation_list = AppAnnotationService.export_annotation_list_by_app_id(str(app_id))
+        annotation_list = AppAnnotationService.export_annotation_list_by_app_id(str(app_id), session=db.session())
         annotation_models = TypeAdapter(list[Annotation]).validate_python(annotation_list, from_attributes=True)
-        response_data = AnnotationExportList(data=annotation_models).model_dump(mode="json")
-
-        # Create response with secure headers for CSV export
-        response = make_response(response_data, 200)
-        response.headers["Content-Type"] = "application/json; charset=utf-8"
-        response.headers["X-Content-Type-Options"] = "nosniff"
-
-        return response
+        return (
+            AnnotationExportList(data=annotation_models).model_dump(mode="json"),
+            200,
+            {
+                "Content-Type": "application/json; charset=utf-8",
+                "X-Content-Type-Options": "nosniff",
+            },
+        )
 
 
 @console_ns.route("/apps/<uuid:app_id>/annotations/<uuid:annotation_id>")
@@ -388,7 +408,9 @@ class AnnotationUpdateDeleteApi(Resource):
             update_args["answer"] = args.answer
         if args.question is not None:
             update_args["question"] = args.question
-        annotation = AppAnnotationService.update_app_annotation_directly(update_args, str(app_id), str(annotation_id))
+        app_ref = _get_app_ref(str(app_id))
+        annotation_ref = AppRefService.create_annotation_ref(app_ref, str(annotation_id))
+        annotation = AppAnnotationService.update_app_annotation_directly(update_args, annotation_ref, db.session())
         return Annotation.model_validate(annotation, from_attributes=True).model_dump(mode="json")
 
     @setup_required
@@ -398,7 +420,9 @@ class AnnotationUpdateDeleteApi(Resource):
     @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_EDIT)
     @console_ns.response(204, "Annotation deleted successfully")
     def delete(self, app_id: UUID, annotation_id: UUID):
-        AppAnnotationService.delete_app_annotation(str(app_id), str(annotation_id))
+        app_ref = _get_app_ref(str(app_id))
+        annotation_ref = AppRefService.create_annotation_ref(app_ref, str(annotation_id))
+        AppAnnotationService.delete_app_annotation(annotation_ref, db.session())
         return "", 204
 
 
@@ -408,9 +432,7 @@ class AnnotationBatchImportApi(Resource):
     @console_ns.doc(description="Batch import annotations from CSV file with rate limiting and security checks")
     @console_ns.doc(params={"app_id": "Application ID"})
     @console_ns.response(
-        200,
-        "Batch import started successfully",
-        console_ns.models[AnnotationJobStatusResponse.__name__],
+        200, "Batch import started successfully", console_ns.models[AnnotationBatchImportResponse.__name__]
     )
     @console_ns.response(403, "Insufficient permissions")
     @console_ns.response(400, "No file uploaded or too many files")
@@ -457,7 +479,10 @@ class AnnotationBatchImportApi(Resource):
         if file_size == 0:
             raise ValueError("The uploaded file is empty")
 
-        return AppAnnotationService.batch_import_app_annotations(str(app_id), file)
+        return dump_response(
+            AnnotationBatchImportResponse,
+            AppAnnotationService.batch_import_app_annotations(str(app_id), file, session=db.session()),
+        )
 
 
 @console_ns.route("/apps/<uuid:app_id>/annotations/batch-import-status/<uuid:job_id>")
@@ -466,9 +491,7 @@ class AnnotationBatchImportStatusApi(Resource):
     @console_ns.doc(description="Get status of batch import job")
     @console_ns.doc(params={"app_id": "Application ID", "job_id": "Job ID"})
     @console_ns.response(
-        200,
-        "Job status retrieved successfully",
-        console_ns.models[AnnotationJobStatusResponse.__name__],
+        200, "Job status retrieved successfully", console_ns.models[AnnotationJobStatusDetailResponse.__name__]
     )
     @console_ns.response(403, "Insufficient permissions")
     @setup_required
@@ -488,7 +511,9 @@ class AnnotationBatchImportStatusApi(Resource):
             indexing_error_msg_key = f"app_annotation_batch_import_error_msg_{str(job_id)}"
             error_msg = redis_client.get(indexing_error_msg_key).decode()
 
-        return {"job_id": job_id, "job_status": job_status, "error_msg": error_msg}, 200
+        return AnnotationJobStatusDetailResponse(
+            job_id=str(job_id), job_status=job_status, error_msg=error_msg
+        ).model_dump(mode="json"), 200
 
 
 @console_ns.route("/apps/<uuid:app_id>/annotations/<uuid:annotation_id>/hit-histories")
@@ -511,17 +536,17 @@ class AnnotationHitHistoryListApi(Resource):
     def get(self, app_id: UUID, annotation_id: UUID):
         page = request.args.get("page", default=1, type=int)
         limit = request.args.get("limit", default=20, type=int)
+        app_ref = _get_app_ref(str(app_id))
+        annotation_ref = AppRefService.create_annotation_ref(app_ref, str(annotation_id))
         annotation_hit_history_list, total = AppAnnotationService.get_annotation_hit_histories(
-            str(app_id), str(annotation_id), page, limit
+            annotation_ref,
+            page,
+            limit,
+            session=db.session(),
         )
         history_models = TypeAdapter(list[AnnotationHitHistory]).validate_python(
             annotation_hit_history_list, from_attributes=True
         )
-        response = AnnotationHitHistoryList(
-            data=history_models,
-            has_more=len(annotation_hit_history_list) == limit,
-            limit=limit,
-            total=total,
-            page=page,
-        )
-        return response.model_dump(mode="json")
+        return AnnotationHitHistoryList(
+            data=history_models, has_more=len(annotation_hit_history_list) == limit, limit=limit, total=total, page=page
+        ).model_dump(mode="json")

api/controllers/console/app/app.py

@@ -1,5 +1,4 @@
 import logging
-import re
 import uuid
 from collections.abc import Sequence
 from datetime import datetime
@@ -10,14 +9,15 @@ from flask_restx import Resource
 from pydantic import AliasChoices, BaseModel, Field, computed_field, field_validator
 from sqlalchemy import select
 from sqlalchemy.orm import Session
-from werkzeug.datastructures import MultiDict
 from werkzeug.exceptions import BadRequest, NotFound
 
 from configs import dify_config
+from controllers.common.app_access import resolve_app_access_filter
 from controllers.common.fields import RedirectUrlResponse, SimpleResultResponse
 from controllers.common.helpers import FileInfo
 from controllers.common.schema import (
     query_params_from_model,
+    query_params_from_request,
     register_enum_models,
     register_response_schema_models,
     register_schema_models,
@@ -74,11 +74,9 @@ ALLOW_CREATE_APP_MODES = ["chat", "agent-chat", "advanced-chat", "workflow", "co
 register_enum_models(console_ns, IconType)
 
 _logger = logging.getLogger(__name__)
-_TAG_IDS_BRACKET_PATTERN = re.compile(r"^tag_ids\[(\d+)\]$")
-_CREATOR_IDS_BRACKET_PATTERN = re.compile(r"^creator_ids\[(\d+)\]$")
 AppListMode = Literal["completion", "chat", "advanced-chat", "workflow", "agent-chat", "agent", "channel", "all"]
 DEFAULT_APP_LIST_MODE: AppListMode = "all"
-APP_LIST_PERMISSION_KEYS = frozenset({"app.preview", "app.acl.preview", "app.full_access"})
+APP_LIST_QUERY_ARRAY_FIELDS = ("tag_ids", "creator_ids")
 
 
 class AppListBaseQuery(BaseModel):
@@ -139,38 +137,6 @@ class StarredAppListQuery(AppListBaseQuery):
     pass
 
 
-def _normalize_app_list_query_args(query_args: MultiDict[str, str]) -> dict[str, str | list[str]]:
-    normalized: dict[str, str | list[str]] = {}
-    indexed_tag_ids: list[tuple[int, str]] = []
-    indexed_creator_ids: list[tuple[int, str]] = []
-
-    for key in query_args:
-        match = _TAG_IDS_BRACKET_PATTERN.fullmatch(key)
-        if match:
-            indexed_tag_ids.extend((int(match.group(1)), value) for value in query_args.getlist(key))
-            continue
-
-        match = _CREATOR_IDS_BRACKET_PATTERN.fullmatch(key)
-        if match:
-            indexed_creator_ids.extend((int(match.group(1)), value) for value in query_args.getlist(key))
-            continue
-
-        value = query_args.get(key)
-        if value is not None:
-            normalized[key] = value
-
-    if indexed_tag_ids:
-        normalized["tag_ids"] = [value for _, value in sorted(indexed_tag_ids)]
-    if indexed_creator_ids:
-        normalized["creator_ids"] = [value for _, value in sorted(indexed_creator_ids)]
-
-    return normalized
-
-
-def _has_app_list_permission(permission_keys: Sequence[str]) -> bool:
-    return any(permission_key in APP_LIST_PERMISSION_KEYS for permission_key in permission_keys)
-
-
 class CreateAppPayload(BaseModel):
     name: str = Field(..., min_length=1, description="App name")
     description: str | None = Field(default=None, description="App description (max 400 chars)", max_length=400)
@@ -234,13 +200,10 @@ class AppTracePayload(BaseModel):
 
 
 class AppTraceResponse(ResponseModel):
-    enabled: bool
+    enabled: bool = False
     tracing_provider: str | None = None
 
 
-type JSONValue = Any
-
-
 class Tag(ResponseModel):
     id: str
     name: str
@@ -261,7 +224,7 @@ class WorkflowPartial(ResponseModel):
 
 
 class ModelConfigPartial(ResponseModel):
-    model: JSONValue | None = Field(default=None, validation_alias=AliasChoices("model_dict", "model"))
+    model: Any | None = Field(default=None, validation_alias=AliasChoices("model_dict", "model"))
     pre_prompt: str | None = None
     created_by: str | None = None
     created_at: int | None = None
@@ -276,54 +239,52 @@ class ModelConfigPartial(ResponseModel):
 
 class ModelConfig(ResponseModel):
     opening_statement: str | None = None
-    suggested_questions: JSONValue | None = Field(
+    suggested_questions: Any | None = Field(
         default=None, validation_alias=AliasChoices("suggested_questions_list", "suggested_questions")
     )
-    suggested_questions_after_answer: JSONValue | None = Field(
+    suggested_questions_after_answer: Any | None = Field(
         default=None,
         validation_alias=AliasChoices("suggested_questions_after_answer_dict", "suggested_questions_after_answer"),
     )
-    speech_to_text: JSONValue | None = Field(
+    speech_to_text: Any | None = Field(
         default=None, validation_alias=AliasChoices("speech_to_text_dict", "speech_to_text")
     )
-    text_to_speech: JSONValue | None = Field(
+    text_to_speech: Any | None = Field(
         default=None, validation_alias=AliasChoices("text_to_speech_dict", "text_to_speech")
     )
-    retriever_resource: JSONValue | None = Field(
+    retriever_resource: Any | None = Field(
         default=None, validation_alias=AliasChoices("retriever_resource_dict", "retriever_resource")
     )
-    annotation_reply: JSONValue | None = Field(
+    annotation_reply: Any | None = Field(
         default=None, validation_alias=AliasChoices("annotation_reply_dict", "annotation_reply")
     )
-    more_like_this: JSONValue | None = Field(
+    more_like_this: Any | None = Field(
         default=None, validation_alias=AliasChoices("more_like_this_dict", "more_like_this")
     )
-    sensitive_word_avoidance: JSONValue | None = Field(
+    sensitive_word_avoidance: Any | None = Field(
         default=None, validation_alias=AliasChoices("sensitive_word_avoidance_dict", "sensitive_word_avoidance")
     )
-    external_data_tools: JSONValue | None = Field(
+    external_data_tools: Any | None = Field(
         default=None, validation_alias=AliasChoices("external_data_tools_list", "external_data_tools")
     )
-    model: JSONValue | None = Field(default=None, validation_alias=AliasChoices("model_dict", "model"))
-    user_input_form: JSONValue | None = Field(
+    model: Any | None = Field(default=None, validation_alias=AliasChoices("model_dict", "model"))
+    user_input_form: Any | None = Field(
         default=None, validation_alias=AliasChoices("user_input_form_list", "user_input_form")
     )
     dataset_query_variable: str | None = None
     pre_prompt: str | None = None
-    agent_mode: JSONValue | None = Field(default=None, validation_alias=AliasChoices("agent_mode_dict", "agent_mode"))
+    agent_mode: Any | None = Field(default=None, validation_alias=AliasChoices("agent_mode_dict", "agent_mode"))
     prompt_type: str | None = None
-    chat_prompt_config: JSONValue | None = Field(
+    chat_prompt_config: Any | None = Field(
         default=None, validation_alias=AliasChoices("chat_prompt_config_dict", "chat_prompt_config")
     )
-    completion_prompt_config: JSONValue | None = Field(
+    completion_prompt_config: Any | None = Field(
         default=None, validation_alias=AliasChoices("completion_prompt_config_dict", "completion_prompt_config")
     )
-    dataset_configs: JSONValue | None = Field(
+    dataset_configs: Any | None = Field(
         default=None, validation_alias=AliasChoices("dataset_configs_dict", "dataset_configs")
     )
-    file_upload: JSONValue | None = Field(
-        default=None, validation_alias=AliasChoices("file_upload_dict", "file_upload")
-    )
+    file_upload: Any | None = Field(default=None, validation_alias=AliasChoices("file_upload_dict", "file_upload"))
     created_by: str | None = None
     created_at: int | None = None
     updated_by: str | None = None
@@ -335,7 +296,7 @@ class ModelConfig(ResponseModel):
         return to_timestamp(value)
 
 
-class Site(ResponseModel):
+class AppDetailSiteResponse(ResponseModel):
     access_token: str | None = Field(default=None, validation_alias="code")
     code: str | None = None
     title: str | None = None
@@ -349,6 +310,7 @@ class Site(ResponseModel):
     customize_domain: str | None = None
     copyright: str | None = None
     privacy_policy: str | None = None
+    input_placeholder: str | None = None
     custom_disclaimer: str | None = None
     customize_token_strategy: str | None = None
     prompt_public: bool | None = None
@@ -443,7 +405,7 @@ class AppDetail(ResponseModel):
         alias="model_config",
     )
     workflow: WorkflowPartial | None = None
-    tracing: JSONValue | None = None
+    tracing: Any | None = None
     use_icon_as_answer_icon: bool | None = None
     created_by: str | None = None
     created_at: int | None = None
@@ -465,7 +427,7 @@ class AppDetailWithSite(AppDetail):
     api_base_url: str | None = None
     max_active_requests: int | None = None
     deleted_tools: list[DeletedTool] = Field(default_factory=list)
-    site: Site | None = None
+    site: AppDetailSiteResponse | None = None
     # For Agent App type: the roster Agent backing this app (None otherwise).
     bound_agent_id: str | None = None
     # For Agent App responses exposed through /agent.
@@ -489,6 +451,16 @@ class AppExportResponse(ResponseModel):
     data: str
 
 
+class AppImportResponse(ResponseModel):
+    id: str
+    status: ImportStatus
+    app_id: str | None = None
+    app_mode: str | None = None
+    current_dsl_version: str
+    imported_dsl_version: str = ""
+    error: str = ""
+
+
 def _enrich_app_list_items(session: Session, *, apps: Sequence[App], tenant_id: str) -> None:
     if FeatureService.get_system_features().webapp_auth.enabled:
         app_ids = [str(app.id) for app in apps]
@@ -531,7 +503,9 @@ def _enrich_app_list_items(session: Session, *, apps: Sequence[App], tenant_id:
 
 
 register_enum_models(console_ns, RetrievalMethod, WorkflowExecutionStatus, DatasetPermissionEnum)
-register_response_schema_models(console_ns, AppTraceResponse, RedirectUrlResponse, SimpleResultResponse)
+register_response_schema_models(
+    console_ns, RedirectUrlResponse, SimpleResultResponse, AppImportResponse, AppTraceResponse
+)
 
 register_schema_models(
     console_ns,
@@ -550,7 +524,7 @@ register_schema_models(
     WorkflowPartial,
     ModelConfigPartial,
     ModelConfig,
-    Site,
+    AppDetailSiteResponse,
     DeletedTool,
     AppDetail,
     AppExportResponse,
@@ -595,7 +569,7 @@ class AppListApi(Resource):
     @with_current_tenant_id
     def get(self, current_tenant_id: str, current_user_id: str, session: Session):
         """Get app list"""
-        args = AppListQuery.model_validate(_normalize_app_list_query_args(request.args))
+        args = query_params_from_request(AppListQuery, list_fields=APP_LIST_QUERY_ARRAY_FIELDS)
         params = AppListParams(
             page=args.page,
             limit=args.limit,
@@ -610,47 +584,22 @@ class AppListApi(Resource):
         permissions = enterprise_rbac_service.RBACService.MyPermissions.get(
             str(current_tenant_id),
             current_user_id,
+            session=db.session(),
         )
         if dify_config.RBAC_ENABLED:
-            whitelist_scope = enterprise_rbac_service.RBACService.AppAccess.whitelist_resources(
+            access_filter = resolve_app_access_filter(
                 str(current_tenant_id),
                 current_user_id,
+                permissions=permissions,
             )
-            can_manage_own_apps = "app.create_and_management" in permissions.workspace.permission_keys
-            has_default_preview = _has_app_list_permission(
-                permissions.app.default_permission_keys
-            ) or _has_app_list_permission(permissions.workspace.permission_keys)
-            permission_app_ids: set[str] | None = None
-            if not has_default_preview:
-                permission_app_ids = {
-                    override.resource_id
-                    for override in permissions.app.overrides
-                    if _has_app_list_permission(override.permission_keys)
-                }
-
-            if getattr(whitelist_scope, "unrestricted", False):
-                accessible_app_ids = permission_app_ids
-            else:
-                accessible_app_ids = set(whitelist_scope.resource_ids)
-                if permission_app_ids is not None:
-                    accessible_app_ids |= permission_app_ids
-                elif has_default_preview:
-                    accessible_app_ids = None
-
-            if accessible_app_ids:
-                params.accessible_app_ids = sorted(accessible_app_ids)
-                params.include_own_apps = can_manage_own_apps
-            elif accessible_app_ids is not None and can_manage_own_apps:
-                params.is_created_by_me = True
-            elif accessible_app_ids is not None:
-                params.accessible_app_ids = []
+            access_filter.apply_to_params(params)
 
         # get app list
         app_service = AppService()
-        app_pagination = app_service.get_paginate_apps(current_user_id, current_tenant_id, params, db.session)
+        app_pagination = app_service.get_paginate_apps(current_user_id, current_tenant_id, params, session)
         if not app_pagination:
-            empty = AppPagination(page=args.page, limit=args.limit, total=0, has_more=False, data=[])
-            return empty.model_dump(mode="json"), 200
+            response = AppPagination(page=args.page, limit=args.limit, total=0, has_more=False, data=[])
+            return response.model_dump(mode="json"), 200
 
         app_ids = [str(app.id) for app in app_pagination.items]
         permission_keys_map = permissions.app.permission_keys_by_resource_ids(app_ids)
@@ -695,11 +644,12 @@ class AppListApi(Resource):
         )
 
         app_service = AppService()
-        app = app_service.create_app(current_tenant_id, params, current_user)
+        app = app_service.create_app(current_tenant_id, params, current_user, session=db.session())
         permission_keys_map = enterprise_rbac_service.RBACService.AppPermissions.batch_get(
             str(current_tenant_id),
             current_user.id,
             [str(app.id)],
+            session=db.session(),
         )
         app_detail = AppDetailWithSite.model_validate(app, from_attributes=True).model_copy(
             update={"permission_keys": permission_keys_map.get(str(app.id), [])}
@@ -721,7 +671,7 @@ class StarredAppListApi(Resource):
     @with_current_user_id
     @with_current_tenant_id
     def get(self, current_tenant_id: str, current_user_id: str, session: Session):
-        args = StarredAppListQuery.model_validate(_normalize_app_list_query_args(request.args))
+        args = query_params_from_request(StarredAppListQuery, list_fields=APP_LIST_QUERY_ARRAY_FIELDS)
         params = StarredAppListParams(
             page=args.page,
             limit=args.limit,
@@ -733,15 +683,13 @@ class StarredAppListApi(Resource):
             is_created_by_me=args.is_created_by_me,
         )
 
-        app_pagination = AppService().get_paginate_starred_apps(current_user_id, current_tenant_id, params, db.session)
+        app_pagination = AppService().get_paginate_starred_apps(current_user_id, current_tenant_id, params, session)
         if not app_pagination:
             empty = AppPagination(page=args.page, limit=args.limit, total=0, has_more=False, data=[])
             return empty.model_dump(mode="json"), 200
 
         _enrich_app_list_items(session, apps=app_pagination.items, tenant_id=current_tenant_id)
-
-        pagination_model = AppPagination.model_validate(app_pagination, from_attributes=True)
-        return pagination_model.model_dump(mode="json"), 200
+        return AppPagination.model_validate(app_pagination, from_attributes=True).model_dump(mode="json"), 200
 
 
 @console_ns.route("/apps/<uuid:app_id>/star")
@@ -760,7 +708,7 @@ class AppStarApi(Resource):
     @get_app_model(mode=None)
     def post(self, session: Session, current_user_id: str, app_model: App):
         AppService.star_app(session, app=app_model, account_id=current_user_id)
-        return dump_response(SimpleResultResponse, {"result": "success"})
+        return SimpleResultResponse(result="success").model_dump(mode="json")
 
     @console_ns.doc("unstar_app")
     @console_ns.doc(description="Remove the current account's star from an application")
@@ -776,7 +724,7 @@ class AppStarApi(Resource):
     @get_app_model(mode=None)
     def delete(self, session: Session, current_user_id: str, app_model: App):
         AppService.unstar_app(session, app=app_model, account_id=current_user_id)
-        return dump_response(SimpleResultResponse, {"result": "success"})
+        return SimpleResultResponse(result="success").model_dump(mode="json")
 
 
 @console_ns.route("/apps/<uuid:app_id>")
@@ -807,6 +755,7 @@ class AppApi(Resource):
             str(current_tenant_id),
             current_user.id,
             app_id=str(app_model.id),
+            session=db.session(),
         )
         permission_keys_map = permissions.app.permission_keys_by_resource_ids([str(app_model.id)])
 
@@ -843,9 +792,8 @@ class AppApi(Resource):
             "use_icon_as_answer_icon": args.use_icon_as_answer_icon or False,
             "max_active_requests": args.max_active_requests or 0,
         }
-        app_model = app_service.update_app(app_model, args_dict)
-        response_model = AppDetailWithSite.model_validate(app_model, from_attributes=True)
-        return response_model.model_dump(mode="json")
+        app_model = app_service.update_app(app_model, args_dict, session=db.session())
+        return dump_response(AppDetailWithSite, app_model)
 
     @console_ns.doc("delete_app")
     @console_ns.doc(description="Delete application")
@@ -861,7 +809,7 @@ class AppApi(Resource):
     def delete(self, app_model: App):
         """Delete app"""
         app_service = AppService()
-        app_service.delete_app(app_model)
+        app_service.delete_app(app_model, session=db.session())
 
         return "", 204
 
@@ -873,6 +821,7 @@ class AppCopyApi(Resource):
     @console_ns.doc(params={"app_id": "Application ID to copy"})
     @console_ns.expect(console_ns.models[CopyAppPayload.__name__])
     @console_ns.response(201, "App copied successfully", console_ns.models[AppDetailWithSite.__name__])
+    @console_ns.response(202, "App copy requires confirmation", console_ns.models[AppImportResponse.__name__])
     @console_ns.response(403, "Insufficient permissions")
     @setup_required
     @login_required
@@ -889,7 +838,7 @@ class AppCopyApi(Resource):
 
         with Session(db.engine, expire_on_commit=False) as session:
             import_service = AppDslService(session)
-            yaml_content = import_service.export_dsl(app_model=app_model, include_secret=True)
+            yaml_content = import_service.export_dsl(app_model=app_model, session=session, include_secret=True)
             result = import_service.import_app(
                 account=current_user,
                 import_mode=ImportMode.YAML_CONTENT,
@@ -902,10 +851,10 @@ class AppCopyApi(Resource):
             )
             if result.status == ImportStatus.FAILED:
                 session.rollback()
-                return result.model_dump(mode="json"), 400
+                return dump_response(AppImportResponse, result), 400
             if result.status == ImportStatus.PENDING:
                 session.rollback()
-                return result.model_dump(mode="json"), 202
+                return dump_response(AppImportResponse, result), 202
             session.commit()
 
             # Inherit web app permission from original app
@@ -931,6 +880,7 @@ class AppCopyApi(Resource):
             str(current_tenant_id),
             current_user.id,
             [str(app.id)],
+            session=db.session(),
         )
         response_model = AppDetailWithSite.model_validate(app, from_attributes=True).model_copy(
             update={"permission_keys": permission_keys_map.get(str(app.id), [])}
@@ -956,14 +906,15 @@ class AppExportApi(Resource):
         """Export app"""
         args = AppExportQuery.model_validate(request.args.to_dict(flat=True))
 
-        payload = AppExportResponse(
+        response = AppExportResponse(
             data=AppDslService.export_dsl(
                 app_model=app_model,
+                session=db.session(),
                 include_secret=args.include_secret,
                 workflow_id=args.workflow_id,
             )
         )
-        return payload.model_dump(mode="json")
+        return response.model_dump(mode="json")
 
 
 @console_ns.route("/apps/<uuid:app_id>/publish-to-creators-platform")
@@ -983,13 +934,13 @@ class AppPublishToCreatorsPlatformApi(Resource):
         if not dify_config.CREATORS_PLATFORM_FEATURES_ENABLED:
             return {"error": "Creators Platform features are not enabled"}, 403
 
-        dsl_content = AppDslService.export_dsl(app_model=app_model, include_secret=False)
+        dsl_content = AppDslService.export_dsl(app_model=app_model, session=db.session(), include_secret=False)
         dsl_bytes = dsl_content.encode("utf-8")
 
         claim_code = upload_dsl(dsl_bytes)
         redirect_url = get_redirect_url(current_user_id, claim_code)
 
-        return {"redirect_url": redirect_url}
+        return RedirectUrlResponse(redirect_url=redirect_url).model_dump(mode="json")
 
 
 @console_ns.route("/apps/<uuid:app_id>/name")
@@ -1009,9 +960,8 @@ class AppNameApi(Resource):
         args = AppNamePayload.model_validate(console_ns.payload)
 
         app_service = AppService()
-        app_model = app_service.update_app_name(app_model, args.name)
-        response_model = AppDetail.model_validate(app_model, from_attributes=True)
-        return response_model.model_dump(mode="json")
+        app_model = app_service.update_app_name(app_model, args.name, session=db.session())
+        return dump_response(AppDetail, app_model)
 
 
 @console_ns.route("/apps/<uuid:app_id>/icon")
@@ -1037,9 +987,9 @@ class AppIconApi(Resource):
             args.icon or "",
             args.icon_background or "",
             args.icon_type,
+            session=db.session(),
         )
-        response_model = AppDetail.model_validate(app_model, from_attributes=True)
-        return response_model.model_dump(mode="json")
+        return dump_response(AppDetail, app_model)
 
 
 @console_ns.route("/apps/<uuid:app_id>/site-enable")
@@ -1060,9 +1010,8 @@ class AppSiteStatus(Resource):
         args = AppSiteStatusPayload.model_validate(console_ns.payload)
 
         app_service = AppService()
-        app_model = app_service.update_app_site_status(app_model, args.enable_site)
-        response_model = AppDetail.model_validate(app_model, from_attributes=True)
-        return response_model.model_dump(mode="json")
+        app_model = app_service.update_app_site_status(app_model, args.enable_site, session=db.session())
+        return dump_response(AppDetail, app_model)
 
 
 @console_ns.route("/apps/<uuid:app_id>/api-enable")
@@ -1083,9 +1032,8 @@ class AppApiStatus(Resource):
         args = AppApiStatusPayload.model_validate(console_ns.payload)
 
         app_service = AppService()
-        app_model = app_service.update_app_api_status(app_model, args.enable_api)
-        response_model = AppDetail.model_validate(app_model, from_attributes=True)
-        return response_model.model_dump(mode="json")
+        app_model = app_service.update_app_api_status(app_model, args.enable_api, session=db.session())
+        return dump_response(AppDetail, app_model)
 
 
 @console_ns.route("/apps/<uuid:app_id>/trace")
@@ -1108,7 +1056,7 @@ class AppTraceApi(Resource):
         """Get app trace"""
         app_trace_config = OpsTraceManager.get_app_tracing_config(app_model.id, session)
 
-        return app_trace_config
+        return dump_response(AppTraceResponse, app_trace_config)
 
     @console_ns.doc("update_app_trace")
     @console_ns.doc(description="Update app tracing configuration")
@@ -1124,7 +1072,7 @@ class AppTraceApi(Resource):
     @login_required
     @account_initialization_required
     @edit_permission_required
-    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR)
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_TRACING_CONFIG)
     @get_app_model
     def post(self, app_model: App):
         # add app trace
@@ -1136,4 +1084,4 @@ class AppTraceApi(Resource):
             tracing_provider=args.tracing_provider,
         )
 
-        return {"result": "success"}
+        return SimpleResultResponse(result="success").model_dump(mode="json")

api/controllers/console/app/audio.py

@@ -1,5 +1,4 @@
 import logging
-from typing import Any
 
 from flask import request
 from flask_restx import Resource
@@ -7,7 +6,6 @@ from pydantic import BaseModel, Field, RootModel
 from werkzeug.exceptions import InternalServerError
 
 import services
-from controllers.common.fields import AudioBinaryResponse
 from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models
 from controllers.console import console_ns
 from controllers.console.app.error import (
@@ -30,9 +28,13 @@ from controllers.console.wraps import (
     setup_required,
 )
 from core.errors.error import ModelCurrentlyNotSupportError, ProviderTokenNotInitError, QuotaExceededError
+from extensions.ext_database import db
+from fields.base import ResponseModel
 from graphon.model_runtime.errors.invoke import InvokeError
-from libs.login import login_required
+from libs.helper import dump_response
+from libs.login import current_user, login_required
 from models import App, AppMode
+from services.app_ref_service import AppRefService
 from services.audio_service import AudioService
 from services.errors.audio import (
     AudioTooLargeServiceError,
@@ -55,16 +57,27 @@ class TextToSpeechVoiceQuery(BaseModel):
     language: str = Field(..., description="Language code")
 
 
-class AudioTranscriptResponse(BaseModel):
+class AudioTranscriptResponse(ResponseModel):
     text: str = Field(description="Transcribed text from audio")
 
 
-class TextToSpeechVoiceListResponse(RootModel[list[dict[str, Any]]]):
-    root: list[dict[str, Any]]
+class TextToSpeechVoiceResponse(ResponseModel):
+    # see api/core/plugin/impl/model.py
+    name: str = Field(description="Voice display name")
+    value: str = Field(description="Voice identifier")
 
 
-register_schema_models(console_ns, AudioTranscriptResponse, TextToSpeechPayload, TextToSpeechVoiceQuery)
-register_response_schema_models(console_ns, AudioBinaryResponse, TextToSpeechVoiceListResponse)
+class TextToSpeechVoiceListResponse(RootModel[list[TextToSpeechVoiceResponse]]):
+    root: list[TextToSpeechVoiceResponse] = Field(description="Available voices")
+
+
+register_schema_models(console_ns, TextToSpeechPayload, TextToSpeechVoiceQuery)
+register_response_schema_models(
+    console_ns,
+    AudioTranscriptResponse,
+    TextToSpeechVoiceResponse,
+    TextToSpeechVoiceListResponse,
+)
 
 
 @console_ns.route("/apps/<uuid:app_id>/audio-to-text")
@@ -93,7 +106,7 @@ class ChatMessageAudioApi(Resource):
                 end_user=None,
             )
 
-            return response
+            return dump_response(AudioTranscriptResponse, response)
         except services.errors.app_model_config.AppModelConfigBrokenError:
             logger.exception("App model config broken.")
             raise AppUnavailableError()
@@ -126,11 +139,8 @@ class ChatMessageTextApi(Resource):
     @console_ns.doc(description="Convert text to speech for chat messages")
     @console_ns.doc(params={"app_id": "App ID"})
     @console_ns.expect(console_ns.models[TextToSpeechPayload.__name__])
-    @console_ns.response(
-        200,
-        "Text to speech conversion successful",
-        console_ns.models[AudioBinaryResponse.__name__],
-    )
+    # TTS returns provider audio bytes, so the success response is intentionally schema-less.
+    @console_ns.response(200, "Text to speech conversion successful")
     @console_ns.response(400, "Bad request - Invalid parameters")
     @setup_required
     @login_required
@@ -139,15 +149,24 @@ class ChatMessageTextApi(Resource):
     def post(self, app_model: App):
         try:
             payload = TextToSpeechPayload.model_validate(console_ns.payload)
+            message_ref = None
+            if payload.message_id:
+                app_ref = AppRefService.create_app_ref(app_model)
+                message_ref = AppRefService.create_message_ref(
+                    app_ref,
+                    payload.message_id,
+                    account_id=current_user.id,
+                )
 
-            response = AudioService.transcript_tts(
+            # response-contract:ignore
+            return AudioService.transcript_tts(
                 app_model=app_model,
+                session=db.session(),
                 text=payload.text,
                 voice=payload.voice,
-                message_id=payload.message_id,
+                message_ref=message_ref,
                 is_draft=True,
             )
-            return response
         except services.errors.app_model_config.AppModelConfigBrokenError:
             logger.exception("App model config broken.")
             raise AppUnavailableError()
@@ -178,8 +197,7 @@ class ChatMessageTextApi(Resource):
 class TextModesApi(Resource):
     @console_ns.doc("get_text_to_speech_voices")
     @console_ns.doc(description="Get available TTS voices for a specific language")
-    @console_ns.doc(params={"app_id": "App ID"})
-    @console_ns.doc(params=query_params_from_model(TextToSpeechVoiceQuery))
+    @console_ns.doc(params={"app_id": "App ID", **query_params_from_model(TextToSpeechVoiceQuery)})
     @console_ns.response(
         200,
         "TTS voices retrieved successfully",
@@ -200,7 +218,7 @@ class TextModesApi(Resource):
                 language=args.language,
             )
 
-            return response
+            return dump_response(TextToSpeechVoiceListResponse, response)
         except services.errors.audio.ProviderNotSupportTextToSpeechLanageServiceError:
             raise AppUnavailableError("Text to audio voices language parameter loss.")
         except NoAudioUploadedServiceError:

api/controllers/console/app/completion.py

@@ -1,17 +1,20 @@
+import json
 import logging
-from typing import Any, Literal
+from collections.abc import Generator, Iterator, Mapping
+from typing import Any, Literal, Protocol, runtime_checkable
 from uuid import UUID
 
 from flask import request
 from flask_restx import Resource
 from pydantic import BaseModel, Field, field_validator
+from sqlalchemy.orm import Session
 from werkzeug.exceptions import BadRequest, InternalServerError, NotFound
 
 import services
-from controllers.common.fields import GeneratedAppResponse, SimpleResultResponse
+from controllers.common.fields import SimpleResultResponse
 from controllers.common.schema import register_response_schema_models, register_schema_models
 from controllers.console import console_ns
-from controllers.console.agent.app_helpers import resolve_agent_app_model
+from controllers.console.agent.app_helpers import resolve_agent_runtime_app_model
 from controllers.console.app.error import (
     AppUnavailableError,
     CompletionRequestError,
@@ -20,7 +23,7 @@ from controllers.console.app.error import (
     ProviderNotInitializeError,
     ProviderQuotaExceededError,
 )
-from controllers.console.app.wraps import get_app_model
+from controllers.console.app.wraps import get_app_model, with_session
 from controllers.console.wraps import (
     RBACPermission,
     RBACResourceScope,
@@ -34,18 +37,22 @@ from controllers.console.wraps import (
 )
 from controllers.web.error import InvokeRateLimitError as InvokeRateLimitHttpError
 from core.app.entities.app_invoke_entities import InvokeFrom
+from core.app.features.rate_limiting.rate_limit import RateLimitGenerator
 from core.errors.error import (
     ModelCurrentlyNotSupportError,
     ProviderTokenNotInitError,
     QuotaExceededError,
 )
 from core.helper.trace_id_helper import get_external_trace_id
+from extensions.ext_database import db
 from graphon.model_runtime.errors.invoke import InvokeError
 from libs import helper
 from libs.helper import uuid_value
 from libs.login import login_required
 from models import Account
 from models.model import App, AppMode
+from services.agent.errors import AgentNotFoundError
+from services.agent.roster_service import AgentRosterService
 from services.app_generate_service import AppGenerateService
 from services.app_task_service import AppTaskService
 from services.errors.llm import InvokeRateLimitError
@@ -53,6 +60,11 @@ from services.errors.llm import InvokeRateLimitError
 logger = logging.getLogger(__name__)
 
 
+@runtime_checkable
+class _ClosableStream(Protocol):
+    def close(self) -> None: ...
+
+
 def _resolve_debugger_chat_streaming(
     *, app_mode: AppMode, response_mode: str, response_mode_provided: bool = True
 ) -> bool:
@@ -90,6 +102,10 @@ class ChatMessagePayload(BaseMessagePayload):
     query: str = Field(..., description="User query")
     conversation_id: str | None = Field(default=None, description="Conversation ID")
     parent_message_id: str | None = Field(default=None, description="Parent message ID")
+    draft_type: Literal["draft", "debug_build"] = Field(
+        default="draft",
+        description="Agent App debug config source. Use debug_build while the Agent is in build mode.",
+    )
 
     @field_validator("conversation_id", "parent_message_id")
     @classmethod
@@ -99,8 +115,32 @@ class ChatMessagePayload(BaseMessagePayload):
         return uuid_value(value)
 
 
+_BUILD_CHAT_FINALIZATION_QUERY = """Finalize this Build chat configuration for the agent.
+
+This step is only for persisting Agent config changes discovered in the current Build chat. Do not install packages,
+edit workspace files, run validation or debugging commands, make exploratory checks, or perform other work.
+
+Use only the current Build chat message history to identify changes that need to be persisted. Do not inspect, test, or
+validate old config unless the message history already shows that the old config is invalid.
+
+Only update the build-draft config note when the current Build chat contains durable context that later runs need.
+Write the config note in the language used by the message history.
+Do not create, update, delete, inspect, or fill gaps in other Agent config resources, including config files, config
+skills, config env, tools, models, knowledge, or prompt settings.
+
+When updating the config note with the Agent config CLI usage provided in the runtime prompt, record only durable
+context needed by later runs, such as:
+
+- what you installed or configured outside the workspace for this agent,
+- where those external updates live, including CLI tools, packages, and persistent $HOME paths,
+- how the agent should use it in later runs,
+- any setup, authentication, or user action still required.
+
+After config persistence completes, respond FINISHED."""
+
+
 register_schema_models(console_ns, CompletionMessagePayload, ChatMessagePayload)
-register_response_schema_models(console_ns, GeneratedAppResponse, SimpleResultResponse)
+register_response_schema_models(console_ns, SimpleResultResponse)
 
 
 # define completion message api for user
@@ -110,7 +150,7 @@ class CompletionMessageApi(Resource):
     @console_ns.doc(description="Generate completion message for debugging")
     @console_ns.doc(params={"app_id": "Application ID"})
     @console_ns.expect(console_ns.models[CompletionMessagePayload.__name__])
-    @console_ns.response(200, "Completion generated successfully", console_ns.models[GeneratedAppResponse.__name__])
+    @console_ns.response(200, "Completion generated successfully")
     @console_ns.response(400, "Invalid request parameters")
     @console_ns.response(404, "App not found")
     @setup_required
@@ -119,7 +159,8 @@ class CompletionMessageApi(Resource):
     @with_current_user
     @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_TEST_AND_RUN)
     @get_app_model(mode=AppMode.COMPLETION)
-    def post(self, current_user: Account, app_model: App):
+    @with_session
+    def post(self, session: Session, current_user: Account, app_model: App):
         args_model = CompletionMessagePayload.model_validate(console_ns.payload)
         args = args_model.model_dump(exclude_none=True, by_alias=True)
 
@@ -128,9 +169,15 @@ class CompletionMessageApi(Resource):
 
         try:
             response = AppGenerateService.generate(
-                app_model=app_model, user=current_user, args=args, invoke_from=InvokeFrom.DEBUGGER, streaming=streaming
+                session=session,
+                app_model=app_model,
+                user=current_user,
+                args=args,
+                invoke_from=InvokeFrom.DEBUGGER,
+                streaming=streaming,
             )
 
+            # response-contract:ignore compact_generate_response
             return helper.compact_generate_response(response)
         except services.errors.conversation.ConversationNotExistsError:
             raise NotFound("Conversation Not Exists.")
@@ -174,7 +221,7 @@ class CompletionMessageStopApi(Resource):
             app_mode=AppMode.value_of(app_model.mode),
         )
 
-        return {"result": "success"}, 200
+        return SimpleResultResponse(result="success").model_dump(mode="json"), 200
 
 
 @console_ns.route("/apps/<uuid:app_id>/chat-messages")
@@ -183,7 +230,7 @@ class ChatMessageApi(Resource):
     @console_ns.doc(description="Generate chat message for debugging")
     @console_ns.doc(params={"app_id": "Application ID"})
     @console_ns.expect(console_ns.models[ChatMessagePayload.__name__])
-    @console_ns.response(200, "Chat message generated successfully", console_ns.models[GeneratedAppResponse.__name__])
+    @console_ns.response(200, "Chat message generated successfully")
     @console_ns.response(400, "Invalid request parameters")
     @console_ns.response(404, "App or conversation not found")
     @setup_required
@@ -191,10 +238,14 @@ class ChatMessageApi(Resource):
     @account_initialization_required
     @edit_permission_required
     @with_current_user
+    @with_current_tenant_id
     @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_TEST_AND_RUN)
     @get_app_model(mode=[AppMode.CHAT, AppMode.AGENT_CHAT, AppMode.AGENT])
-    def post(self, current_user: Account, app_model: App):
-        return _create_chat_message(current_user=current_user, app_model=app_model)
+    @with_session
+    def post(self, session: Session, current_tenant_id: str, current_user: Account, app_model: App):
+        return _create_chat_message(
+            session=session, current_tenant_id=current_tenant_id, current_user=current_user, app_model=app_model
+        )
 
 
 @console_ns.route("/agent/<uuid:agent_id>/chat-messages")
@@ -203,7 +254,7 @@ class AgentChatMessageApi(Resource):
     @console_ns.doc(description="Generate an Agent App chat message for debugging")
     @console_ns.doc(params={"agent_id": "Agent ID"})
     @console_ns.expect(console_ns.models[ChatMessagePayload.__name__])
-    @console_ns.response(200, "Chat message generated successfully", console_ns.models[GeneratedAppResponse.__name__])
+    @console_ns.response(200, "Chat message generated successfully")
     @console_ns.response(400, "Invalid request parameters")
     @console_ns.response(404, "Agent or conversation not found")
     @setup_required
@@ -213,9 +264,43 @@ class AgentChatMessageApi(Resource):
     @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_TEST_AND_RUN)
     @with_current_user
     @with_current_tenant_id
-    def post(self, current_tenant_id: str, current_user: Account, agent_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
-        return _create_chat_message(current_user=current_user, app_model=app_model)
+    @with_session
+    def post(self, session: Session, current_tenant_id: str, current_user: Account, agent_id: UUID):
+        app_model = resolve_agent_runtime_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
+        return _create_chat_message(
+            session=session,
+            current_tenant_id=current_tenant_id,
+            current_user=current_user,
+            app_model=app_model,
+            agent_id=str(agent_id),
+        )
+
+
+@console_ns.route("/agent/<uuid:agent_id>/build-chat/finalize")
+class AgentBuildChatFinalizeApi(Resource):
+    @console_ns.doc("finalize_agent_build_chat")
+    @console_ns.doc(description="Run a build-draft Agent App turn that asks the agent to push config updates")
+    @console_ns.doc(params={"agent_id": "Agent ID"})
+    @console_ns.response(200, "Success", console_ns.models[SimpleResultResponse.__name__])
+    @console_ns.response(400, "Invalid request parameters")
+    @console_ns.response(404, "Agent, build draft, or conversation not found")
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @edit_permission_required
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_TEST_AND_RUN)
+    @with_current_user
+    @with_current_tenant_id
+    @with_session
+    def post(self, session: Session, current_tenant_id: str, current_user: Account, agent_id: UUID):
+        app_model = resolve_agent_runtime_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
+        return _create_build_chat_finalization_message(
+            session=session,
+            current_tenant_id=current_tenant_id,
+            current_user=current_user,
+            app_model=app_model,
+            agent_id=str(agent_id),
+        )
 
 
 @console_ns.route("/apps/<uuid:app_id>/chat-messages/<string:task_id>/stop")
@@ -245,15 +330,54 @@ class AgentChatMessageStopApi(Resource):
     @with_current_user_id
     @with_current_tenant_id
     def post(self, current_tenant_id: str, current_user_id: str, agent_id: UUID, task_id: str):
-        app_model = resolve_agent_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
         return _stop_chat_message(current_user_id=current_user_id, app_model=app_model, task_id=task_id)
 
 
-def _create_chat_message(*, current_user: Account, app_model: App):
+def _resolve_current_user_agent_debug_conversation_id(
+    *, current_tenant_id: str, current_user: Account, app_model: App, agent_id: str | None
+) -> str:
+    roster_service = AgentRosterService(db.session)
+    if agent_id:
+        return roster_service.get_or_create_agent_app_debug_conversation_id(
+            tenant_id=current_tenant_id,
+            agent_id=agent_id,
+            account_id=current_user.id,
+        )
+
+    agent = roster_service.get_app_backing_agent(tenant_id=current_tenant_id, app_id=str(app_model.id))
+    if agent is None:
+        raise AgentNotFoundError()
+    return roster_service.get_or_create_agent_app_debug_conversation_id(
+        tenant_id=current_tenant_id,
+        agent_id=agent.id,
+        account_id=current_user.id,
+    )
+
+
+def _create_chat_message(
+    *,
+    session: Session,
+    current_user: Account,
+    app_model: App,
+    current_tenant_id: str | None = None,
+    agent_id: str | None = None,
+):
     raw_payload = console_ns.payload or {}
     args_model = ChatMessagePayload.model_validate(raw_payload)
     args = args_model.model_dump(exclude_none=True, by_alias=True)
 
+    if AppMode.value_of(app_model.mode) == AppMode.AGENT:
+        debug_conversation_id = _resolve_current_user_agent_debug_conversation_id(
+            current_tenant_id=current_tenant_id or app_model.tenant_id,
+            current_user=current_user,
+            app_model=app_model,
+            agent_id=agent_id,
+        )
+        if args_model.conversation_id and args_model.conversation_id != debug_conversation_id:
+            raise NotFound("Conversation Not Exists.")
+        args["conversation_id"] = debug_conversation_id
+
     streaming = _resolve_debugger_chat_streaming(
         app_mode=AppMode.value_of(app_model.mode),
         response_mode=args_model.response_mode,
@@ -267,12 +391,107 @@ def _create_chat_message(*, current_user: Account, app_model: App):
     if external_trace_id:
         args["external_trace_id"] = external_trace_id
 
-    try:
-        response = AppGenerateService.generate(
-            app_model=app_model, user=current_user, args=args, invoke_from=InvokeFrom.DEBUGGER, streaming=streaming
-        )
+    return _generate_chat_message_response(
+        session=session,
+        current_user=current_user,
+        app_model=app_model,
+        args=args,
+        streaming=streaming,
+    )
 
-        return helper.compact_generate_response(response)
+
+def _create_build_chat_finalization_message(
+    *, session: Session, current_user: Account, app_model: App, current_tenant_id: str, agent_id: str
+):
+    debug_conversation_id = _resolve_current_user_agent_debug_conversation_id(
+        current_tenant_id=current_tenant_id,
+        current_user=current_user,
+        app_model=app_model,
+        agent_id=agent_id,
+    )
+    args: dict[str, Any] = {
+        "query": _BUILD_CHAT_FINALIZATION_QUERY,
+        "inputs": {},
+        "response_mode": "streaming",
+        "draft_type": "debug_build",
+        "conversation_id": debug_conversation_id,
+        "auto_generate_name": False,
+    }
+    external_trace_id = get_external_trace_id(request)
+    if external_trace_id:
+        args["external_trace_id"] = external_trace_id
+
+    response = _generate_chat_message(
+        session=session,
+        current_user=current_user,
+        app_model=app_model,
+        args=args,
+        streaming=True,
+    )
+    _drain_streaming_generate_response(response)
+    return {"result": "success"}, 200
+
+
+def _drain_streaming_generate_response(response: RateLimitGenerator | Generator[str, None, None]) -> None:
+    """Consume a streamed app-generate response until a terminal message event arrives.
+
+    Finalize keeps the normal Agent App streaming path so the existing queue,
+    persistence, and runtime-session behavior stay intact. The console API only
+    changes the HTTP boundary: it drains the SSE stream server-side and returns
+    success after the generated build-chat message reaches ``message_end``.
+    """
+    try:
+        for chunk in response:
+            for raw_event in chunk.split("\n\n"):
+                if not raw_event.strip():
+                    continue
+
+                event_name: str | None = None
+                data_lines: list[str] = []
+                for line in raw_event.splitlines():
+                    if line.startswith("event: "):
+                        event_name = line.removeprefix("event: ").strip()
+                    elif line.startswith("data: "):
+                        data_lines.append(line.removeprefix("data: "))
+
+                if not data_lines:
+                    if event_name == "ping":
+                        continue
+                    continue
+
+                payload = json.loads("\n".join(data_lines))
+                if not isinstance(payload, dict):
+                    continue
+
+                payload_event = payload.get("event")
+                if payload_event == "message_end":
+                    return
+                if payload_event == "error":
+                    raise CompletionRequestError(str(payload.get("message") or "Build chat finalization failed."))
+    finally:
+        if isinstance(response, _ClosableStream):
+            response.close()
+
+    raise CompletionRequestError("Build chat finalization did not complete.")
+
+
+def _generate_chat_message(
+    *,
+    session: Session,
+    current_user: Account,
+    app_model: App,
+    args: dict[str, Any],
+    streaming: bool,
+):
+    try:
+        return AppGenerateService.generate(
+            session=session,
+            app_model=app_model,
+            user=current_user,
+            args=args,
+            invoke_from=InvokeFrom.DEBUGGER,
+            streaming=streaming,
+        )
     except services.errors.conversation.ConversationNotExistsError:
         raise NotFound("Conversation Not Exists.")
     except services.errors.conversation.ConversationCompletedError:
@@ -288,6 +507,8 @@ def _create_chat_message(*, current_user: Account, app_model: App):
         raise ProviderModelCurrentlyNotSupportError()
     except InvokeRateLimitError as ex:
         raise InvokeRateLimitHttpError(ex.description)
+    except CompletionRequestError:
+        raise
     except InvokeError as e:
         raise CompletionRequestError(e.description)
     except ValueError as e:
@@ -297,6 +518,26 @@ def _create_chat_message(*, current_user: Account, app_model: App):
         raise InternalServerError()
 
 
+def _generate_chat_message_response(
+    *,
+    session: Session,
+    current_user: Account,
+    app_model: App,
+    args: dict[str, Any],
+    streaming: bool,
+):
+    response = _generate_chat_message(
+        session=session,
+        current_user=current_user,
+        app_model=app_model,
+        args=args,
+        streaming=streaming,
+    )
+    if AppMode.value_of(app_model.mode) == AppMode.AGENT and streaming:
+        response = _raise_agent_stream_error_before_response(response)
+    return helper.compact_generate_response(response)
+
+
 def _stop_chat_message(*, current_user_id: str, app_model: App, task_id: str):
     AppTaskService.stop_task(
         task_id=task_id,
@@ -305,4 +546,68 @@ def _stop_chat_message(*, current_user_id: str, app_model: App, task_id: str):
         app_mode=AppMode.value_of(app_model.mode),
     )
 
-    return {"result": "success"}, 200
+    return SimpleResultResponse(result="success").model_dump(mode="json"), 200
+
+
+def _raise_agent_stream_error_before_response(response):
+    """Surface immediate Agent App stream errors as HTTP errors before SSE starts.
+
+    The shared streaming helper always returns HTTP 200 once the SSE response is
+    created. Agent v2 configuration errors, such as an invalid model API key,
+    can be the first real stream event after the initial ping; pre-reading that
+    first non-ping event lets the console API return the existing 400 error
+    contract instead of a successful HTTP response carrying only an SSE error.
+    """
+    if isinstance(response, Mapping):
+        return response
+
+    buffered: list[str] = []
+    iterator = iter(response)
+    while True:
+        try:
+            chunk = next(iterator)
+        except StopIteration:
+            return iter(buffered)
+
+        if not isinstance(chunk, str):
+            return _prepend_stream_chunks(buffered, chunk, iterator)
+
+        if _is_sse_ping(chunk):
+            buffered.append(chunk)
+            continue
+
+        error_payload = _extract_sse_error_payload(chunk)
+        if error_payload is not None:
+            if isinstance(response, _ClosableStream):
+                response.close()
+            message = error_payload.get("message")
+            raise CompletionRequestError(str(message or "Agent App chat failed."))
+
+        return _prepend_stream_chunks(buffered, chunk, iterator)
+
+
+def _prepend_stream_chunks(buffered: list[Any], first: Any, iterator: Iterator[Any]) -> Generator[Any, None, None]:
+    yield from buffered
+    yield first
+    yield from iterator
+
+
+def _is_sse_ping(chunk: str) -> bool:
+    return chunk.strip() == "event: ping"
+
+
+def _extract_sse_error_payload(chunk: str) -> dict[str, Any] | None:
+    for raw_event in chunk.split("\n\n"):
+        data_lines: list[str] = []
+        for line in raw_event.splitlines():
+            if line.startswith("data: "):
+                data_lines.append(line.removeprefix("data: "))
+        if not data_lines:
+            continue
+        try:
+            payload = json.loads("\n".join(data_lines))
+        except json.JSONDecodeError:
+            continue
+        if isinstance(payload, dict) and payload.get("event") == "error":
+            return payload
+    return None

api/controllers/console/app/conversation.py

@@ -9,7 +9,7 @@ from sqlalchemy import func, or_
 from sqlalchemy.orm import selectinload
 from werkzeug.exceptions import NotFound
 
-from controllers.common.schema import query_params_from_model, register_schema_models
+from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models
 from controllers.console import console_ns
 from controllers.console.app.wraps import get_app_model
 from controllers.console.wraps import (
@@ -39,7 +39,9 @@ from fields.conversation_fields import (
     ConversationWithSummaryPagination as ConversationWithSummaryPaginationResponse,
 )
 from libs.datetime_utils import naive_utc_now, parse_time_range
+from libs.helper import dump_response
 from libs.login import login_required
+from libs.pagination import paginate_query
 from models import Conversation, EndUser, Message, MessageAnnotation
 from models.account import Account
 from models.model import App, AppMode
@@ -79,13 +81,14 @@ register_schema_models(
     console_ns,
     CompletionConversationQuery,
     ChatConversationQuery,
+)
+register_response_schema_models(
+    console_ns,
     ConversationResponse,
     ConversationPaginationResponse,
     ConversationMessageDetailResponse,
     ConversationWithSummaryPaginationResponse,
     ConversationDetailResponse,
-    CompletionConversationQuery,
-    ChatConversationQuery,
 )
 
 
@@ -93,8 +96,7 @@ register_schema_models(
 class CompletionConversationApi(Resource):
     @console_ns.doc("list_completion_conversations")
     @console_ns.doc(description="Get completion conversations with pagination and filtering")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(CompletionConversationQuery))
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(CompletionConversationQuery)})
     @console_ns.response(200, "Success", console_ns.models[ConversationPaginationResponse.__name__])
     @console_ns.response(403, "Insufficient permissions")
     @setup_required
@@ -155,11 +157,9 @@ class CompletionConversationApi(Resource):
 
         query = query.order_by(Conversation.created_at.desc())
 
-        conversations = db.paginate(query, page=args.page, per_page=args.limit, error_out=False)
+        conversations = paginate_query(query, page=args.page, per_page=args.limit)
 
-        return ConversationPaginationResponse.model_validate(conversations, from_attributes=True).model_dump(
-            mode="json"
-        )
+        return dump_response(ConversationPaginationResponse, conversations)
 
 
 @console_ns.route("/apps/<uuid:app_id>/completion-conversations/<uuid:conversation_id>")
@@ -179,9 +179,9 @@ class CompletionConversationDetailApi(Resource):
     @get_app_model(mode=AppMode.COMPLETION)
     def get(self, current_user: Account, app_model: App, conversation_id: UUID):
         conversation_id_str = str(conversation_id)
-        return ConversationMessageDetailResponse.model_validate(
-            _get_conversation(current_user, app_model, conversation_id_str), from_attributes=True
-        ).model_dump(mode="json")
+        return dump_response(
+            ConversationMessageDetailResponse, _get_conversation(current_user, app_model, conversation_id_str)
+        )
 
     @console_ns.doc("delete_completion_conversation")
     @console_ns.doc(description="Delete a completion conversation")
@@ -200,7 +200,7 @@ class CompletionConversationDetailApi(Resource):
         conversation_id_str = str(conversation_id)
 
         try:
-            ConversationService.delete(app_model, conversation_id_str, current_user)
+            ConversationService.delete(app_model, conversation_id_str, current_user, session=db.session())
         except ConversationNotExistsError:
             raise NotFound("Conversation Not Exists.")
 
@@ -211,8 +211,7 @@ class CompletionConversationDetailApi(Resource):
 class ChatConversationApi(Resource):
     @console_ns.doc("list_chat_conversations")
     @console_ns.doc(description="Get chat conversations with pagination, filtering and summary")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(ChatConversationQuery))
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(ChatConversationQuery)})
     @console_ns.response(200, "Success", console_ns.models[ConversationWithSummaryPaginationResponse.__name__])
     @console_ns.response(403, "Insufficient permissions")
     @setup_required
@@ -312,11 +311,9 @@ class ChatConversationApi(Resource):
             case _:
                 query = query.order_by(Conversation.created_at.desc())
 
-        conversations = db.paginate(query, page=args.page, per_page=args.limit, error_out=False)
+        conversations = paginate_query(query, page=args.page, per_page=args.limit)
 
-        return ConversationWithSummaryPaginationResponse.model_validate(conversations, from_attributes=True).model_dump(
-            mode="json"
-        )
+        return dump_response(ConversationWithSummaryPaginationResponse, conversations)
 
 
 @console_ns.route("/apps/<uuid:app_id>/chat-conversations/<uuid:conversation_id>")
@@ -336,9 +333,9 @@ class ChatConversationDetailApi(Resource):
     @get_app_model(mode=[AppMode.CHAT, AppMode.AGENT_CHAT, AppMode.ADVANCED_CHAT, AppMode.AGENT])
     def get(self, current_user: Account, app_model: App, conversation_id: UUID):
         conversation_id_str = str(conversation_id)
-        return ConversationDetailResponse.model_validate(
-            _get_conversation(current_user, app_model, conversation_id_str), from_attributes=True
-        ).model_dump(mode="json")
+        return dump_response(
+            ConversationDetailResponse, _get_conversation(current_user, app_model, conversation_id_str)
+        )
 
     @console_ns.doc("delete_chat_conversation")
     @console_ns.doc(description="Delete a chat conversation")
@@ -357,7 +354,7 @@ class ChatConversationDetailApi(Resource):
         conversation_id_str = str(conversation_id)
 
         try:
-            ConversationService.delete(app_model, conversation_id_str, current_user)
+            ConversationService.delete(app_model, conversation_id_str, current_user, session=db.session())
         except ConversationNotExistsError:
             raise NotFound("Conversation Not Exists.")
 

api/controllers/console/app/conversation_variables.py

@@ -22,7 +22,7 @@ from controllers.console.wraps import (
 from extensions.ext_database import db
 from fields._value_type_serializer import serialize_value_type
 from fields.base import ResponseModel
-from libs.helper import to_timestamp
+from libs.helper import dump_response, to_timestamp
 from libs.login import login_required
 from models import ConversationVariable
 from models.model import App, AppMode
@@ -119,7 +119,8 @@ class ConversationVariablesApi(Resource):
         with sessionmaker(db.engine, expire_on_commit=False).begin() as session:
             rows = session.scalars(stmt).all()
 
-        response = PaginatedConversationVariableResponse.model_validate(
+        return dump_response(
+            PaginatedConversationVariableResponse,
             {
                 "page": page,
                 "limit": page_size,
@@ -135,6 +136,5 @@ class ConversationVariablesApi(Resource):
                     )
                     for row in rows
                 ],
-            }
+            },
         )
-        return response.model_dump(mode="json")

api/controllers/console/app/generator.py

@@ -1,8 +1,10 @@
-from collections.abc import Sequence
+import json
+from collections.abc import Generator, Sequence
 from typing import Any, Literal
 
 from flask_restx import Resource
 from pydantic import BaseModel, Field, RootModel
+from sqlalchemy import select
 from sqlalchemy.orm import Session
 
 from controllers.common.fields import SimpleDataResponse
@@ -23,8 +25,10 @@ from core.helper.code_executor.javascript.javascript_code_provider import Javasc
 from core.helper.code_executor.python3.python3_code_provider import Python3CodeProvider
 from core.llm_generator.entities import RuleCodeGeneratePayload, RuleGeneratePayload, RuleStructuredOutputPayload
 from core.llm_generator.llm_generator import LLMGenerator
+from core.workflow.generator.types import WorkflowGenerateErrorCode
 from graphon.model_runtime.entities.llm_entities import LLMMode
 from graphon.model_runtime.errors.invoke import InvokeError
+from libs.helper import compact_generate_response
 from libs.login import login_required
 from models import App
 from services.workflow_generator_service import WorkflowGeneratorService
@@ -64,7 +68,10 @@ class WorkflowGeneratePayload(BaseModel):
     can reuse its existing handler.
     """
 
-    mode: Literal["workflow", "advanced-chat"] = Field(..., description="Target app mode for the generated graph")
+    mode: Literal["workflow", "advanced-chat", "auto"] = Field(
+        ...,
+        description="Target app mode for the generated graph; 'auto' lets the backend classify the instruction",
+    )
     instruction: str = Field(..., description="Natural-language workflow description")
     ideal_output: str = Field(default="", description="Optional sample output for grounding")
     model_config_data: ModelConfig = Field(
@@ -78,6 +85,19 @@ class WorkflowGeneratePayload(BaseModel):
     )
 
 
+class WorkflowInstructionSuggestionsPayload(BaseModel):
+    """Payload for the workflow-generator instruction-suggestions endpoint.
+
+    Runs before the user picks a model, so the suggestions come from the
+    tenant's default model. The underlying generator never raises — an empty
+    ``suggestions`` list is a valid 200 (soft-fail).
+    """
+
+    mode: Literal["workflow", "advanced-chat"] = Field(..., description="Target app mode for the suggestions")
+    language: str | None = Field(default=None, description="Optional language to write the suggestions in")
+    count: int = Field(default=4, ge=1, le=6, description="Number of suggestions to return (1-6)")
+
+
 class GeneratorResponse(RootModel[Any]):
     root: Any
 
@@ -91,6 +111,7 @@ register_schema_models(
     InstructionGeneratePayload,
     InstructionTemplatePayload,
     WorkflowGeneratePayload,
+    WorkflowInstructionSuggestionsPayload,
     ModelConfig,
 )
 register_response_schema_models(console_ns, GeneratorResponse, SimpleDataResponse)
@@ -216,7 +237,9 @@ class InstructionGenerateApi(Resource):
         try:
             # Generate from nothing for a workflow node
             if (args.current in (code_template, "")) and args.node_id != "":
-                app = session.get(App, args.flow_id)
+                app = session.scalar(
+                    select(App).where(App.id == args.flow_id, App.tenant_id == current_tenant_id).limit(1)
+                )
                 if not app:
                     return {"error": f"app {args.flow_id} not found"}, 400
                 workflow = WorkflowService().get_draft_workflow(app_model=app, session=session)
@@ -313,6 +336,34 @@ class InstructionGenerationTemplateApi(Resource):
                 raise ValueError(f"Invalid type: {args.type}")
 
 
+def _workflow_instruction_guard(args: WorkflowGeneratePayload) -> tuple[dict, int] | None:
+    """Shared boundary guard for the workflow-generate endpoints.
+
+    Returns a ``(body, 400)`` tuple when the instruction is empty / whitespace
+    or either free-text field exceeds the cap, else ``None``. Pydantic only
+    validates the field is a str; a whitespace-only or pasted-document input
+    would otherwise waste a slow planner+builder roundtrip on a response the
+    validator rejects anyway. Both the blocking and streaming endpoints call
+    this so they reject identical inputs.
+    """
+    if not args.instruction.strip():
+        return {
+            "error": "Instruction is required",
+            "errors": [{"code": WorkflowGenerateErrorCode.EMPTY_INSTRUCTION, "detail": "Instruction is required"}],
+        }, 400
+    if len(args.instruction) > _MAX_INSTRUCTION_LENGTH or len(args.ideal_output) > _MAX_INSTRUCTION_LENGTH:
+        return {
+            "error": "Instruction is too long",
+            "errors": [
+                {
+                    "code": WorkflowGenerateErrorCode.INSTRUCTION_TOO_LONG,
+                    "detail": f"Instruction and ideal output must each be at most {_MAX_INSTRUCTION_LENGTH} characters",
+                }
+            ],
+        }, 400
+    return None
+
+
 @console_ns.route("/workflow-generate")
 class WorkflowGenerateApi(Resource):
     """Generate a Workflow / Chatflow draft graph from a natural-language description.
@@ -335,31 +386,11 @@ class WorkflowGenerateApi(Resource):
     def post(self, current_tenant_id: str):
         args = WorkflowGeneratePayload.model_validate(console_ns.payload)
 
-        # Reject obviously-empty instructions at the boundary — Pydantic only
-        # validates ``instruction`` is a str, but a whitespace-only string
-        # would still hit the LLM and waste a planner+builder roundtrip on a
-        # response that the postprocess validator would reject anyway.
-        if not args.instruction.strip():
-            return {
-                "error": "Instruction is required",
-                "errors": [{"code": "EMPTY_INSTRUCTION", "detail": "Instruction is required"}],
-            }, 400
-
-        # Bound the prompt at the boundary too: an arbitrarily long
-        # instruction (or pasted document) blows the planner/builder context
-        # window and fails with an opaque provider error after two slow LLM
-        # calls. The cap matches the frontend textarea's maxLength.
-        if len(args.instruction) > _MAX_INSTRUCTION_LENGTH or len(args.ideal_output) > _MAX_INSTRUCTION_LENGTH:
-            return {
-                "error": "Instruction is too long",
-                "errors": [
-                    {
-                        "code": "INSTRUCTION_TOO_LONG",
-                        "detail": f"Instruction and ideal output must each be at most "
-                        f"{_MAX_INSTRUCTION_LENGTH} characters",
-                    }
-                ],
-            }, 400
+        # Reject empty / over-length instructions at the boundary (shared with
+        # the streaming endpoint) before spending a planner+builder roundtrip.
+        guard = _workflow_instruction_guard(args)
+        if guard is not None:
+            return guard
 
         try:
             result = WorkflowGeneratorService.generate_workflow_graph(
@@ -380,3 +411,93 @@ class WorkflowGenerateApi(Resource):
             raise CompletionRequestError(e.description)
 
         return result
+
+
+@console_ns.route("/workflow-generate/suggestions")
+class WorkflowInstructionSuggestionsApi(Resource):
+    """Suggest short, buildable example instructions for the cmd+k generator.
+
+    Runs before a model is selected (uses the tenant's default model). The
+    underlying generator never raises, so an empty list is a valid 200 — the
+    frontend renders "no suggestions" rather than an error, so no provider-error
+    mapping is needed here.
+    """
+
+    @console_ns.doc("generate_workflow_instruction_suggestions")
+    @console_ns.doc(description="Suggest example workflow-generator instructions for the tenant")
+    @console_ns.expect(console_ns.models[WorkflowInstructionSuggestionsPayload.__name__])
+    @console_ns.response(200, "Suggestions generated successfully", console_ns.models[GeneratorResponse.__name__])
+    @console_ns.response(400, "Invalid request parameters")
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @with_current_tenant_id
+    def post(self, current_tenant_id: str):
+        args = WorkflowInstructionSuggestionsPayload.model_validate(console_ns.payload)
+        suggestions = LLMGenerator.generate_workflow_instruction_suggestions(
+            tenant_id=current_tenant_id,
+            mode=args.mode,
+            language=args.language,
+            count=args.count,
+        )
+        return {"suggestions": suggestions}
+
+
+@console_ns.route("/workflow-generate/stream")
+class WorkflowGenerateStreamApi(Resource):
+    """Plan-first streaming variant of ``/workflow-generate`` (Server-Sent Events).
+
+    Emits a ``plan`` event (high-level node list + app metadata) as soon as the
+    planner returns, then a final ``result`` event with the full graph — the
+    SAME envelope ``/workflow-generate`` returns. Provider-init / invoke errors
+    are surfaced as a single ``result`` event (code ``MODEL_ERROR``) so the
+    frontend's stream parser always receives a result rather than a non-SSE HTTP
+    error.
+    """
+
+    @console_ns.doc("generate_workflow_graph_stream")
+    @console_ns.doc(description="Stream a Dify workflow graph (plan then result) via SSE")
+    @console_ns.expect(console_ns.models[WorkflowGeneratePayload.__name__])
+    @console_ns.response(200, "Server-Sent Events stream of plan/result events")
+    @console_ns.response(400, "Invalid request parameters")
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @with_current_tenant_id
+    def post(self, current_tenant_id: str):
+        args = WorkflowGeneratePayload.model_validate(console_ns.payload)
+
+        # Same boundary guards as the blocking endpoint — return a normal 400
+        # JSON for these BEFORE opening the stream.
+        guard = _workflow_instruction_guard(args)
+        if guard is not None:
+            return guard
+
+        def generate() -> Generator[str, None, None]:
+            try:
+                for event_name, payload in WorkflowGeneratorService.generate_workflow_graph_stream(
+                    tenant_id=current_tenant_id,
+                    mode=args.mode,
+                    instruction=args.instruction,
+                    model_config=args.model_config_data,
+                    ideal_output=args.ideal_output,
+                    current_graph=args.current_graph,
+                ):
+                    body = {"event": event_name, **payload}
+                    yield f"data: {json.dumps(body)}\n\n"
+            except (ProviderTokenNotInitError, QuotaExceededError, ModelCurrentlyNotSupportError, InvokeError) as e:
+                # The model instance is resolved inside the service (lazily, on
+                # first iteration), so a provider / init error surfaces here.
+                # Emit it as a single SSE result event rather than a non-SSE
+                # error response so the frontend's stream parser always gets a
+                # result it can render.
+                detail = getattr(e, "description", None) or str(e) or "Model invocation failed"
+                error_body = {
+                    "event": "result",
+                    "graph": {"nodes": [], "edges": [], "viewport": {"x": 0.0, "y": 0.0, "zoom": 0.7}},
+                    "error": detail,
+                    "errors": [{"code": WorkflowGenerateErrorCode.MODEL_ERROR, "detail": detail}],
+                }
+                yield f"data: {json.dumps(error_body)}\n\n"
+
+        return compact_generate_response(generate())

api/controllers/console/app/mcp_server.py

@@ -22,10 +22,11 @@ from controllers.console.wraps import (
 )
 from extensions.ext_database import db
 from fields.base import ResponseModel
-from libs.helper import to_timestamp
+from libs.helper import dump_response, to_timestamp
 from libs.login import login_required
 from models.enums import AppMCPServerStatus
 from models.model import App, AppMCPServer
+from services.app_ref_service import AppRefService
 
 
 class MCPServerCreatePayload(BaseModel):
@@ -92,7 +93,7 @@ class AppMCPServerController(Resource):
         server = db.session.scalar(select(AppMCPServer).where(AppMCPServer.app_id == app_model.id).limit(1))
         if server is None:
             return {}
-        return AppMCPServerResponse.model_validate(server, from_attributes=True).model_dump(mode="json")
+        return dump_response(AppMCPServerResponse, server)
 
     @console_ns.doc("create_app_mcp_server")
     @console_ns.doc(description="Create MCP server configuration for an application")
@@ -127,7 +128,7 @@ class AppMCPServerController(Resource):
         )
         db.session.add(server)
         db.session.commit()
-        return AppMCPServerResponse.model_validate(server, from_attributes=True).model_dump(mode="json"), 201
+        return dump_response(AppMCPServerResponse, server), 201
 
     @console_ns.doc("update_app_mcp_server")
     @console_ns.doc(description="Update MCP server configuration for an application")
@@ -146,7 +147,17 @@ class AppMCPServerController(Resource):
     @get_app_model
     def put(self, app_model: App):
         payload = MCPServerUpdatePayload.model_validate(console_ns.payload or {})
-        server = db.session.get(AppMCPServer, payload.id)
+        app_ref = AppRefService.create_app_ref(app_model)
+        server_ref = AppRefService.create_mcp_server_ref(app_ref, payload.id)
+        server = db.session.scalar(
+            select(AppMCPServer)
+            .where(
+                AppMCPServer.id == server_ref.server_id,
+                AppMCPServer.tenant_id == server_ref.tenant_id,
+                AppMCPServer.app_id == server_ref.app_id,
+            )
+            .limit(1)
+        )
         if not server:
             raise NotFound()
 
@@ -165,7 +176,7 @@ class AppMCPServerController(Resource):
             except ValueError:
                 raise ValueError("Invalid status")
         db.session.commit()
-        return AppMCPServerResponse.model_validate(server, from_attributes=True).model_dump(mode="json")
+        return dump_response(AppMCPServerResponse, server)
 
 
 @console_ns.route("/apps/<uuid:server_id>/server/refresh")
@@ -192,4 +203,4 @@ class AppMCPServerRefreshController(Resource):
             raise NotFound()
         server.server_code = AppMCPServer.generate_server_code(16)
         db.session.commit()
-        return AppMCPServerResponse.model_validate(server, from_attributes=True).model_dump(mode="json")
+        return dump_response(AppMCPServerResponse, server)

api/controllers/console/app/message.py

@@ -1,5 +1,4 @@
 import logging
-from datetime import datetime
 from typing import Literal
 from uuid import UUID
 
@@ -13,7 +12,7 @@ from controllers.common.controller_schemas import MessageFeedbackPayload as _Mes
 from controllers.common.fields import SimpleResultResponse, TextFileResponse
 from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models
 from controllers.console import console_ns
-from controllers.console.agent.app_helpers import resolve_agent_app_model
+from controllers.console.agent.app_helpers import resolve_agent_runtime_app_model
 from controllers.console.app.error import (
     CompletionRequestError,
     ProviderModelCurrentlyNotSupportError,
@@ -38,21 +37,16 @@ from core.errors.error import ModelCurrentlyNotSupportError, ProviderTokenNotIni
 from extensions.ext_database import db
 from fields.base import ResponseModel
 from fields.conversation_fields import (
-    AgentThought,
-    ConversationAnnotation,
-    ConversationAnnotationHitHistory,
-    Feedback,
-    JSONValue,
-    MessageFile,
-    format_files_contained,
+    MessageDetail as BaseMessageDetailResponse,
 )
 from graphon.model_runtime.errors.invoke import InvokeError
-from libs.helper import to_timestamp, uuid_value
+from libs.helper import dump_response, uuid_value
 from libs.infinite_scroll_pagination import InfiniteScrollPagination
 from libs.login import login_required
 from models.account import Account
 from models.enums import FeedbackFromSource, FeedbackRating
 from models.model import App, AppMode, Conversation, Message, MessageAnnotation, MessageFeedback
+from services.conversation_service import ConversationService
 from services.errors.conversation import ConversationNotExistsError
 from services.errors.message import MessageNotExistsError, SuggestedQuestionsAfterAnswerDisabledError
 from services.message_service import MessageService, attach_message_extra_contents
@@ -110,49 +104,16 @@ class FeedbackExportQuery(BaseModel):
         raise ValueError("has_comment must be a boolean value")
 
 
-class AnnotationCountResponse(BaseModel):
+class AnnotationCountResponse(ResponseModel):
     count: int = Field(description="Number of annotations")
 
 
-class SuggestedQuestionsResponse(BaseModel):
+class SuggestedQuestionsResponse(ResponseModel):
     data: list[str] = Field(description="Suggested question")
 
 
-class MessageDetailResponse(ResponseModel):
-    id: str
-    conversation_id: str
-    inputs: dict[str, JSONValue]
-    query: str
-    message: JSONValue | None = None
-    message_tokens: int | None = None
-    answer: str = Field(validation_alias="re_sign_file_url_answer")
-    answer_tokens: int | None = None
-    provider_response_latency: float | None = None
-    from_source: str
-    from_end_user_id: str | None = None
-    from_account_id: str | None = None
-    feedbacks: list[Feedback] = Field(default_factory=list)
-    workflow_run_id: str | None = None
-    annotation: ConversationAnnotation | None = None
-    annotation_hit_history: ConversationAnnotationHitHistory | None = None
-    created_at: int | None = None
-    agent_thoughts: list[AgentThought] = Field(default_factory=list)
-    message_files: list[MessageFile] = Field(default_factory=list)
+class MessageDetailResponse(BaseMessageDetailResponse):
     extra_contents: list[ExecutionExtraContentDomainModel] = Field(default_factory=list)
-    metadata: JSONValue | None = Field(default=None, validation_alias="message_metadata_dict")
-    status: str
-    error: str | None = None
-    parent_message_id: str | None = None
-
-    @field_validator("inputs", mode="before")
-    @classmethod
-    def _normalize_inputs(cls, value: JSONValue) -> JSONValue:
-        return format_files_contained(value)
-
-    @field_validator("created_at", mode="before")
-    @classmethod
-    def _normalize_created_at(cls, value: datetime | int | None) -> int | None:
-        return to_timestamp(value)
 
 
 class MessageInfiniteScrollPaginationResponse(ResponseModel):
@@ -166,30 +127,34 @@ register_schema_models(
     ChatMessagesQuery,
     MessageFeedbackPayload,
     FeedbackExportQuery,
+)
+register_response_schema_models(
+    console_ns,
     AnnotationCountResponse,
     SuggestedQuestionsResponse,
     MessageDetailResponse,
     MessageInfiniteScrollPaginationResponse,
+    SimpleResultResponse,
+    TextFileResponse,
 )
-register_response_schema_models(console_ns, SimpleResultResponse, TextFileResponse)
 
 
 @console_ns.route("/apps/<uuid:app_id>/chat-messages")
 class ChatMessageListApi(Resource):
     @console_ns.doc("list_chat_messages")
     @console_ns.doc(description="Get chat messages for a conversation with pagination")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(ChatMessagesQuery))
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(ChatMessagesQuery)})
     @console_ns.response(200, "Success", console_ns.models[MessageInfiniteScrollPaginationResponse.__name__])
     @console_ns.response(404, "Conversation not found")
     @login_required
     @account_initialization_required
     @setup_required
     @edit_permission_required
+    @with_current_user
     @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_VIEW_LAYOUT)
     @get_app_model(mode=[AppMode.CHAT, AppMode.AGENT_CHAT, AppMode.ADVANCED_CHAT, AppMode.AGENT])
-    def get(self, app_model: App):
-        return _list_chat_messages(app_model=app_model)
+    def get(self, current_user: Account, app_model: App):
+        return _list_chat_messages(app_model=app_model, current_user=current_user)
 
 
 @console_ns.route("/agent/<uuid:agent_id>/chat-messages")
@@ -205,10 +170,11 @@ class AgentChatMessageListApi(Resource):
     @setup_required
     @edit_permission_required
     @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_VIEW_LAYOUT)
+    @with_current_user
     @with_current_tenant_id
-    def get(self, current_tenant_id: str, agent_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
-        return _list_chat_messages(app_model=app_model)
+    def get(self, current_tenant_id: str, current_user: Account, agent_id: UUID):
+        app_model = resolve_agent_runtime_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
+        return _list_chat_messages(app_model=app_model, current_user=current_user)
 
 
 @console_ns.route("/apps/<uuid:app_id>/feedbacks")
@@ -243,7 +209,7 @@ class AgentMessageFeedbackApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def post(self, current_tenant_id: str, current_user: Account, agent_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
         return _update_message_feedback(current_user=current_user, app_model=app_model)
 
 
@@ -267,7 +233,7 @@ class MessageAnnotationCountApi(Resource):
             select(func.count(MessageAnnotation.id)).where(MessageAnnotation.app_id == app_model.id)
         )
 
-        return {"count": count}
+        return AnnotationCountResponse(count=count or 0).model_dump(mode="json")
 
 
 @console_ns.route("/apps/<uuid:app_id>/chat-messages/<uuid:message_id>/suggested-questions")
@@ -308,7 +274,7 @@ class AgentMessageSuggestedQuestionApi(Resource):
     @with_current_user
     @with_current_tenant_id
     def get(self, current_tenant_id: str, current_user: Account, agent_id: UUID, message_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
         return _get_message_suggested_questions(current_user=current_user, app_model=app_model, message_id=message_id)
 
 
@@ -316,13 +282,12 @@ class AgentMessageSuggestedQuestionApi(Resource):
 class MessageFeedbackExportApi(Resource):
     @console_ns.doc("export_feedbacks")
     @console_ns.doc(description="Export user feedback data for Google Sheets")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(FeedbackExportQuery))
     @console_ns.response(
         200,
         "Feedback data exported successfully",
         console_ns.models[TextFileResponse.__name__],
     )
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(FeedbackExportQuery)})
     @console_ns.response(400, "Invalid parameters")
     @console_ns.response(500, "Internal server error")
     @setup_required
@@ -338,8 +303,8 @@ class MessageFeedbackExportApi(Resource):
 
         try:
             export_data = FeedbackService.export_feedbacks(
-                db.session(),
-                app_id=app_model.id,
+                app_model.id,
+                session=db.session(),
                 from_source=args.from_source,
                 rating=args.rating,
                 has_comment=args.has_comment,
@@ -347,7 +312,6 @@ class MessageFeedbackExportApi(Resource):
                 end_date=args.end_date,
                 format_type=args.format,
             )
-
             return export_data
 
         except ValueError as e:
@@ -386,18 +350,29 @@ class AgentMessageApi(Resource):
     @account_initialization_required
     @with_current_tenant_id
     def get(self, current_tenant_id: str, agent_id: UUID, message_id: UUID):
-        app_model = resolve_agent_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
+        app_model = resolve_agent_runtime_app_model(tenant_id=current_tenant_id, agent_id=agent_id)
         return _get_message_detail(app_model=app_model, message_id=message_id)
 
 
-def _list_chat_messages(*, app_model: App):
+def _list_chat_messages(*, app_model: App, current_user: Account | None = None):
     args = ChatMessagesQuery.model_validate(request.args.to_dict())
 
-    conversation = db.session.scalar(
-        select(Conversation)
-        .where(Conversation.id == args.conversation_id, Conversation.app_id == app_model.id)
-        .limit(1)
-    )
+    if AppMode.value_of(app_model.mode) == AppMode.AGENT and current_user is not None:
+        try:
+            conversation = ConversationService.get_conversation(
+                app_model=app_model,
+                conversation_id=args.conversation_id,
+                user=current_user,
+                session=db.session(),
+            )
+        except ConversationNotExistsError:
+            raise NotFound("Conversation Not Exists.")
+    else:
+        conversation = db.session.scalar(
+            select(Conversation)
+            .where(Conversation.id == args.conversation_id, Conversation.app_id == app_model.id)
+            .limit(1)
+        )
 
     if not conversation:
         raise NotFound("Conversation Not Exists.")
@@ -448,16 +423,16 @@ def _list_chat_messages(*, app_model: App):
     history_messages = list(reversed(history_messages))
     attach_message_extra_contents(history_messages)
 
-    return MessageInfiniteScrollPaginationResponse.model_validate(
+    return dump_response(
+        MessageInfiniteScrollPaginationResponse,
         InfiniteScrollPagination(data=history_messages, limit=args.limit, has_more=has_more),
-        from_attributes=True,
-    ).model_dump(mode="json")
+    )
 
 
 def _update_message_feedback(*, current_user: Account, app_model: App):
     args = MessageFeedbackPayload.model_validate(console_ns.payload)
 
-    message_id = str(args.message_id)
+    message_id = args.message_id
 
     message = db.session.scalar(
         select(Message).where(Message.id == message_id, Message.app_id == app_model.id).limit(1)
@@ -492,7 +467,7 @@ def _update_message_feedback(*, current_user: Account, app_model: App):
 
     db.session.commit()
 
-    return {"result": "success"}
+    return SimpleResultResponse(result="success").model_dump(mode="json")
 
 
 def _get_message_suggested_questions(*, current_user: Account, app_model: App, message_id: UUID):
@@ -500,7 +475,11 @@ def _get_message_suggested_questions(*, current_user: Account, app_model: App, m
 
     try:
         questions = MessageService.get_suggested_questions_after_answer(
-            app_model=app_model, message_id=message_id_str, user=current_user, invoke_from=InvokeFrom.DEBUGGER
+            app_model=app_model,
+            message_id=message_id_str,
+            user=current_user,
+            invoke_from=InvokeFrom.DEBUGGER,
+            session=db.session(),
         )
     except MessageNotExistsError:
         raise NotFound("Message not found")
@@ -520,7 +499,7 @@ def _get_message_suggested_questions(*, current_user: Account, app_model: App, m
         logger.exception("internal server error.")
         raise InternalServerError()
 
-    return {"data": questions}
+    return dump_response(SuggestedQuestionsResponse, {"data": questions})
 
 
 def _get_message_detail(*, app_model: App, message_id: UUID):
@@ -534,4 +513,4 @@ def _get_message_detail(*, app_model: App, message_id: UUID):
         raise NotFound("Message Not Exists.")
 
     attach_message_extra_contents([message])
-    return MessageDetailResponse.model_validate(message, from_attributes=True).model_dump(mode="json")
+    return dump_response(MessageDetailResponse, message)

api/controllers/console/app/ops_trace.py

@@ -13,9 +13,11 @@ from controllers.console.wraps import (
     RBACPermission,
     RBACResourceScope,
     account_initialization_required,
+    edit_permission_required,
     rbac_permission_required,
     setup_required,
 )
+from extensions.ext_database import db
 from fields.base import ResponseModel
 from libs.login import login_required
 from models import App
@@ -70,14 +72,14 @@ class TraceAppConfigApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR)
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_TRACING_CONFIG)
     @get_app_model
     def get(self, app_model: App):
         args = TraceProviderQuery.model_validate(request.args.to_dict(flat=True))  # type: ignore
 
         try:
             trace_config = OpsService.get_tracing_app_config(
-                app_id=app_model.id, tracing_provider=args.tracing_provider
+                app_id=app_model.id, tracing_provider=args.tracing_provider, session=db.session()
             )
             if not trace_config:
                 return {"has_not_configured": True}
@@ -95,9 +97,12 @@ class TraceAppConfigApi(Resource):
         console_ns.models[TraceAppConfigResponse.__name__],
     )
     @console_ns.response(400, "Invalid request parameters or configuration already exists")
+    @console_ns.response(403, "Insufficient permissions")
     @setup_required
     @login_required
     @account_initialization_required
+    @edit_permission_required
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_TRACING_CONFIG)
     @get_app_model
     def post(self, app_model: App):
         """Create a new trace app configuration"""
@@ -105,7 +110,10 @@ class TraceAppConfigApi(Resource):
 
         try:
             result = OpsService.create_tracing_app_config(
-                app_id=app_model.id, tracing_provider=args.tracing_provider, tracing_config=args.tracing_config
+                app_id=app_model.id,
+                tracing_provider=args.tracing_provider,
+                tracing_config=args.tracing_config,
+                session=db.session(),
             )
             if not result:
                 raise TracingConfigIsExist()
@@ -125,9 +133,12 @@ class TraceAppConfigApi(Resource):
         console_ns.models[TraceAppConfigResponse.__name__],
     )
     @console_ns.response(400, "Invalid request parameters or configuration not found")
+    @console_ns.response(403, "Insufficient permissions")
     @setup_required
     @login_required
     @account_initialization_required
+    @edit_permission_required
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_TRACING_CONFIG)
     @get_app_model
     def patch(self, app_model: App):
         """Update an existing trace app configuration"""
@@ -135,7 +146,10 @@ class TraceAppConfigApi(Resource):
 
         try:
             result = OpsService.update_tracing_app_config(
-                app_id=app_model.id, tracing_provider=args.tracing_provider, tracing_config=args.tracing_config
+                app_id=app_model.id,
+                tracing_provider=args.tracing_provider,
+                tracing_config=args.tracing_config,
+                session=db.session(),
             )
             if not result:
                 raise TracingConfigNotExist()
@@ -149,16 +163,21 @@ class TraceAppConfigApi(Resource):
     @console_ns.doc(params=query_params_from_model(TraceProviderQuery))
     @console_ns.response(204, "Tracing configuration deleted successfully")
     @console_ns.response(400, "Invalid request parameters or configuration not found")
+    @console_ns.response(403, "Insufficient permissions")
     @setup_required
     @login_required
     @account_initialization_required
+    @edit_permission_required
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_TRACING_CONFIG)
     @get_app_model
     def delete(self, app_model: App):
         """Delete an existing trace app configuration"""
         args = TraceProviderQuery.model_validate(request.args.to_dict(flat=True))
 
         try:
-            result = OpsService.delete_tracing_app_config(app_id=app_model.id, tracing_provider=args.tracing_provider)
+            result = OpsService.delete_tracing_app_config(
+                app_id=app_model.id, tracing_provider=args.tracing_provider, session=db.session()
+            )
             if not result:
                 raise TracingConfigNotExist()
             return "", 204

api/controllers/console/app/permission_keys.py

@@ -1,6 +1,9 @@
+from extensions.ext_database import db
 from services.enterprise import rbac_service as enterprise_rbac_service
 
 
 def get_app_permission_keys(tenant_id: str, account_id: str | None, app_id: str) -> list[str]:
-    permission_keys_map = enterprise_rbac_service.RBACService.AppPermissions.batch_get(tenant_id, account_id, [app_id])
+    permission_keys_map = enterprise_rbac_service.RBACService.AppPermissions.batch_get(
+        tenant_id, account_id, [app_id], session=db.session()
+    )
     return permission_keys_map.get(app_id, [])

api/controllers/console/app/site.py

@@ -22,6 +22,7 @@ from controllers.console.wraps import (
 from extensions.ext_database import db
 from fields.base import ResponseModel
 from libs.datetime_utils import naive_utc_now
+from libs.helper import dump_response
 from libs.login import login_required
 from models import Site
 from models.account import Account
@@ -40,6 +41,7 @@ class AppSiteUpdatePayload(BaseModel):
     customize_domain: str | None = Field(default=None)
     copyright: str | None = Field(default=None)
     privacy_policy: str | None = Field(default=None)
+    input_placeholder: str | None = Field(default=None)
     custom_disclaimer: str | None = Field(default=None)
     customize_token_strategy: Literal["must", "allow", "not_allow"] | None = Field(default=None)
     prompt_public: bool | None = Field(default=None)
@@ -66,6 +68,7 @@ class AppSiteResponse(ResponseModel):
     customize_domain: str | None = None
     copyright: str | None = None
     privacy_policy: str | None = None
+    input_placeholder: str | None = None
     custom_disclaimer: str | None = None
     customize_token_strategy: str
     prompt_public: bool
@@ -110,6 +113,7 @@ class AppSite(Resource):
             "customize_domain",
             "copyright",
             "privacy_policy",
+            "input_placeholder",
             "custom_disclaimer",
             "customize_token_strategy",
             "prompt_public",
@@ -124,7 +128,7 @@ class AppSite(Resource):
         site.updated_at = naive_utc_now()
         db.session.commit()
 
-        return AppSiteResponse.model_validate(site, from_attributes=True).model_dump(mode="json")
+        return dump_response(AppSiteResponse, site)
 
 
 @console_ns.route("/apps/<uuid:app_id>/site/access-token-reset")
@@ -153,4 +157,4 @@ class AppSiteAccessTokenReset(Resource):
         site.updated_at = naive_utc_now()
         db.session.commit()
 
-        return AppSiteResponse.model_validate(site, from_attributes=True).model_dump(mode="json")
+        return dump_response(AppSiteResponse, site)

api/controllers/console/app/statistic.py

@@ -1,7 +1,7 @@
 from decimal import Decimal
 
 import sqlalchemy as sa
-from flask import abort, jsonify, request
+from flask import abort, request
 from flask_restx import Resource
 from pydantic import BaseModel, Field, field_validator
 
@@ -20,7 +20,7 @@ from core.app.entities.app_invoke_entities import InvokeFrom
 from extensions.ext_database import db
 from fields.base import ResponseModel
 from libs.datetime_utils import parse_time_range
-from libs.helper import convert_datetime_to_date
+from libs.helper import convert_datetime_to_date, dump_response
 from libs.login import login_required
 from models import AppMode
 from models.account import Account
@@ -44,8 +44,15 @@ class DailyMessageStatisticItem(ResponseModel):
     message_count: int
 
 
-class DailyMessageStatisticResponse(ResponseModel):
-    data: list[DailyMessageStatisticItem]
+register_schema_models(console_ns, StatisticTimeRangeQuery)
+
+
+class StatisticDataResponse[T](ResponseModel):
+    data: list[T]
+
+
+class DailyMessageStatisticResponse(StatisticDataResponse[DailyMessageStatisticItem]):
+    pass
 
 
 class DailyConversationStatisticItem(ResponseModel):
@@ -53,8 +60,8 @@ class DailyConversationStatisticItem(ResponseModel):
     conversation_count: int
 
 
-class DailyConversationStatisticResponse(ResponseModel):
-    data: list[DailyConversationStatisticItem]
+class DailyConversationStatisticResponse(StatisticDataResponse[DailyConversationStatisticItem]):
+    pass
 
 
 class DailyTerminalStatisticItem(ResponseModel):
@@ -62,19 +69,19 @@ class DailyTerminalStatisticItem(ResponseModel):
     terminal_count: int
 
 
-class DailyTerminalStatisticResponse(ResponseModel):
-    data: list[DailyTerminalStatisticItem]
+class DailyTerminalStatisticResponse(StatisticDataResponse[DailyTerminalStatisticItem]):
+    pass
 
 
 class DailyTokenCostStatisticItem(ResponseModel):
     date: str
-    token_count: int
-    total_price: str | float
-    currency: str
+    token_count: int | None = None
+    total_price: Decimal | None = None
+    currency: str | None = None
 
 
-class DailyTokenCostStatisticResponse(ResponseModel):
-    data: list[DailyTokenCostStatisticItem]
+class DailyTokenCostStatisticResponse(StatisticDataResponse[DailyTokenCostStatisticItem]):
+    pass
 
 
 class AverageSessionInteractionStatisticItem(ResponseModel):
@@ -82,8 +89,8 @@ class AverageSessionInteractionStatisticItem(ResponseModel):
     interactions: float
 
 
-class AverageSessionInteractionStatisticResponse(ResponseModel):
-    data: list[AverageSessionInteractionStatisticItem]
+class AverageSessionInteractionStatisticResponse(StatisticDataResponse[AverageSessionInteractionStatisticItem]):
+    pass
 
 
 class UserSatisfactionRateStatisticItem(ResponseModel):
@@ -91,8 +98,8 @@ class UserSatisfactionRateStatisticItem(ResponseModel):
     rate: float
 
 
-class UserSatisfactionRateStatisticResponse(ResponseModel):
-    data: list[UserSatisfactionRateStatisticItem]
+class UserSatisfactionRateStatisticResponse(StatisticDataResponse[UserSatisfactionRateStatisticItem]):
+    pass
 
 
 class AverageResponseTimeStatisticItem(ResponseModel):
@@ -100,8 +107,8 @@ class AverageResponseTimeStatisticItem(ResponseModel):
     latency: float
 
 
-class AverageResponseTimeStatisticResponse(ResponseModel):
-    data: list[AverageResponseTimeStatisticItem]
+class AverageResponseTimeStatisticResponse(StatisticDataResponse[AverageResponseTimeStatisticItem]):
+    pass
 
 
 class TokensPerSecondStatisticItem(ResponseModel):
@@ -109,20 +116,27 @@ class TokensPerSecondStatisticItem(ResponseModel):
     tps: float
 
 
-class TokensPerSecondStatisticResponse(ResponseModel):
-    data: list[TokensPerSecondStatisticItem]
+class TokensPerSecondStatisticResponse(StatisticDataResponse[TokensPerSecondStatisticItem]):
+    pass
 
 
-register_schema_models(console_ns, StatisticTimeRangeQuery)
 register_response_schema_models(
     console_ns,
+    DailyMessageStatisticItem,
     DailyMessageStatisticResponse,
+    DailyConversationStatisticItem,
     DailyConversationStatisticResponse,
+    DailyTerminalStatisticItem,
     DailyTerminalStatisticResponse,
+    DailyTokenCostStatisticItem,
     DailyTokenCostStatisticResponse,
+    AverageSessionInteractionStatisticItem,
     AverageSessionInteractionStatisticResponse,
+    UserSatisfactionRateStatisticItem,
     UserSatisfactionRateStatisticResponse,
+    AverageResponseTimeStatisticItem,
     AverageResponseTimeStatisticResponse,
+    TokensPerSecondStatisticItem,
     TokensPerSecondStatisticResponse,
 )
 
@@ -131,8 +145,7 @@ register_response_schema_models(
 class DailyMessageStatistic(Resource):
     @console_ns.doc("get_daily_message_statistics")
     @console_ns.doc(description="Get daily message statistics for an application")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(StatisticTimeRangeQuery))
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(StatisticTimeRangeQuery)})
     @console_ns.response(
         200,
         "Daily message statistics retrieved successfully",
@@ -185,15 +198,14 @@ WHERE
             for i in rs:
                 response_data.append({"date": str(i.date), "message_count": i.message_count})
 
-        return jsonify({"data": response_data})
+        return dump_response(DailyMessageStatisticResponse, {"data": response_data})
 
 
 @console_ns.route("/apps/<uuid:app_id>/statistics/daily-conversations")
 class DailyConversationStatistic(Resource):
     @console_ns.doc("get_daily_conversation_statistics")
     @console_ns.doc(description="Get daily conversation statistics for an application")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(StatisticTimeRangeQuery))
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(StatisticTimeRangeQuery)})
     @console_ns.response(
         200,
         "Daily conversation statistics retrieved successfully",
@@ -245,15 +257,14 @@ WHERE
             for i in rs:
                 response_data.append({"date": str(i.date), "conversation_count": i.conversation_count})
 
-        return jsonify({"data": response_data})
+        return dump_response(DailyConversationStatisticResponse, {"data": response_data})
 
 
 @console_ns.route("/apps/<uuid:app_id>/statistics/daily-end-users")
 class DailyTerminalsStatistic(Resource):
     @console_ns.doc("get_daily_terminals_statistics")
     @console_ns.doc(description="Get daily terminal/end-user statistics for an application")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(StatisticTimeRangeQuery))
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(StatisticTimeRangeQuery)})
     @console_ns.response(
         200,
         "Daily terminal statistics retrieved successfully",
@@ -306,15 +317,14 @@ WHERE
             for i in rs:
                 response_data.append({"date": str(i.date), "terminal_count": i.terminal_count})
 
-        return jsonify({"data": response_data})
+        return dump_response(DailyTerminalStatisticResponse, {"data": response_data})
 
 
 @console_ns.route("/apps/<uuid:app_id>/statistics/token-costs")
 class DailyTokenCostStatistic(Resource):
     @console_ns.doc("get_daily_token_cost_statistics")
     @console_ns.doc(description="Get daily token cost statistics for an application")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(StatisticTimeRangeQuery))
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(StatisticTimeRangeQuery)})
     @console_ns.response(
         200,
         "Daily token cost statistics retrieved successfully",
@@ -370,15 +380,14 @@ WHERE
                     {"date": str(i.date), "token_count": i.token_count, "total_price": i.total_price, "currency": "USD"}
                 )
 
-        return jsonify({"data": response_data})
+        return dump_response(DailyTokenCostStatisticResponse, {"data": response_data})
 
 
 @console_ns.route("/apps/<uuid:app_id>/statistics/average-session-interactions")
 class AverageSessionInteractionStatistic(Resource):
     @console_ns.doc("get_average_session_interaction_statistics")
     @console_ns.doc(description="Get average session interaction statistics for an application")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(StatisticTimeRangeQuery))
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(StatisticTimeRangeQuery)})
     @console_ns.response(
         200,
         "Average session interaction statistics retrieved successfully",
@@ -450,15 +459,14 @@ ORDER BY
                     {"date": str(i.date), "interactions": float(i.interactions.quantize(Decimal("0.01")))}
                 )
 
-        return jsonify({"data": response_data})
+        return dump_response(AverageSessionInteractionStatisticResponse, {"data": response_data})
 
 
 @console_ns.route("/apps/<uuid:app_id>/statistics/user-satisfaction-rate")
 class UserSatisfactionRateStatistic(Resource):
     @console_ns.doc("get_user_satisfaction_rate_statistics")
     @console_ns.doc(description="Get user satisfaction rate statistics for an application")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(StatisticTimeRangeQuery))
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(StatisticTimeRangeQuery)})
     @console_ns.response(
         200,
         "User satisfaction rate statistics retrieved successfully",
@@ -520,15 +528,14 @@ WHERE
                     }
                 )
 
-        return jsonify({"data": response_data})
+        return dump_response(UserSatisfactionRateStatisticResponse, {"data": response_data})
 
 
 @console_ns.route("/apps/<uuid:app_id>/statistics/average-response-time")
 class AverageResponseTimeStatistic(Resource):
     @console_ns.doc("get_average_response_time_statistics")
     @console_ns.doc(description="Get average response time statistics for an application")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(StatisticTimeRangeQuery))
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(StatisticTimeRangeQuery)})
     @console_ns.response(
         200,
         "Average response time statistics retrieved successfully",
@@ -581,15 +588,14 @@ WHERE
             for i in rs:
                 response_data.append({"date": str(i.date), "latency": round(i.latency * 1000, 4)})
 
-        return jsonify({"data": response_data})
+        return dump_response(AverageResponseTimeStatisticResponse, {"data": response_data})
 
 
 @console_ns.route("/apps/<uuid:app_id>/statistics/tokens-per-second")
 class TokensPerSecondStatistic(Resource):
     @console_ns.doc("get_tokens_per_second_statistics")
     @console_ns.doc(description="Get tokens per second statistics for an application")
-    @console_ns.doc(params={"app_id": "Application ID"})
-    @console_ns.doc(params=query_params_from_model(StatisticTimeRangeQuery))
+    @console_ns.doc(params={"app_id": "Application ID", **query_params_from_model(StatisticTimeRangeQuery)})
     @console_ns.response(
         200,
         "Tokens per second statistics retrieved successfully",
@@ -645,4 +651,4 @@ WHERE
             for i in rs:
                 response_data.append({"date": str(i.date), "tps": round(i.tokens_per_second, 4)})
 
-        return jsonify({"data": response_data})
+        return dump_response(TokensPerSecondStatisticResponse, {"data": response_data})

api/controllers/console/app/workflow.py

@@ -2,11 +2,11 @@ import json
 import logging
 from collections.abc import Sequence
 from datetime import datetime
-from typing import Any, NotRequired, TypedDict, cast
+from typing import Any, NotRequired, TypedDict
 
 from flask import abort, request
 from flask_restx import Resource, fields
-from pydantic import AliasChoices, BaseModel, Field, RootModel, ValidationError, field_validator
+from pydantic import AliasChoices, BaseModel, ConfigDict, Field, RootModel, ValidationError, field_validator
 from sqlalchemy.orm import Session, sessionmaker
 from werkzeug.exceptions import BadRequest, Forbidden, InternalServerError, NotFound
 
@@ -27,7 +27,7 @@ from controllers.console.app.error import (
     DraftWorkflowNotSync,
 )
 from controllers.console.app.permission_keys import get_app_permission_keys
-from controllers.console.app.wraps import get_app_model
+from controllers.console.app.wraps import get_app_model, with_session
 from controllers.console.wraps import (
     RBACPermission,
     RBACResourceScope,
@@ -78,6 +78,7 @@ from repositories.workflow_collaboration_repository import WORKFLOW_ONLINE_USERS
 from services.app_generate_service import AppGenerateService
 from services.errors.app import IsDraftWorkflowError, WorkflowHashNotEqualError, WorkflowNotFoundError
 from services.errors.llm import InvokeRateLimitError
+from services.workflow_ref_service import WorkflowRefService
 from services.workflow_service import DraftWorkflowDeletionError, WorkflowInUseError, WorkflowService
 
 logger = logging.getLogger(__name__)
@@ -158,8 +159,71 @@ class ConvertToWorkflowPayload(BaseModel):
     icon_background: str | None = None
 
 
+class WorkflowFeatureTogglePayload(BaseModel):
+    model_config = ConfigDict(extra="allow")
+
+    enabled: bool | None = None
+
+
+class WorkflowSuggestedQuestionsAfterAnswerPayload(WorkflowFeatureTogglePayload):
+    model: dict[str, Any] | None = None
+    prompt: str | None = None
+
+
+class WorkflowTextToSpeechPayload(WorkflowFeatureTogglePayload):
+    language: str | None = None
+    voice: str | None = None
+    autoPlay: str | None = None
+
+
+class WorkflowSensitiveWordAvoidancePayload(WorkflowFeatureTogglePayload):
+    type: str | None = None
+    config: dict[str, Any] | None = None
+
+
+class WorkflowFileUploadTransferPayload(WorkflowFeatureTogglePayload):
+    number_limits: int | None = None
+    transfer_methods: list[str] | None = None
+
+
+class WorkflowFileUploadImagePayload(WorkflowFileUploadTransferPayload):
+    detail: str | None = None
+
+
+class WorkflowFileUploadPreviewConfigPayload(BaseModel):
+    mode: str | None = None
+    file_type_list: list[str] | None = None
+
+
+class WorkflowFileUploadPayload(WorkflowFeatureTogglePayload):
+    allowed_file_types: list[str] | None = None
+    allowed_file_extensions: list[str] | None = None
+    allowed_file_upload_methods: list[str] | None = None
+    number_limits: int | None = None
+    image: WorkflowFileUploadImagePayload | None = None
+    document: WorkflowFileUploadTransferPayload | None = None
+    audio: WorkflowFileUploadTransferPayload | None = None
+    video: WorkflowFileUploadTransferPayload | None = None
+    custom: WorkflowFileUploadTransferPayload | None = None
+    preview_config: WorkflowFileUploadPreviewConfigPayload | None = None
+    fileUploadConfig: dict[str, Any] | None = None
+
+
+class WorkflowFeaturesConfigPayload(BaseModel):
+    model_config = ConfigDict(extra="allow")
+
+    opening_statement: str | None = None
+    suggested_questions: list[str] | None = None
+    suggested_questions_after_answer: WorkflowSuggestedQuestionsAfterAnswerPayload | None = None
+    text_to_speech: WorkflowTextToSpeechPayload | None = None
+    speech_to_text: WorkflowFeatureTogglePayload | None = None
+    retriever_resource: WorkflowFeatureTogglePayload | None = None
+    sensitive_word_avoidance: WorkflowSensitiveWordAvoidancePayload | None = None
+    file_upload: WorkflowFileUploadPayload | None = None
+
+
 class WorkflowFeaturesPayload(BaseModel):
-    features: dict[str, Any] = Field(
+    features: WorkflowFeaturesConfigPayload = Field(
         ...,
         description="Workflow feature configuration",
     )
@@ -343,6 +407,15 @@ register_schema_models(
     ConvertToWorkflowPayload,
     WorkflowListQuery,
     WorkflowUpdatePayload,
+    WorkflowFeatureTogglePayload,
+    WorkflowSuggestedQuestionsAfterAnswerPayload,
+    WorkflowTextToSpeechPayload,
+    WorkflowSensitiveWordAvoidancePayload,
+    WorkflowFileUploadTransferPayload,
+    WorkflowFileUploadImagePayload,
+    WorkflowFileUploadPreviewConfigPayload,
+    WorkflowFileUploadPayload,
+    WorkflowFeaturesConfigPayload,
     WorkflowFeaturesPayload,
     WorkflowOnlineUsersPayload,
     DraftWorkflowTriggerRunPayload,
@@ -449,7 +522,7 @@ class DraftWorkflowApi(Resource):
         """
         # fetch draft workflow by app_model
         workflow_service = WorkflowService()
-        workflow = workflow_service.get_draft_workflow(app_model=app_model)
+        workflow = workflow_service.get_draft_workflow(app_model=app_model, session=db.session())
 
         if not workflow:
             raise DraftWorkflowNotExist()
@@ -460,7 +533,7 @@ class DraftWorkflowApi(Resource):
         # front-end can treat draft graph node data as the editing source.
         response = WorkflowResponse.model_validate(workflow, from_attributes=True).model_dump(mode="json")
         response["graph"] = WorkflowAgentPublishService.project_draft_bindings_to_graph(
-            session=cast(Session, db.session),
+            session=db.session(),
             draft_workflow=workflow,
         )
         return response
@@ -529,6 +602,7 @@ class DraftWorkflowApi(Resource):
                 account=current_user,
                 environment_variables=environment_variables,
                 conversation_variables=conversation_variables,
+                session=db.session(),
             )
         except WorkflowHashNotEqualError:
             raise DraftWorkflowNotSync()
@@ -558,7 +632,8 @@ class AdvancedChatDraftWorkflowRunApi(Resource):
     @get_app_model(mode=[AppMode.ADVANCED_CHAT])
     @with_current_user
     @edit_permission_required
-    def post(self, current_user: Account, app_model: App):
+    @with_session
+    def post(self, session: Session, current_user: Account, app_model: App):
         """
         Run draft workflow
         """
@@ -571,7 +646,12 @@ class AdvancedChatDraftWorkflowRunApi(Resource):
 
         try:
             response = AppGenerateService.generate(
-                app_model=app_model, user=current_user, args=args, invoke_from=InvokeFrom.DEBUGGER, streaming=True
+                session=session,
+                app_model=app_model,
+                user=current_user,
+                args=args,
+                invoke_from=InvokeFrom.DEBUGGER,
+                streaming=True,
             )
 
             return helper.compact_generate_response(response)
@@ -616,7 +696,12 @@ class AdvancedChatDraftRunIterationNodeApi(Resource):
 
         try:
             response = AppGenerateService.generate_single_iteration(
-                app_model=app_model, user=current_user, node_id=node_id, args=args, streaming=True
+                app_model=app_model,
+                user=current_user,
+                node_id=node_id,
+                args=args,
+                session=db.session(),
+                streaming=True,
             )
 
             return helper.compact_generate_response(response)
@@ -659,7 +744,12 @@ class WorkflowDraftRunIterationNodeApi(Resource):
 
         try:
             response = AppGenerateService.generate_single_iteration(
-                app_model=app_model, user=current_user, node_id=node_id, args=args, streaming=True
+                app_model=app_model,
+                user=current_user,
+                node_id=node_id,
+                args=args,
+                session=db.session(),
+                streaming=True,
             )
 
             return helper.compact_generate_response(response)
@@ -698,7 +788,12 @@ class AdvancedChatDraftRunLoopNodeApi(Resource):
 
         try:
             response = AppGenerateService.generate_single_loop(
-                app_model=app_model, user=current_user, node_id=node_id, args=args, streaming=True
+                app_model=app_model,
+                user=current_user,
+                node_id=node_id,
+                args=args,
+                session=db.session(),
+                streaming=True,
             )
 
             return helper.compact_generate_response(response)
@@ -741,7 +836,12 @@ class WorkflowDraftRunLoopNodeApi(Resource):
 
         try:
             response = AppGenerateService.generate_single_loop(
-                app_model=app_model, user=current_user, node_id=node_id, args=args, streaming=True
+                app_model=app_model,
+                user=current_user,
+                node_id=node_id,
+                args=args,
+                session=db.session(),
+                streaming=True,
             )
 
             return helper.compact_generate_response(response)
@@ -818,6 +918,7 @@ class AdvancedChatDraftHumanInputFormPreviewApi(Resource):
             account=current_user,
             node_id=node_id,
             inputs=inputs,
+            session=db.session(),
         )
         return jsonable_encoder(preview)
 
@@ -853,6 +954,7 @@ class AdvancedChatDraftHumanInputFormRunApi(Resource):
             form_inputs=args.form_inputs,
             inputs=args.inputs,
             action=args.action,
+            session=db.session(),
         )
         return jsonable_encoder(result)
 
@@ -884,6 +986,7 @@ class WorkflowDraftHumanInputFormPreviewApi(Resource):
             account=current_user,
             node_id=node_id,
             inputs=inputs,
+            session=db.session(),
         )
         return jsonable_encoder(preview)
 
@@ -919,6 +1022,7 @@ class WorkflowDraftHumanInputFormRunApi(Resource):
             form_inputs=args.form_inputs,
             inputs=args.inputs,
             action=args.action,
+            session=db.session(),
         )
         return jsonable_encoder(result)
 
@@ -949,6 +1053,7 @@ class WorkflowDraftHumanInputDeliveryTestApi(Resource):
             node_id=node_id,
             delivery_method_id=args.delivery_method_id,
             inputs=args.inputs,
+            session=db.session(),
         )
         return jsonable_encoder({})
 
@@ -972,7 +1077,8 @@ class DraftWorkflowRunApi(Resource):
     @get_app_model(mode=[AppMode.WORKFLOW])
     @with_current_user
     @edit_permission_required
-    def post(self, current_user: Account, app_model: App):
+    @with_session
+    def post(self, session: Session, current_user: Account, app_model: App):
         """
         Run draft workflow
         """
@@ -984,6 +1090,7 @@ class DraftWorkflowRunApi(Resource):
 
         try:
             response = AppGenerateService.generate(
+                session=session,
                 app_model=app_model,
                 user=current_user,
                 args=args,
@@ -1057,7 +1164,7 @@ class DraftWorkflowNodeRunApi(Resource):
 
         workflow_srv = WorkflowService()
         # fetch draft workflow by app_model
-        draft_workflow = workflow_srv.get_draft_workflow(app_model=app_model)
+        draft_workflow = workflow_srv.get_draft_workflow(app_model=app_model, session=db.session())
         if not draft_workflow:
             raise ValueError("Workflow not initialized")
         files = _parse_file(draft_workflow, args.get("files"))
@@ -1100,7 +1207,7 @@ class PublishedWorkflowApi(Resource):
         """
         # fetch published workflow by app_model
         workflow_service = WorkflowService()
-        workflow = workflow_service.get_published_workflow(app_model=app_model)
+        workflow = workflow_service.get_published_workflow(app_model=app_model, session=db.session())
 
         # return workflow, if not found, return None
         if workflow is None:
@@ -1242,7 +1349,9 @@ class ConvertToWorkflowApi(Resource):
 
         # convert to workflow mode
         workflow_service = WorkflowService()
-        new_app_model = workflow_service.convert_to_workflow(app_model=app_model, account=current_user, args=args)
+        new_app_model = workflow_service.convert_to_workflow(
+            app_model=app_model, account=current_user, args=args, session=db.session()
+        )
 
         # return app id
         return {
@@ -1274,10 +1383,12 @@ class WorkflowFeaturesApi(Resource):
     def post(self, current_user: Account, app_model: App):
 
         args = WorkflowFeaturesPayload.model_validate(console_ns.payload or {})
-        features = args.features
+        features = args.features.model_dump(mode="json", exclude_unset=True)
 
         workflow_service = WorkflowService()
-        workflow_service.update_draft_workflow_features(app_model=app_model, features=features, account=current_user)
+        workflow_service.update_draft_workflow_features(
+            app_model=app_model, features=features, account=current_user, session=db.session()
+        )
 
         return {"result": "success"}
 
@@ -1358,6 +1469,7 @@ class DraftWorkflowRestoreApi(Resource):
                 app_model=app_model,
                 workflow_id=workflow_id,
                 account=current_user,
+                session=db.session(),
             )
         except IsDraftWorkflowError as exc:
             raise BadRequest(RESTORE_SOURCE_WORKFLOW_MUST_BE_PUBLISHED_MESSAGE) from exc
@@ -1406,15 +1518,15 @@ class WorkflowByIdApi(Resource):
             return {"message": "No valid fields to update"}, 400
 
         workflow_service = WorkflowService()
+        workflow_ref = WorkflowRefService.create_app_workflow_ref(app_model, workflow_id)
 
         # Create a session and manage the transaction
         with sessionmaker(db.engine, expire_on_commit=False).begin() as session:
             workflow = workflow_service.update_workflow(
                 session=session,
-                workflow_id=workflow_id,
-                tenant_id=app_model.tenant_id,
                 account_id=current_user.id,
                 data=update_data,
+                workflow_ref=workflow_ref,
             )
 
             if not workflow:
@@ -1434,12 +1546,14 @@ class WorkflowByIdApi(Resource):
         Delete workflow
         """
         workflow_service = WorkflowService()
+        workflow_ref = WorkflowRefService.create_app_workflow_ref(app_model, workflow_id)
 
         # Create a session and manage the transaction
         with sessionmaker(db.engine).begin() as session:
             try:
                 workflow_service.delete_workflow(
-                    session=session, workflow_id=workflow_id, tenant_id=app_model.tenant_id
+                    session=session,
+                    workflow_ref=workflow_ref,
                 )
             except WorkflowInUseError as e:
                 abort(400, description=str(e))
@@ -1470,7 +1584,7 @@ class DraftWorkflowNodeLastRunApi(Resource):
     @get_app_model(mode=[AppMode.ADVANCED_CHAT, AppMode.WORKFLOW])
     def get(self, app_model: App, node_id: str):
         srv = WorkflowService()
-        workflow = srv.get_draft_workflow(app_model)
+        workflow = srv.get_draft_workflow(app_model, session=db.session())
         if not workflow:
             raise NotFound("Workflow not found")
         node_exec = srv.get_node_last_run(
@@ -1515,14 +1629,15 @@ class DraftWorkflowTriggerRunApi(Resource):
     @get_app_model(mode=[AppMode.WORKFLOW])
     @with_current_user
     @edit_permission_required
-    def post(self, current_user: Account, app_model: App):
+    @with_session
+    def post(self, session: Session, current_user: Account, app_model: App):
         """
         Poll for trigger events and execute full workflow when event arrives
         """
         args = DraftWorkflowTriggerRunPayload.model_validate(console_ns.payload or {})
         node_id = args.node_id
         workflow_service = WorkflowService()
-        draft_workflow = workflow_service.get_draft_workflow(app_model)
+        draft_workflow = workflow_service.get_draft_workflow(app_model, session=db.session())
         if not draft_workflow:
             raise ValueError("Workflow not found")
 
@@ -1543,6 +1658,7 @@ class DraftWorkflowTriggerRunApi(Resource):
             workflow_args[SKIP_PREPARE_USER_INPUTS_KEY] = True
             return helper.compact_generate_response(
                 AppGenerateService.generate(
+                    session=session,
                     app_model=app_model,
                     user=current_user,
                     args=workflow_args,
@@ -1590,7 +1706,7 @@ class DraftWorkflowTriggerNodeApi(Resource):
         """
 
         workflow_service = WorkflowService()
-        draft_workflow = workflow_service.get_draft_workflow(app_model)
+        draft_workflow = workflow_service.get_draft_workflow(app_model, session=db.session())
         if not draft_workflow:
             raise ValueError("Workflow not found")
 
@@ -1665,7 +1781,8 @@ class DraftWorkflowTriggerRunAllApi(Resource):
     @with_current_user
     @edit_permission_required
     @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_TEST_AND_RUN)
-    def post(self, current_user: Account, app_model: App):
+    @with_session
+    def post(self, session: Session, current_user: Account, app_model: App):
         """
         Full workflow debug when the start node is a trigger
         """
@@ -1673,7 +1790,7 @@ class DraftWorkflowTriggerRunAllApi(Resource):
         args = DraftWorkflowTriggerRunAllPayload.model_validate(console_ns.payload or {})
         node_ids = args.node_ids
         workflow_service = WorkflowService()
-        draft_workflow = workflow_service.get_draft_workflow(app_model)
+        draft_workflow = workflow_service.get_draft_workflow(app_model, session=db.session())
         if not draft_workflow:
             raise ValueError("Workflow not found")
 
@@ -1697,6 +1814,7 @@ class DraftWorkflowTriggerRunAllApi(Resource):
 
             workflow_args[SKIP_PREPARE_USER_INPUTS_KEY] = True
             response = AppGenerateService.generate(
+                session=session,
                 app_model=app_model,
                 user=current_user,
                 args=workflow_args,
@@ -1741,7 +1859,7 @@ class WorkflowOnlineUsersApi(Resource):
             return {"data": []}
 
         workflow_service = WorkflowService()
-        accessible_app_ids = workflow_service.get_accessible_app_ids(app_ids, current_tenant_id)
+        accessible_app_ids = workflow_service.get_accessible_app_ids(app_ids, current_tenant_id, session=db.session())
         ordered_accessible_app_ids = [app_id for app_id in app_ids if app_id in accessible_app_ids]
 
         users_json_by_app_id: dict[str, Any] = {}

api/controllers/console/app/workflow_app_log.py

@@ -181,7 +181,7 @@ class WorkflowAppLogApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR)
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_LOG_AND_ANNOTATION)
     @get_app_model(mode=[AppMode.WORKFLOW])
     def get(self, app_model: App):
         """
@@ -225,7 +225,7 @@ class WorkflowArchivedLogApi(Resource):
     @setup_required
     @login_required
     @account_initialization_required
-    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR)
+    @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_LOG_AND_ANNOTATION)
     @get_app_model(mode=[AppMode.WORKFLOW])
     def get(self, app_model: App):
         """

api/controllers/console/app/workflow_comment.py

@@ -17,6 +17,7 @@ from controllers.console.wraps import (
     with_current_tenant_id,
     with_current_user,
 )
+from extensions.ext_database import db
 from fields.base import ResponseModel
 from fields.member_fields import AccountWithRole
 from libs.helper import build_avatar_url, dump_response, to_timestamp
@@ -188,14 +189,14 @@ class WorkflowCommentReplyUpdate(ResponseModel):
 
 register_schema_models(
     console_ns,
-    AccountWithRole,
-    WorkflowCommentMentionUsersPayload,
     WorkflowCommentCreatePayload,
     WorkflowCommentUpdatePayload,
     WorkflowCommentReplyPayload,
 )
 register_response_schema_models(
     console_ns,
+    AccountWithRole,
+    WorkflowCommentMentionUsersPayload,
     WorkflowCommentAccount,
     WorkflowCommentReply,
     WorkflowCommentMention,
@@ -489,7 +490,7 @@ class WorkflowCommentMentionUsersApi(Resource):
         current_tenant = current_user.current_tenant  # need the tenant object here
         if current_tenant is None:
             raise ValueError("current tenant is required")
-        members = TenantService.get_tenant_members(current_tenant)
+        members = TenantService.get_tenant_members(current_tenant, session=db.session())
         users = TypeAdapter(list[AccountWithRole]).validate_python(members, from_attributes=True)
         response = WorkflowCommentMentionUsersPayload(users=users)
         return response.model_dump(mode="json"), 200

api/controllers/console/app/workflow_draft_variable.py

@@ -6,7 +6,7 @@ from uuid import UUID
 
 from flask import Response, request
 from flask_restx import Resource, fields, marshal, marshal_with
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, ConfigDict, Field
 from sqlalchemy.orm import sessionmaker
 
 from controllers.common.errors import InvalidArgumentError, NotFoundError
@@ -79,15 +79,33 @@ class WorkflowDraftVariableUpdatePayload(BaseModel):
     value: Any | None = Field(default=None, description="Variable value")
 
 
+class WorkflowVariableItemPayload(BaseModel):
+    model_config = ConfigDict(extra="allow")
+
+    id: str | None = None
+    name: str | None = None
+    value_type: str | None = None
+    value: Any | None = None
+    description: str | None = None
+
+
+class ConversationVariableItemPayload(WorkflowVariableItemPayload):
+    pass
+
+
+class EnvironmentVariableItemPayload(WorkflowVariableItemPayload):
+    pass
+
+
 class ConversationVariableUpdatePayload(BaseModel):
-    conversation_variables: list[dict[str, Any]] = Field(
+    conversation_variables: list[ConversationVariableItemPayload] = Field(
         ...,
         description="Conversation variables for the draft workflow",
     )
 
 
 class EnvironmentVariableUpdatePayload(BaseModel):
-    environment_variables: list[dict[str, Any]] = Field(
+    environment_variables: list[EnvironmentVariableItemPayload] = Field(
         ...,
         description="Environment variables for the draft workflow",
     )
@@ -114,7 +132,9 @@ register_schema_models(
     console_ns,
     WorkflowDraftVariableListQuery,
     WorkflowDraftVariableUpdatePayload,
+    ConversationVariableItemPayload,
     ConversationVariableUpdatePayload,
+    EnvironmentVariableItemPayload,
     EnvironmentVariableUpdatePayload,
 )
 register_response_schema_models(console_ns, SimpleResultResponse, EnvironmentVariableListResponse)
@@ -317,7 +337,7 @@ class WorkflowVariableCollectionApi(Resource):
 
         # fetch draft workflow by app_model
         workflow_service = WorkflowService()
-        workflow_exist = workflow_service.is_workflow_exist(app_model=app_model)
+        workflow_exist = workflow_service.is_workflow_exist(app_model=app_model, session=db.session())
         if not workflow_exist:
             raise DraftWorkflowNotExist()
 
@@ -533,7 +553,7 @@ class VariableResetApi(Resource):
         )
 
         workflow_srv = WorkflowService()
-        draft_workflow = workflow_srv.get_draft_workflow(app_model)
+        draft_workflow = workflow_srv.get_draft_workflow(app_model, session=db.session())
         if draft_workflow is None:
             raise NotFoundError(
                 f"Draft workflow not found, app_id={app_model.id}",
@@ -586,7 +606,7 @@ class ConversationVariableCollectionApi(Resource):
         # NOTE(QuantumGhost): Prefill conversation variables into the draft variables table
         # so their IDs can be returned to the caller.
         workflow_srv = WorkflowService()
-        draft_workflow = workflow_srv.get_draft_workflow(app_model)
+        draft_workflow = workflow_srv.get_draft_workflow(app_model, session=db.session())
         if draft_workflow is None:
             raise NotFoundError(description=f"draft workflow not found, id={app_model.id}")
         draft_var_srv = WorkflowDraftVariableService(db.session())
@@ -615,7 +635,9 @@ class ConversationVariableCollectionApi(Resource):
 
         workflow_service = WorkflowService()
 
-        conversation_variables_list = payload.conversation_variables
+        conversation_variables_list = [
+            variable.model_dump(mode="json", exclude_unset=True) for variable in payload.conversation_variables
+        ]
         conversation_variables = [
             variable_factory.build_conversation_variable_from_mapping(obj) for obj in conversation_variables_list
         ]
@@ -624,6 +646,7 @@ class ConversationVariableCollectionApi(Resource):
             app_model=app_model,
             account=current_user,
             conversation_variables=conversation_variables,
+            session=db.session(),
         )
 
         return {"result": "success"}
@@ -661,7 +684,7 @@ class EnvironmentVariableCollectionApi(Resource):
         """
         # fetch draft workflow by app_model
         workflow_service = WorkflowService()
-        workflow = workflow_service.get_draft_workflow(app_model=app_model)
+        workflow = workflow_service.get_draft_workflow(app_model=app_model, session=db.session())
         if workflow is None:
             raise DraftWorkflowNotExist()
 
@@ -707,7 +730,9 @@ class EnvironmentVariableCollectionApi(Resource):
 
         workflow_service = WorkflowService()
 
-        environment_variables_list = payload.environment_variables
+        environment_variables_list = [
+            variable.model_dump(mode="json", exclude_unset=True) for variable in payload.environment_variables
+        ]
         environment_variables = [
             variable_factory.build_environment_variable_from_mapping(obj) for obj in environment_variables_list
         ]
@@ -716,6 +741,7 @@ class EnvironmentVariableCollectionApi(Resource):
             app_model=app_model,
             account=current_user,
             environment_variables=environment_variables,
+            session=db.session(),
         )
 
         return {"result": "success"}

api/controllers/console/app/workflow_node_output_inspector.py

@@ -41,6 +41,7 @@ from controllers.console.wraps import (
     rbac_permission_required,
     setup_required,
 )
+from extensions.ext_database import db
 from libs.exception import BaseHTTPException
 from libs.login import login_required
 from models import App, AppMode
@@ -92,7 +93,9 @@ def _serve_snapshot(app_model: App, run_id: UUID) -> dict:
     Flask request context.
     """
     try:
-        snapshot = _service().snapshot_workflow_run(app_model=app_model, workflow_run_id=str(run_id))
+        snapshot = _service().snapshot_workflow_run(
+            app_model=app_model, workflow_run_id=str(run_id), session=db.session()
+        )
     except NodeOutputInspectorError as error:
         raise _InspectorNotFound(error) from error
     return snapshot.model_dump(mode="json")
@@ -105,6 +108,7 @@ def _serve_node_detail(app_model: App, run_id: UUID, node_id: str) -> dict:
             app_model=app_model,
             workflow_run_id=str(run_id),
             node_id=node_id,
+            session=db.session(),
         )
     except NodeOutputInspectorError as error:
         raise _InspectorNotFound(error) from error
@@ -119,6 +123,7 @@ def _serve_output_preview(app_model: App, run_id: UUID, node_id: str, output_nam
             workflow_run_id=str(run_id),
             node_id=node_id,
             output_name=output_name,
+            session=db.session(),
         )
     except NodeOutputInspectorError as error:
         raise _InspectorNotFound(error) from error
@@ -245,7 +250,7 @@ def _stream_inspector_events(app_model: App, run_id: UUID) -> Iterator[str]:
     # if the run is gone (raised before yielding any bytes, so Flask turns it
     # into the normal HTTP 404 path).
     try:
-        snapshot = service.snapshot_workflow_run(app_model=app_model, workflow_run_id=run_id_str)
+        snapshot = service.snapshot_workflow_run(app_model=app_model, workflow_run_id=run_id_str, session=db.session())
     except NodeOutputInspectorError as error:
         raise _InspectorNotFound(error) from error
 
@@ -308,6 +313,7 @@ def _stream_inspector_events(app_model: App, run_id: UUID) -> Iterator[str]:
                 app_model=app_model,
                 workflow_run_id=run_id_str,
                 node_id=message.node_id,
+                session=db.session(),
             )
         except NodeOutputInspectorError:
             # Node may not appear in the graph yet (race with persistence); skip.

api/controllers/console/app/workflow_run.py

@@ -23,6 +23,7 @@ from controllers.console.wraps import (
     with_current_user,
 )
 from core.workflow.human_input_forms import load_form_tokens_by_form_id as _load_form_tokens_by_form_id
+from core.workflow.nodes.human_input.pause_reason import HumanInputRequired
 from extensions.ext_database import db
 from fields.base import ResponseModel
 from fields.workflow_run_fields import (
@@ -33,7 +34,6 @@ from fields.workflow_run_fields import (
     WorkflowRunNodeExecutionResponse,
     WorkflowRunPaginationResponse,
 )
-from graphon.entities.pause_reason import HumanInputRequired
 from graphon.enums import WorkflowExecutionStatus
 from libs.archive_storage import ArchiveStorageNotConfiguredError, get_archive_storage
 from libs.custom_inputs import time_duration

api/controllers/console/app/workflow_trigger.py

@@ -12,6 +12,7 @@ from configs import dify_config
 from controllers.common.schema import query_params_from_model, register_schema_models
 from extensions.ext_database import db
 from fields.base import ResponseModel
+from libs.helper import dump_response
 from libs.login import login_required
 from models.enums import AppTriggerStatus
 from models.model import App, AppMode
@@ -121,7 +122,7 @@ class WebhookTriggerApi(Resource):
             if not webhook_trigger:
                 raise NotFound("Webhook trigger not found for this node")
 
-            return WebhookTriggerResponse.model_validate(webhook_trigger, from_attributes=True).model_dump(mode="json")
+            return dump_response(WebhookTriggerResponse, webhook_trigger)
 
 
 @console_ns.route("/apps/<uuid:app_id>/triggers")
@@ -160,9 +161,7 @@ class AppTriggersApi(Resource):
             else:
                 trigger.icon = ""  # type: ignore
 
-        return WorkflowTriggerListResponse.model_validate({"data": triggers}, from_attributes=True).model_dump(
-            mode="json"
-        )
+        return dump_response(WorkflowTriggerListResponse, {"data": triggers})
 
 
 @console_ns.route("/apps/<uuid:app_id>/trigger-enable")
@@ -204,4 +203,4 @@ class AppTriggerEnableApi(Resource):
         else:
             trigger.icon = ""  # type: ignore
 
-        return WorkflowTriggerResponse.model_validate(trigger, from_attributes=True).model_dump(mode="json")
+        return dump_response(WorkflowTriggerResponse, trigger)

api/controllers/console/app/wraps.py

@@ -1,26 +1,26 @@
 """Controller decorators for console app resources.
 
-`with_session` opens one SQLAlchemy session for a request handler and injects it
-as the first argument after `self`. Handlers use a transaction by default so
-migrated write paths keep commit/rollback handling; pure read handlers may opt
-out with `write=False`. App-loading decorators prefer that injected session when
-present, while still supporting existing handlers that have not been migrated
-yet and still rely on Flask-SQLAlchemy's scoped `db.session`.
+App-loading decorators prefer a session injected by
+`controllers.common.session.with_session` when present, while still supporting
+existing handlers that have not been migrated yet and still rely on
+Flask-SQLAlchemy's scoped `db.session`.
 """
 
 from collections.abc import Callable
 from functools import wraps
-from typing import Concatenate, cast, overload
+from typing import cast, overload
 
 from sqlalchemy import select
 from sqlalchemy.orm import Session
 
+from controllers.common.session import with_session
 from controllers.console.app.error import AppNotFoundError
-from core.db.session_factory import session_factory
 from extensions.ext_database import db
 from libs.login import current_account_with_tenant
 from models import App, AppMode
 
+__all__ = ["get_app_model", "get_app_model_with_trial", "with_session"]
+
 
 def _load_app_model(session: Session, app_id: str) -> App | None:
     """Load the tenant-scoped app row with the request session owned by `with_session`."""
@@ -45,48 +45,6 @@ def _load_app_model_with_trial(app_id: str) -> App | None:
     return app_model
 
 
-@overload
-def with_session[T, **P, R](
-    view: Callable[Concatenate[T, Session, P], R],
-    *,
-    write: bool = True,
-) -> Callable[Concatenate[T, P], R]: ...
-
-
-@overload
-def with_session[T, **P, R](
-    view: None = None,
-    *,
-    write: bool = True,
-) -> Callable[[Callable[Concatenate[T, Session, P], R]], Callable[Concatenate[T, P], R]]: ...
-
-
-def with_session[T, **P, R](
-    view: Callable[Concatenate[T, Session, P], R] | None = None,
-    *,
-    write: bool = True,
-) -> (
-    Callable[Concatenate[T, P], R] | Callable[[Callable[Concatenate[T, Session, P], R]], Callable[Concatenate[T, P], R]]
-):
-    """Inject a request-scoped session, using a transaction only for write handlers."""
-
-    def decorator(view: Callable[Concatenate[T, Session, P], R]) -> Callable[Concatenate[T, P], R]:
-        @wraps(view)
-        def wrapper(self: T, *args: P.args, **kwargs: P.kwargs) -> R:
-            if write:
-                with session_factory.get_session_maker().begin() as session:
-                    return view(self, session, *args, **kwargs)
-
-            with session_factory.create_session() as session:
-                return view(self, session, *args, **kwargs)
-
-        return wrapper
-
-    if view is None:
-        return decorator
-    return decorator(view)
-
-
 def _get_injected_session(args: tuple[object, ...]) -> Session | None:
     """Return the request session inserted by `with_session`, if this handler has been migrated."""
     if len(args) < 2:

api/controllers/console/auth/activate.py

@@ -89,7 +89,9 @@ class ActivateCheckApi(Resource):
         workspaceId = args.workspace_id
         token = args.token
 
-        invitation = RegisterService.get_invitation_with_case_fallback(workspaceId, args.email, token)
+        invitation = RegisterService.get_invitation_with_case_fallback(
+            workspaceId, args.email, token, session=db.session()
+        )
         if invitation:
             data = invitation.get("data", {})
             tenant = invitation.get("tenant", None)
@@ -137,7 +139,9 @@ class ActivateApi(Resource):
         args = ActivatePayload.model_validate(console_ns.payload)
 
         normalized_request_email = args.email.lower() if args.email else None
-        invitation = RegisterService.get_invitation_with_case_fallback(args.workspace_id, args.email, args.token)
+        invitation = RegisterService.get_invitation_with_case_fallback(
+            args.workspace_id, args.email, args.token, session=db.session()
+        )
         if invitation is None:
             raise AlreadyActivateError()
 
@@ -174,7 +178,7 @@ class ActivateApi(Resource):
         RegisterService.revoke_token(args.workspace_id, normalized_request_email, args.token)
 
         if membership_id is None:
-            TenantService.create_tenant_member(tenant, account, db.session, role=role)
+            TenantService.create_tenant_member(tenant, account, db.session(), role=role)
 
         if setup_fields:
             account.name = setup_fields[0]
@@ -184,6 +188,6 @@ class ActivateApi(Resource):
             account.status = AccountStatus.ACTIVE
             account.initialized_at = naive_utc_now()
 
-        TenantService.switch_tenant(account, tenant.id)
+        TenantService.switch_tenant(account, tenant.id, session=db.session())
 
         return {"result": "success"}

api/controllers/console/auth/data_source_bearer_auth.py

@@ -5,6 +5,7 @@ from pydantic import BaseModel, Field
 
 from controllers.common.fields import SimpleResultResponse
 from controllers.common.schema import register_response_schema_models, register_schema_models
+from extensions.ext_database import db
 from fields.base import ResponseModel
 from libs.login import login_required
 from services.auth.api_key_auth_service import ApiKeyAuthService
@@ -58,7 +59,7 @@ class ApiKeyAuthDataSource(Resource):
     @account_initialization_required
     @with_current_tenant_id
     def get(self, current_tenant_id: str):
-        data_source_api_key_bindings = ApiKeyAuthService.get_provider_auth_list(current_tenant_id)
+        data_source_api_key_bindings = ApiKeyAuthService.get_provider_auth_list(db.session(), current_tenant_id)
         if data_source_api_key_bindings:
             return {
                 "sources": [
@@ -83,7 +84,7 @@ class ApiKeyAuthDataSourceBinding(Resource):
     @login_required
     @account_initialization_required
     @is_admin_or_owner_required
-    @rbac_permission_required(RBACResourceScope.WORKSPACE, RBACPermission.CREDENTIAL_MANAGE, resource_required=False)
+    @rbac_permission_required(RBACResourceScope.WORKSPACE, RBACPermission.CREDENTIAL_CREATE, resource_required=False)
     @console_ns.expect(console_ns.models[ApiKeyAuthBindingPayload.__name__])
     @with_current_tenant_id
     def post(self, current_tenant_id: str):
@@ -92,7 +93,7 @@ class ApiKeyAuthDataSourceBinding(Resource):
         data = payload.model_dump()
         ApiKeyAuthService.validate_api_key_auth_args(data)
         try:
-            ApiKeyAuthService.create_provider_auth(current_tenant_id, data)
+            ApiKeyAuthService.create_provider_auth(db.session(), current_tenant_id, data)
         except Exception as e:
             raise ApiKeyAuthFailedError(str(e))
         return {"result": "success"}, 200
@@ -109,6 +110,6 @@ class ApiKeyAuthDataSourceBindingDelete(Resource):
     @with_current_tenant_id
     def delete(self, current_tenant_id: str, binding_id: UUID):
         # The role of the current user in the table must be admin or owner
-        ApiKeyAuthService.delete_provider_auth(current_tenant_id, str(binding_id))
+        ApiKeyAuthService.delete_provider_auth(db.session(), current_tenant_id, str(binding_id))
 
         return "", 204

api/controllers/console/auth/email_register.py

@@ -15,6 +15,7 @@ from controllers.console.auth.error import (
     InvalidTokenError,
     PasswordMismatchError,
 )
+from extensions.ext_database import db
 from fields.base import ResponseModel
 from libs.helper import EmailStr, extract_remote_ip
 from libs.helper import timezone as validate_timezone_string
@@ -100,7 +101,7 @@ class EmailRegisterSendEmailApi(Resource):
         if dify_config.BILLING_ENABLED and BillingService.is_email_in_freeze(normalized_email):
             raise AccountInFreezeError()
 
-        account = AccountService.get_account_by_email_with_case_fallback(args.email)
+        account = AccountService.get_account_by_email_with_case_fallback(db.session(), args.email)
         token = AccountService.send_email_register_email(email=normalized_email, account=account, language=language)
         return {"result": "success", "data": token}
 
@@ -175,7 +176,7 @@ class EmailRegisterResetApi(Resource):
         email = register_data.get("email", "")
         normalized_email = email.lower()
 
-        account = AccountService.get_account_by_email_with_case_fallback(email)
+        account = AccountService.get_account_by_email_with_case_fallback(db.session(), email)
 
         if account:
             raise EmailAlreadyInUseError()
@@ -186,7 +187,7 @@ class EmailRegisterResetApi(Resource):
             timezone=args.timezone,
             language=args.language,
         )
-        token_pair = AccountService.login(account=account, ip_address=extract_remote_ip(request))
+        token_pair = AccountService.login(account=account, session=db.session(), ip_address=extract_remote_ip(request))
         AccountService.reset_login_error_rate_limit(normalized_email)
 
         return {"result": "success", "data": token_pair.model_dump()}
@@ -205,6 +206,7 @@ class EmailRegisterResetApi(Resource):
                 password=password,
                 interface_language=get_valid_language(language),
                 timezone=timezone,
+                session=db.session(),
             )
         except AccountRegisterError:
             raise AccountInFreezeError()

api/controllers/console/auth/forgot_password.py

@@ -82,7 +82,7 @@ class ForgotPasswordSendEmailApi(Resource):
         else:
             language = "en-US"
 
-        account = AccountService.get_account_by_email_with_case_fallback(args.email)
+        account = AccountService.get_account_by_email_with_case_fallback(db.session(), args.email)
 
         token = AccountService.send_reset_password_email(
             account=account,
@@ -180,7 +180,7 @@ class ForgotPasswordResetApi(Resource):
         password_hashed = hash_password(args.new_password, salt)
 
         email = reset_data.get("email", "")
-        account = AccountService.get_account_by_email_with_case_fallback(email)
+        account = AccountService.get_account_by_email_with_case_fallback(db.session(), email)
 
         if account:
             account = db.session.merge(account)
@@ -198,10 +198,10 @@ class ForgotPasswordResetApi(Resource):
 
         # Create workspace if needed
         if (
-            not TenantService.get_join_tenants(account)
+            not TenantService.get_join_tenants(account, session=db.session())
             and FeatureService.get_system_features().is_allow_create_workspace
         ):
-            tenant = TenantService.create_tenant(f"{account.name}'s Workspace")
-            TenantService.create_tenant_member(tenant, account, db.session, role="owner")
+            tenant = TenantService.create_tenant(f"{account.name}'s Workspace", session=db.session())
+            TenantService.create_tenant_member(tenant, account, db.session(), role="owner")
             account.current_tenant = tenant
             tenant_was_created.send(tenant)

api/controllers/console/auth/login.py

@@ -9,7 +9,12 @@ from werkzeug.exceptions import Unauthorized
 import services
 from configs import dify_config
 from constants.languages import get_valid_language
-from controllers.common.fields import SimpleResultDataResponse, SimpleResultOptionalDataResponse, SimpleResultResponse
+from controllers.common.fields import (
+    SimpleResultDataResponse,
+    SimpleResultMessageResponse,
+    SimpleResultOptionalDataResponse,
+    SimpleResultResponse,
+)
 from controllers.common.schema import register_response_schema_models, register_schema_models
 from controllers.console import console_ns
 from controllers.console.auth.error import (
@@ -51,7 +56,7 @@ from models.account import Account
 from services.account_service import AccountService, InvitationDetailDict, RegisterService, TenantService
 from services.billing_service import BillingService
 from services.entities.auth_entities import LoginFailureReason, LoginPayloadBase
-from services.errors.account import AccountRegisterError
+from services.errors.account import AccountRegisterError, RefreshTokenAccountNotFoundError, RefreshTokenNotFoundError
 from services.errors.workspace import WorkSpaceNotAllowedCreateError, WorkspacesLimitExceededError
 from services.feature_service import FeatureService
 
@@ -87,6 +92,7 @@ register_schema_models(console_ns, LoginPayload, EmailPayload, EmailCodeLoginPay
 register_response_schema_models(
     console_ns,
     SimpleResultDataResponse,
+    SimpleResultMessageResponse,
     SimpleResultOptionalDataResponse,
     SimpleResultResponse,
 )
@@ -119,7 +125,9 @@ class LoginApi(Resource):
         invite_token = args.invite_token
         invitation_data: InvitationDetailDict | None = None
         if invite_token:
-            invitation_data = RegisterService.get_invitation_with_case_fallback(None, request_email, invite_token)
+            invitation_data = RegisterService.get_invitation_with_case_fallback(
+                None, request_email, invite_token, session=db.session()
+            )
             if invitation_data is None:
                 invite_token = None
 
@@ -145,23 +153,26 @@ class LoginApi(Resource):
             _log_console_login_failure(email=normalized_email, reason=LoginFailureReason.INVALID_CREDENTIALS)
             raise AuthenticationFailedError() from exc
         # SELF_HOSTED only have one workspace
-        tenants = TenantService.get_join_tenants(account)
+        tenants = TenantService.get_join_tenants(account, session=db.session())
         if len(tenants) == 0:
             system_features = FeatureService.get_system_features()
 
             if system_features.is_allow_create_workspace and not system_features.license.workspaces.is_available():
                 raise WorkspacesLimitExceeded()
             else:
-                return {
-                    "result": "fail",
-                    "data": "workspace not found, please contact system admin to invite you to join in a workspace",
-                }
+                return SimpleResultOptionalDataResponse(
+                    result="fail",
+                    data="workspace not found, please contact system admin to invite you to join in a workspace",
+                ).model_dump(mode="json")
 
-        token_pair = AccountService.login(account=account, ip_address=extract_remote_ip(request))
+        token_pair = AccountService.login(account=account, session=db.session(), ip_address=extract_remote_ip(request))
         AccountService.reset_login_error_rate_limit(normalized_email)
 
         # Create response with cookies instead of returning tokens in body
-        response = make_response({"result": "success"})
+        # response-contract:ignore cookie-bearing Flask response
+        response = make_response(
+            SimpleResultOptionalDataResponse(result="success").model_dump(mode="json", exclude_none=True)
+        )
 
         set_access_token_to_cookie(request, response, token_pair.access_token)
         set_refresh_token_to_cookie(request, response, token_pair.refresh_token)
@@ -176,12 +187,11 @@ class LogoutApi(Resource):
     @console_ns.response(200, "Success", console_ns.models[SimpleResultResponse.__name__])
     @with_current_user
     def post(self, account: Account):
-        if isinstance(account, flask_login.AnonymousUserMixin):
-            response = make_response({"result": "success"})
-        else:
+        # response-contract:ignore cookie-bearing Flask response
+        response = make_response(SimpleResultResponse(result="success").model_dump(mode="json"))
+        if not isinstance(account, flask_login.AnonymousUserMixin):
             AccountService.logout(account=account)
             flask_login.logout_user()
-            response = make_response({"result": "success"})
 
         # Clear cookies on logout
         clear_access_token_from_cookie(response)
@@ -217,7 +227,7 @@ class ResetPasswordSendEmailApi(Resource):
             is_allow_register=FeatureService.get_system_features().is_allow_register,
         )
 
-        return {"result": "success", "data": token}
+        return SimpleResultDataResponse(result="success", data=token).model_dump(mode="json")
 
 
 @console_ns.route("/email-code-login")
@@ -250,7 +260,7 @@ class EmailCodeLoginSendEmailApi(Resource):
         else:
             token = AccountService.send_email_code_login_email(account=account, language=language)
 
-        return {"result": "success", "data": token}
+        return SimpleResultDataResponse(result="success", data=token).model_dump(mode="json")
 
 
 @console_ns.route("/email-code-login/validity")
@@ -291,7 +301,7 @@ class EmailCodeLoginApi(Resource):
             _log_console_login_failure(email=user_email, reason=LoginFailureReason.ACCOUNT_IN_FREEZE)
             raise AccountInFreezeError()
         if account:
-            tenants = TenantService.get_join_tenants(account)
+            tenants = TenantService.get_join_tenants(account, session=db.session())
             if not tenants:
                 workspaces = FeatureService.get_system_features().license.workspaces
                 if not workspaces.is_available():
@@ -299,8 +309,8 @@ class EmailCodeLoginApi(Resource):
                 if not FeatureService.get_system_features().is_allow_create_workspace:
                     raise NotAllowedCreateWorkspace()
                 else:
-                    new_tenant = TenantService.create_tenant(f"{account.name}'s Workspace")
-                    TenantService.create_tenant_member(new_tenant, account, db.session, role="owner")
+                    new_tenant = TenantService.create_tenant(f"{account.name}'s Workspace", session=db.session())
+                    TenantService.create_tenant_member(new_tenant, account, db.session(), role="owner")
                     account.current_tenant = new_tenant
                     tenant_was_created.send(new_tenant)
 
@@ -311,6 +321,7 @@ class EmailCodeLoginApi(Resource):
                     name=user_email,
                     interface_language=get_valid_language(language),
                     timezone=args.timezone,
+                    session=db.session(),
                 )
             except WorkSpaceNotAllowedCreateError:
                 raise NotAllowedCreateWorkspace()
@@ -319,11 +330,12 @@ class EmailCodeLoginApi(Resource):
                 raise AccountInFreezeError()
             except WorkspacesLimitExceededError:
                 raise WorkspacesLimitExceeded()
-        token_pair = AccountService.login(account, ip_address=extract_remote_ip(request))
+        token_pair = AccountService.login(account, session=db.session(), ip_address=extract_remote_ip(request))
         AccountService.reset_login_error_rate_limit(user_email)
 
         # Create response with cookies instead of returning tokens in body
-        response = make_response({"result": "success"})
+        # response-contract:ignore cookie-bearing Flask response
+        response = make_response(SimpleResultResponse(result="success").model_dump(mode="json"))
 
         set_csrf_token_to_cookie(request, response, token_pair.csrf_token)
         # Set HTTP-only secure cookies for tokens
@@ -335,45 +347,53 @@ class EmailCodeLoginApi(Resource):
 @console_ns.route("/refresh-token")
 class RefreshTokenApi(Resource):
     @console_ns.response(200, "Success", console_ns.models[SimpleResultResponse.__name__])
+    @console_ns.response(401, "Unauthorized", console_ns.models[SimpleResultMessageResponse.__name__])
     def post(self):
         # Get refresh token from cookie instead of request body
         refresh_token = extract_refresh_token(request)
 
         if not refresh_token:
-            return {"result": "fail", "message": "No refresh token provided"}, 401
+            return SimpleResultMessageResponse(result="fail", message="No refresh token provided").model_dump(
+                mode="json"
+            ), 401
 
         try:
-            new_token_pair = AccountService.refresh_token(refresh_token)
+            new_token_pair = AccountService.refresh_token(refresh_token, session=db.session())
+        except Unauthorized as exc:
+            return SimpleResultMessageResponse(result="fail", message=exc.description or "Unauthorized.").model_dump(
+                mode="json"
+            ), 401
+        except (RefreshTokenNotFoundError, RefreshTokenAccountNotFoundError) as exc:
+            return SimpleResultMessageResponse(result="fail", message=str(exc)).model_dump(mode="json"), 401
 
-            # Create response with new cookies
-            response = make_response({"result": "success"})
+        # Create response with new cookies
+        # response-contract:ignore cookie-bearing Flask response
+        response = make_response(SimpleResultResponse(result="success").model_dump(mode="json"))
 
-            # Update cookies with new tokens
-            set_csrf_token_to_cookie(request, response, new_token_pair.csrf_token)
-            set_access_token_to_cookie(request, response, new_token_pair.access_token)
-            set_refresh_token_to_cookie(request, response, new_token_pair.refresh_token)
-            return response
-        except Exception as e:
-            return {"result": "fail", "message": str(e)}, 401
+        # Update cookies with new tokens
+        set_csrf_token_to_cookie(request, response, new_token_pair.csrf_token)
+        set_access_token_to_cookie(request, response, new_token_pair.access_token)
+        set_refresh_token_to_cookie(request, response, new_token_pair.refresh_token)
+        return response
 
 
 def _get_account_with_case_fallback(email: str):
-    account = AccountService.get_user_through_email(email)
+    account = AccountService.get_user_through_email(email, session=db.session())
     if account or email == email.lower():
         return account
 
-    return AccountService.get_user_through_email(email.lower())
+    return AccountService.get_user_through_email(email.lower(), session=db.session())
 
 
 def _authenticate_account_with_case_fallback(
     original_email: str, normalized_email: str, password: str, invite_token: str | None
 ):
     try:
-        return AccountService.authenticate(original_email, password, invite_token)
+        return AccountService.authenticate(original_email, password, invite_token, session=db.session())
     except services.errors.account.AccountPasswordError:
         if original_email == normalized_email:
             raise
-        return AccountService.authenticate(normalized_email, password, invite_token)
+        return AccountService.authenticate(normalized_email, password, invite_token, session=db.session())
 
 
 def _log_console_login_failure(*, email: str, reason: LoginFailureReason) -> None:

api/controllers/console/auth/oauth.py

@@ -195,7 +195,7 @@ class OAuthCallback(Resource):
             db.session.commit()
 
         try:
-            TenantService.create_owner_tenant_if_not_exist(account)
+            TenantService.create_owner_tenant_if_not_exist(account, session=db.session())
         except Unauthorized:
             return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?message=Workspace not found.")
         except WorkSpaceNotAllowedCreateError:
@@ -206,6 +206,7 @@ class OAuthCallback(Resource):
 
         token_pair = AccountService.login(
             account=account,
+            session=db.session(),
             ip_address=extract_remote_ip(request),
         )
 
@@ -224,7 +225,7 @@ def _get_account_by_openid_or_email(provider: str, user_info: OAuthUserInfo) ->
     account: Account | None = Account.get_by_openid(provider, user_info.id)
 
     if not account:
-        account = AccountService.get_account_by_email_with_case_fallback(user_info.email)
+        account = AccountService.get_account_by_email_with_case_fallback(db.session(), user_info.email)
 
     return account
 
@@ -240,13 +241,13 @@ def _generate_account(
     oauth_new_user = False
 
     if account:
-        tenants = TenantService.get_join_tenants(account)
+        tenants = TenantService.get_join_tenants(account, session=db.session())
         if not tenants:
             if not FeatureService.get_system_features().is_allow_create_workspace:
                 raise WorkSpaceNotAllowedCreateError()
             else:
-                new_tenant = TenantService.create_tenant(f"{account.name}'s Workspace")
-                TenantService.create_tenant_member(new_tenant, account, db.session, role="owner")
+                new_tenant = TenantService.create_tenant(f"{account.name}'s Workspace", session=db.session())
+                TenantService.create_tenant_member(new_tenant, account, db.session(), role="owner")
                 account.current_tenant = new_tenant
                 tenant_was_created.send(new_tenant)
 
@@ -272,9 +273,10 @@ def _generate_account(
             provider=provider,
             language=interface_language,
             timezone=timezone,
+            session=db.session(),
         )
 
     # Link account
-    AccountService.link_account_integrate(provider, user_info.id, account)
+    AccountService.link_account_integrate(provider, user_info.id, account, session=db.session())
 
     return account, oauth_new_user

api/controllers/console/auth/oauth_server.py

@@ -10,6 +10,7 @@ from werkzeug.exceptions import BadRequest, NotFound
 
 from controllers.common.schema import register_response_schema_models, register_schema_models
 from controllers.console.wraps import account_initialization_required, setup_required, with_current_user
+from extensions.ext_database import db
 from graphon.model_runtime.utils.encoders import jsonable_encoder
 from libs.login import login_required
 from models import Account
@@ -131,7 +132,9 @@ def oauth_server_access_token_required[T, **P, R](
             response.headers["WWW-Authenticate"] = "Bearer"
             return response
 
-        account = OAuthServerService.validate_oauth_access_token(oauth_provider_app.client_id, access_token)
+        account = OAuthServerService.validate_oauth_access_token(
+            oauth_provider_app.client_id, access_token, db.session()
+        )
         if not account:
             response = jsonify({"error": "access_token or client_id is invalid"})
             response.status_code = 401

api/controllers/console/billing/billing.py

@@ -16,6 +16,8 @@ from controllers.console.wraps import (
     with_current_user,
 )
 from enums.cloud_plan import CloudPlan
+from extensions.ext_database import db
+from fields.base import ResponseModel
 from libs.login import login_required
 from models import Account
 from services.billing_service import BillingService
@@ -34,8 +36,12 @@ class BillingResponse(RootModel[dict[str, Any]]):
     root: dict[str, Any]
 
 
+class BillingInvoiceResponse(ResponseModel):
+    url: str
+
+
 register_schema_models(console_ns, SubscriptionQuery, PartnerTenantsPayload)
-register_response_schema_models(console_ns, BillingResponse)
+register_response_schema_models(console_ns, BillingResponse, BillingInvoiceResponse)
 
 
 @console_ns.route("/billing/subscription")
@@ -50,13 +56,13 @@ class Subscription(Resource):
     @with_current_tenant_id
     def get(self, current_tenant_id: str, current_user: Account):
         args = SubscriptionQuery.model_validate(request.args.to_dict(flat=True))
-        BillingService.is_tenant_owner_or_admin(current_user)
+        BillingService.is_tenant_owner_or_admin(db.session(), current_user)
         return BillingService.get_subscription(args.plan, args.interval, current_user.email, current_tenant_id)
 
 
 @console_ns.route("/billing/invoices")
 class Invoices(Resource):
-    @console_ns.response(200, "Success", console_ns.models[BillingResponse.__name__])
+    @console_ns.response(200, "Success", console_ns.models[BillingInvoiceResponse.__name__])
     @setup_required
     @login_required
     @account_initialization_required
@@ -64,7 +70,7 @@ class Invoices(Resource):
     @with_current_user
     @with_current_tenant_id
     def get(self, current_tenant_id: str, current_user: Account):
-        BillingService.is_tenant_owner_or_admin(current_user)
+        BillingService.is_tenant_owner_or_admin(db.session(), current_user)
         return BillingService.get_invoices(current_user.email, current_tenant_id)
 
 

api/controllers/console/datasets/data_source.py

@@ -243,72 +243,71 @@ class DataSourceNotionListApi(Resource):
         if not credential:
             raise NotFound("Credential not found.")
         exist_page_ids = []
-        with sessionmaker(db.engine).begin() as session:
-            # import notion in the exist dataset
-            if query.dataset_id:
-                dataset = DatasetService.get_dataset(query.dataset_id)
-                if not dataset:
-                    raise NotFound("Dataset not found.")
-                if dataset.data_source_type != "notion_import":
-                    raise ValueError("Dataset is not notion type.")
+        # import notion in the exist dataset
+        if query.dataset_id:
+            dataset = DatasetService.get_dataset(query.dataset_id, db.session())
+            if not dataset:
+                raise NotFound("Dataset not found.")
+            if dataset.data_source_type != "notion_import":
+                raise ValueError("Dataset is not notion type.")
 
-                documents = session.scalars(
-                    select(Document).where(
-                        Document.dataset_id == query.dataset_id,
-                        Document.tenant_id == current_tenant_id,
-                        Document.data_source_type == "notion_import",
-                        Document.enabled.is_(True),
-                    )
-                ).all()
-                if documents:
-                    for document in documents:
-                        data_source_info = json.loads(document.data_source_info)
-                        exist_page_ids.append(data_source_info["notion_page_id"])
-            # get all authorized pages
-            from core.datasource.datasource_manager import DatasourceManager
-
-            datasource_runtime = DatasourceManager.get_datasource_runtime(
-                provider_id="langgenius/notion_datasource/notion_datasource",
-                datasource_name="notion_datasource",
-                tenant_id=current_tenant_id,
-                datasource_type=DatasourceProviderType.ONLINE_DOCUMENT,
-            )
-            datasource_provider_service = DatasourceProviderService()
-            if credential:
-                datasource_runtime.runtime.credentials = credential
-            datasource_runtime = cast(OnlineDocumentDatasourcePlugin, datasource_runtime)
-            online_document_result: Generator[OnlineDocumentPagesMessage, None, None] = (
-                datasource_runtime.get_online_document_pages(
-                    user_id=current_user.id,
-                    datasource_parameters={},
-                    provider_type=datasource_runtime.datasource_provider_type(),
+            documents = db.session.scalars(
+                select(Document).where(
+                    Document.dataset_id == query.dataset_id,
+                    Document.tenant_id == current_tenant_id,
+                    Document.data_source_type == "notion_import",
+                    Document.enabled.is_(True),
                 )
+            ).all()
+            if documents:
+                for document in documents:
+                    data_source_info = json.loads(document.data_source_info)
+                    exist_page_ids.append(data_source_info["notion_page_id"])
+        # get all authorized pages
+        from core.datasource.datasource_manager import DatasourceManager
+
+        datasource_runtime = DatasourceManager.get_datasource_runtime(
+            provider_id="langgenius/notion_datasource/notion_datasource",
+            datasource_name="notion_datasource",
+            tenant_id=current_tenant_id,
+            datasource_type=DatasourceProviderType.ONLINE_DOCUMENT,
+        )
+        datasource_provider_service = DatasourceProviderService()
+        if credential:
+            datasource_runtime.runtime.credentials = credential
+        datasource_runtime = cast(OnlineDocumentDatasourcePlugin, datasource_runtime)
+        online_document_result: Generator[OnlineDocumentPagesMessage, None, None] = (
+            datasource_runtime.get_online_document_pages(
+                user_id=current_user.id,
+                datasource_parameters={},
+                provider_type=datasource_runtime.datasource_provider_type(),
             )
-            try:
-                pages = []
-                workspace_info = {}
-                for message in online_document_result:
-                    result = message.result
-                    for info in result:
-                        workspace_info = {
-                            "workspace_id": info.workspace_id,
-                            "workspace_name": info.workspace_name,
-                            "workspace_icon": info.workspace_icon,
+        )
+        try:
+            pages = []
+            workspace_info = {}
+            for message in online_document_result:
+                result = message.result
+                for info in result:
+                    workspace_info = {
+                        "workspace_id": info.workspace_id,
+                        "workspace_name": info.workspace_name,
+                        "workspace_icon": info.workspace_icon,
+                    }
+                    for page in info.pages:
+                        page_info = {
+                            "page_id": page.page_id,
+                            "page_name": page.page_name,
+                            "type": page.type,
+                            "parent_id": page.parent_id,
+                            "is_bound": page.page_id in exist_page_ids,
+                            "page_icon": page.page_icon,
                         }
-                        for page in info.pages:
-                            page_info = {
-                                "page_id": page.page_id,
-                                "page_name": page.page_name,
-                                "type": page.type,
-                                "parent_id": page.parent_id,
-                                "is_bound": page.page_id in exist_page_ids,
-                                "page_icon": page.page_icon,
-                            }
-                            pages.append(page_info)
-            except Exception as e:
-                raise e
-            notion_info = [{**workspace_info, "pages": pages}] if workspace_info else []
-            return dump_response(NotionIntegrateInfoListResponse, {"notion_info": notion_info}), 200
+                        pages.append(page_info)
+        except Exception as e:
+            raise e
+        notion_info = [{**workspace_info, "pages": pages}] if workspace_info else []
+        return dump_response(NotionIntegrateInfoListResponse, {"notion_info": notion_info}), 200
 
 
 @console_ns.route("/notion/pages/<uuid:page_id>/<string:page_type>/preview")
@@ -401,11 +400,11 @@ class DataSourceNotionDatasetSyncApi(Resource):
     @rbac_permission_required(RBACResourceScope.DATASET, RBACPermission.DATASET_CREATE_AND_MANAGEMENT)
     def get(self, dataset_id: UUID) -> tuple[dict[str, str], int]:
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
 
-        documents = DocumentService.get_document_by_dataset_id(dataset_id_str)
+        documents = DocumentService.get_document_by_dataset_id(dataset_id_str, db.session())
         for document in documents:
             document_indexing_sync_task.delay(dataset_id_str, document.id)
         return {"result": "success"}, 200
@@ -421,11 +420,11 @@ class DataSourceNotionDocumentSyncApi(Resource):
     def get(self, dataset_id: UUID, document_id: UUID) -> tuple[dict[str, str], int]:
         dataset_id_str = str(dataset_id)
         document_id_str = str(document_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
 
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if document is None:
             raise NotFound("Document not found.")
         document_indexing_sync_task.delay(dataset_id_str, document_id_str)

api/controllers/console/datasets/datasets.py

@@ -6,6 +6,7 @@ from flask import request
 from flask_restx import Resource
 from pydantic import BaseModel, Field, field_validator, model_validator
 from sqlalchemy import func, select
+from sqlalchemy.orm import Session
 from werkzeug.exceptions import Forbidden, NotFound
 
 import services
@@ -15,6 +16,7 @@ from controllers.common.schema import query_params_from_model, register_response
 from controllers.console import console_ns
 from controllers.console.apikey import ApiKeyItem, ApiKeyList
 from controllers.console.app.error import ProviderNotInitializeError
+from controllers.console.app.wraps import with_session
 from controllers.console.datasets.error import DatasetInUseError, DatasetNameDuplicateError, IndexingEstimateError
 from controllers.console.wraps import (
     RBACPermission,
@@ -416,6 +418,7 @@ class DatasetListApi(Resource):
         permissions = enterprise_rbac_service.RBACService.MyPermissions.get(
             str(current_tenant_id),
             current_user.id,
+            session=db.session(),
         )
 
         accessible_dataset_ids: list[str] | None = None
@@ -459,7 +462,7 @@ class DatasetListApi(Resource):
             datasets, total = DatasetService.get_datasets(
                 query.page,
                 query.limit,
-                db.session,
+                db.session(),
                 current_tenant_id,
                 current_user,
                 query.keyword,
@@ -538,7 +541,8 @@ class DatasetListApi(Resource):
     @cloud_edition_billing_rate_limit_check("knowledge")
     @with_current_user
     @with_current_tenant_id
-    def post(self, current_tenant_id: str, current_user: Account):
+    @with_session
+    def post(self, session: Session, current_tenant_id: str, current_user: Account):
         payload = DatasetCreatePayload.model_validate(console_ns.payload or {})
 
         # The role of the current user in the ta table must be admin, owner, or editor, or dataset_operator
@@ -552,6 +556,7 @@ class DatasetListApi(Resource):
 
         try:
             dataset = DatasetService.create_empty_dataset(
+                session=session,
                 tenant_id=current_tenant_id,
                 name=payload.name,
                 description=payload.description,
@@ -566,15 +571,16 @@ class DatasetListApi(Resource):
             raise DatasetNameDuplicateError()
 
         permission_keys_map = enterprise_rbac_service.RBACService.DatasetPermissions.batch_get(
-            str(current_tenant_id),
+            current_tenant_id,
             current_user.id,
-            [str(dataset.id)],
+            [dataset.id],
+            session=session,
         )
 
         item = DatasetDetailWithPartialMembersResponse.model_validate(dataset, from_attributes=True).model_dump(
             mode="json"
         )
-        item["permission_keys"] = permission_keys_map.get(str(dataset.id), [])
+        item["permission_keys"] = permission_keys_map.get(dataset.id, [])
         return item, 201
 
 
@@ -598,17 +604,18 @@ class DatasetApi(Resource):
     @with_current_tenant_id
     def get(self, current_tenant_id: str, current_user: Account, dataset_id: UUID):
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
         permissions = enterprise_rbac_service.RBACService.MyPermissions.get(
-            str(current_tenant_id),
+            current_tenant_id,
             current_user.id,
             dataset_id=dataset_id_str,
+            session=db.session(),
         )
         permission_keys_map = permissions.dataset.permission_keys_by_resource_ids([dataset_id_str])
         data = dump_response(DatasetDetailResponse, dataset)
@@ -618,7 +625,7 @@ class DatasetApi(Resource):
                 provider_id = ModelProviderID(dataset.embedding_model_provider)
                 data["embedding_model_provider"] = str(provider_id)
         if data.get("permission") == "partial_members":
-            part_users_list = DatasetPermissionService.get_dataset_partial_member_list(dataset_id_str)
+            part_users_list = DatasetPermissionService.get_dataset_partial_member_list(dataset_id_str, db.session())
             data.update({"partial_member_list": part_users_list})
 
         # check embedding setting
@@ -659,9 +666,10 @@ class DatasetApi(Resource):
     @with_current_user
     @with_current_tenant_id
     @rbac_permission_required(RBACResourceScope.DATASET, RBACPermission.DATASET_EDIT)
-    def patch(self, current_tenant_id: str, current_user: Account, dataset_id: UUID):
+    @with_session
+    def patch(self, session: Session, current_tenant_id: str, current_user: Account, dataset_id: UUID):
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
 
@@ -680,30 +688,33 @@ class DatasetApi(Resource):
         # The role of the current user in the ta table must be admin, owner, editor, or dataset_operator
         if not dify_config.RBAC_ENABLED:
             DatasetPermissionService.check_permission(
-                current_user, dataset, payload.permission, payload.partial_member_list
+                session, current_user, dataset, payload.permission, payload.partial_member_list
             )
 
-        dataset = DatasetService.update_dataset(dataset_id_str, payload_data, current_user)
+        dataset = DatasetService.update_dataset(session, dataset_id_str, payload_data, current_user)
 
         if dataset is None:
             raise NotFound("Dataset not found.")
 
         permission_keys_map = enterprise_rbac_service.RBACService.DatasetPermissions.batch_get(
-            str(current_tenant_id),
+            current_tenant_id,
             current_user.id,
             [dataset_id_str],
+            session=session,
         )
         result_data = dump_response(DatasetDetailResponse, dataset)
         result_data["permission_keys"] = permission_keys_map.get(dataset_id_str, [])
         tenant_id = current_tenant_id
 
         if payload.partial_member_list is not None and payload.permission == DatasetPermissionEnum.PARTIAL_TEAM:
-            DatasetPermissionService.update_partial_member_list(tenant_id, dataset_id_str, payload.partial_member_list)
+            DatasetPermissionService.update_partial_member_list(
+                tenant_id, dataset_id_str, payload.partial_member_list, db.session()
+            )
         # clear partial member list when permission is only_me or all_team_members
         elif payload.permission in {DatasetPermissionEnum.ONLY_ME, DatasetPermissionEnum.ALL_TEAM}:
-            DatasetPermissionService.clear_partial_member_list(dataset_id_str)
+            DatasetPermissionService.clear_partial_member_list(dataset_id_str, db.session())
 
-        partial_member_list = DatasetPermissionService.get_dataset_partial_member_list(dataset_id_str)
+        partial_member_list = DatasetPermissionService.get_dataset_partial_member_list(dataset_id_str, db.session())
         result_data.update({"partial_member_list": partial_member_list})
 
         return result_data, 200
@@ -722,8 +733,8 @@ class DatasetApi(Resource):
             raise Forbidden()
 
         try:
-            if DatasetService.delete_dataset(dataset_id_str, current_user):
-                DatasetPermissionService.clear_partial_member_list(dataset_id_str)
+            if DatasetService.delete_dataset(dataset_id_str, current_user, db.session()):
+                DatasetPermissionService.clear_partial_member_list(dataset_id_str, db.session())
                 return "", 204
             else:
                 raise NotFound("Dataset not found.")
@@ -748,7 +759,7 @@ class DatasetUseCheckApi(Resource):
     def get(self, dataset_id: UUID):
         dataset_id_str = str(dataset_id)
 
-        dataset_is_using = DatasetService.dataset_use_check(dataset_id_str)
+        dataset_is_using = DatasetService.dataset_use_check(dataset_id_str, db.session())
         return {"is_using": dataset_is_using}, 200
 
 
@@ -769,12 +780,12 @@ class DatasetQueryApi(Resource):
     @rbac_permission_required(RBACResourceScope.DATASET, RBACPermission.DATASET_READONLY)
     def get(self, current_user: Account, dataset_id: UUID):
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
 
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
 
@@ -910,16 +921,16 @@ class DatasetRelatedAppListApi(Resource):
     @rbac_permission_required(RBACResourceScope.DATASET, RBACPermission.DATASET_READONLY)
     def get(self, current_user: Account, dataset_id: UUID):
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
 
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
 
-        app_dataset_joins = DatasetService.get_related_apps(dataset.id)
+        app_dataset_joins = DatasetService.get_related_apps(dataset.id, db.session())
 
         related_apps = []
         for app_dataset_join in app_dataset_joins:
@@ -1094,7 +1105,7 @@ class DatasetEnableApiApi(Resource):
     def post(self, dataset_id: UUID, status: str):
         dataset_id_str = str(dataset_id)
 
-        DatasetService.update_dataset_api_status(dataset_id_str, status == "enable")
+        DatasetService.update_dataset_api_status(dataset_id_str, status == "enable", db.session())
 
         return {"result": "success"}, 200
 
@@ -1163,10 +1174,10 @@ class DatasetErrorDocs(Resource):
     @rbac_permission_required(RBACResourceScope.DATASET, RBACPermission.DATASET_READONLY)
     def get(self, dataset_id: UUID):
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
-        results = DocumentService.get_error_documents_by_dataset_id(dataset_id_str)
+        results = DocumentService.get_error_documents_by_dataset_id(dataset_id_str, db.session())
 
         return dump_response(ErrorDocsResponse, {"data": results, "total": len(results)}), 200
 
@@ -1190,15 +1201,15 @@ class DatasetPermissionUserListApi(Resource):
     @rbac_permission_required(RBACResourceScope.DATASET, RBACPermission.DATASET_READONLY)
     def get(self, current_user: Account, dataset_id: UUID):
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
 
-        partial_members_list = DatasetPermissionService.get_dataset_partial_member_list(dataset_id_str)
+        partial_members_list = DatasetPermissionService.get_dataset_partial_member_list(dataset_id_str, db.session())
 
         return dump_response(PartialMemberListResponse, {"data": partial_members_list}), 200
 
@@ -1220,7 +1231,8 @@ class DatasetAutoDisableLogApi(Resource):
     @rbac_permission_required(RBACResourceScope.DATASET, RBACPermission.DATASET_READONLY)
     def get(self, dataset_id: UUID):
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
-        return dump_response(AutoDisableLogsResponse, DatasetService.get_dataset_auto_disable_logs(dataset_id_str)), 200
+        auto_disable_logs = DatasetService.get_dataset_auto_disable_logs(dataset_id_str, db.session())
+        return dump_response(AutoDisableLogsResponse, auto_disable_logs), 200

api/controllers/console/datasets/datasets_document.py

@@ -46,9 +46,11 @@ from graphon.model_runtime.errors.invoke import InvokeAuthorizationError
 from libs.datetime_utils import naive_utc_now
 from libs.helper import dump_response, to_timestamp
 from libs.login import login_required
+from libs.pagination import paginate_query
 from models import Account, DatasetProcessRule, Document, DocumentSegment, UploadFile
 from models.dataset import DocumentPipelineExecutionLog
 from models.enums import IndexingStatus, SegmentStatus
+from services.dataset_ref_service import DatasetRefService
 from services.dataset_service import DatasetService, DocumentService
 from services.entities.knowledge_entities.knowledge_entities import KnowledgeConfig, ProcessRule, RetrievalModel
 from services.file_service import FileService
@@ -181,16 +183,16 @@ class DocumentResource(Resource):
     def get_document(
         self, dataset_id: str, document_id: str, current_user: Account, current_tenant_id: str
     ) -> Document:
-        dataset = DatasetService.get_dataset(dataset_id)
+        dataset = DatasetService.get_dataset(dataset_id, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
 
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
 
-        document = DocumentService.get_document(dataset_id, document_id)
+        document = DocumentService.get_document(dataset_id, document_id, session=db.session())
 
         if not document:
             raise NotFound("Document not found.")
@@ -201,16 +203,16 @@ class DocumentResource(Resource):
         return document
 
     def get_batch_documents(self, dataset_id: str, batch: str, current_user: Account) -> Sequence[Document]:
-        dataset = DatasetService.get_dataset(dataset_id)
+        dataset = DatasetService.get_dataset(dataset_id, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
 
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
 
-        documents = DocumentService.get_batch_documents(dataset_id, batch)
+        documents = DocumentService.get_batch_documents(dataset_id, batch, db.session())
 
         if not documents:
             raise NotFound("Documents not found.")
@@ -241,13 +243,13 @@ class GetProcessRuleApi(Resource):
             # get the latest process rule
             document = db.get_or_404(Document, document_id)
 
-            dataset = DatasetService.get_dataset(document.dataset_id)
+            dataset = DatasetService.get_dataset(document.dataset_id, db.session())
 
             if not dataset:
                 raise NotFound("Dataset not found.")
 
             try:
-                DatasetService.check_dataset_permission(dataset, current_user)
+                DatasetService.check_dataset_permission(dataset, current_user, db.session())
             except services.errors.account.NoPermissionError as e:
                 raise Forbidden(str(e))
 
@@ -317,12 +319,12 @@ class DatasetDocumentListApi(Resource):
                     )
         except (ArgumentTypeError, ValueError, Exception):
             fetch = False
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
 
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
 
@@ -367,13 +369,14 @@ class DatasetDocumentListApi(Resource):
                     desc(Document.position),
                 )
 
-        paginated_documents = db.paginate(select=query, page=page, per_page=limit, max_per_page=100, error_out=False)
+        paginated_documents = paginate_query(query, page=page, per_page=limit, max_per_page=100)
         documents = paginated_documents.items
 
         DocumentService.enrich_documents_with_summary_index_status(
             documents=documents,
             dataset=dataset,
             tenant_id=current_tenant_id,
+            session=db.session(),
         )
 
         if fetch:
@@ -421,7 +424,7 @@ class DatasetDocumentListApi(Resource):
     def post(self, current_user: Account, dataset_id: UUID):
         dataset_id_str = str(dataset_id)
 
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
 
         if not dataset:
             raise NotFound("Dataset not found.")
@@ -431,7 +434,7 @@ class DatasetDocumentListApi(Resource):
             raise Forbidden()
 
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
 
@@ -444,8 +447,10 @@ class DatasetDocumentListApi(Resource):
         DocumentService.document_create_args_validate(knowledge_config)
 
         try:
-            documents, batch = DocumentService.save_document_with_dataset_id(dataset, knowledge_config, current_user)
-            dataset = DatasetService.get_dataset(dataset_id_str)
+            documents, batch = DocumentService.save_document_with_dataset_id(
+                dataset, knowledge_config, current_user, session=db.session()
+            )
+            dataset = DatasetService.get_dataset(dataset_id_str, db.session())
 
         except ProviderTokenNotInitError as ex:
             raise ProviderNotInitializeError(ex.description)
@@ -464,7 +469,7 @@ class DatasetDocumentListApi(Resource):
     @rbac_permission_required(RBACResourceScope.DATASET, RBACPermission.DATASET_EDIT)
     def delete(self, dataset_id: UUID):
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
         # check user's model setting
@@ -472,7 +477,8 @@ class DatasetDocumentListApi(Resource):
 
         try:
             document_ids = request.args.getlist("document_id")
-            DocumentService.delete_documents(dataset, document_ids)
+            dataset_ref = DatasetRefService.create_dataset_ref(dataset)
+            DocumentService.delete_documents(dataset_ref, document_ids, dataset.doc_form, db.session())
         except services.errors.document.DocumentIndexingError:
             raise DocumentIndexingError("Cannot delete document during indexing.")
 
@@ -531,6 +537,7 @@ class DatasetInitApi(Resource):
                 tenant_id=current_tenant_id,
                 knowledge_config=knowledge_config,
                 account=current_user,
+                session=db.session(),
             )
         except ProviderTokenNotInitError as ex:
             raise ProviderNotInitializeError(ex.description)
@@ -867,7 +874,7 @@ class DocumentApi(DocumentResource):
         if metadata == "only":
             response = {"id": document.id, "doc_type": document.doc_type, "doc_metadata": document.doc_metadata_details}
         elif metadata == "without":
-            dataset_process_rules = DatasetService.get_process_rules(dataset_id_str)
+            dataset_process_rules = DatasetService.get_process_rules(dataset_id_str, db.session())
             document_process_rules = document.dataset_process_rule.to_dict() if document.dataset_process_rule else {}
             response = {
                 "id": document.id,
@@ -901,7 +908,7 @@ class DocumentApi(DocumentResource):
                 "need_summary": document.need_summary if document.need_summary is not None else False,
             }
         else:
-            dataset_process_rules = DatasetService.get_process_rules(dataset_id_str)
+            dataset_process_rules = DatasetService.get_process_rules(dataset_id_str, db.session())
             document_process_rules = document.dataset_process_rule.to_dict() if document.dataset_process_rule else {}
             response = {
                 "id": document.id,
@@ -950,7 +957,7 @@ class DocumentApi(DocumentResource):
     def delete(self, current_tenant_id: str, current_user: Account, dataset_id: UUID, document_id: UUID):
         dataset_id_str = str(dataset_id)
         document_id_str = str(document_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
         # check user's model setting
@@ -959,7 +966,7 @@ class DocumentApi(DocumentResource):
         document = self.get_document(dataset_id_str, document_id_str, current_user, current_tenant_id)
 
         try:
-            DocumentService.delete_document(document)
+            DocumentService.delete_document(document, db.session())
         except services.errors.document.DocumentIndexingError:
             raise DocumentIndexingError("Cannot delete document during indexing.")
 
@@ -983,7 +990,7 @@ class DocumentDownloadApi(DocumentResource):
     def get(self, current_tenant_id: str, current_user: Account, dataset_id: UUID, document_id: UUID) -> dict[str, Any]:
         # Reuse the shared permission/tenant checks implemented in DocumentResource.
         document = self.get_document(str(dataset_id), str(document_id), current_user, current_tenant_id)
-        return {"url": DocumentService.get_document_download_url(document)}
+        return {"url": DocumentService.get_document_download_url(document, db.session())}
 
 
 @console_ns.route("/datasets/<uuid:dataset_id>/documents/download-zip")
@@ -1013,6 +1020,7 @@ class DocumentBatchDownloadZipApi(DocumentResource):
             document_ids=document_ids,
             tenant_id=current_tenant_id,
             current_user=current_user,
+            session=db.session(),
         )
 
         # Delegate ZIP packing to FileService, but keep Flask response+cleanup in the route.
@@ -1161,7 +1169,7 @@ class DocumentStatusApi(DocumentResource):
         self, current_user: Account, dataset_id: UUID, action: Literal["enable", "disable", "archive", "un_archive"]
     ):
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if dataset is None:
             raise NotFound("Dataset not found.")
 
@@ -1173,12 +1181,12 @@ class DocumentStatusApi(DocumentResource):
         DatasetService.check_dataset_model_setting(dataset)
 
         # check user's permission
-        DatasetService.check_dataset_permission(dataset, current_user)
+        DatasetService.check_dataset_permission(dataset, current_user, db.session())
 
         document_ids = request.args.getlist("document_id")
 
         try:
-            DocumentService.batch_update_document_status(dataset, document_ids, action, current_user)
+            DocumentService.batch_update_document_status(dataset, document_ids, action, current_user, db.session())
         except services.errors.document.DocumentIndexingError as e:
             raise InvalidActionError(str(e))
         except ValueError as e:
@@ -1202,11 +1210,11 @@ class DocumentPauseApi(DocumentResource):
         dataset_id_str = str(dataset_id)
         document_id_str = str(document_id)
 
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
 
-        document = DocumentService.get_document(dataset.id, document_id_str)
+        document = DocumentService.get_document(dataset.id, document_id_str, session=db.session())
 
         # 404 if document not found
         if document is None:
@@ -1218,7 +1226,7 @@ class DocumentPauseApi(DocumentResource):
 
         try:
             # pause document
-            DocumentService.pause_document(document)
+            DocumentService.pause_document(document, db.session())
         except services.errors.document.DocumentIndexingError:
             raise DocumentIndexingError("Cannot pause completed document.")
 
@@ -1237,10 +1245,10 @@ class DocumentRecoverApi(DocumentResource):
         """recover document."""
         dataset_id_str = str(dataset_id)
         document_id_str = str(document_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
-        document = DocumentService.get_document(dataset.id, document_id_str)
+        document = DocumentService.get_document(dataset.id, document_id_str, session=db.session())
 
         # 404 if document not found
         if document is None:
@@ -1251,7 +1259,7 @@ class DocumentRecoverApi(DocumentResource):
             raise ArchivedDocumentImmutableError()
         try:
             # pause document
-            DocumentService.recover_document(document)
+            DocumentService.recover_document(document, db.session())
         except services.errors.document.DocumentIndexingError:
             raise DocumentIndexingError("Document is not in paused status.")
 
@@ -1271,13 +1279,13 @@ class DocumentRetryApi(DocumentResource):
         """retry document."""
         payload = DocumentRetryPayload.model_validate(console_ns.payload or {})
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         retry_documents = []
         if not dataset:
             raise NotFound("Dataset not found.")
         for document_id in payload.document_ids:
             try:
-                document = DocumentService.get_document(dataset.id, document_id)
+                document = DocumentService.get_document(dataset.id, document_id, session=db.session())
 
                 # 404 if document not found
                 if document is None:
@@ -1295,7 +1303,7 @@ class DocumentRetryApi(DocumentResource):
                 logger.exception("Failed to retry document, document id: %s", document_id)
                 continue
         # retry document
-        DocumentService.retry_document(dataset_id_str, retry_documents)
+        DocumentService.retry_document(dataset_id_str, retry_documents, db.session())
 
         return "", 204
 
@@ -1313,14 +1321,14 @@ class DocumentRenameApi(DocumentResource):
         # The role of the current user in the ta table must be admin, owner, editor, or dataset_operator
         if not current_user.is_dataset_editor:
             raise Forbidden()
-        dataset = DatasetService.get_dataset(dataset_id)
+        dataset = DatasetService.get_dataset(dataset_id, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
-        DatasetService.check_dataset_operator_permission(current_user, dataset)
+        DatasetService.check_dataset_operator_permission(current_user, dataset, session=db.session())
         payload = DocumentRenamePayload.model_validate(console_ns.payload or {})
 
         try:
-            document = DocumentService.rename_document(str(dataset_id), str(document_id), payload.name)
+            document = DocumentService.rename_document(str(dataset_id), str(document_id), payload.name, db.session())
         except services.errors.document.DocumentIndexingError:
             raise DocumentIndexingError("Cannot delete document during indexing.")
 
@@ -1338,11 +1346,11 @@ class WebsiteDocumentSyncApi(DocumentResource):
     def get(self, current_tenant_id: str, dataset_id: UUID, document_id: UUID):
         """sync website document."""
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset.id, document_id_str)
+        document = DocumentService.get_document(dataset.id, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
         if document.tenant_id != current_tenant_id:
@@ -1353,7 +1361,7 @@ class WebsiteDocumentSyncApi(DocumentResource):
         if DocumentService.check_archived(document):
             raise ArchivedDocumentImmutableError()
         # sync document
-        DocumentService.sync_website_document(dataset_id_str, document)
+        DocumentService.sync_website_document(dataset_id_str, document, db.session())
 
         return {"result": "success"}, 200
 
@@ -1373,10 +1381,10 @@ class DocumentPipelineExecutionLogApi(DocumentResource):
         dataset_id_str = str(dataset_id)
         document_id_str = str(document_id)
 
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
-        document = DocumentService.get_document(dataset.id, document_id_str)
+        document = DocumentService.get_document(dataset.id, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
         log = db.session.scalar(
@@ -1431,7 +1439,7 @@ class DocumentGenerateSummaryApi(Resource):
         dataset_id_str = str(dataset_id)
 
         # Get dataset
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
 
@@ -1440,7 +1448,7 @@ class DocumentGenerateSummaryApi(Resource):
             raise Forbidden()
 
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
 
@@ -1465,7 +1473,7 @@ class DocumentGenerateSummaryApi(Resource):
             raise ValueError("Summary index is not enabled for this dataset. Please enable it in the dataset settings.")
 
         # Verify all documents exist and belong to the dataset
-        documents = DocumentService.get_documents_by_ids(dataset_id_str, document_list)
+        documents = DocumentService.get_documents_by_ids(dataset_id_str, document_list, db.session())
 
         if len(documents) != len(document_list):
             found_ids = {doc.id for doc in documents}
@@ -1481,6 +1489,7 @@ class DocumentGenerateSummaryApi(Resource):
             DocumentService.update_documents_need_summary(
                 dataset_id=dataset_id_str,
                 document_ids=document_ids_to_update,
+                session=db.session(),
                 need_summary=True,
             )
 
@@ -1531,13 +1540,13 @@ class DocumentSummaryStatusApi(DocumentResource):
         document_id_str = str(document_id)
 
         # Get dataset
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
 
         # Check permissions
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
 
@@ -1547,6 +1556,7 @@ class DocumentSummaryStatusApi(DocumentResource):
         result = SummaryIndexService.get_document_summary_status_detail(
             document_id=document_id_str,
             dataset_id=dataset_id_str,
+            session=db.session(),
         )
 
         return result, 200

api/controllers/console/datasets/datasets_segments.py

@@ -57,9 +57,11 @@ from fields.segment_fields import (
 from graphon.model_runtime.entities.model_entities import ModelType
 from libs.helper import dump_response, escape_like_pattern
 from libs.login import login_required
+from libs.pagination import paginate_query
 from models import Account
-from models.dataset import ChildChunk, DocumentSegment
+from models.dataset import Dataset, Document, DocumentSegment
 from models.model import UploadFile
+from services.dataset_ref_service import DatasetRefService, SegmentRef
 from services.dataset_service import DatasetService, DocumentService, SegmentService
 from services.entities.knowledge_entities.knowledge_entities import ChildChunkUpdateArgs, SegmentUpdateArgs
 from services.errors.chunk import ChildChunkDeleteIndexError as ChildChunkDeleteIndexServiceError
@@ -162,6 +164,21 @@ register_response_schema_models(
 )
 
 
+def _get_segment_for_document(
+    dataset: Dataset, document: Document, segment_id: str
+) -> tuple[SegmentRef, DocumentSegment]:
+    dataset_ref = DatasetRefService.create_dataset_ref(dataset)
+    document_ref = DatasetRefService.create_document_ref(dataset_ref, document)
+    if document_ref is None:
+        raise NotFound("Document not found.")
+
+    segment_ref = DatasetRefService.create_segment_ref(document_ref, segment_id)
+    segment = SegmentService.get_segment_by_ref(segment_ref, db.session())
+    if not segment:
+        raise NotFound("Segment not found.")
+    return segment_ref, segment
+
+
 @console_ns.route("/datasets/<uuid:dataset_id>/documents/<uuid:document_id>/segments")
 class DatasetDocumentSegmentListApi(Resource):
     @console_ns.doc(params=SegmentDocParams.DATASET_DOCUMENT)
@@ -176,16 +193,16 @@ class DatasetDocumentSegmentListApi(Resource):
     def get(self, current_tenant_id: str, current_user: Account, dataset_id: UUID, document_id: UUID):
         dataset_id_str = str(dataset_id)
         document_id_str = str(document_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
 
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
 
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
 
         if not document:
             raise NotFound("Document not found.")
@@ -254,14 +271,14 @@ class DatasetDocumentSegmentListApi(Resource):
             elif args.enabled.lower() == "false":
                 query = query.where(DocumentSegment.enabled == False)
 
-        segments = db.paginate(select=query, page=page, per_page=limit, max_per_page=100, error_out=False)
+        segments = paginate_query(query, page=page, per_page=limit, max_per_page=100)
 
         segment_list = list(segments.items)
         segment_ids = [segment.id for segment in segment_list]
         summaries: dict[str, str | None] = {}
         if segment_ids:
             summary_records = SummaryIndexService.get_segments_summaries(
-                segment_ids=segment_ids, dataset_id=dataset_id_str
+                segment_ids=segment_ids, dataset_id=dataset_id_str, session=db.session()
             )
             summaries = {chunk_id: summary.summary_content for chunk_id, summary in summary_records.items()}
 
@@ -286,14 +303,14 @@ class DatasetDocumentSegmentListApi(Resource):
     def delete(self, current_user: Account, dataset_id: UUID, document_id: UUID):
         # check dataset
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         # check user's model setting
         DatasetService.check_dataset_model_setting(dataset)
         # check document
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
         segment_ids = request.args.getlist("segment_id")
@@ -302,10 +319,10 @@ class DatasetDocumentSegmentListApi(Resource):
         if not current_user.is_dataset_editor:
             raise Forbidden()
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
-        SegmentService.delete_segments(segment_ids, document, dataset)
+        SegmentService.delete_segments(segment_ids, document, dataset, db.session())
         return "", 204
 
 
@@ -331,11 +348,11 @@ class DatasetDocumentSegmentApi(Resource):
         action: Literal["enable", "disable"],
     ):
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
         # check user's model setting
@@ -345,7 +362,7 @@ class DatasetDocumentSegmentApi(Resource):
             raise Forbidden()
 
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
         if dataset.indexing_technique == IndexTechniqueType.HIGH_QUALITY:
@@ -371,7 +388,7 @@ class DatasetDocumentSegmentApi(Resource):
         if cache_result is not None:
             raise InvalidActionError("Document is being indexed, please try again later")
         try:
-            SegmentService.update_segments_status(segment_ids, action, dataset, document)
+            SegmentService.update_segments_status(segment_ids, action, dataset, document, db.session())
         except Exception as e:
             raise InvalidActionError(str(e))
         return dump_response(SimpleResultResponse, {"result": "success"}), 200
@@ -394,12 +411,12 @@ class DatasetDocumentSegmentAddApi(Resource):
     def post(self, current_tenant_id: str, current_user: Account, dataset_id: UUID, document_id: UUID):
         # check dataset
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         # check document
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
         if not current_user.is_dataset_editor:
@@ -421,15 +438,20 @@ class DatasetDocumentSegmentAddApi(Resource):
             except ProviderTokenNotInitError as ex:
                 raise ProviderNotInitializeError(ex.description)
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
         # validate args
         payload = SegmentCreatePayload.model_validate(console_ns.payload or {})
         payload_dict = payload.model_dump(exclude_none=True)
         SegmentService.segment_create_args_validate(payload_dict, document)
-        segment = type_cast(DocumentSegment, SegmentService.create_segment(payload_dict, document, dataset))
-        summary = SummaryIndexService.get_segment_summary(segment_id=segment.id, dataset_id=dataset_id_str)
+        segment = type_cast(
+            DocumentSegment,
+            SegmentService.create_segment(payload_dict, document, dataset, db.session()),
+        )
+        summary = SummaryIndexService.get_segment_summary(
+            segment_id=segment.id, dataset_id=dataset_id_str, session=db.session()
+        )
         response = {
             "data": segment_response_with_summary(segment, summary.summary_content if summary else None),
             "doc_form": document.doc_form,
@@ -455,16 +477,23 @@ class DatasetDocumentSegmentUpdateApi(Resource):
     ):
         # check dataset
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         # check user's model setting
         DatasetService.check_dataset_model_setting(dataset)
         # check document
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
+        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
+        if not current_user.is_dataset_editor:
+            raise Forbidden()
+        try:
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
+        except services.errors.account.NoPermissionError as e:
+            raise Forbidden(str(e))
         if dataset.indexing_technique == IndexTechniqueType.HIGH_QUALITY:
             # check embedding model setting
             try:
@@ -481,22 +510,8 @@ class DatasetDocumentSegmentUpdateApi(Resource):
                 )
             except ProviderTokenNotInitError as ex:
                 raise ProviderNotInitializeError(ex.description)
-            # check segment
         segment_id_str = str(segment_id)
-        segment = db.session.scalar(
-            select(DocumentSegment)
-            .where(DocumentSegment.id == segment_id_str, DocumentSegment.tenant_id == current_tenant_id)
-            .limit(1)
-        )
-        if not segment:
-            raise NotFound("Segment not found.")
-        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
-        if not current_user.is_dataset_editor:
-            raise Forbidden()
-        try:
-            DatasetService.check_dataset_permission(dataset, current_user)
-        except services.errors.account.NoPermissionError as e:
-            raise Forbidden(str(e))
+        _, segment = _get_segment_for_document(dataset, document, segment_id_str)
         # validate args
         payload = SegmentUpdatePayload.model_validate(console_ns.payload or {})
         payload_dict = payload.model_dump(exclude_none=True)
@@ -504,9 +519,15 @@ class DatasetDocumentSegmentUpdateApi(Resource):
 
         # Update segment (summary update with change detection is handled in SegmentService.update_segment)
         segment = SegmentService.update_segment(
-            SegmentUpdateArgs.model_validate(payload.model_dump(exclude_none=True)), segment, document, dataset
+            SegmentUpdateArgs.model_validate(payload.model_dump(exclude_none=True)),
+            segment,
+            document,
+            dataset,
+            db.session(),
+        )
+        summary = SummaryIndexService.get_segment_summary(
+            segment_id=segment.id, dataset_id=dataset_id_str, session=db.session()
         )
-        summary = SummaryIndexService.get_segment_summary(segment_id=segment.id, dataset_id=dataset_id_str)
         response = {
             "data": segment_response_with_summary(segment, summary.summary_content if summary else None),
             "doc_form": document.doc_form,
@@ -527,33 +548,26 @@ class DatasetDocumentSegmentUpdateApi(Resource):
     ):
         # check dataset
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         # check user's model setting
         DatasetService.check_dataset_model_setting(dataset)
         # check document
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
-        # check segment
-        segment_id_str = str(segment_id)
-        segment = db.session.scalar(
-            select(DocumentSegment)
-            .where(DocumentSegment.id == segment_id_str, DocumentSegment.tenant_id == current_tenant_id)
-            .limit(1)
-        )
-        if not segment:
-            raise NotFound("Segment not found.")
         # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
         if not current_user.is_dataset_editor:
             raise Forbidden()
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
-        SegmentService.delete_segment(segment, document, dataset)
+        segment_id_str = str(segment_id)
+        _, segment = _get_segment_for_document(dataset, document, segment_id_str)
+        SegmentService.delete_segment(segment, document, dataset, db.session())
         return "", 204
 
 
@@ -576,12 +590,12 @@ class DatasetDocumentSegmentBatchImportApi(Resource):
     def post(self, current_tenant_id: str, current_user: Account, dataset_id: UUID, document_id: UUID):
         # check dataset
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         # check document
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
 
@@ -651,25 +665,20 @@ class ChildChunkAddApi(Resource):
     ):
         # check dataset
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         # check document
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
-        # check segment
-        segment_id_str = str(segment_id)
-        segment = db.session.scalar(
-            select(DocumentSegment)
-            .where(DocumentSegment.id == segment_id_str, DocumentSegment.tenant_id == current_tenant_id)
-            .limit(1)
-        )
-        if not segment:
-            raise NotFound("Segment not found.")
         if not current_user.is_dataset_editor:
             raise Forbidden()
+        try:
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
+        except services.errors.account.NoPermissionError as e:
+            raise Forbidden(str(e))
         # check embedding model setting
         if dataset.indexing_technique == IndexTechniqueType.HIGH_QUALITY:
             try:
@@ -686,14 +695,12 @@ class ChildChunkAddApi(Resource):
                 )
             except ProviderTokenNotInitError as ex:
                 raise ProviderNotInitializeError(ex.description)
-        try:
-            DatasetService.check_dataset_permission(dataset, current_user)
-        except services.errors.account.NoPermissionError as e:
-            raise Forbidden(str(e))
+        segment_id_str = str(segment_id)
+        _, segment = _get_segment_for_document(dataset, document, segment_id_str)
         # validate args
         try:
             payload = ChildChunkCreatePayload.model_validate(console_ns.payload or {})
-            child_chunk = SegmentService.create_child_chunk(payload.content, segment, document, dataset)
+            child_chunk = SegmentService.create_child_chunk(payload.content, segment, document, dataset, db.session())
         except ChildChunkIndexingServiceError as e:
             raise ChildChunkIndexingError(str(e))
         return dump_response(ChildChunkDetailResponse, {"data": child_chunk}), 200
@@ -709,25 +716,18 @@ class ChildChunkAddApi(Resource):
     def get(self, current_tenant_id: str, dataset_id: UUID, document_id: UUID, segment_id: UUID):
         # check dataset
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         # check user's model setting
         DatasetService.check_dataset_model_setting(dataset)
         # check document
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
-        # check segment
         segment_id_str = str(segment_id)
-        segment = db.session.scalar(
-            select(DocumentSegment)
-            .where(DocumentSegment.id == segment_id_str, DocumentSegment.tenant_id == current_tenant_id)
-            .limit(1)
-        )
-        if not segment:
-            raise NotFound("Segment not found.")
+        _get_segment_for_document(dataset, document, segment_id_str)
         args = query_params_from_request(ChildChunkListQuery, use_defaults_for_malformed_ints=True)
 
         page = args.page
@@ -766,36 +766,29 @@ class ChildChunkAddApi(Resource):
     ):
         # check dataset
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         # check user's model setting
         DatasetService.check_dataset_model_setting(dataset)
         # check document
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
-            # check segment
-        segment_id_str = str(segment_id)
-        segment = db.session.scalar(
-            select(DocumentSegment)
-            .where(DocumentSegment.id == segment_id_str, DocumentSegment.tenant_id == current_tenant_id)
-            .limit(1)
-        )
-        if not segment:
-            raise NotFound("Segment not found.")
         # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
         if not current_user.is_dataset_editor:
             raise Forbidden()
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
+        segment_id_str = str(segment_id)
+        _, segment = _get_segment_for_document(dataset, document, segment_id_str)
         # validate args
         payload = ChildChunkBatchUpdatePayload.model_validate(console_ns.payload or {})
         try:
-            child_chunks = SegmentService.update_child_chunks(payload.chunks, segment, document, dataset)
+            child_chunks = SegmentService.update_child_chunks(payload.chunks, segment, document, dataset, db.session())
         except ChildChunkIndexingServiceError as e:
             raise ChildChunkIndexingError(str(e))
         return dump_response(ChildChunkBatchUpdateResponse, {"data": child_chunks}), 200
@@ -825,48 +818,31 @@ class ChildChunkUpdateApi(Resource):
     ):
         # check dataset
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         # check user's model setting
         DatasetService.check_dataset_model_setting(dataset)
         # check document
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
-        # check segment
-        segment_id_str = str(segment_id)
-        segment = db.session.scalar(
-            select(DocumentSegment)
-            .where(DocumentSegment.id == segment_id_str, DocumentSegment.tenant_id == current_tenant_id)
-            .limit(1)
-        )
-        if not segment:
-            raise NotFound("Segment not found.")
-        # check child chunk
-        child_chunk_id_str = str(child_chunk_id)
-        child_chunk = db.session.scalar(
-            select(ChildChunk)
-            .where(
-                ChildChunk.id == child_chunk_id_str,
-                ChildChunk.tenant_id == current_tenant_id,
-                ChildChunk.segment_id == segment.id,
-                ChildChunk.document_id == document_id_str,
-            )
-            .limit(1)
-        )
-        if not child_chunk:
-            raise NotFound("Child chunk not found.")
         # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
         if not current_user.is_dataset_editor:
             raise Forbidden()
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
+        segment_id_str = str(segment_id)
+        segment_ref, _ = _get_segment_for_document(dataset, document, segment_id_str)
+        child_chunk_id_str = str(child_chunk_id)
+        child_chunk = SegmentService.get_child_chunk_by_segment_ref(child_chunk_id_str, segment_ref, db.session())
+        if not child_chunk:
+            raise NotFound("Child chunk not found.")
         try:
-            SegmentService.delete_child_chunk(child_chunk, dataset)
+            SegmentService.delete_child_chunk(child_chunk, dataset, db.session())
         except ChildChunkDeleteIndexServiceError as e:
             raise ChildChunkDeleteIndexError(str(e))
         return "", 204
@@ -893,50 +869,35 @@ class ChildChunkUpdateApi(Resource):
     ):
         # check dataset
         dataset_id_str = str(dataset_id)
-        dataset = DatasetService.get_dataset(dataset_id_str)
+        dataset = DatasetService.get_dataset(dataset_id_str, db.session())
         if not dataset:
             raise NotFound("Dataset not found.")
         # check user's model setting
         DatasetService.check_dataset_model_setting(dataset)
         # check document
         document_id_str = str(document_id)
-        document = DocumentService.get_document(dataset_id_str, document_id_str)
+        document = DocumentService.get_document(dataset_id_str, document_id_str, session=db.session())
         if not document:
             raise NotFound("Document not found.")
-            # check segment
-        segment_id_str = str(segment_id)
-        segment = db.session.scalar(
-            select(DocumentSegment)
-            .where(DocumentSegment.id == segment_id_str, DocumentSegment.tenant_id == current_tenant_id)
-            .limit(1)
-        )
-        if not segment:
-            raise NotFound("Segment not found.")
-        # check child chunk
-        child_chunk_id_str = str(child_chunk_id)
-        child_chunk = db.session.scalar(
-            select(ChildChunk)
-            .where(
-                ChildChunk.id == child_chunk_id_str,
-                ChildChunk.tenant_id == current_tenant_id,
-                ChildChunk.segment_id == segment.id,
-                ChildChunk.document_id == document_id_str,
-            )
-            .limit(1)
-        )
-        if not child_chunk:
-            raise NotFound("Child chunk not found.")
         # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
         if not current_user.is_dataset_editor:
             raise Forbidden()
         try:
-            DatasetService.check_dataset_permission(dataset, current_user)
+            DatasetService.check_dataset_permission(dataset, current_user, db.session())
         except services.errors.account.NoPermissionError as e:
             raise Forbidden(str(e))
+        segment_id_str = str(segment_id)
+        segment_ref, segment = _get_segment_for_document(dataset, document, segment_id_str)
+        child_chunk_id_str = str(child_chunk_id)
+        child_chunk = SegmentService.get_child_chunk_by_segment_ref(child_chunk_id_str, segment_ref, db.session())
+        if not child_chunk:
+            raise NotFound("Child chunk not found.")
         # validate args
         try:
             payload = ChildChunkUpdatePayload.model_validate(console_ns.payload or {})
-            child_chunk = SegmentService.update_child_chunk(payload.content, child_chunk, segment, document, dataset)
+            child_chunk = SegmentService.update_child_chunk(
+                payload.content, child_chunk, segment, document, dataset, db.session()
+            )
         except ChildChunkIndexingServiceError as e:
             raise ChildChunkIndexingError(str(e))
         return dump_response(ChildChunkDetailResponse, {"data": child_chunk}), 200