chore: bump version to 1.1.0 (#16128 )

Signed-off-by: -LAN- <laipz8200@outlook.com>
fix chatflow metadata field name (#16130 )
2025-03-18 20:01:24 +08:00 · 2025-03-18 19:40:42 +08:00 · 2025-03-18 16:42:19 +08:00 · 2025-03-18 16:41:35 +08:00 · 2025-03-18 16:21:53 +08:00 · 2025-03-18 15:07:53 +08:00
628 changed files with 27559 additions and 7248 deletions
@@ -0,0 +1,13 @@
+name: "👾 Tracker"
+description: For inner usages, please donot use this template.
+title: "[Tracker] "
+labels:
+  - tracker
+body:
+  - type: textarea
+    id: content
+    attributes:
+      label: Blockers
+      placeholder: "- [ ] ..."
+    validations:
+      required: true
@@ -5,6 +5,7 @@ on:
    branches:
      - "main"
      - "deploy/dev"
+      - "deploy/enterprise"
  release:
    types: [published]

@@ -0,0 +1,29 @@
+name: Deploy Enterprise
+
+permissions:
+  contents: read
+
+on:
+  workflow_run:
+    workflows: ["Build and Push API & Web"]
+    branches:
+      - "deploy/enterprise"
+    types:
+      - completed
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    if: |
+      github.event.workflow_run.conclusion == 'success' &&
+      github.event.workflow_run.head_branch == 'deploy/enterprise'
+
+    steps:
+      - name: Deploy to server
+        uses: appleboy/ssh-action@v0.1.8
+        with:
+          host: ${{ secrets.ENTERPRISE_SSH_HOST }}
+          username: ${{ secrets.ENTERPRISE_SSH_USER }}
+          password: ${{ secrets.ENTERPRISE_SSH_PASSWORD }}
+          script: |
+            ${{ vars.ENTERPRISE_SSH_SCRIPT || secrets.ENTERPRISE_SSH_SCRIPT }}
@@ -10,5 +10,6 @@ yq eval '.services["elasticsearch"].ports += ["9200:9200"]' -i docker/docker-com
 yq eval '.services.couchbase-server.ports += ["8091-8096:8091-8096"]' -i docker/docker-compose.yaml
 yq eval '.services.couchbase-server.ports += ["11210:11210"]' -i docker/docker-compose.yaml
 yq eval '.services.tidb.ports += ["4000:4000"]' -i docker/tidb/docker-compose.yaml
+yq eval '.services.opengauss.ports += ["6600:6600"]' -i docker/docker-compose.yaml

-echo "Ports exposed for sandbox, weaviate, tidb, qdrant, chroma, milvus, pgvector, pgvecto-rs, elasticsearch, couchbase"
+echo "Ports exposed for sandbox, weaviate, tidb, qdrant, chroma, milvus, pgvector, pgvecto-rs, elasticsearch, couchbase, opengauss"
@@ -76,6 +76,7 @@ jobs:
            milvus-standalone
            pgvecto-rs
            pgvector
+            opengauss
            chroma
            elasticsearch

@@ -183,6 +183,7 @@ docker/nginx/conf.d/default.conf
 docker/nginx/ssl/*
 !docker/nginx/ssl/.gitkeep
 docker/middleware.env
+docker/docker-compose.override.yaml

 sdks/python-client/build
 sdks/python-client/dist
@@ -201,3 +202,6 @@ api/.vscode

 # plugin migrate
 plugins.jsonl
+
+# mise
+mise.toml
@@ -26,7 +26,7 @@
  | [@jyong](https://github.com/JohnJyong)                       | RAG 流水线设计                                  |
  | [@GarfieldDai](https://github.com/GarfieldDai)               | 构建 workflow 编排                     |
  | [@iamjoel](https://github.com/iamjoel) & [@zxhlyh](https://github.com/zxhlyh) | 让我们的前端更易用                  |
-  | [@guchenhe](https://github.com/guchenhe) & [@crazywoola](https://github.com/crazywoola) | 开发人员体验, 综合事项联系人 |
+  | [@guchenhe](https://github.com/guchenhe) & [@crazywoola](https://github.com/crazywoola) | 开发人员体验，综合事项联系人 |
  | [@takatost](https://github.com/takatost)                     | 产品整体方向和架构           |

  事项优先级：
@@ -47,7 +47,7 @@
  | ------------------------------------------------------------ | --------------- |
  | 核心功能的 Bugs（例如无法登录、应用无法工作、安全漏洞） | 紧急        |
  | 非紧急 bugs, 性能提升                        | 中等优先级 |
-  | 小幅修复(错别字, 能正常工作但存在误导的 UI)                | 低优先级    |
+  | 小幅修复 (错别字，能正常工作但存在误导的 UI)                | 低优先级    |

 ## 安装

@@ -0,0 +1,153 @@
+# 貢獻指南
+
+您想為 Dify 做出貢獻 - 這太棒了，我們迫不及待地想看看您的成果。作為一家人力和資金有限的初創公司，我們有宏大的抱負，希望設計出最直觀的工作流程來構建和管理 LLM 應用程式。來自社群的任何幫助都非常珍貴，真的。
+
+鑑於我們的現狀，我們需要靈活且快速地發展，但同時也希望確保像您這樣的貢獻者能夠獲得盡可能順暢的貢獻體驗。我們編寫了這份貢獻指南，目的是幫助您熟悉代碼庫以及我們如何與貢獻者合作，讓您可以更快地進入有趣的部分。
+
+這份指南，就像 Dify 本身一樣，是不斷發展的。如果有時它落後於實際項目，我們非常感謝您的理解，也歡迎任何改進的反饋。
+
+關於授權，請花一分鐘閱讀我們簡短的[授權和貢獻者協議](./LICENSE)。社群也遵守[行為準則](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)。
+
+## 在開始之前
+
+[尋找](https://github.com/langgenius/dify/issues?q=is:issue+is:open)現有的 issue，或[創建](https://github.com/langgenius/dify/issues/new/choose)一個新的。我們將 issues 分為 2 種類型：
+
+### 功能請求
+
+- 如果您要開啟新的功能請求，我們希望您能解釋所提議的功能要達成什麼目標，並且盡可能包含更多的相關背景資訊。[@perzeusss](https://github.com/perzeuss) 已經製作了一個實用的[功能請求輔助工具](https://udify.app/chat/MK2kVSnw1gakVwMX)，能幫助您草擬您的需求。歡迎試用。
+
+- 如果您想從現有問題中選擇一個來處理，只需在其下方留言表示即可。
+
+  相關方向的團隊成員會加入討論。如果一切順利，他們會同意您開始編寫代碼。我們要求您在得到許可前先不要開始處理該功能，以免我們提出變更時您的工作成果被浪費。
+
+  根據所提議功能的領域不同，您可能會與不同的團隊成員討論。以下是目前每位團隊成員所負責的領域概述：
+
+  | 成員                                                                                    | 負責領域                       |
+  | --------------------------------------------------------------------------------------- | ------------------------------ |
+  | [@yeuoly](https://github.com/Yeuoly)                                                    | 設計 Agents 架構               |
+  | [@jyong](https://github.com/JohnJyong)                                                  | RAG 管道設計                   |
+  | [@GarfieldDai](https://github.com/GarfieldDai)                                          | 建構工作流程編排               |
+  | [@iamjoel](https://github.com/iamjoel) & [@zxhlyh](https://github.com/zxhlyh)           | 打造易用的前端界面             |
+  | [@guchenhe](https://github.com/guchenhe) & [@crazywoola](https://github.com/crazywoola) | 開發者體驗，各類問題的聯絡窗口 |
+  | [@takatost](https://github.com/takatost)                                                | 整體產品方向與架構             |
+
+  我們如何排定優先順序：
+
+  | 功能類型                                                                                                | 優先級   |
+  | ------------------------------------------------------------------------------------------------------- | -------- |
+  | 被團隊成員標記為高優先級的功能                                                                          | 高優先級 |
+  | 來自我們[社群回饋版](https://github.com/langgenius/dify/discussions/categories/feedbacks)的熱門功能請求 | 中優先級 |
+  | 非核心功能和次要增強                                                                                    | 低優先級 |
+  | 有價值但非急迫的功能                                                                                    | 未來功能 |
+
+### 其他事項 (例如錯誤回報、效能優化、錯字更正)
+
+- 可以直接開始編寫程式碼。
+
+  我們如何排定優先順序：
+
+  | 問題類型                                              | 優先級   |
+  | ----------------------------------------------------- | -------- |
+  | 核心功能的錯誤 (無法登入、應用程式無法運行、安全漏洞) | 重要     |
+  | 非關鍵性錯誤、效能提升                                | 中優先級 |
+  | 小修正 (錯字、令人困惑但仍可運作的使用者界面)         | 低優先級 |
+
+## 安裝
+
+以下是設置 Dify 開發環境的步驟：
+
+### 1. 分叉此存儲庫
+
+### 2. 複製代碼庫
+
+從您的終端機複製分叉的代碼庫：
+
+```shell
+git clone git@github.com:<github_username>/dify.git
+```
+
+- [Docker](https://www.docker.com/)
+- [Docker Compose](https://docs.docker.com/compose/install/)
+- [Node.js v18.x (LTS)](http://nodejs.org)
+- [pnpm](https://pnpm.io/)
+- [Python](https://www.python.org/) version 3.11.x or 3.12.x
+
+### 4. 安裝
+
+Dify 由後端和前端組成。透過 `cd api/` 導航至後端目錄，然後按照[後端 README](api/README.md)進行安裝。在另一個終端機視窗中，透過 `cd web/` 導航至前端目錄，然後按照[前端 README](web/README.md)進行安裝。
+
+查閱[安裝常見問題](https://docs.dify.ai/learn-more/faq/install-faq)了解常見問題和故障排除步驟的列表。
+
+### 5. 在瀏覽器中訪問 Dify
+
+要驗證您的設置，請在瀏覽器中訪問 [http://localhost:3000](http://localhost:3000)（預設值，或您自行設定的 URL 和埠號）。現在您應該能看到 Dify 已啟動並運行。
+
+## 開發
+
+如果您要添加模型提供者，請參考[此指南](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/README.md)。
+
+如果您要為 Agent 或工作流程添加工具提供者，請參考[此指南](./api/core/tools/README.md)。
+
+為了幫助您快速找到您的貢獻適合的位置，以下是 Dify 後端和前端的簡要註解大綱：
+
+### 後端
+
+Dify 的後端使用 Python 的 [Flask](https://flask.palletsprojects.com/en/3.0.x/) 框架編寫。它使用 [SQLAlchemy](https://www.sqlalchemy.org/) 作為 ORM 工具，使用 [Celery](https://docs.celeryq.dev/en/stable/getting-started/introduction.html) 進行任務佇列處理。授權邏輯則透過 Flask-login 實現。
+
+```text
+[api/]
+├── constants             // 整個專案中使用的常數與設定值
+├── controllers           // API 路由定義與請求處理邏輯
+├── core                  // 核心應用服務、模型整合與工具實現
+├── docker                // Docker 容器化相關設定檔案
+├── events                // 事件處理與流程管理機制
+├── extensions            // 與第三方框架或平台的整合擴充功能
+├── fields                // 資料序列化與結構定義欄位
+├── libs                  // 可重複使用的共用程式庫與輔助工具
+├── migrations            // 資料庫結構變更與遷移腳本
+├── models                // 資料庫模型與資料結構定義
+├── services              // 核心業務邏輯與功能實現
+├── storage               // 私鑰與敏感資訊儲存機制
+├── tasks                 // 非同步任務與背景作業處理器
+└── tests
+```
+
+### 前端
+
+網站基於 [Next.js](https://nextjs.org/) 的 Typescript 樣板，並使用 [Tailwind CSS](https://tailwindcss.com/) 進行樣式設計。[React-i18next](https://react.i18next.com/) 用於國際化。
+
+```text
+[web/]
+├── app                   // 頁面佈局與介面元件
+│   ├── (commonLayout)    // 應用程式共用佈局結構
+│   ├── (shareLayout)     // Token 會話專用共享佈局
+│   ├── activate          // 帳號啟用頁面
+│   ├── components        // 頁面與佈局共用元件
+│   ├── install           // 系統安裝頁面
+│   ├── signin            // 使用者登入頁面
+│   └── styles            // 全域共用樣式定義
+├── assets                // 靜態資源檔案庫
+├── bin                   // 建構流程執行腳本
+├── config                // 系統可調整設定與選項
+├── context               // 應用程式狀態共享上下文
+├── dictionaries          // 多語系翻譯詞彙庫
+├── docker                // Docker 容器設定檔
+├── hooks                 // 可重複使用的 React Hooks
+├── i18n                  // 國際化與本地化設定
+├── models                // 資料結構與 API 回應模型
+├── public                // 靜態資源與網站圖標
+├── service               // API 操作介面定義
+├── test                  // 測試用例與測試框架
+├── types                 // TypeScript 型別定義
+└── utils                 // 共用輔助功能函式庫
+```
+
+## 提交您的 PR
+
+最後，是時候向我們的存儲庫開啟拉取請求（PR）了。對於主要功能，我們會先將它們合併到 `deploy/dev` 分支進行測試，然後才會進入 `main` 分支。如果您遇到合併衝突或不知道如何開啟拉取請求等問題，請查看 [GitHub 的拉取請求教學](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests)。
+
+就是這樣！一旦您的 PR 被合併，您將作為貢獻者出現在我們的 [README](https://github.com/langgenius/dify/blob/main/README.md) 中。
+
+## 獲取幫助
+
+如果您在貢獻過程中遇到困難或有迫切的問題，只需通過相關的 GitHub issue 向我們提問，或加入我們的 [Discord](https://discord.gg/8Tpq4AcN9c) 進行快速交流。
@@ -40,6 +40,7 @@

 <p align="center">
  <a href="./README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
+  <a href="./README_TW.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
  <a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
  <a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
  <a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
@@ -53,14 +54,14 @@
  <a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
 </p>

-
-Dify is an open-source LLM app development platform. Its intuitive interface combines agentic AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production. 
+Dify is an open-source LLM app development platform. Its intuitive interface combines agentic AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.

 ## Quick start
+
 > Before installing Dify, make sure your machine meets the following minimum system requirements:
-> 
->- CPU >= 2 Core
->- RAM >= 4 GiB
+>
+> - CPU >= 2 Core
+> - RAM >= 4 GiB

 </br>

@@ -76,41 +77,40 @@ docker compose up -d
 After running, you can access the Dify dashboard in your browser at [http://localhost/install](http://localhost/install) and start the initialization process.

 #### Seeking help
+
 Please refer to our [FAQ](https://docs.dify.ai/getting-started/install-self-hosted/faqs) if you encounter problems setting up Dify. Reach out to [the community and us](#community--contact) if you are still having issues.

 > If you'd like to contribute to Dify or do additional development, refer to our [guide to deploying from source code](https://docs.dify.ai/getting-started/install-self-hosted/local-source-code)

 ## Key features
-**1. Workflow**: 
-  Build and test powerful AI workflows on a visual canvas, leveraging all the following features and beyond.

+**1. Workflow**:
+Build and test powerful AI workflows on a visual canvas, leveraging all the following features and beyond.

-  https://github.com/langgenius/dify/assets/13230914/356df23e-1604-483d-80a6-9517ece318aa
+https://github.com/langgenius/dify/assets/13230914/356df23e-1604-483d-80a6-9517ece318aa

-
-
-**2. Comprehensive model support**: 
-  Seamless integration with hundreds of proprietary / open-source LLMs from dozens of inference providers and self-hosted solutions, covering GPT, Mistral, Llama3, and any OpenAI API-compatible models. A full list of supported model providers can be found [here](https://docs.dify.ai/getting-started/readme/model-providers).
+**2. Comprehensive model support**:
+Seamless integration with hundreds of proprietary / open-source LLMs from dozens of inference providers and self-hosted solutions, covering GPT, Mistral, Llama3, and any OpenAI API-compatible models. A full list of supported model providers can be found [here](https://docs.dify.ai/getting-started/readme/model-providers).

 ![providers-v5](https://github.com/langgenius/dify/assets/13230914/5a17bdbe-097a-4100-8363-40255b70f6e3)

+**3. Prompt IDE**:
+Intuitive interface for crafting prompts, comparing model performance, and adding additional features such as text-to-speech to a chat-based app.

-**3. Prompt IDE**: 
-  Intuitive interface for crafting prompts, comparing model performance, and adding additional features such as text-to-speech to a chat-based app. 
+**4. RAG Pipeline**:
+Extensive RAG capabilities that cover everything from document ingestion to retrieval, with out-of-box support for text extraction from PDFs, PPTs, and other common document formats.

-**4. RAG Pipeline**: 
-  Extensive RAG capabilities that cover everything from document ingestion to retrieval, with out-of-box support for text extraction from PDFs, PPTs, and other common document formats.
+**5. Agent capabilities**:
+You can define agents based on LLM Function Calling or ReAct, and add pre-built or custom tools for the agent. Dify provides 50+ built-in tools for AI agents, such as Google Search, DALL·E, Stable Diffusion and WolframAlpha.

-**5. Agent capabilities**: 
-  You can define agents based on LLM Function Calling or ReAct, and add pre-built or custom tools for the agent. Dify provides 50+ built-in tools for AI agents, such as Google Search, DALL·E, Stable Diffusion and WolframAlpha.
+**6. LLMOps**:
+Monitor and analyze application logs and performance over time. You could continuously improve prompts, datasets, and models based on production data and annotations.

-**6. LLMOps**: 
-  Monitor and analyze application logs and performance over time. You could continuously improve prompts, datasets, and models based on production data and annotations.
-
-**7. Backend-as-a-Service**: 
-  All of Dify's offerings come with corresponding APIs, so you could effortlessly integrate Dify into your own business logic.
+**7. Backend-as-a-Service**:
+All of Dify's offerings come with corresponding APIs, so you could effortlessly integrate Dify into your own business logic.

 ## Feature Comparison
+
 <table style="width: 100%;">
  <tr>
    <th align="center">Feature</th>
@@ -180,24 +180,22 @@ Please refer to our [FAQ](https://docs.dify.ai/getting-started/install-self-host
 ## Using Dify

 - **Cloud </br>**
-We host a [Dify Cloud](https://dify.ai) service for anyone to try with zero setup. It provides all the capabilities of the self-deployed version, and includes 200 free GPT-4 calls in the sandbox plan.
+  We host a [Dify Cloud](https://dify.ai) service for anyone to try with zero setup. It provides all the capabilities of the self-deployed version, and includes 200 free GPT-4 calls in the sandbox plan.

 - **Self-hosting Dify Community Edition</br>**
-Quickly get Dify running in your environment with this [starter guide](#quick-start).
-Use our [documentation](https://docs.dify.ai) for further references and more in-depth instructions.
+  Quickly get Dify running in your environment with this [starter guide](#quick-start).
+  Use our [documentation](https://docs.dify.ai) for further references and more in-depth instructions.

 - **Dify for enterprise / organizations</br>**
-We provide additional enterprise-centric features. [Log your questions for us through this chatbot](https://udify.app/chat/22L1zSxg6yW1cWQg) or [send us an email](mailto:business@dify.ai?subject=[GitHub]Business%20License%20Inquiry) to discuss enterprise needs. </br>
+  We provide additional enterprise-centric features. [Log your questions for us through this chatbot](https://udify.app/chat/22L1zSxg6yW1cWQg) or [send us an email](mailto:business@dify.ai?subject=[GitHub]Business%20License%20Inquiry) to discuss enterprise needs. </br>
  > For startups and small businesses using AWS, check out [Dify Premium on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-t22mebxzwjhu6) and deploy it to your own AWS VPC with one-click. It's an affordable AMI offering with the option to create apps with custom logo and branding.

-
 ## Staying ahead

 Star Dify on GitHub and be instantly notified of new releases.

 ![star-us](https://github.com/langgenius/dify/assets/13230914/b823edc1-6388-4e25-ad45-2f6b187adbb4)

-
 ## Advanced Setup

 If you need to customize the configuration, please refer to the comments in our [.env.example](docker/.env.example) file and update the corresponding values in your `.env` file. Additionally, you might need to make adjustments to the `docker-compose.yaml` file itself, such as changing image versions, port mappings, or volume mounts, based on your specific deployment environment and requirements. After making any changes, please re-run `docker-compose up -d`. You can find the full list of available environment variables [here](https://docs.dify.ai/getting-started/install-self-hosted/environments).
@@ -213,32 +211,34 @@ If you'd like to configure a highly-available setup, there are community-contrib
 Deploy Dify to Cloud Platform with a single click using [terraform](https://www.terraform.io/)

 ##### Azure Global
+
 - [Azure Terraform by @nikawang](https://github.com/nikawang/dify-azure-terraform)

 ##### Google Cloud
+
 - [Google Cloud Terraform by @sotazum](https://github.com/DeNA/dify-google-cloud-terraform)

 #### Using AWS CDK for Deployment

 Deploy Dify to AWS with [CDK](https://aws.amazon.com/cdk/)

-##### AWS 
+##### AWS
+
 - [AWS CDK by @KevinZhao](https://github.com/aws-samples/solution-for-deploying-dify-on-aws)

 ## Contributing

-For those who'd like to contribute code, see our [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md). 
+For those who'd like to contribute code, see our [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
 At the same time, please consider supporting Dify by sharing it on social media and at events and conferences.

-
 > We are looking for contributors to help with translating Dify to languages other than Mandarin or English. If you are interested in helping, please see the [i18n README](https://github.com/langgenius/dify/blob/main/web/i18n/README.md) for more information, and leave us a comment in the `global-users` channel of our [Discord Community Server](https://discord.gg/8Tpq4AcN9c).

 ## Community & contact

-* [Github Discussion](https://github.com/langgenius/dify/discussions). Best for: sharing feedback and asking questions.
-* [GitHub Issues](https://github.com/langgenius/dify/issues). Best for: bugs you encounter using Dify.AI, and feature proposals. See our [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
-* [Discord](https://discord.gg/FngNHpbcY7). Best for: sharing your applications and hanging out with the community.
-* [X(Twitter)](https://twitter.com/dify_ai). Best for: sharing your applications and hanging out with the community.
+- [Github Discussion](https://github.com/langgenius/dify/discussions). Best for: sharing feedback and asking questions.
+- [GitHub Issues](https://github.com/langgenius/dify/issues). Best for: bugs you encounter using Dify.AI, and feature proposals. See our [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
+- [Discord](https://discord.gg/FngNHpbcY7). Best for: sharing your applications and hanging out with the community.
+- [X(Twitter)](https://twitter.com/dify_ai). Best for: sharing your applications and hanging out with the community.

 **Contributors**

@@ -250,7 +250,6 @@ At the same time, please consider supporting Dify by sharing it on social media

 [![Star History Chart](https://api.star-history.com/svg?repos=langgenius/dify&type=Date)](https://star-history.com/#langgenius/dify&Date)

-
 ## Security disclosure

 To protect your privacy, please avoid posting security issues on GitHub. Instead, send your questions to security@dify.ai and we will provide you with a more detailed answer.
@@ -258,4 +257,3 @@ To protect your privacy, please avoid posting security issues on GitHub. Instead
 ## License

 This repository is available under the [Dify Open Source License](LICENSE), which is essentially Apache 2.0 with a few additional restrictions.
-
@@ -79,7 +79,7 @@ Dify 是一个开源的 LLM 应用开发平台。其直观的界面结合了 AI
  广泛的 RAG 功能，涵盖从文档摄入到检索的所有内容，支持从 PDF、PPT 和其他常见文档格式中提取文本的开箱即用的支持。

 **5. Agent 智能体**: 
-  您可以基于 LLM 函数调用或 ReAct 定义 Agent，并为 Agent 添加预构建或自定义工具。Dify 为 AI Agent 提供了50多种内置工具，如谷歌搜索、DALL·E、Stable Diffusion 和 WolframAlpha 等。
+  您可以基于 LLM 函数调用或 ReAct 定义 Agent，并为 Agent 添加预构建或自定义工具。Dify 为 AI Agent 提供了 50 多种内置工具，如谷歌搜索、DALL·E、Stable Diffusion 和 WolframAlpha 等。

 **6. LLMOps**: 
  随时间监视和分析应用程序日志和性能。您可以根据生产数据和标注持续改进提示、数据集和模型。
@@ -112,7 +112,7 @@ Dify 是一个开源的 LLM 应用开发平台。其直观的界面结合了 AI
    <td align="center">仅限 OpenAI</td>
  </tr>
  <tr>
-    <td align="center">RAG引擎</td>
+    <td align="center">RAG 引擎</td>
    <td align="center">✅</td>
    <td align="center">✅</td>
    <td align="center">✅</td>
@@ -234,7 +234,7 @@ docker compose up -d
 对于那些想要贡献代码的人，请参阅我们的[贡献指南](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md)。
 同时，请考虑通过社交媒体、活动和会议来支持 Dify 的分享。

-> 我们正在寻找贡献者来帮助将Dify翻译成除了中文和英文之外的其他语言。如果您有兴趣帮助，请参阅我们的[i18n README](https://github.com/langgenius/dify/blob/main/web/i18n/README.md)获取更多信息，并在我们的[Discord社区服务器](https://discord.gg/8Tpq4AcN9c)的`global-users`频道中留言。
+> 我们正在寻找贡献者来帮助将 Dify 翻译成除了中文和英文之外的其他语言。如果您有兴趣帮助，请参阅我们的[i18n README](https://github.com/langgenius/dify/blob/main/web/i18n/README.md)获取更多信息，并在我们的[Discord 社区服务器](https://discord.gg/8Tpq4AcN9c)的`global-users`频道中留言。

 **Contributors**

@@ -0,0 +1,258 @@
+![cover-v5-optimized](https://github.com/langgenius/dify/assets/13230914/f9e19af5-61ba-4119-b926-d10c4c06ebab)
+
+<p align="center">
+  📌 <a href="https://dify.ai/blog/introducing-dify-workflow-file-upload-a-demo-on-ai-podcast">介紹 Dify 工作流程檔案上傳功能：重現 Google NotebookLM Podcast</a>
+</p>
+
+<p align="center">
+  <a href="https://cloud.dify.ai">Dify 雲端服務</a> ·
+  <a href="https://docs.dify.ai/getting-started/install-self-hosted">自行託管</a> ·
+  <a href="https://docs.dify.ai">說明文件</a> ·
+  <a href="https://udify.app/chat/22L1zSxg6yW1cWQg">企業諮詢</a>
+</p>
+
+<p align="center">
+    <a href="https://dify.ai" target="_blank">
+        <img alt="Static Badge" src="https://img.shields.io/badge/Product-F04438"></a>
+    <a href="https://dify.ai/pricing" target="_blank">
+        <img alt="Static Badge" src="https://img.shields.io/badge/free-pricing?logo=free&color=%20%23155EEF&label=pricing&labelColor=%20%23528bff"></a>
+    <a href="https://discord.gg/FngNHpbcY7" target="_blank">
+        <img src="https://img.shields.io/discord/1082486657678311454?logo=discord&labelColor=%20%235462eb&logoColor=%20%23f5f5f5&color=%20%235462eb"
+            alt="chat on Discord"></a>
+    <a href="https://reddit.com/r/difyai" target="_blank">  
+        <img src="https://img.shields.io/reddit/subreddit-subscribers/difyai?style=plastic&logo=reddit&label=r%2Fdifyai&labelColor=white"
+            alt="join Reddit"></a>
+    <a href="https://twitter.com/intent/follow?screen_name=dify_ai" target="_blank">
+        <img src="https://img.shields.io/twitter/follow/dify_ai?logo=X&color=%20%23f5f5f5"
+            alt="follow on X(Twitter)"></a>
+    <a href="https://www.linkedin.com/company/langgenius/" target="_blank">
+        <img src="https://custom-icon-badges.demolab.com/badge/LinkedIn-0A66C2?logo=linkedin-white&logoColor=fff"
+            alt="follow on LinkedIn"></a>
+    <a href="https://hub.docker.com/u/langgenius" target="_blank">
+        <img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/langgenius/dify-web?labelColor=%20%23FDB062&color=%20%23f79009"></a>
+    <a href="https://github.com/langgenius/dify/graphs/commit-activity" target="_blank">
+        <img alt="Commits last month" src="https://img.shields.io/github/commit-activity/m/langgenius/dify?labelColor=%20%2332b583&color=%20%2312b76a"></a>
+    <a href="https://github.com/langgenius/dify/" target="_blank">
+        <img alt="Issues closed" src="https://img.shields.io/github/issues-search?query=repo%3Alanggenius%2Fdify%20is%3Aclosed&label=issues%20closed&labelColor=%20%237d89b0&color=%20%235d6b98"></a>
+    <a href="https://github.com/langgenius/dify/discussions/" target="_blank">
+        <img alt="Discussion posts" src="https://img.shields.io/github/discussions/langgenius/dify?labelColor=%20%239b8afb&color=%20%237a5af8"></a>
+</p>
+
+<p align="center">
+  <a href="./README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
+  <a href="./README_TW.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
+  <a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
+  <a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
+  <a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
+  <a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
+  <a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
+  <a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
+  <a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
+  <a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
+  <a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
+  <a href="./README_DE.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
+</p>
+
+Dify 是一個開源的 LLM 應用程式開發平台。其直觀的界面結合了智能代理工作流程、RAG 管道、代理功能、模型管理、可觀察性功能等，讓您能夠快速從原型進展到生產環境。
+
+## 快速開始
+
+> 安裝 Dify 之前，請確保您的機器符合以下最低系統要求：
+>
+> - CPU >= 2 核心
+> - 記憶體 >= 4 GiB
+
+</br>
+
+啟動 Dify 伺服器最簡單的方式是透過 [docker compose](docker/docker-compose.yaml)。在使用以下命令運行 Dify 之前，請確保您的機器已安裝 [Docker](https://docs.docker.com/get-docker/) 和 [Docker Compose](https://docs.docker.com/compose/install/)：
+
+```bash
+cd dify
+cd docker
+cp .env.example .env
+docker compose up -d
+```
+
+運行後，您可以在瀏覽器中通過 [http://localhost/install](http://localhost/install) 訪問 Dify 儀表板並開始初始化過程。
+
+### 尋求幫助
+
+如果您在設置 Dify 時遇到問題，請參考我們的 [常見問題](https://docs.dify.ai/getting-started/install-self-hosted/faqs)。如果仍有疑問，請聯絡 [社區和我們](#community--contact)。
+
+> 如果您想為 Dify 做出貢獻或進行額外開發，請參考我們的 [從原始碼部署指南](https://docs.dify.ai/getting-started/install-self-hosted/local-source-code)
+
+## 核心功能
+
+**1. 工作流程**：
+在視覺化畫布上建立和測試強大的 AI 工作流程，利用以下所有功能及更多。
+
+https://github.com/langgenius/dify/assets/13230914/356df23e-1604-483d-80a6-9517ece318aa
+
+**2. 全面的模型支援**：
+無縫整合來自數十個推理提供商和自託管解決方案的數百個專有/開源 LLM，涵蓋 GPT、Mistral、Llama3 和任何與 OpenAI API 兼容的模型。您可以在[此處](https://docs.dify.ai/getting-started/readme/model-providers)找到支援的模型提供商完整列表。
+
+![providers-v5](https://github.com/langgenius/dify/assets/13230914/5a17bdbe-097a-4100-8363-40255b70f6e3)
+
+**3. 提示詞 IDE**：
+直觀的界面，用於編寫提示詞、比較模型性能，以及為聊天型應用程式添加文字轉語音等額外功能。
+
+**4. RAG 管道**：
+廣泛的 RAG 功能，涵蓋從文件擷取到檢索的全部流程，內建支援從 PDF、PPT 和其他常見文件格式提取文本。
+
+**5. 代理功能**：
+您可以基於 LLM 函數調用或 ReAct 定義代理，並為代理添加預構建或自定義工具。Dify 為 AI 代理提供 50 多種內建工具，如 Google 搜尋、DALL·E、Stable Diffusion 和 WolframAlpha。
+
+**6. LLMOps**：
+監控並分析應用程式日誌和長期效能。您可以根據生產數據和標註持續改進提示詞、數據集和模型。
+
+**7. 後端即服務**：
+Dify 的所有功能都提供相應的 API，因此您可以輕鬆地將 Dify 整合到您自己的業務邏輯中。
+
+## 功能比較
+
+<table style="width: 100%;">
+  <tr>
+    <th align="center">功能</th>
+    <th align="center">Dify.AI</th>
+    <th align="center">LangChain</th>
+    <th align="center">Flowise</th>
+    <th align="center">OpenAI Assistants API</th>
+  </tr>
+  <tr>
+    <td align="center">程式設計方法</td>
+    <td align="center">API + 應用導向</td>
+    <td align="center">Python 代碼</td>
+    <td align="center">應用導向</td>
+    <td align="center">API 導向</td>
+  </tr>
+  <tr>
+    <td align="center">支援的 LLM 模型</td>
+    <td align="center">豐富多樣</td>
+    <td align="center">豐富多樣</td>
+    <td align="center">豐富多樣</td>
+    <td align="center">僅限 OpenAI</td>
+  </tr>
+  <tr>
+    <td align="center">RAG 引擎</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+  </tr>
+  <tr>
+    <td align="center">代理功能</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">✅</td>
+  </tr>
+  <tr>
+    <td align="center">工作流程</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+  </tr>
+  <tr>
+    <td align="center">可觀察性</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">❌</td>
+  </tr>
+  <tr>
+    <td align="center">企業級功能 (SSO/存取控制)</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+    <td align="center">❌</td>
+    <td align="center">❌</td>
+  </tr>
+  <tr>
+    <td align="center">本地部署</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">✅</td>
+    <td align="center">❌</td>
+  </tr>
+</table>
+
+## 使用 Dify
+
+- **雲端服務 </br>**
+  我們提供 [Dify Cloud](https://dify.ai) 服務，任何人都可以零配置嘗試。它提供與自部署版本相同的所有功能，並在沙盒計劃中包含 200 次免費 GPT-4 調用。
+
+- **自託管 Dify 社區版</br>**
+  使用這份[快速指南](#快速開始)在您的環境中快速運行 Dify。
+  使用我們的[文檔](https://docs.dify.ai)獲取更多參考和深入指導。
+
+- **企業/組織版 Dify</br>**
+  我們提供額外的企業中心功能。[通過這個聊天機器人記錄您的問題](https://udify.app/chat/22L1zSxg6yW1cWQg)或[發送電子郵件給我們](mailto:business@dify.ai?subject=[GitHub]Business%20License%20Inquiry)討論企業需求。</br>
+  > 對於使用 AWS 的初創企業和小型企業，請查看 [AWS Marketplace 上的 Dify Premium](https://aws.amazon.com/marketplace/pp/prodview-t22mebxzwjhu6)，並一鍵部署到您自己的 AWS VPC。這是一個經濟實惠的 AMI 產品，可選擇使用自定義徽標和品牌創建應用。
+
+## 保持領先
+
+在 GitHub 上為 Dify 加星，即時獲取新版本通知。
+
+![star-us](https://github.com/langgenius/dify/assets/13230914/b823edc1-6388-4e25-ad45-2f6b187adbb4)
+
+## 進階設定
+
+如果您需要自定義配置，請參考我們的 [.env.example](docker/.env.example) 文件中的註釋，並在您的 `.env` 文件中更新相應的值。此外，根據您特定的部署環境和需求，您可能需要調整 `docker-compose.yaml` 文件本身，例如更改映像版本、端口映射或卷掛載。進行任何更改後，請重新運行 `docker-compose up -d`。您可以在[這裡](https://docs.dify.ai/getting-started/install-self-hosted/environments)找到可用環境變數的完整列表。
+
+如果您想配置高可用性設置，社區貢獻的 [Helm Charts](https://helm.sh/) 和 YAML 文件允許在 Kubernetes 上部署 Dify。
+
+- [由 @LeoQuote 提供的 Helm Chart](https://github.com/douban/charts/tree/master/charts/dify)
+- [由 @BorisPolonsky 提供的 Helm Chart](https://github.com/BorisPolonsky/dify-helm)
+- [由 @Winson-030 提供的 YAML 文件](https://github.com/Winson-030/dify-kubernetes)
+
+### 使用 Terraform 進行部署
+
+使用 [terraform](https://www.terraform.io/) 一鍵部署 Dify 到雲端平台
+
+### Azure 全球
+
+- [由 @nikawang 提供的 Azure Terraform](https://github.com/nikawang/dify-azure-terraform)
+
+### Google Cloud
+
+- [由 @sotazum 提供的 Google Cloud Terraform](https://github.com/DeNA/dify-google-cloud-terraform)
+
+### 使用 AWS CDK 進行部署
+
+使用 [CDK](https://aws.amazon.com/cdk/) 部署 Dify 到 AWS
+
+### AWS
+
+- [由 @KevinZhao 提供的 AWS CDK](https://github.com/aws-samples/solution-for-deploying-dify-on-aws)
+
+## 貢獻
+
+對於想要貢獻程式碼的開發者，請參閱我們的[貢獻指南](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md)。
+同時，也請考慮透過在社群媒體和各種活動與會議上分享 Dify 來支持我們。
+
+> 我們正在尋找貢獻者協助將 Dify 翻譯成中文和英文以外的語言。如果您有興趣幫忙，請查看 [i18n README](https://github.com/langgenius/dify/blob/main/web/i18n/README.md) 獲取更多資訊，並在我們的 [Discord 社群伺服器](https://discord.gg/8Tpq4AcN9c) 的 `global-users` 頻道留言給我們。
+
+## 社群與聯絡方式
+
+- [Github Discussion](https://github.com/langgenius/dify/discussions)：最適合分享反饋和提問。
+- [GitHub Issues](https://github.com/langgenius/dify/issues)：最適合報告使用 Dify.AI 時遇到的問題和提出功能建議。請參閱我們的[貢獻指南](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md)。
+- [Discord](https://discord.gg/FngNHpbcY7)：最適合分享您的應用程式並與社群互動。
+- [X(Twitter)](https://twitter.com/dify_ai)：最適合分享您的應用程式並與社群互動。
+
+**貢獻者**
+
+<a href="https://github.com/langgenius/dify/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=langgenius/dify" />
+</a>
+
+## 星星歷史
+
+[![Star History Chart](https://api.star-history.com/svg?repos=langgenius/dify&type=Date)](https://star-history.com/#langgenius/dify&Date)
+
+## 安全揭露
+
+為保護您的隱私，請避免在 GitHub 上發布安全性問題。請將您的問題發送至 security@dify.ai，我們將為您提供更詳細的答覆。
+
+## 授權條款
+
+本代碼庫採用 [Dify 開源授權](LICENSE)，這基本上是 Apache 2.0 授權加上一些額外限制條款。
@@ -137,7 +137,7 @@ WEB_API_CORS_ALLOW_ORIGINS=http://127.0.0.1:3000,*
 CONSOLE_CORS_ALLOW_ORIGINS=http://127.0.0.1:3000,*

 # Vector database configuration
-# support: weaviate, qdrant, milvus, myscale, relyt, pgvecto_rs, pgvector, pgvector, chroma, opensearch, tidb_vector, couchbase, vikingdb, upstash, lindorm, oceanbase
+# support: weaviate, qdrant, milvus, myscale, relyt, pgvecto_rs, pgvector, pgvector, chroma, opensearch, tidb_vector, couchbase, vikingdb, upstash, lindorm, oceanbase, opengauss
 VECTOR_STORE=weaviate

 # Weaviate configuration
@@ -298,6 +298,14 @@ OCEANBASE_VECTOR_PASSWORD=difyai123456
 OCEANBASE_VECTOR_DATABASE=test
 OCEANBASE_MEMORY_LIMIT=6G

+# openGauss configuration
+OPENGAUSS_HOST=127.0.0.1
+OPENGAUSS_PORT=6600
+OPENGAUSS_USER=postgres
+OPENGAUSS_PASSWORD=Dify@123
+OPENGAUSS_DATABASE=dify
+OPENGAUSS_MIN_CONNECTION=1
+OPENGAUSS_MAX_CONNECTION=5

 # Upload configuration
 UPLOAD_FILE_SIZE_LIMIT=15
@@ -378,6 +386,7 @@ HTTP_REQUEST_MAX_READ_TIMEOUT=600
 HTTP_REQUEST_MAX_WRITE_TIMEOUT=600
 HTTP_REQUEST_NODE_MAX_BINARY_SIZE=10485760
 HTTP_REQUEST_NODE_MAX_TEXT_SIZE=1048576
+HTTP_REQUEST_NODE_SSL_VERIFY=True

 # Respect X-* headers to redirect clients
 RESPECT_XFORWARD_HEADERS_ENABLED=false
@@ -444,4 +453,4 @@ CREATE_TIDB_SERVICE_JOB_ENABLED=false
 # Maximum number of submitted thread count in a ThreadPool for parallel node execution
 MAX_SUBMIT_COUNT=100
 # Lockout duration in seconds
-LOGIN_LOCKOUT_DURATION=86400
+LOGIN_LOCKOUT_DURATION=86400
@@ -56,8 +56,6 @@ RUN \
        curl nodejs libgmp-dev libmpfr-dev libmpc-dev \
        # For Security
        expat libldap-2.5-0 perl libsqlite3-0 zlib1g \
-        # install a chinese font to support the use of tools like matplotlib
-        fonts-noto-cjk \
        # install a package to improve the accuracy of guessing mime type and file extension
        media-types \
        # install libmagic to support the use of python-magic guess MIMETYPE
@@ -160,11 +160,17 @@ def migrate_annotation_vector_database():
    while True:
        try:
            # get apps info
+            per_page = 50
            apps = (
-                App.query.filter(App.status == "normal")
+                db.session.query(App)
+                .filter(App.status == "normal")
                .order_by(App.created_at.desc())
-                .paginate(page=page, per_page=50)
+                .limit(per_page)
+                .offset((page - 1) * per_page)
+                .all()
            )
+            if not apps:
+                break
        except NotFound:
            break

@@ -267,6 +273,7 @@ def migrate_knowledge_vector_database():
        VectorType.WEAVIATE,
        VectorType.ORACLE,
        VectorType.ELASTICSEARCH,
+        VectorType.OPENGAUSS,
    }
    lower_collection_vector_types = {
        VectorType.ANALYTICDB,
@@ -332,6 +332,11 @@ class HttpConfig(BaseSettings):
        default=1 * 1024 * 1024,
    )

+    HTTP_REQUEST_NODE_SSL_VERIFY: bool = Field(
+        description="Enable or disable SSL verification for HTTP requests",
+        default=True,
+    )
+
    SSRF_DEFAULT_MAX_RETRIES: PositiveInt = Field(
        description="Maximum number of retries for network requests (SSRF)",
        default=3,
@@ -26,6 +26,7 @@ from .vdb.lindorm_config import LindormConfig
 from .vdb.milvus_config import MilvusConfig
 from .vdb.myscale_config import MyScaleConfig
 from .vdb.oceanbase_config import OceanBaseVectorConfig
+from .vdb.opengauss_config import OpenGaussConfig
 from .vdb.opensearch_config import OpenSearchConfig
 from .vdb.oracle_config import OracleConfig
 from .vdb.pgvector_config import PGVectorConfig
@@ -281,5 +282,6 @@ class MiddlewareConfig(
    LindormConfig,
    OceanBaseVectorConfig,
    BaiduVectorDBConfig,
+    OpenGaussConfig,
 ):
    pass
@@ -0,0 +1,45 @@
+from typing import Optional
+
+from pydantic import Field, PositiveInt
+from pydantic_settings import BaseSettings
+
+
+class OpenGaussConfig(BaseSettings):
+    """
+    Configuration settings for OpenGauss
+    """
+
+    OPENGAUSS_HOST: Optional[str] = Field(
+        description="Hostname or IP address of the OpenGauss server(e.g., 'localhost')",
+        default=None,
+    )
+
+    OPENGAUSS_PORT: PositiveInt = Field(
+        description="Port number on which the OpenGauss server is listening (default is 6600)",
+        default=6600,
+    )
+
+    OPENGAUSS_USER: Optional[str] = Field(
+        description="Username for authenticating with the OpenGauss database",
+        default=None,
+    )
+
+    OPENGAUSS_PASSWORD: Optional[str] = Field(
+        description="Password for authenticating with the OpenGauss database",
+        default=None,
+    )
+
+    OPENGAUSS_DATABASE: Optional[str] = Field(
+        description="Name of the OpenGauss database to connect to",
+        default=None,
+    )
+
+    OPENGAUSS_MIN_CONNECTION: PositiveInt = Field(
+        description="Min connection of the OpenGauss database",
+        default=1,
+    )
+
+    OPENGAUSS_MAX_CONNECTION: PositiveInt = Field(
+        description="Max connection of the OpenGauss database",
+        default=5,
+    )
@@ -43,3 +43,8 @@ class PGVectorConfig(BaseSettings):
        description="Max connection of the PostgreSQL database",
        default=5,
    )
+
+    PGVECTOR_PG_BIGM: bool = Field(
+        description="Whether to use pg_bigm module for full text search",
+        default=False,
+    )
@@ -9,7 +9,7 @@ class PackagingInfo(BaseSettings):

    CURRENT_VERSION: str = Field(
        description="Dify version",
-        default="1.0.0",
+        default="1.1.0",
    )

    COMMIT_SHA: str = Field(
@@ -71,7 +71,7 @@ from .app import (
 from .auth import activate, data_source_bearer_auth, data_source_oauth, forgot_password, login, oauth

 # Import billing controllers
-from .billing import billing
+from .billing import billing, compliance

 # Import datasets controllers
 from .datasets import (
@@ -81,6 +81,7 @@ from .datasets import (
    datasets_segments,
    external,
    hit_testing,
+    metadata,
    website,
 )

@@ -316,7 +316,7 @@ class AppTraceApi(Resource):
    @account_initialization_required
    def post(self, app_id):
        # add app trace
-        if not current_user.is_admin_or_owner:
+        if not current_user.is_editor:
            raise Forbidden()
        parser = reqparse.RequestParser()
        parser.add_argument("enabled", type=bool, required=True, location="json")
@@ -1,8 +1,10 @@
 import json
 import logging
+from typing import cast

 from flask import abort, request
 from flask_restful import Resource, inputs, marshal_with, reqparse  # type: ignore
+from sqlalchemy.orm import Session
 from werkzeug.exceptions import Forbidden, InternalServerError, NotFound

 import services
@@ -13,6 +15,7 @@ from controllers.console.app.wraps import get_app_model
 from controllers.console.wraps import account_initialization_required, setup_required
 from core.app.apps.base_app_queue_manager import AppQueueManager
 from core.app.entities.app_invoke_entities import InvokeFrom
+from extensions.ext_database import db
 from factories import variable_factory
 from fields.workflow_fields import workflow_fields, workflow_pagination_fields
 from fields.workflow_run_fields import workflow_run_node_execution_fields
@@ -24,7 +27,7 @@ from models.account import Account
 from models.model import AppMode
 from services.app_generate_service import AppGenerateService
 from services.errors.app import WorkflowHashNotEqualError
-from services.workflow_service import WorkflowService
+from services.workflow_service import DraftWorkflowDeletionError, WorkflowInUseError, WorkflowService

 logger = logging.getLogger(__name__)

@@ -246,6 +249,80 @@ class WorkflowDraftRunIterationNodeApi(Resource):
            raise InternalServerError()


+class AdvancedChatDraftRunLoopNodeApi(Resource):
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @get_app_model(mode=[AppMode.ADVANCED_CHAT])
+    def post(self, app_model: App, node_id: str):
+        """
+        Run draft workflow loop node
+        """
+        # The role of the current user in the ta table must be admin, owner, or editor
+        if not current_user.is_editor:
+            raise Forbidden()
+
+        if not isinstance(current_user, Account):
+            raise Forbidden()
+
+        parser = reqparse.RequestParser()
+        parser.add_argument("inputs", type=dict, location="json")
+        args = parser.parse_args()
+
+        try:
+            response = AppGenerateService.generate_single_loop(
+                app_model=app_model, user=current_user, node_id=node_id, args=args, streaming=True
+            )
+
+            return helper.compact_generate_response(response)
+        except services.errors.conversation.ConversationNotExistsError:
+            raise NotFound("Conversation Not Exists.")
+        except services.errors.conversation.ConversationCompletedError:
+            raise ConversationCompletedError()
+        except ValueError as e:
+            raise e
+        except Exception:
+            logging.exception("internal server error.")
+            raise InternalServerError()
+
+
+class WorkflowDraftRunLoopNodeApi(Resource):
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @get_app_model(mode=[AppMode.WORKFLOW])
+    def post(self, app_model: App, node_id: str):
+        """
+        Run draft workflow loop node
+        """
+        # The role of the current user in the ta table must be admin, owner, or editor
+        if not current_user.is_editor:
+            raise Forbidden()
+
+        if not isinstance(current_user, Account):
+            raise Forbidden()
+
+        parser = reqparse.RequestParser()
+        parser.add_argument("inputs", type=dict, location="json")
+        args = parser.parse_args()
+
+        try:
+            response = AppGenerateService.generate_single_loop(
+                app_model=app_model, user=current_user, node_id=node_id, args=args, streaming=True
+            )
+
+            return helper.compact_generate_response(response)
+        except services.errors.conversation.ConversationNotExistsError:
+            raise NotFound("Conversation Not Exists.")
+        except services.errors.conversation.ConversationCompletedError:
+            raise ConversationCompletedError()
+        except ValueError as e:
+            raise e
+        except Exception:
+            logging.exception("internal server error.")
+            raise InternalServerError()
+
+
 class DraftWorkflowRunApi(Resource):
    @setup_required
    @login_required
@@ -365,10 +442,38 @@ class PublishedWorkflowApi(Resource):
        if not isinstance(current_user, Account):
            raise Forbidden()

-        workflow_service = WorkflowService()
-        workflow = workflow_service.publish_workflow(app_model=app_model, account=current_user)
+        parser = reqparse.RequestParser()
+        parser.add_argument("marked_name", type=str, required=False, default="", location="json")
+        parser.add_argument("marked_comment", type=str, required=False, default="", location="json")
+        args = parser.parse_args()

-        return {"result": "success", "created_at": TimestampField().format(workflow.created_at)}
+        # Validate name and comment length
+        if args.marked_name and len(args.marked_name) > 20:
+            raise ValueError("Marked name cannot exceed 20 characters")
+        if args.marked_comment and len(args.marked_comment) > 100:
+            raise ValueError("Marked comment cannot exceed 100 characters")
+
+        workflow_service = WorkflowService()
+        with Session(db.engine) as session:
+            workflow = workflow_service.publish_workflow(
+                session=session,
+                app_model=app_model,
+                account=current_user,
+                marked_name=args.marked_name or "",
+                marked_comment=args.marked_comment or "",
+            )
+
+            app_model.workflow_id = workflow.id
+            db.session.commit()
+
+            workflow_created_at = TimestampField().format(workflow.created_at)
+
+            session.commit()
+
+        return {
+            "result": "success",
+            "created_at": workflow_created_at,
+        }


 class DefaultBlockConfigsApi(Resource):
@@ -490,32 +595,193 @@ class PublishedAllWorkflowApi(Resource):
        parser = reqparse.RequestParser()
        parser.add_argument("page", type=inputs.int_range(1, 99999), required=False, default=1, location="args")
        parser.add_argument("limit", type=inputs.int_range(1, 100), required=False, default=20, location="args")
+        parser.add_argument("user_id", type=str, required=False, location="args")
+        parser.add_argument("named_only", type=inputs.boolean, required=False, default=False, location="args")
        args = parser.parse_args()
-        page = args.get("page")
-        limit = args.get("limit")
+        page = int(args.get("page", 1))
+        limit = int(args.get("limit", 10))
+        user_id = args.get("user_id")
+        named_only = args.get("named_only", False)
+
+        if user_id:
+            if user_id != current_user.id:
+                raise Forbidden()
+            user_id = cast(str, user_id)
+
        workflow_service = WorkflowService()
-        workflows, has_more = workflow_service.get_all_published_workflow(app_model=app_model, page=page, limit=limit)
+        with Session(db.engine) as session:
+            workflows, has_more = workflow_service.get_all_published_workflow(
+                session=session,
+                app_model=app_model,
+                page=page,
+                limit=limit,
+                user_id=user_id,
+                named_only=named_only,
+            )

-        return {"items": workflows, "page": page, "limit": limit, "has_more": has_more}
+            return {
+                "items": workflows,
+                "page": page,
+                "limit": limit,
+                "has_more": has_more,
+            }


-api.add_resource(DraftWorkflowApi, "/apps/<uuid:app_id>/workflows/draft")
-api.add_resource(WorkflowConfigApi, "/apps/<uuid:app_id>/workflows/draft/config")
-api.add_resource(AdvancedChatDraftWorkflowRunApi, "/apps/<uuid:app_id>/advanced-chat/workflows/draft/run")
-api.add_resource(DraftWorkflowRunApi, "/apps/<uuid:app_id>/workflows/draft/run")
-api.add_resource(WorkflowTaskStopApi, "/apps/<uuid:app_id>/workflow-runs/tasks/<string:task_id>/stop")
-api.add_resource(DraftWorkflowNodeRunApi, "/apps/<uuid:app_id>/workflows/draft/nodes/<string:node_id>/run")
+class WorkflowByIdApi(Resource):
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @get_app_model(mode=[AppMode.ADVANCED_CHAT, AppMode.WORKFLOW])
+    @marshal_with(workflow_fields)
+    def patch(self, app_model: App, workflow_id: str):
+        """
+        Update workflow attributes
+        """
+        # Check permission
+        if not current_user.is_editor:
+            raise Forbidden()
+
+        if not isinstance(current_user, Account):
+            raise Forbidden()
+
+        parser = reqparse.RequestParser()
+        parser.add_argument("marked_name", type=str, required=False, location="json")
+        parser.add_argument("marked_comment", type=str, required=False, location="json")
+        args = parser.parse_args()
+
+        # Validate name and comment length
+        if args.marked_name and len(args.marked_name) > 20:
+            raise ValueError("Marked name cannot exceed 20 characters")
+        if args.marked_comment and len(args.marked_comment) > 100:
+            raise ValueError("Marked comment cannot exceed 100 characters")
+        args = parser.parse_args()
+
+        # Prepare update data
+        update_data = {}
+        if args.get("marked_name") is not None:
+            update_data["marked_name"] = args["marked_name"]
+        if args.get("marked_comment") is not None:
+            update_data["marked_comment"] = args["marked_comment"]
+
+        if not update_data:
+            return {"message": "No valid fields to update"}, 400
+
+        workflow_service = WorkflowService()
+
+        # Create a session and manage the transaction
+        with Session(db.engine, expire_on_commit=False) as session:
+            workflow = workflow_service.update_workflow(
+                session=session,
+                workflow_id=workflow_id,
+                tenant_id=app_model.tenant_id,
+                account_id=current_user.id,
+                data=update_data,
+            )
+
+            if not workflow:
+                raise NotFound("Workflow not found")
+
+            # Commit the transaction in the controller
+            session.commit()
+
+        return workflow
+
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @get_app_model(mode=[AppMode.ADVANCED_CHAT, AppMode.WORKFLOW])
+    def delete(self, app_model: App, workflow_id: str):
+        """
+        Delete workflow
+        """
+        # Check permission
+        if not current_user.is_editor:
+            raise Forbidden()
+
+        if not isinstance(current_user, Account):
+            raise Forbidden()
+
+        workflow_service = WorkflowService()
+
+        # Create a session and manage the transaction
+        with Session(db.engine) as session:
+            try:
+                workflow_service.delete_workflow(
+                    session=session, workflow_id=workflow_id, tenant_id=app_model.tenant_id
+                )
+                # Commit the transaction in the controller
+                session.commit()
+            except WorkflowInUseError as e:
+                abort(400, description=str(e))
+            except DraftWorkflowDeletionError as e:
+                abort(400, description=str(e))
+            except ValueError as e:
+                raise NotFound(str(e))
+
+        return None, 204
+
+
+api.add_resource(
+    DraftWorkflowApi,
+    "/apps/<uuid:app_id>/workflows/draft",
+)
+api.add_resource(
+    WorkflowConfigApi,
+    "/apps/<uuid:app_id>/workflows/draft/config",
+)
+api.add_resource(
+    AdvancedChatDraftWorkflowRunApi,
+    "/apps/<uuid:app_id>/advanced-chat/workflows/draft/run",
+)
+api.add_resource(
+    DraftWorkflowRunApi,
+    "/apps/<uuid:app_id>/workflows/draft/run",
+)
+api.add_resource(
+    WorkflowTaskStopApi,
+    "/apps/<uuid:app_id>/workflow-runs/tasks/<string:task_id>/stop",
+)
+api.add_resource(
+    DraftWorkflowNodeRunApi,
+    "/apps/<uuid:app_id>/workflows/draft/nodes/<string:node_id>/run",
+)
 api.add_resource(
    AdvancedChatDraftRunIterationNodeApi,
    "/apps/<uuid:app_id>/advanced-chat/workflows/draft/iteration/nodes/<string:node_id>/run",
 )
 api.add_resource(
-    WorkflowDraftRunIterationNodeApi, "/apps/<uuid:app_id>/workflows/draft/iteration/nodes/<string:node_id>/run"
+    WorkflowDraftRunIterationNodeApi,
+    "/apps/<uuid:app_id>/workflows/draft/iteration/nodes/<string:node_id>/run",
 )
-api.add_resource(PublishedWorkflowApi, "/apps/<uuid:app_id>/workflows/publish")
-api.add_resource(PublishedAllWorkflowApi, "/apps/<uuid:app_id>/workflows")
-api.add_resource(DefaultBlockConfigsApi, "/apps/<uuid:app_id>/workflows/default-workflow-block-configs")
 api.add_resource(
-    DefaultBlockConfigApi, "/apps/<uuid:app_id>/workflows/default-workflow-block-configs/<string:block_type>"
+    AdvancedChatDraftRunLoopNodeApi,
+    "/apps/<uuid:app_id>/advanced-chat/workflows/draft/loop/nodes/<string:node_id>/run",
+)
+api.add_resource(
+    WorkflowDraftRunLoopNodeApi,
+    "/apps/<uuid:app_id>/workflows/draft/loop/nodes/<string:node_id>/run",
+)
+api.add_resource(
+    PublishedWorkflowApi,
+    "/apps/<uuid:app_id>/workflows/publish",
+)
+api.add_resource(
+    PublishedAllWorkflowApi,
+    "/apps/<uuid:app_id>/workflows",
+)
+api.add_resource(
+    DefaultBlockConfigsApi,
+    "/apps/<uuid:app_id>/workflows/default-workflow-block-configs",
+)
+api.add_resource(
+    DefaultBlockConfigApi,
+    "/apps/<uuid:app_id>/workflows/default-workflow-block-configs/<string:block_type>",
+)
+api.add_resource(
+    ConvertToWorkflowApi,
+    "/apps/<uuid:app_id>/convert-to-workflow",
+)
+api.add_resource(
+    WorkflowByIdApi,
+    "/apps/<uuid:app_id>/workflows/<string:workflow_id>",
 )
-api.add_resource(ConvertToWorkflowApi, "/apps/<uuid:app_id>/convert-to-workflow")
@@ -1,13 +1,18 @@
+from datetime import datetime
+
 from flask_restful import Resource, marshal_with, reqparse  # type: ignore
 from flask_restful.inputs import int_range  # type: ignore
+from sqlalchemy.orm import Session

 from controllers.console import api
 from controllers.console.app.wraps import get_app_model
 from controllers.console.wraps import account_initialization_required, setup_required
+from extensions.ext_database import db
 from fields.workflow_app_log_fields import workflow_app_log_pagination_fields
 from libs.login import login_required
 from models import App
 from models.model import AppMode
+from models.workflow import WorkflowRunStatus
 from services.workflow_app_service import WorkflowAppService


@@ -24,17 +29,38 @@ class WorkflowAppLogApi(Resource):
        parser = reqparse.RequestParser()
        parser.add_argument("keyword", type=str, location="args")
        parser.add_argument("status", type=str, choices=["succeeded", "failed", "stopped"], location="args")
+        parser.add_argument(
+            "created_at__before", type=str, location="args", help="Filter logs created before this timestamp"
+        )
+        parser.add_argument(
+            "created_at__after", type=str, location="args", help="Filter logs created after this timestamp"
+        )
        parser.add_argument("page", type=int_range(1, 99999), default=1, location="args")
        parser.add_argument("limit", type=int_range(1, 100), default=20, location="args")
        args = parser.parse_args()

+        args.status = WorkflowRunStatus(args.status) if args.status else None
+        if args.created_at__before:
+            args.created_at__before = datetime.fromisoformat(args.created_at__before.replace("Z", "+00:00"))
+
+        if args.created_at__after:
+            args.created_at__after = datetime.fromisoformat(args.created_at__after.replace("Z", "+00:00"))
+
        # get paginate workflow app logs
        workflow_app_service = WorkflowAppService()
-        workflow_app_log_pagination = workflow_app_service.get_paginate_workflow_app_logs(
-            app_model=app_model, args=args
-        )
+        with Session(db.engine) as session:
+            workflow_app_log_pagination = workflow_app_service.get_paginate_workflow_app_logs(
+                session=session,
+                app_model=app_model,
+                keyword=args.keyword,
+                status=args.status,
+                created_at_before=args.created_at__before,
+                created_at_after=args.created_at__after,
+                page=args.page,
+                limit=args.limit,
+            )

-        return workflow_app_log_pagination
+            return workflow_app_log_pagination


 api.add_resource(WorkflowAppLogApi, "/apps/<uuid:app_id>/workflow-app-logs")
@@ -0,0 +1,35 @@
+from flask import request
+from flask_login import current_user  # type: ignore
+from flask_restful import Resource, reqparse  # type: ignore
+
+from libs.helper import extract_remote_ip
+from libs.login import login_required
+from services.billing_service import BillingService
+
+from .. import api
+from ..wraps import account_initialization_required, only_edition_cloud, setup_required
+
+
+class ComplianceApi(Resource):
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @only_edition_cloud
+    def get(self):
+        parser = reqparse.RequestParser()
+        parser.add_argument("doc_name", type=str, required=True, location="args")
+        args = parser.parse_args()
+
+        ip_address = extract_remote_ip(request)
+        device_info = request.headers.get("User-Agent", "Unknown device")
+
+        return BillingService.get_compliance_download_link(
+            doc_name=args.doc_name,
+            account_id=current_user.id,
+            tenant_id=current_user.current_tenant_id,
+            ip=ip_address,
+            device_info=device_info,
+        )
+
+
+api.add_resource(ComplianceApi, "/compliance/download")
@@ -122,7 +122,7 @@ class DataSourceNotionListApi(Resource):
                if dataset.data_source_type != "notion_import":
                    raise ValueError("Dataset is not notion type.")

-                documents = session.execute(
+                documents = session.scalars(
                    select(Document).filter_by(
                        dataset_id=dataset_id,
                        tenant_id=current_user.current_tenant_id,
@@ -10,7 +10,12 @@ from controllers.console import api
 from controllers.console.apikey import api_key_fields, api_key_list
 from controllers.console.app.error import ProviderNotInitializeError
 from controllers.console.datasets.error import DatasetInUseError, DatasetNameDuplicateError, IndexingEstimateError
-from controllers.console.wraps import account_initialization_required, enterprise_license_required, setup_required
+from controllers.console.wraps import (
+    account_initialization_required,
+    cloud_edition_billing_rate_limit_check,
+    enterprise_license_required,
+    setup_required,
+)
 from core.errors.error import LLMBadRequestError, ProviderTokenNotInitError
 from core.indexing_runner import IndexingRunner
 from core.model_runtime.entities.model_entities import ModelType
@@ -96,6 +101,7 @@ class DatasetListApi(Resource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def post(self):
        parser = reqparse.RequestParser()
        parser.add_argument(
@@ -178,6 +184,10 @@ class DatasetApi(Resource):
        except services.errors.account.NoPermissionError as e:
            raise Forbidden(str(e))
        data = marshal(dataset, dataset_detail_fields)
+        if dataset.indexing_technique == "high_quality":
+            if dataset.embedding_model_provider:
+                provider_id = ModelProviderID(dataset.embedding_model_provider)
+                data["embedding_model_provider"] = str(provider_id)
        if data.get("permission") == "partial_members":
            part_users_list = DatasetPermissionService.get_dataset_partial_member_list(dataset_id_str)
            data.update({"partial_member_list": part_users_list})
@@ -210,6 +220,7 @@ class DatasetApi(Resource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def patch(self, dataset_id):
        dataset_id_str = str(dataset_id)
        dataset = DatasetService.get_dataset(dataset_id_str)
@@ -276,7 +287,11 @@ class DatasetApi(Resource):
        data = request.get_json()

        # check embedding model setting
-        if data.get("indexing_technique") == "high_quality":
+        if (
+            data.get("indexing_technique") == "high_quality"
+            and data.get("embedding_model_provider") is not None
+            and data.get("embedding_model") is not None
+        ):
            DatasetService.check_embedding_model_setting(
                dataset.tenant_id, data.get("embedding_model_provider"), data.get("embedding_model")
            )
@@ -313,6 +328,7 @@ class DatasetApi(Resource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def delete(self, dataset_id):
        dataset_id_str = str(dataset_id)

@@ -647,6 +663,7 @@ class DatasetRetrievalSettingApi(Resource):
                | VectorType.LINDORM
                | VectorType.COUCHBASE
                | VectorType.MILVUS
+                | VectorType.OPENGAUSS
            ):
                return {
                    "retrieval_method": [
@@ -690,6 +707,7 @@ class DatasetRetrievalSettingMockApi(Resource):
                | VectorType.COUCHBASE
                | VectorType.PGVECTOR
                | VectorType.LINDORM
+                | VectorType.OPENGAUSS
            ):
                return {
                    "retrieval_method": [
@@ -26,6 +26,7 @@ from controllers.console.datasets.error import (
 )
 from controllers.console.wraps import (
    account_initialization_required,
+    cloud_edition_billing_rate_limit_check,
    cloud_edition_billing_resource_check,
    setup_required,
 )
@@ -242,6 +243,7 @@ class DatasetDocumentListApi(Resource):
    @account_initialization_required
    @marshal_with(documents_and_batch_fields)
    @cloud_edition_billing_resource_check("vector_space")
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def post(self, dataset_id):
        dataset_id = str(dataset_id)

@@ -297,6 +299,7 @@ class DatasetDocumentListApi(Resource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def delete(self, dataset_id):
        dataset_id = str(dataset_id)
        dataset = DatasetService.get_dataset(dataset_id)
@@ -320,9 +323,10 @@ class DatasetInitApi(Resource):
    @account_initialization_required
    @marshal_with(dataset_and_document_fields)
    @cloud_edition_billing_resource_check("vector_space")
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def post(self):
-        # The role of the current user in the ta table must be admin, owner, or editor
-        if not current_user.is_editor:
+        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
+        if not current_user.is_dataset_editor:
            raise Forbidden()

        parser = reqparse.RequestParser()
@@ -617,7 +621,7 @@ class DocumentDetailApi(DocumentResource):
            raise InvalidMetadataError(f"Invalid metadata value: {metadata}")

        if metadata == "only":
-            response = {"id": document.id, "doc_type": document.doc_type, "doc_metadata": document.doc_metadata}
+            response = {"id": document.id, "doc_type": document.doc_type, "doc_metadata": document.doc_metadata_details}
        elif metadata == "without":
            dataset_process_rules = DatasetService.get_process_rules(dataset_id)
            document_process_rules = document.dataset_process_rule.to_dict()
@@ -678,7 +682,7 @@ class DocumentDetailApi(DocumentResource):
                "disabled_by": document.disabled_by,
                "archived": document.archived,
                "doc_type": document.doc_type,
-                "doc_metadata": document.doc_metadata,
+                "doc_metadata": document.doc_metadata_details,
                "segment_count": document.segment_count,
                "average_segment_length": document.average_segment_length,
                "hit_count": document.hit_count,
@@ -694,13 +698,14 @@ class DocumentProcessingApi(DocumentResource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def patch(self, dataset_id, document_id, action):
        dataset_id = str(dataset_id)
        document_id = str(document_id)
        document = self.get_document(dataset_id, document_id)

-        # The role of the current user in the ta table must be admin, owner, or editor
-        if not current_user.is_editor:
+        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
+        if not current_user.is_dataset_editor:
            raise Forbidden()

        if action == "pause":
@@ -730,6 +735,7 @@ class DocumentDeleteApi(DocumentResource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def delete(self, dataset_id, document_id):
        dataset_id = str(dataset_id)
        document_id = str(document_id)
@@ -763,8 +769,8 @@ class DocumentMetadataApi(DocumentResource):
        doc_type = req_data.get("doc_type")
        doc_metadata = req_data.get("doc_metadata")

-        # The role of the current user in the ta table must be admin, owner, or editor
-        if not current_user.is_editor:
+        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
+        if not current_user.is_dataset_editor:
            raise Forbidden()

        if doc_type is None or doc_metadata is None:
@@ -798,6 +804,7 @@ class DocumentStatusApi(DocumentResource):
    @login_required
    @account_initialization_required
    @cloud_edition_billing_resource_check("vector_space")
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def patch(self, dataset_id, action):
        dataset_id = str(dataset_id)
        dataset = DatasetService.get_dataset(dataset_id)
@@ -893,6 +900,7 @@ class DocumentPauseApi(DocumentResource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def patch(self, dataset_id, document_id):
        """pause document."""
        dataset_id = str(dataset_id)
@@ -925,6 +933,7 @@ class DocumentRecoverApi(DocumentResource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def patch(self, dataset_id, document_id):
        """recover document."""
        dataset_id = str(dataset_id)
@@ -954,6 +963,7 @@ class DocumentRetryApi(DocumentResource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def post(self, dataset_id):
        """retry document."""

@@ -19,6 +19,7 @@ from controllers.console.datasets.error import (
 from controllers.console.wraps import (
    account_initialization_required,
    cloud_edition_billing_knowledge_limit_check,
+    cloud_edition_billing_rate_limit_check,
    cloud_edition_billing_resource_check,
    setup_required,
 )
@@ -106,6 +107,7 @@ class DatasetDocumentSegmentListApi(Resource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def delete(self, dataset_id, document_id):
        # check dataset
        dataset_id = str(dataset_id)
@@ -121,8 +123,8 @@ class DatasetDocumentSegmentListApi(Resource):
            raise NotFound("Document not found.")
        segment_ids = request.args.getlist("segment_id")

-        # The role of the current user in the ta table must be admin or owner
-        if not current_user.is_editor:
+        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
+        if not current_user.is_dataset_editor:
            raise Forbidden()
        try:
            DatasetService.check_dataset_permission(dataset, current_user)
@@ -137,6 +139,7 @@ class DatasetDocumentSegmentApi(Resource):
    @login_required
    @account_initialization_required
    @cloud_edition_billing_resource_check("vector_space")
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def patch(self, dataset_id, document_id, action):
        dataset_id = str(dataset_id)
        dataset = DatasetService.get_dataset(dataset_id)
@@ -148,8 +151,8 @@ class DatasetDocumentSegmentApi(Resource):
            raise NotFound("Document not found.")
        # check user's model setting
        DatasetService.check_dataset_model_setting(dataset)
-        # The role of the current user in the ta table must be admin, owner, or editor
-        if not current_user.is_editor:
+        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
+        if not current_user.is_dataset_editor:
            raise Forbidden()

        try:
@@ -191,6 +194,7 @@ class DatasetDocumentSegmentAddApi(Resource):
    @account_initialization_required
    @cloud_edition_billing_resource_check("vector_space")
    @cloud_edition_billing_knowledge_limit_check("add_segment")
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def post(self, dataset_id, document_id):
        # check dataset
        dataset_id = str(dataset_id)
@@ -202,7 +206,7 @@ class DatasetDocumentSegmentAddApi(Resource):
        document = DocumentService.get_document(dataset_id, document_id)
        if not document:
            raise NotFound("Document not found.")
-        if not current_user.is_editor:
+        if not current_user.is_dataset_editor:
            raise Forbidden()
        # check embedding model setting
        if dataset.indexing_technique == "high_quality":
@@ -240,6 +244,7 @@ class DatasetDocumentSegmentUpdateApi(Resource):
    @login_required
    @account_initialization_required
    @cloud_edition_billing_resource_check("vector_space")
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def patch(self, dataset_id, document_id, segment_id):
        # check dataset
        dataset_id = str(dataset_id)
@@ -276,8 +281,8 @@ class DatasetDocumentSegmentUpdateApi(Resource):
        ).first()
        if not segment:
            raise NotFound("Segment not found.")
-        # The role of the current user in the ta table must be admin, owner, or editor
-        if not current_user.is_editor:
+        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
+        if not current_user.is_dataset_editor:
            raise Forbidden()
        try:
            DatasetService.check_dataset_permission(dataset, current_user)
@@ -299,6 +304,7 @@ class DatasetDocumentSegmentUpdateApi(Resource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def delete(self, dataset_id, document_id, segment_id):
        # check dataset
        dataset_id = str(dataset_id)
@@ -319,8 +325,8 @@ class DatasetDocumentSegmentUpdateApi(Resource):
        ).first()
        if not segment:
            raise NotFound("Segment not found.")
-        # The role of the current user in the ta table must be admin or owner
-        if not current_user.is_editor:
+        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
+        if not current_user.is_dataset_editor:
            raise Forbidden()
        try:
            DatasetService.check_dataset_permission(dataset, current_user)
@@ -336,6 +342,7 @@ class DatasetDocumentSegmentBatchImportApi(Resource):
    @account_initialization_required
    @cloud_edition_billing_resource_check("vector_space")
    @cloud_edition_billing_knowledge_limit_check("add_segment")
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def post(self, dataset_id, document_id):
        # check dataset
        dataset_id = str(dataset_id)
@@ -402,6 +409,7 @@ class ChildChunkAddApi(Resource):
    @account_initialization_required
    @cloud_edition_billing_resource_check("vector_space")
    @cloud_edition_billing_knowledge_limit_check("add_segment")
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def post(self, dataset_id, document_id, segment_id):
        # check dataset
        dataset_id = str(dataset_id)
@@ -420,7 +428,7 @@ class ChildChunkAddApi(Resource):
        ).first()
        if not segment:
            raise NotFound("Segment not found.")
-        if not current_user.is_editor:
+        if not current_user.is_dataset_editor:
            raise Forbidden()
        # check embedding model setting
        if dataset.indexing_technique == "high_quality":
@@ -499,6 +507,7 @@ class ChildChunkAddApi(Resource):
    @login_required
    @account_initialization_required
    @cloud_edition_billing_resource_check("vector_space")
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def patch(self, dataset_id, document_id, segment_id):
        # check dataset
        dataset_id = str(dataset_id)
@@ -519,8 +528,8 @@ class ChildChunkAddApi(Resource):
        ).first()
        if not segment:
            raise NotFound("Segment not found.")
-        # The role of the current user in the ta table must be admin, owner, or editor
-        if not current_user.is_editor:
+        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
+        if not current_user.is_dataset_editor:
            raise Forbidden()
        try:
            DatasetService.check_dataset_permission(dataset, current_user)
@@ -542,6 +551,7 @@ class ChildChunkUpdateApi(Resource):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def delete(self, dataset_id, document_id, segment_id, child_chunk_id):
        # check dataset
        dataset_id = str(dataset_id)
@@ -569,8 +579,8 @@ class ChildChunkUpdateApi(Resource):
        ).first()
        if not child_chunk:
            raise NotFound("Child chunk not found.")
-        # The role of the current user in the ta table must be admin or owner
-        if not current_user.is_editor:
+        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
+        if not current_user.is_dataset_editor:
            raise Forbidden()
        try:
            DatasetService.check_dataset_permission(dataset, current_user)
@@ -586,6 +596,7 @@ class ChildChunkUpdateApi(Resource):
    @login_required
    @account_initialization_required
    @cloud_edition_billing_resource_check("vector_space")
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def patch(self, dataset_id, document_id, segment_id, child_chunk_id):
        # check dataset
        dataset_id = str(dataset_id)
@@ -613,8 +624,8 @@ class ChildChunkUpdateApi(Resource):
        ).first()
        if not child_chunk:
            raise NotFound("Child chunk not found.")
-        # The role of the current user in the ta table must be admin or owner
-        if not current_user.is_editor:
+        # The role of the current user in the ta table must be admin, owner, dataset_operator, or editor
+        if not current_user.is_dataset_editor:
            raise Forbidden()
        try:
            DatasetService.check_dataset_permission(dataset, current_user)
@@ -2,7 +2,11 @@ from flask_restful import Resource  # type: ignore

 from controllers.console import api
 from controllers.console.datasets.hit_testing_base import DatasetsHitTestingBase
-from controllers.console.wraps import account_initialization_required, setup_required
+from controllers.console.wraps import (
+    account_initialization_required,
+    cloud_edition_billing_rate_limit_check,
+    setup_required,
+)
 from libs.login import login_required


@@ -10,6 +14,7 @@ class HitTestingApi(Resource, DatasetsHitTestingBase):
    @setup_required
    @login_required
    @account_initialization_required
+    @cloud_edition_billing_rate_limit_check("knowledge")
    def post(self, dataset_id):
        dataset_id_str = str(dataset_id)

@@ -0,0 +1,155 @@
+from flask_login import current_user  # type: ignore  # type: ignore
+from flask_restful import Resource, marshal_with, reqparse  # type: ignore
+from werkzeug.exceptions import NotFound
+
+from controllers.console import api
+from controllers.console.wraps import account_initialization_required, enterprise_license_required, setup_required
+from fields.dataset_fields import dataset_metadata_fields
+from libs.login import login_required
+from services.dataset_service import DatasetService
+from services.entities.knowledge_entities.knowledge_entities import (
+    MetadataArgs,
+    MetadataOperationData,
+)
+from services.metadata_service import MetadataService
+
+
+def _validate_name(name):
+    if not name or len(name) < 1 or len(name) > 40:
+        raise ValueError("Name must be between 1 to 40 characters.")
+    return name
+
+
+def _validate_description_length(description):
+    if len(description) > 400:
+        raise ValueError("Description cannot exceed 400 characters.")
+    return description
+
+
+class DatasetMetadataCreateApi(Resource):
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @enterprise_license_required
+    @marshal_with(dataset_metadata_fields)
+    def post(self, dataset_id):
+        parser = reqparse.RequestParser()
+        parser.add_argument("type", type=str, required=True, nullable=True, location="json")
+        parser.add_argument("name", type=str, required=True, nullable=True, location="json")
+        args = parser.parse_args()
+        metadata_args = MetadataArgs(**args)
+
+        dataset_id_str = str(dataset_id)
+        dataset = DatasetService.get_dataset(dataset_id_str)
+        if dataset is None:
+            raise NotFound("Dataset not found.")
+        DatasetService.check_dataset_permission(dataset, current_user)
+
+        metadata = MetadataService.create_metadata(dataset_id_str, metadata_args)
+        return metadata, 201
+
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @enterprise_license_required
+    def get(self, dataset_id):
+        dataset_id_str = str(dataset_id)
+        dataset = DatasetService.get_dataset(dataset_id_str)
+        if dataset is None:
+            raise NotFound("Dataset not found.")
+        return MetadataService.get_dataset_metadatas(dataset), 200
+
+
+class DatasetMetadataApi(Resource):
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @enterprise_license_required
+    @marshal_with(dataset_metadata_fields)
+    def patch(self, dataset_id, metadata_id):
+        parser = reqparse.RequestParser()
+        parser.add_argument("name", type=str, required=True, nullable=True, location="json")
+        args = parser.parse_args()
+
+        dataset_id_str = str(dataset_id)
+        metadata_id_str = str(metadata_id)
+        dataset = DatasetService.get_dataset(dataset_id_str)
+        if dataset is None:
+            raise NotFound("Dataset not found.")
+        DatasetService.check_dataset_permission(dataset, current_user)
+
+        metadata = MetadataService.update_metadata_name(dataset_id_str, metadata_id_str, args.get("name"))
+        return metadata, 200
+
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @enterprise_license_required
+    def delete(self, dataset_id, metadata_id):
+        dataset_id_str = str(dataset_id)
+        metadata_id_str = str(metadata_id)
+        dataset = DatasetService.get_dataset(dataset_id_str)
+        if dataset is None:
+            raise NotFound("Dataset not found.")
+        DatasetService.check_dataset_permission(dataset, current_user)
+
+        MetadataService.delete_metadata(dataset_id_str, metadata_id_str)
+        return 200
+
+
+class DatasetMetadataBuiltInFieldApi(Resource):
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @enterprise_license_required
+    def get(self):
+        built_in_fields = MetadataService.get_built_in_fields()
+        return {"fields": built_in_fields}, 200
+
+
+class DatasetMetadataBuiltInFieldActionApi(Resource):
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @enterprise_license_required
+    def post(self, dataset_id, action):
+        dataset_id_str = str(dataset_id)
+        dataset = DatasetService.get_dataset(dataset_id_str)
+        if dataset is None:
+            raise NotFound("Dataset not found.")
+        DatasetService.check_dataset_permission(dataset, current_user)
+
+        if action == "enable":
+            MetadataService.enable_built_in_field(dataset)
+        elif action == "disable":
+            MetadataService.disable_built_in_field(dataset)
+        return 200
+
+
+class DocumentMetadataEditApi(Resource):
+    @setup_required
+    @login_required
+    @account_initialization_required
+    @enterprise_license_required
+    def post(self, dataset_id):
+        dataset_id_str = str(dataset_id)
+        dataset = DatasetService.get_dataset(dataset_id_str)
+        if dataset is None:
+            raise NotFound("Dataset not found.")
+        DatasetService.check_dataset_permission(dataset, current_user)
+
+        parser = reqparse.RequestParser()
+        parser.add_argument("operation_data", type=list, required=True, nullable=True, location="json")
+        args = parser.parse_args()
+        metadata_args = MetadataOperationData(**args)
+
+        MetadataService.update_documents_metadata(dataset, metadata_args)
+
+        return 200
+
+
+api.add_resource(DatasetMetadataCreateApi, "/datasets/<uuid:dataset_id>/metadata")
+api.add_resource(DatasetMetadataApi, "/datasets/<uuid:dataset_id>/metadata/<uuid:metadata_id>")
+api.add_resource(DatasetMetadataBuiltInFieldApi, "/datasets/metadata/built-in")
+api.add_resource(DatasetMetadataBuiltInFieldActionApi, "/datasets/<uuid:dataset_id>/metadata/built-in/<string:action>")
+api.add_resource(DocumentMetadataEditApi, "/datasets/<uuid:dataset_id>/documents/metadata")
@@ -101,3 +101,9 @@ class AccountInFreezeError(BaseHTTPException):
        "This email account has been deleted within the past 30 days"
        "and is temporarily unavailable for new account registration."
    )
+
+
+class CompilanceRateLimitError(BaseHTTPException):
+    error_code = "compilance_rate_limit"
+    description = "Rate limit exceeded for downloading compliance report."
+    code = 429
@@ -26,6 +26,7 @@ from libs.helper import TimestampField
 from libs.login import login_required
 from models.account import Tenant, TenantStatus
 from services.account_service import TenantService
+from services.feature_service import FeatureService
 from services.file_service import FileService
 from services.workspace_service import WorkspaceService

@@ -68,6 +69,11 @@ class TenantListApi(Resource):
        tenants = TenantService.get_join_tenants(current_user)

        for tenant in tenants:
+            features = FeatureService.get_features(tenant.id)
+            if features.billing.enabled:
+                tenant.plan = features.billing.subscription.plan
+            else:
+                tenant.plan = "sandbox"
            if tenant.id == current_user.current_tenant_id:
                tenant.current = True  # Set current=True for current tenant
        return {"workspaces": marshal(tenants, tenants_fields)}, 200
@@ -82,28 +88,20 @@ class WorkspaceListApi(Resource):
        parser.add_argument("limit", type=inputs.int_range(1, 100), required=False, default=20, location="args")
        args = parser.parse_args()

-        tenants = Tenant.query.order_by(Tenant.created_at.desc()).paginate(page=args["page"], per_page=args["limit"])
-
+        tenants = Tenant.query.order_by(Tenant.created_at.desc()).paginate(
+            page=args["page"], per_page=args["limit"], error_out=False
+        )
        has_more = False
-        if len(tenants.items) == args["limit"]:
-            current_page_first_tenant = tenants[-1]
-            rest_count = (
-                db.session.query(Tenant)
-                .filter(
-                    Tenant.created_at < current_page_first_tenant.created_at, Tenant.id != current_page_first_tenant.id
-                )
-                .count()
-            )

-            if rest_count > 0:
-                has_more = True
-        total = db.session.query(Tenant).count()
+        if tenants.has_next:
+            has_more = True
+
        return {
            "data": marshal(tenants.items, workspace_fields),
            "has_more": has_more,
            "limit": args["limit"],
            "page": args["page"],
-            "total": total,
+            "total": tenants.total,
        }, 200


@@ -1,5 +1,6 @@
 import json
 import os
+import time
 from functools import wraps

 from flask import abort, request
@@ -8,6 +9,8 @@ from flask_login import current_user  # type: ignore
 from configs import dify_config
 from controllers.console.workspace.error import AccountNotInitializedError
 from extensions.ext_database import db
+from extensions.ext_redis import redis_client
+from models.dataset import RateLimitLog
 from models.model import DifySetup
 from services.feature_service import FeatureService, LicenseStatus
 from services.operation_service import OperationService
@@ -67,7 +70,9 @@ def cloud_edition_billing_resource_check(resource: str):
                elif resource == "apps" and 0 < apps.limit <= apps.size:
                    abort(403, "The number of apps has reached the limit of your subscription.")
                elif resource == "vector_space" and 0 < vector_space.limit <= vector_space.size:
-                    abort(403, "The capacity of the vector space has reached the limit of your subscription.")
+                    abort(
+                        403, "The capacity of the knowledge storage space has reached the limit of your subscription."
+                    )
                elif resource == "documents" and 0 < documents_upload_quota.limit <= documents_upload_quota.size:
                    # The api of file upload is used in the multiple places,
                    # so we need to check the source of the request from datasets
@@ -112,6 +117,41 @@ def cloud_edition_billing_knowledge_limit_check(resource: str):
    return interceptor


+def cloud_edition_billing_rate_limit_check(resource: str):
+    def interceptor(view):
+        @wraps(view)
+        def decorated(*args, **kwargs):
+            if resource == "knowledge":
+                knowledge_rate_limit = FeatureService.get_knowledge_rate_limit(current_user.current_tenant_id)
+                if knowledge_rate_limit.enabled:
+                    current_time = int(time.time() * 1000)
+                    key = f"rate_limit_{current_user.current_tenant_id}"
+
+                    redis_client.zadd(key, {current_time: current_time})
+
+                    redis_client.zremrangebyscore(key, 0, current_time - 60000)
+
+                    request_count = redis_client.zcard(key)
+
+                    if request_count > knowledge_rate_limit.limit:
+                        # add ratelimit record
+                        rate_limit_log = RateLimitLog(
+                            tenant_id=current_user.current_tenant_id,
+                            subscription_plan=knowledge_rate_limit.subscription_plan,
+                            operation="knowledge",
+                        )
+                        db.session.add(rate_limit_log)
+                        db.session.commit()
+                        abort(
+                            403, "Sorry, you have reached the knowledge base request rate limit of your subscription."
+                        )
+            return view(*args, **kwargs)
+
+        return decorated
+
+    return interceptor
+
+
 def cloud_utm_record(view):
    @wraps(view)
    def decorated(*args, **kwargs):
@@ -10,7 +10,7 @@ from controllers.service_api.app.error import NotChatAppError
 from controllers.service_api.wraps import FetchUserArg, WhereisUserArg, validate_app_token
 from core.app.entities.app_invoke_entities import InvokeFrom
 from fields.conversation_fields import message_file_fields
-from fields.message_fields import feedback_fields, retriever_resource_fields
+from fields.message_fields import agent_thought_fields, feedback_fields, retriever_resource_fields
 from fields.raws import FilesContainedField
 from libs.helper import TimestampField, uuid_value
 from models.model import App, AppMode, EndUser
@@ -19,20 +19,6 @@ from services.message_service import MessageService


 class MessageListApi(Resource):
-    agent_thought_fields = {
-        "id": fields.String,
-        "chain_id": fields.String,
-        "message_id": fields.String,
-        "position": fields.Integer,
-        "thought": fields.String,
-        "tool": fields.String,
-        "tool_labels": fields.Raw,
-        "tool_input": fields.String,
-        "created_at": TimestampField,
-        "observation": fields.String,
-        "message_files": fields.List(fields.Nested(message_file_fields)),
-    }
-
    message_fields = {
        "id": fields.String,
        "conversation_id": fields.String,
@@ -70,7 +56,7 @@ class MessageListApi(Resource):

        try:
            return MessageService.pagination_by_first_id(
-                app_model, end_user, args["conversation_id"], args["first_id"], args["limit"], "desc"
+                app_model, end_user, args["conversation_id"], args["first_id"], args["limit"]
            )
        except services.errors.conversation.ConversationNotExistsError:
            raise NotFound("Conversation Not Exists.")
@@ -1,7 +1,9 @@
 import logging
+from datetime import datetime

 from flask_restful import Resource, fields, marshal_with, reqparse  # type: ignore
 from flask_restful.inputs import int_range  # type: ignore
+from sqlalchemy.orm import Session
 from werkzeug.exceptions import InternalServerError

 from controllers.service_api import api
@@ -25,7 +27,7 @@ from extensions.ext_database import db
 from fields.workflow_app_log_fields import workflow_app_log_pagination_fields
 from libs import helper
 from models.model import App, AppMode, EndUser
-from models.workflow import WorkflowRun
+from models.workflow import WorkflowRun, WorkflowRunStatus
 from services.app_generate_service import AppGenerateService
 from services.workflow_app_service import WorkflowAppService

@@ -125,17 +127,34 @@ class WorkflowAppLogApi(Resource):
        parser = reqparse.RequestParser()
        parser.add_argument("keyword", type=str, location="args")
        parser.add_argument("status", type=str, choices=["succeeded", "failed", "stopped"], location="args")
+        parser.add_argument("created_at__before", type=str, location="args")
+        parser.add_argument("created_at__after", type=str, location="args")
        parser.add_argument("page", type=int_range(1, 99999), default=1, location="args")
        parser.add_argument("limit", type=int_range(1, 100), default=20, location="args")
        args = parser.parse_args()

+        args.status = WorkflowRunStatus(args.status) if args.status else None
+        if args.created_at__before:
+            args.created_at__before = datetime.fromisoformat(args.created_at__before.replace("Z", "+00:00"))
+
+        if args.created_at__after:
+            args.created_at__after = datetime.fromisoformat(args.created_at__after.replace("Z", "+00:00"))
+
        # get paginate workflow app logs
        workflow_app_service = WorkflowAppService()
-        workflow_app_log_pagination = workflow_app_service.get_paginate_workflow_app_logs(
-            app_model=app_model, args=args
-        )
+        with Session(db.engine) as session:
+            workflow_app_log_pagination = workflow_app_service.get_paginate_workflow_app_logs(
+                session=session,
+                app_model=app_model,
+                keyword=args.keyword,
+                status=args.status,
+                created_at_before=args.created_at__before,
+                created_at_after=args.created_at__after,
+                page=args.page,
+                limit=args.limit,
+            )

-        return workflow_app_log_pagination
+            return workflow_app_log_pagination


 api.add_resource(WorkflowRunApi, "/workflows/run")
@@ -1,3 +1,4 @@
+import time
 from collections.abc import Callable
 from datetime import UTC, datetime, timedelta
 from enum import Enum
@@ -13,8 +14,10 @@ from sqlalchemy.orm import Session
 from werkzeug.exceptions import Forbidden, Unauthorized

 from extensions.ext_database import db
+from extensions.ext_redis import redis_client
 from libs.login import _get_user
 from models.account import Account, Tenant, TenantAccountJoin, TenantStatus
+from models.dataset import RateLimitLog
 from models.model import ApiToken, App, EndUser
 from services.feature_service import FeatureService

@@ -139,6 +142,43 @@ def cloud_edition_billing_knowledge_limit_check(resource: str, api_token_type: s
    return interceptor


+def cloud_edition_billing_rate_limit_check(resource: str, api_token_type: str):
+    def interceptor(view):
+        @wraps(view)
+        def decorated(*args, **kwargs):
+            api_token = validate_and_get_api_token(api_token_type)
+
+            if resource == "knowledge":
+                knowledge_rate_limit = FeatureService.get_knowledge_rate_limit(api_token.tenant_id)
+                if knowledge_rate_limit.enabled:
+                    current_time = int(time.time() * 1000)
+                    key = f"rate_limit_{api_token.tenant_id}"
+
+                    redis_client.zadd(key, {current_time: current_time})
+
+                    redis_client.zremrangebyscore(key, 0, current_time - 60000)
+
+                    request_count = redis_client.zcard(key)
+
+                    if request_count > knowledge_rate_limit.limit:
+                        # add ratelimit record
+                        rate_limit_log = RateLimitLog(
+                            tenant_id=api_token.tenant_id,
+                            subscription_plan=knowledge_rate_limit.subscription_plan,
+                            operation="knowledge",
+                        )
+                        db.session.add(rate_limit_log)
+                        db.session.commit()
+                        raise Forbidden(
+                            "Sorry, you have reached the knowledge base request rate limit of your subscription."
+                        )
+            return view(*args, **kwargs)
+
+        return decorated
+
+    return interceptor
+
+
 def validate_dataset_token(view=None):
    def decorator(view):
        @wraps(view)
@@ -1,7 +1,12 @@
 import uuid
 from typing import Optional

-from core.app.app_config.entities import DatasetEntity, DatasetRetrieveConfigEntity
+from core.app.app_config.entities import (
+    DatasetEntity,
+    DatasetRetrieveConfigEntity,
+    MetadataFilteringCondition,
+    ModelConfig,
+)
 from core.entities.agent_entities import PlanningStrategy
 from models.model import AppMode
 from services.dataset_service import DatasetService
@@ -78,6 +83,15 @@ class DatasetConfigManager:
                    retrieve_strategy=DatasetRetrieveConfigEntity.RetrieveStrategy.value_of(
                        dataset_configs["retrieval_model"]
                    ),
+                    metadata_filtering_mode=dataset_configs.get("metadata_filtering_mode", "disabled"),
+                    metadata_model_config=ModelConfig(**dataset_configs.get("metadata_model_config"))
+                    if dataset_configs.get("metadata_model_config")
+                    else None,
+                    metadata_filtering_conditions=MetadataFilteringCondition(
+                        **dataset_configs.get("metadata_filtering_conditions", {})
+                    )
+                    if dataset_configs.get("metadata_filtering_conditions")
+                    else None,
                ),
            )
        else:
@@ -89,11 +103,22 @@ class DatasetConfigManager:
                        dataset_configs["retrieval_model"]
                    ),
                    top_k=dataset_configs.get("top_k", 4),
-                    score_threshold=dataset_configs.get("score_threshold"),
+                    score_threshold=dataset_configs.get("score_threshold")
+                    if dataset_configs.get("score_threshold_enabled", False)
+                    else None,
                    reranking_model=dataset_configs.get("reranking_model"),
                    weights=dataset_configs.get("weights"),
                    reranking_enabled=dataset_configs.get("reranking_enabled", True),
                    rerank_mode=dataset_configs.get("reranking_mode", "reranking_model"),
+                    metadata_filtering_mode=dataset_configs.get("metadata_filtering_mode", "disabled"),
+                    metadata_model_config=ModelConfig(**dataset_configs.get("metadata_model_config"))
+                    if dataset_configs.get("metadata_model_config")
+                    else None,
+                    metadata_filtering_conditions=MetadataFilteringCondition(
+                        **dataset_configs.get("metadata_filtering_conditions", {})
+                    )
+                    if dataset_configs.get("metadata_filtering_conditions")
+                    else None,
                ),
            )

@@ -1,10 +1,11 @@
 from collections.abc import Sequence
 from enum import Enum, StrEnum
-from typing import Any, Optional
+from typing import Any, Literal, Optional

 from pydantic import BaseModel, Field, field_validator

 from core.file import FileTransferMethod, FileType, FileUploadConfig
+from core.model_runtime.entities.llm_entities import LLMMode
 from core.model_runtime.entities.message_entities import PromptMessageRole
 from models.model import AppMode

@@ -135,6 +136,55 @@ class ExternalDataVariableEntity(BaseModel):
    config: dict[str, Any] = Field(default_factory=dict)


+SupportedComparisonOperator = Literal[
+    # for string or array
+    "contains",
+    "not contains",
+    "start with",
+    "end with",
+    "is",
+    "is not",
+    "empty",
+    "not empty",
+    # for number
+    "=",
+    "≠",
+    ">",
+    "<",
+    "≥",
+    "≤",
+    # for time
+    "before",
+    "after",
+]
+
+
+class ModelConfig(BaseModel):
+    provider: str
+    name: str
+    mode: LLMMode
+    completion_params: dict[str, Any] = {}
+
+
+class Condition(BaseModel):
+    """
+    Conditon detail
+    """
+
+    name: str
+    comparison_operator: SupportedComparisonOperator
+    value: str | Sequence[str] | None | int | float = None
+
+
+class MetadataFilteringCondition(BaseModel):
+    """
+    Metadata Filtering Condition.
+    """
+
+    logical_operator: Optional[Literal["and", "or"]] = "and"
+    conditions: Optional[list[Condition]] = Field(default=None, deprecated=True)
+
+
 class DatasetRetrieveConfigEntity(BaseModel):
    """
    Dataset Retrieve Config Entity.
@@ -171,6 +221,9 @@ class DatasetRetrieveConfigEntity(BaseModel):
    reranking_model: Optional[dict] = None
    weights: Optional[dict] = None
    reranking_enabled: Optional[bool] = True
+    metadata_filtering_mode: Optional[Literal["disabled", "automatic", "manual"]] = "disabled"
+    metadata_model_config: Optional[ModelConfig] = None
+    metadata_filtering_conditions: Optional[MetadataFilteringCondition] = None


 class DatasetEntity(BaseModel):
@@ -17,17 +17,15 @@ class FileUploadConfigManager:
        if file_upload_dict:
            if file_upload_dict.get("enabled"):
                transform_methods = file_upload_dict.get("allowed_file_upload_methods", [])
-                data = {
-                    "image_config": {
-                        "number_limits": file_upload_dict["number_limits"],
-                        "transfer_methods": transform_methods,
-                    }
+                file_upload_dict["image_config"] = {
+                    "number_limits": file_upload_dict.get("number_limits", 1),
+                    "transfer_methods": transform_methods,
                }

                if is_vision:
-                    data["image_config"]["detail"] = file_upload_dict.get("image", {}).get("detail", "low")
+                    file_upload_dict["image_config"]["detail"] = file_upload_dict.get("image", {}).get("detail", "high")

-                return FileUploadConfig.model_validate(data)
+                return FileUploadConfig.model_validate(file_upload_dict)

    @classmethod
    def validate_and_set_defaults(cls, config: dict) -> tuple[dict, list[str]]:
@@ -223,6 +223,61 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
            stream=streaming,
        )

+    def single_loop_generate(
+        self,
+        app_model: App,
+        workflow: Workflow,
+        node_id: str,
+        user: Account | EndUser,
+        args: Mapping,
+        streaming: bool = True,
+    ) -> Mapping[str, Any] | Generator[str | Mapping[str, Any], Any, None]:
+        """
+        Generate App response.
+
+        :param app_model: App
+        :param workflow: Workflow
+        :param user: account or end user
+        :param args: request args
+        :param invoke_from: invoke from source
+        :param stream: is stream
+        """
+        if not node_id:
+            raise ValueError("node_id is required")
+
+        if args.get("inputs") is None:
+            raise ValueError("inputs is required")
+
+        # convert to app config
+        app_config = AdvancedChatAppConfigManager.get_app_config(app_model=app_model, workflow=workflow)
+
+        # init application generate entity
+        application_generate_entity = AdvancedChatAppGenerateEntity(
+            task_id=str(uuid.uuid4()),
+            app_config=app_config,
+            conversation_id=None,
+            inputs={},
+            query="",
+            files=[],
+            user_id=user.id,
+            stream=streaming,
+            invoke_from=InvokeFrom.DEBUGGER,
+            extras={"auto_generate_conversation_name": False},
+            single_loop_run=AdvancedChatAppGenerateEntity.SingleLoopRunEntity(node_id=node_id, inputs=args["inputs"]),
+        )
+        contexts.tenant_id.set(application_generate_entity.app_config.tenant_id)
+        contexts.plugin_tool_providers.set({})
+        contexts.plugin_tool_providers_lock.set(threading.Lock())
+
+        return self._generate(
+            workflow=workflow,
+            user=user,
+            invoke_from=InvokeFrom.DEBUGGER,
+            application_generate_entity=application_generate_entity,
+            conversation=None,
+            stream=streaming,
+        )
+
    def _generate(
        self,
        *,
@@ -79,6 +79,13 @@ class AdvancedChatAppRunner(WorkflowBasedAppRunner):
                node_id=self.application_generate_entity.single_iteration_run.node_id,
                user_inputs=dict(self.application_generate_entity.single_iteration_run.inputs),
            )
+        elif self.application_generate_entity.single_loop_run:
+            # if only single loop run is requested
+            graph, variable_pool = self._get_graph_and_variable_pool_of_single_loop(
+                workflow=workflow,
+                node_id=self.application_generate_entity.single_loop_run.node_id,
+                user_inputs=dict(self.application_generate_entity.single_loop_run.inputs),
+            )
        else:
            inputs = self.application_generate_entity.inputs
            query = self.application_generate_entity.query
@@ -23,10 +23,14 @@ from core.app.entities.queue_entities import (
    QueueIterationCompletedEvent,
    QueueIterationNextEvent,
    QueueIterationStartEvent,
+    QueueLoopCompletedEvent,
+    QueueLoopNextEvent,
+    QueueLoopStartEvent,
    QueueMessageReplaceEvent,
    QueueNodeExceptionEvent,
    QueueNodeFailedEvent,
    QueueNodeInIterationFailedEvent,
+    QueueNodeInLoopFailedEvent,
    QueueNodeRetryEvent,
    QueueNodeStartedEvent,
    QueueNodeSucceededEvent,
@@ -372,7 +376,13 @@ class AdvancedChatAppGenerateTaskPipeline:

                if node_finish_resp:
                    yield node_finish_resp
-            elif isinstance(event, QueueNodeFailedEvent | QueueNodeInIterationFailedEvent | QueueNodeExceptionEvent):
+            elif isinstance(
+                event,
+                QueueNodeFailedEvent
+                | QueueNodeInIterationFailedEvent
+                | QueueNodeInLoopFailedEvent
+                | QueueNodeExceptionEvent,
+            ):
                with Session(db.engine, expire_on_commit=False) as session:
                    workflow_node_execution = self._workflow_cycle_manager._handle_workflow_node_execution_failed(
                        session=session, event=event
@@ -472,6 +482,54 @@ class AdvancedChatAppGenerateTaskPipeline:
                    )

                yield iter_finish_resp
+            elif isinstance(event, QueueLoopStartEvent):
+                if not self._workflow_run_id:
+                    raise ValueError("workflow run not initialized.")
+
+                with Session(db.engine, expire_on_commit=False) as session:
+                    workflow_run = self._workflow_cycle_manager._get_workflow_run(
+                        session=session, workflow_run_id=self._workflow_run_id
+                    )
+                    loop_start_resp = self._workflow_cycle_manager._workflow_loop_start_to_stream_response(
+                        session=session,
+                        task_id=self._application_generate_entity.task_id,
+                        workflow_run=workflow_run,
+                        event=event,
+                    )
+
+                yield loop_start_resp
+            elif isinstance(event, QueueLoopNextEvent):
+                if not self._workflow_run_id:
+                    raise ValueError("workflow run not initialized.")
+
+                with Session(db.engine, expire_on_commit=False) as session:
+                    workflow_run = self._workflow_cycle_manager._get_workflow_run(
+                        session=session, workflow_run_id=self._workflow_run_id
+                    )
+                    loop_next_resp = self._workflow_cycle_manager._workflow_loop_next_to_stream_response(
+                        session=session,
+                        task_id=self._application_generate_entity.task_id,
+                        workflow_run=workflow_run,
+                        event=event,
+                    )
+
+                yield loop_next_resp
+            elif isinstance(event, QueueLoopCompletedEvent):
+                if not self._workflow_run_id:
+                    raise ValueError("workflow run not initialized.")
+
+                with Session(db.engine, expire_on_commit=False) as session:
+                    workflow_run = self._workflow_cycle_manager._get_workflow_run(
+                        session=session, workflow_run_id=self._workflow_run_id
+                    )
+                    loop_finish_resp = self._workflow_cycle_manager._workflow_loop_completed_to_stream_response(
+                        session=session,
+                        task_id=self._application_generate_entity.task_id,
+                        workflow_run=workflow_run,
+                        event=event,
+                    )
+
+                yield loop_finish_resp
            elif isinstance(event, QueueWorkflowSucceededEvent):
                if not self._workflow_run_id:
                    raise ValueError("workflow run not initialized.")
@@ -151,7 +151,7 @@ class BaseAppGenerator:

            def gen():
                for message in generator:
-                    if isinstance(message, (Mapping, dict)):
+                    if isinstance(message, Mapping | dict):
                        yield f"data: {json.dumps(message)}\n\n"
                    else:
                        yield f"event: {message}\n\n"
@@ -17,7 +17,11 @@ from core.external_data_tool.external_data_fetch import ExternalDataFetch
 from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_manager import ModelInstance
 from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta, LLMUsage
-from core.model_runtime.entities.message_entities import AssistantPromptMessage, PromptMessage
+from core.model_runtime.entities.message_entities import (
+    AssistantPromptMessage,
+    ImagePromptMessageContent,
+    PromptMessage,
+)
 from core.model_runtime.entities.model_entities import ModelPropertyKey
 from core.model_runtime.errors.invoke import InvokeBadRequestError
 from core.moderation.input_moderation import InputModeration
@@ -141,6 +145,7 @@ class AppRunner:
        query: Optional[str] = None,
        context: Optional[str] = None,
        memory: Optional[TokenBufferMemory] = None,
+        image_detail_config: Optional[ImagePromptMessageContent.DETAIL] = None,
    ) -> tuple[list[PromptMessage], Optional[list[str]]]:
        """
        Organize prompt messages
@@ -167,6 +172,7 @@ class AppRunner:
                context=context,
                memory=memory,
                model_config=model_config,
+                image_detail_config=image_detail_config,
            )
        else:
            memory_config = MemoryConfig(window=MemoryConfig.WindowConfig(enabled=False))
@@ -201,6 +207,7 @@ class AppRunner:
                memory_config=memory_config,
                memory=memory,
                model_config=model_config,
+                image_detail_config=image_detail_config,
            )
            stop = model_config.stop

@@ -11,6 +11,7 @@ from core.app.entities.queue_entities import QueueAnnotationReplyEvent
 from core.callback_handler.index_tool_callback_handler import DatasetIndexToolCallbackHandler
 from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_manager import ModelInstance
+from core.model_runtime.entities.message_entities import ImagePromptMessageContent
 from core.moderation.base import ModerationError
 from core.rag.retrieval.dataset_retrieval import DatasetRetrieval
 from extensions.ext_database import db
@@ -50,6 +51,16 @@ class ChatAppRunner(AppRunner):
        query = application_generate_entity.query
        files = application_generate_entity.files

+        image_detail_config = (
+            application_generate_entity.file_upload_config.image_config.detail
+            if (
+                application_generate_entity.file_upload_config
+                and application_generate_entity.file_upload_config.image_config
+            )
+            else None
+        )
+        image_detail_config = image_detail_config or ImagePromptMessageContent.DETAIL.LOW
+
        # Pre-calculate the number of tokens of the prompt messages,
        # and return the rest number of tokens by model context token size limit and max token size limit.
        # If the rest number of tokens is not enough, raise exception.
@@ -85,6 +96,7 @@ class ChatAppRunner(AppRunner):
            files=files,
            query=query,
            memory=memory,
+            image_detail_config=image_detail_config,
        )

        # moderation
@@ -168,6 +180,7 @@ class ChatAppRunner(AppRunner):
                hit_callback=hit_callback,
                memory=memory,
                message_id=message.id,
+                inputs=inputs,
            )

        # reorganize all inputs and template to prompt messages
@@ -182,6 +195,7 @@ class ChatAppRunner(AppRunner):
            query=query,
            context=context,
            memory=memory,
+            image_detail_config=image_detail_config,
        )

        # check hosting moderation
@@ -9,6 +9,7 @@ from core.app.entities.app_invoke_entities import (
 )
 from core.callback_handler.index_tool_callback_handler import DatasetIndexToolCallbackHandler
 from core.model_manager import ModelInstance
+from core.model_runtime.entities.message_entities import ImagePromptMessageContent
 from core.moderation.base import ModerationError
 from core.rag.retrieval.dataset_retrieval import DatasetRetrieval
 from extensions.ext_database import db
@@ -43,6 +44,16 @@ class CompletionAppRunner(AppRunner):
        query = application_generate_entity.query
        files = application_generate_entity.files

+        image_detail_config = (
+            application_generate_entity.file_upload_config.image_config.detail
+            if (
+                application_generate_entity.file_upload_config
+                and application_generate_entity.file_upload_config.image_config
+            )
+            else None
+        )
+        image_detail_config = image_detail_config or ImagePromptMessageContent.DETAIL.LOW
+
        # Pre-calculate the number of tokens of the prompt messages,
        # and return the rest number of tokens by model context token size limit and max token size limit.
        # If the rest number of tokens is not enough, raise exception.
@@ -66,6 +77,7 @@ class CompletionAppRunner(AppRunner):
            inputs=inputs,
            files=files,
            query=query,
+            image_detail_config=image_detail_config,
        )

        # moderation
@@ -127,6 +139,7 @@ class CompletionAppRunner(AppRunner):
                show_retrieve_source=app_config.additional_features.show_retrieve_source,
                hit_callback=hit_callback,
                message_id=message.id,
+                inputs=inputs,
            )

        # reorganize all inputs and template to prompt messages
@@ -140,6 +153,7 @@ class CompletionAppRunner(AppRunner):
            files=files,
            query=query,
            context=context,
+            image_detail_config=image_detail_config,
        )

        # check hosting moderation
@@ -250,6 +250,60 @@ class WorkflowAppGenerator(BaseAppGenerator):
            streaming=streaming,
        )

+    def single_loop_generate(
+        self,
+        app_model: App,
+        workflow: Workflow,
+        node_id: str,
+        user: Account | EndUser,
+        args: Mapping[str, Any],
+        streaming: bool = True,
+    ) -> Mapping[str, Any] | Generator[str | Mapping[str, Any], None, None]:
+        """
+        Generate App response.
+
+        :param app_model: App
+        :param workflow: Workflow
+        :param user: account or end user
+        :param args: request args
+        :param invoke_from: invoke from source
+        :param stream: is stream
+        """
+        if not node_id:
+            raise ValueError("node_id is required")
+
+        if args.get("inputs") is None:
+            raise ValueError("inputs is required")
+
+        # convert to app config
+        app_config = WorkflowAppConfigManager.get_app_config(app_model=app_model, workflow=workflow)
+
+        # init application generate entity
+        application_generate_entity = WorkflowAppGenerateEntity(
+            task_id=str(uuid.uuid4()),
+            app_config=app_config,
+            inputs={},
+            files=[],
+            user_id=user.id,
+            stream=streaming,
+            invoke_from=InvokeFrom.DEBUGGER,
+            extras={"auto_generate_conversation_name": False},
+            single_loop_run=WorkflowAppGenerateEntity.SingleLoopRunEntity(node_id=node_id, inputs=args["inputs"]),
+            workflow_run_id=str(uuid.uuid4()),
+        )
+        contexts.tenant_id.set(application_generate_entity.app_config.tenant_id)
+        contexts.plugin_tool_providers.set({})
+        contexts.plugin_tool_providers_lock.set(threading.Lock())
+
+        return self._generate(
+            app_model=app_model,
+            workflow=workflow,
+            user=user,
+            invoke_from=InvokeFrom.DEBUGGER,
+            application_generate_entity=application_generate_entity,
+            streaming=streaming,
+        )
+
    def _generate_worker(
        self,
        flask_app: Flask,
@@ -81,6 +81,13 @@ class WorkflowAppRunner(WorkflowBasedAppRunner):
                node_id=self.application_generate_entity.single_iteration_run.node_id,
                user_inputs=self.application_generate_entity.single_iteration_run.inputs,
            )
+        elif self.application_generate_entity.single_loop_run:
+            # if only single loop run is requested
+            graph, variable_pool = self._get_graph_and_variable_pool_of_single_loop(
+                workflow=workflow,
+                node_id=self.application_generate_entity.single_loop_run.node_id,
+                user_inputs=self.application_generate_entity.single_loop_run.inputs,
+            )
        else:
            inputs = self.application_generate_entity.inputs
            files = self.application_generate_entity.files
@@ -18,9 +18,13 @@ from core.app.entities.queue_entities import (
    QueueIterationCompletedEvent,
    QueueIterationNextEvent,
    QueueIterationStartEvent,
+    QueueLoopCompletedEvent,
+    QueueLoopNextEvent,
+    QueueLoopStartEvent,
    QueueNodeExceptionEvent,
    QueueNodeFailedEvent,
    QueueNodeInIterationFailedEvent,
+    QueueNodeInLoopFailedEvent,
    QueueNodeRetryEvent,
    QueueNodeStartedEvent,
    QueueNodeSucceededEvent,
@@ -323,7 +327,13 @@ class WorkflowAppGenerateTaskPipeline:

                if node_success_response:
                    yield node_success_response
-            elif isinstance(event, QueueNodeFailedEvent | QueueNodeInIterationFailedEvent | QueueNodeExceptionEvent):
+            elif isinstance(
+                event,
+                QueueNodeFailedEvent
+                | QueueNodeInIterationFailedEvent
+                | QueueNodeInLoopFailedEvent
+                | QueueNodeExceptionEvent,
+            ):
                with Session(db.engine, expire_on_commit=False) as session:
                    workflow_node_execution = self._workflow_cycle_manager._handle_workflow_node_execution_failed(
                        session=session,
@@ -429,6 +439,57 @@ class WorkflowAppGenerateTaskPipeline:

                yield iter_finish_resp

+            elif isinstance(event, QueueLoopStartEvent):
+                if not self._workflow_run_id:
+                    raise ValueError("workflow run not initialized.")
+
+                with Session(db.engine, expire_on_commit=False) as session:
+                    workflow_run = self._workflow_cycle_manager._get_workflow_run(
+                        session=session, workflow_run_id=self._workflow_run_id
+                    )
+                    loop_start_resp = self._workflow_cycle_manager._workflow_loop_start_to_stream_response(
+                        session=session,
+                        task_id=self._application_generate_entity.task_id,
+                        workflow_run=workflow_run,
+                        event=event,
+                    )
+
+                yield loop_start_resp
+
+            elif isinstance(event, QueueLoopNextEvent):
+                if not self._workflow_run_id:
+                    raise ValueError("workflow run not initialized.")
+
+                with Session(db.engine, expire_on_commit=False) as session:
+                    workflow_run = self._workflow_cycle_manager._get_workflow_run(
+                        session=session, workflow_run_id=self._workflow_run_id
+                    )
+                    loop_next_resp = self._workflow_cycle_manager._workflow_loop_next_to_stream_response(
+                        session=session,
+                        task_id=self._application_generate_entity.task_id,
+                        workflow_run=workflow_run,
+                        event=event,
+                    )
+
+                yield loop_next_resp
+
+            elif isinstance(event, QueueLoopCompletedEvent):
+                if not self._workflow_run_id:
+                    raise ValueError("workflow run not initialized.")
+
+                with Session(db.engine, expire_on_commit=False) as session:
+                    workflow_run = self._workflow_cycle_manager._get_workflow_run(
+                        session=session, workflow_run_id=self._workflow_run_id
+                    )
+                    loop_finish_resp = self._workflow_cycle_manager._workflow_loop_completed_to_stream_response(
+                        session=session,
+                        task_id=self._application_generate_entity.task_id,
+                        workflow_run=workflow_run,
+                        event=event,
+                    )
+
+                yield loop_finish_resp
+
            elif isinstance(event, QueueWorkflowSucceededEvent):
                if not self._workflow_run_id:
                    raise ValueError("workflow run not initialized.")
@@ -9,9 +9,13 @@ from core.app.entities.queue_entities import (
    QueueIterationCompletedEvent,
    QueueIterationNextEvent,
    QueueIterationStartEvent,
+    QueueLoopCompletedEvent,
+    QueueLoopNextEvent,
+    QueueLoopStartEvent,
    QueueNodeExceptionEvent,
    QueueNodeFailedEvent,
    QueueNodeInIterationFailedEvent,
+    QueueNodeInLoopFailedEvent,
    QueueNodeRetryEvent,
    QueueNodeStartedEvent,
    QueueNodeSucceededEvent,
@@ -38,7 +42,12 @@ from core.workflow.graph_engine.entities.event import (
    IterationRunNextEvent,
    IterationRunStartedEvent,
    IterationRunSucceededEvent,
+    LoopRunFailedEvent,
+    LoopRunNextEvent,
+    LoopRunStartedEvent,
+    LoopRunSucceededEvent,
    NodeInIterationFailedEvent,
+    NodeInLoopFailedEvent,
    NodeRunExceptionEvent,
    NodeRunFailedEvent,
    NodeRunRetrieverResourceEvent,
@@ -173,6 +182,96 @@ class WorkflowBasedAppRunner(AppRunner):

        return graph, variable_pool

+    def _get_graph_and_variable_pool_of_single_loop(
+        self,
+        workflow: Workflow,
+        node_id: str,
+        user_inputs: dict,
+    ) -> tuple[Graph, VariablePool]:
+        """
+        Get variable pool of single loop
+        """
+        # fetch workflow graph
+        graph_config = workflow.graph_dict
+        if not graph_config:
+            raise ValueError("workflow graph not found")
+
+        graph_config = cast(dict[str, Any], graph_config)
+
+        if "nodes" not in graph_config or "edges" not in graph_config:
+            raise ValueError("nodes or edges not found in workflow graph")
+
+        if not isinstance(graph_config.get("nodes"), list):
+            raise ValueError("nodes in workflow graph must be a list")
+
+        if not isinstance(graph_config.get("edges"), list):
+            raise ValueError("edges in workflow graph must be a list")
+
+        # filter nodes only in loop
+        node_configs = [
+            node
+            for node in graph_config.get("nodes", [])
+            if node.get("id") == node_id or node.get("data", {}).get("loop_id", "") == node_id
+        ]
+
+        graph_config["nodes"] = node_configs
+
+        node_ids = [node.get("id") for node in node_configs]
+
+        # filter edges only in loop
+        edge_configs = [
+            edge
+            for edge in graph_config.get("edges", [])
+            if (edge.get("source") is None or edge.get("source") in node_ids)
+            and (edge.get("target") is None or edge.get("target") in node_ids)
+        ]
+
+        graph_config["edges"] = edge_configs
+
+        # init graph
+        graph = Graph.init(graph_config=graph_config, root_node_id=node_id)
+
+        if not graph:
+            raise ValueError("graph not found in workflow")
+
+        # fetch node config from node id
+        loop_node_config = None
+        for node in node_configs:
+            if node.get("id") == node_id:
+                loop_node_config = node
+                break
+
+        if not loop_node_config:
+            raise ValueError("loop node id not found in workflow graph")
+
+        # Get node class
+        node_type = NodeType(loop_node_config.get("data", {}).get("type"))
+        node_version = loop_node_config.get("data", {}).get("version", "1")
+        node_cls = NODE_TYPE_CLASSES_MAPPING[node_type][node_version]
+
+        # init variable pool
+        variable_pool = VariablePool(
+            system_variables={},
+            user_inputs={},
+            environment_variables=workflow.environment_variables,
+        )
+
+        try:
+            variable_mapping = node_cls.extract_variable_selector_to_variable_mapping(
+                graph_config=workflow.graph_dict, config=loop_node_config
+            )
+        except NotImplementedError:
+            variable_mapping = {}
+
+        WorkflowEntry.mapping_user_inputs_to_variable_pool(
+            variable_mapping=variable_mapping,
+            user_inputs=user_inputs,
+            variable_pool=variable_pool,
+            tenant_id=workflow.tenant_id,
+        )
+
+        return graph, variable_pool
+
    def _handle_event(self, workflow_entry: WorkflowEntry, event: GraphEngineEvent) -> None:
        """
        Handle event
@@ -216,6 +315,7 @@ class WorkflowBasedAppRunner(AppRunner):
                    node_run_index=event.route_node_state.index,
                    predecessor_node_id=event.predecessor_node_id,
                    in_iteration_id=event.in_iteration_id,
+                    in_loop_id=event.in_loop_id,
                    parallel_mode_run_id=event.parallel_mode_run_id,
                    inputs=inputs,
                    process_data=process_data,
@@ -240,6 +340,7 @@ class WorkflowBasedAppRunner(AppRunner):
                    node_run_index=event.route_node_state.index,
                    predecessor_node_id=event.predecessor_node_id,
                    in_iteration_id=event.in_iteration_id,
+                    in_loop_id=event.in_loop_id,
                    parallel_mode_run_id=event.parallel_mode_run_id,
                    agent_strategy=event.agent_strategy,
                )
@@ -272,6 +373,7 @@ class WorkflowBasedAppRunner(AppRunner):
                    outputs=outputs,
                    execution_metadata=execution_metadata,
                    in_iteration_id=event.in_iteration_id,
+                    in_loop_id=event.in_loop_id,
                )
            )
        elif isinstance(event, NodeRunFailedEvent):
@@ -302,6 +404,7 @@ class WorkflowBasedAppRunner(AppRunner):
                    if event.route_node_state.node_run_result
                    else {},
                    in_iteration_id=event.in_iteration_id,
+                    in_loop_id=event.in_loop_id,
                )
            )
        elif isinstance(event, NodeRunExceptionEvent):
@@ -332,6 +435,7 @@ class WorkflowBasedAppRunner(AppRunner):
                    if event.route_node_state.node_run_result
                    else {},
                    in_iteration_id=event.in_iteration_id,
+                    in_loop_id=event.in_loop_id,
                )
            )
        elif isinstance(event, NodeInIterationFailedEvent):
@@ -362,18 +466,49 @@ class WorkflowBasedAppRunner(AppRunner):
                    error=event.error,
                )
            )
+        elif isinstance(event, NodeInLoopFailedEvent):
+            self._publish_event(
+                QueueNodeInLoopFailedEvent(
+                    node_execution_id=event.id,
+                    node_id=event.node_id,
+                    node_type=event.node_type,
+                    node_data=event.node_data,
+                    parallel_id=event.parallel_id,
+                    parallel_start_node_id=event.parallel_start_node_id,
+                    parent_parallel_id=event.parent_parallel_id,
+                    parent_parallel_start_node_id=event.parent_parallel_start_node_id,
+                    start_at=event.route_node_state.start_at,
+                    inputs=event.route_node_state.node_run_result.inputs
+                    if event.route_node_state.node_run_result
+                    else {},
+                    process_data=event.route_node_state.node_run_result.process_data
+                    if event.route_node_state.node_run_result
+                    else {},
+                    outputs=event.route_node_state.node_run_result.outputs or {}
+                    if event.route_node_state.node_run_result
+                    else {},
+                    execution_metadata=event.route_node_state.node_run_result.metadata
+                    if event.route_node_state.node_run_result
+                    else {},
+                    in_loop_id=event.in_loop_id,
+                    error=event.error,
+                )
+            )
        elif isinstance(event, NodeRunStreamChunkEvent):
            self._publish_event(
                QueueTextChunkEvent(
                    text=event.chunk_content,
                    from_variable_selector=event.from_variable_selector,
                    in_iteration_id=event.in_iteration_id,
+                    in_loop_id=event.in_loop_id,
                )
            )
        elif isinstance(event, NodeRunRetrieverResourceEvent):
            self._publish_event(
                QueueRetrieverResourcesEvent(
-                    retriever_resources=event.retriever_resources, in_iteration_id=event.in_iteration_id
+                    retriever_resources=event.retriever_resources,
+                    in_iteration_id=event.in_iteration_id,
+                    in_loop_id=event.in_loop_id,
                )
            )
        elif isinstance(event, AgentLogEvent):
@@ -387,6 +522,7 @@ class WorkflowBasedAppRunner(AppRunner):
                    status=event.status,
                    data=event.data,
                    metadata=event.metadata,
+                    node_id=event.node_id,
                )
            )
        elif isinstance(event, ParallelBranchRunStartedEvent):
@@ -397,6 +533,7 @@ class WorkflowBasedAppRunner(AppRunner):
                    parent_parallel_id=event.parent_parallel_id,
                    parent_parallel_start_node_id=event.parent_parallel_start_node_id,
                    in_iteration_id=event.in_iteration_id,
+                    in_loop_id=event.in_loop_id,
                )
            )
        elif isinstance(event, ParallelBranchRunSucceededEvent):
@@ -407,6 +544,7 @@ class WorkflowBasedAppRunner(AppRunner):
                    parent_parallel_id=event.parent_parallel_id,
                    parent_parallel_start_node_id=event.parent_parallel_start_node_id,
                    in_iteration_id=event.in_iteration_id,
+                    in_loop_id=event.in_loop_id,
                )
            )
        elif isinstance(event, ParallelBranchRunFailedEvent):
@@ -417,6 +555,7 @@ class WorkflowBasedAppRunner(AppRunner):
                    parent_parallel_id=event.parent_parallel_id,
                    parent_parallel_start_node_id=event.parent_parallel_start_node_id,
                    in_iteration_id=event.in_iteration_id,
+                    in_loop_id=event.in_loop_id,
                    error=event.error,
                )
            )
@@ -476,6 +615,62 @@ class WorkflowBasedAppRunner(AppRunner):
                    error=event.error if isinstance(event, IterationRunFailedEvent) else None,
                )
            )
+        elif isinstance(event, LoopRunStartedEvent):
+            self._publish_event(
+                QueueLoopStartEvent(
+                    node_execution_id=event.loop_id,
+                    node_id=event.loop_node_id,
+                    node_type=event.loop_node_type,
+                    node_data=event.loop_node_data,
+                    parallel_id=event.parallel_id,
+                    parallel_start_node_id=event.parallel_start_node_id,
+                    parent_parallel_id=event.parent_parallel_id,
+                    parent_parallel_start_node_id=event.parent_parallel_start_node_id,
+                    start_at=event.start_at,
+                    node_run_index=workflow_entry.graph_engine.graph_runtime_state.node_run_steps,
+                    inputs=event.inputs,
+                    predecessor_node_id=event.predecessor_node_id,
+                    metadata=event.metadata,
+                )
+            )
+        elif isinstance(event, LoopRunNextEvent):
+            self._publish_event(
+                QueueLoopNextEvent(
+                    node_execution_id=event.loop_id,
+                    node_id=event.loop_node_id,
+                    node_type=event.loop_node_type,
+                    node_data=event.loop_node_data,
+                    parallel_id=event.parallel_id,
+                    parallel_start_node_id=event.parallel_start_node_id,
+                    parent_parallel_id=event.parent_parallel_id,
+                    parent_parallel_start_node_id=event.parent_parallel_start_node_id,
+                    index=event.index,
+                    node_run_index=workflow_entry.graph_engine.graph_runtime_state.node_run_steps,
+                    output=event.pre_loop_output,
+                    parallel_mode_run_id=event.parallel_mode_run_id,
+                    duration=event.duration,
+                )
+            )
+        elif isinstance(event, (LoopRunSucceededEvent | LoopRunFailedEvent)):
+            self._publish_event(
+                QueueLoopCompletedEvent(
+                    node_execution_id=event.loop_id,
+                    node_id=event.loop_node_id,
+                    node_type=event.loop_node_type,
+                    node_data=event.loop_node_data,
+                    parallel_id=event.parallel_id,
+                    parallel_start_node_id=event.parallel_start_node_id,
+                    parent_parallel_id=event.parent_parallel_id,
+                    parent_parallel_start_node_id=event.parent_parallel_start_node_id,
+                    start_at=event.start_at,
+                    node_run_index=workflow_entry.graph_engine.graph_runtime_state.node_run_steps,
+                    inputs=event.inputs,
+                    outputs=event.outputs,
+                    metadata=event.metadata,
+                    steps=event.steps,
+                    error=event.error if isinstance(event, LoopRunFailedEvent) else None,
+                )
+            )

    def get_workflow(self, app_model: App, workflow_id: str) -> Optional[Workflow]:
        """
@@ -187,6 +187,16 @@ class AdvancedChatAppGenerateEntity(ConversationAppGenerateEntity):

    single_iteration_run: Optional[SingleIterationRunEntity] = None

+    class SingleLoopRunEntity(BaseModel):
+        """
+        Single Loop Run Entity.
+        """
+
+        node_id: str
+        inputs: Mapping
+
+    single_loop_run: Optional[SingleLoopRunEntity] = None
+

 class WorkflowAppGenerateEntity(AppGenerateEntity):
    """
@@ -206,3 +216,13 @@ class WorkflowAppGenerateEntity(AppGenerateEntity):
        inputs: dict

    single_iteration_run: Optional[SingleIterationRunEntity] = None
+
+    class SingleLoopRunEntity(BaseModel):
+        """
+        Single Loop Run Entity.
+        """
+
+        node_id: str
+        inputs: dict
+
+    single_loop_run: Optional[SingleLoopRunEntity] = None
@@ -30,6 +30,9 @@ class QueueEvent(StrEnum):
    ITERATION_START = "iteration_start"
    ITERATION_NEXT = "iteration_next"
    ITERATION_COMPLETED = "iteration_completed"
+    LOOP_START = "loop_start"
+    LOOP_NEXT = "loop_next"
+    LOOP_COMPLETED = "loop_completed"
    NODE_STARTED = "node_started"
    NODE_SUCCEEDED = "node_succeeded"
    NODE_FAILED = "node_failed"
@@ -149,6 +152,89 @@ class QueueIterationCompletedEvent(AppQueueEvent):
    error: Optional[str] = None


+class QueueLoopStartEvent(AppQueueEvent):
+    """
+    QueueLoopStartEvent entity
+    """
+
+    event: QueueEvent = QueueEvent.LOOP_START
+    node_execution_id: str
+    node_id: str
+    node_type: NodeType
+    node_data: BaseNodeData
+    parallel_id: Optional[str] = None
+    """parallel id if node is in parallel"""
+    parallel_start_node_id: Optional[str] = None
+    """parallel start node id if node is in parallel"""
+    parent_parallel_id: Optional[str] = None
+    """parent parallel id if node is in parallel"""
+    parent_parallel_start_node_id: Optional[str] = None
+    """parent parallel start node id if node is in parallel"""
+    start_at: datetime
+
+    node_run_index: int
+    inputs: Optional[Mapping[str, Any]] = None
+    predecessor_node_id: Optional[str] = None
+    metadata: Optional[Mapping[str, Any]] = None
+
+
+class QueueLoopNextEvent(AppQueueEvent):
+    """
+    QueueLoopNextEvent entity
+    """
+
+    event: QueueEvent = QueueEvent.LOOP_NEXT
+
+    index: int
+    node_execution_id: str
+    node_id: str
+    node_type: NodeType
+    node_data: BaseNodeData
+    parallel_id: Optional[str] = None
+    """parallel id if node is in parallel"""
+    parallel_start_node_id: Optional[str] = None
+    """parallel start node id if node is in parallel"""
+    parent_parallel_id: Optional[str] = None
+    """parent parallel id if node is in parallel"""
+    parent_parallel_start_node_id: Optional[str] = None
+    """parent parallel start node id if node is in parallel"""
+    parallel_mode_run_id: Optional[str] = None
+    """iteratoin run in parallel mode run id"""
+    node_run_index: int
+    output: Optional[Any] = None  # output for the current loop
+    duration: Optional[float] = None
+
+
+class QueueLoopCompletedEvent(AppQueueEvent):
+    """
+    QueueLoopCompletedEvent entity
+    """
+
+    event: QueueEvent = QueueEvent.LOOP_COMPLETED
+
+    node_execution_id: str
+    node_id: str
+    node_type: NodeType
+    node_data: BaseNodeData
+    parallel_id: Optional[str] = None
+    """parallel id if node is in parallel"""
+    parallel_start_node_id: Optional[str] = None
+    """parallel start node id if node is in parallel"""
+    parent_parallel_id: Optional[str] = None
+    """parent parallel id if node is in parallel"""
+    parent_parallel_start_node_id: Optional[str] = None
+    """parent parallel start node id if node is in parallel"""
+    start_at: datetime
+
+    node_run_index: int
+    inputs: Optional[Mapping[str, Any]] = None
+    outputs: Optional[Mapping[str, Any]] = None
+    metadata: Optional[Mapping[str, Any]] = None
+    steps: int = 0
+
+    error: Optional[str] = None
+
+
 class QueueTextChunkEvent(AppQueueEvent):
    """
    QueueTextChunkEvent entity
@@ -160,6 +246,8 @@ class QueueTextChunkEvent(AppQueueEvent):
    """from variable selector"""
    in_iteration_id: Optional[str] = None
    """iteration id if node is in iteration"""
+    in_loop_id: Optional[str] = None
+    """loop id if node is in loop"""


 class QueueAgentMessageEvent(AppQueueEvent):
@@ -189,6 +277,8 @@ class QueueRetrieverResourcesEvent(AppQueueEvent):
    retriever_resources: list[dict]
    in_iteration_id: Optional[str] = None
    """iteration id if node is in iteration"""
+    in_loop_id: Optional[str] = None
+    """loop id if node is in loop"""


 class QueueAnnotationReplyEvent(AppQueueEvent):
@@ -278,6 +368,8 @@ class QueueNodeStartedEvent(AppQueueEvent):
    """parent parallel start node id if node is in parallel"""
    in_iteration_id: Optional[str] = None
    """iteration id if node is in iteration"""
+    in_loop_id: Optional[str] = None
+    """loop id if node is in loop"""
    start_at: datetime
    parallel_mode_run_id: Optional[str] = None
    """iteratoin run in parallel mode run id"""
@@ -305,6 +397,8 @@ class QueueNodeSucceededEvent(AppQueueEvent):
    """parent parallel start node id if node is in parallel"""
    in_iteration_id: Optional[str] = None
    """iteration id if node is in iteration"""
+    in_loop_id: Optional[str] = None
+    """loop id if node is in loop"""
    start_at: datetime

    inputs: Optional[Mapping[str, Any]] = None
@@ -315,6 +409,8 @@ class QueueNodeSucceededEvent(AppQueueEvent):
    error: Optional[str] = None
    """single iteration duration map"""
    iteration_duration_map: Optional[dict[str, float]] = None
+    """single loop duration map"""
+    loop_duration_map: Optional[dict[str, float]] = None


 class QueueAgentLogEvent(AppQueueEvent):
@@ -331,6 +427,7 @@ class QueueAgentLogEvent(AppQueueEvent):
    status: str
    data: Mapping[str, Any]
    metadata: Optional[Mapping[str, Any]] = None
+    node_id: str


 class QueueNodeRetryEvent(QueueNodeStartedEvent):
@@ -368,6 +465,41 @@ class QueueNodeInIterationFailedEvent(AppQueueEvent):
    """parent parallel start node id if node is in parallel"""
    in_iteration_id: Optional[str] = None
    """iteration id if node is in iteration"""
+    in_loop_id: Optional[str] = None
+    """loop id if node is in loop"""
+    start_at: datetime
+
+    inputs: Optional[Mapping[str, Any]] = None
+    process_data: Optional[Mapping[str, Any]] = None
+    outputs: Optional[Mapping[str, Any]] = None
+    execution_metadata: Optional[Mapping[NodeRunMetadataKey, Any]] = None
+
+    error: str
+
+
+class QueueNodeInLoopFailedEvent(AppQueueEvent):
+    """
+    QueueNodeInLoopFailedEvent entity
+    """
+
+    event: QueueEvent = QueueEvent.NODE_FAILED
+
+    node_execution_id: str
+    node_id: str
+    node_type: NodeType
+    node_data: BaseNodeData
+    parallel_id: Optional[str] = None
+    """parallel id if node is in parallel"""
+    parallel_start_node_id: Optional[str] = None
+    """parallel start node id if node is in parallel"""
+    parent_parallel_id: Optional[str] = None
+    """parent parallel id if node is in parallel"""
+    parent_parallel_start_node_id: Optional[str] = None
+    """parent parallel start node id if node is in parallel"""
+    in_iteration_id: Optional[str] = None
+    """iteration id if node is in iteration"""
+    in_loop_id: Optional[str] = None
+    """loop id if node is in loop"""
    start_at: datetime

    inputs: Optional[Mapping[str, Any]] = None
@@ -399,6 +531,8 @@ class QueueNodeExceptionEvent(AppQueueEvent):
    """parent parallel start node id if node is in parallel"""
    in_iteration_id: Optional[str] = None
    """iteration id if node is in iteration"""
+    in_loop_id: Optional[str] = None
+    """loop id if node is in loop"""
    start_at: datetime

    inputs: Optional[Mapping[str, Any]] = None
@@ -430,6 +564,8 @@ class QueueNodeFailedEvent(AppQueueEvent):
    """parent parallel start node id if node is in parallel"""
    in_iteration_id: Optional[str] = None
    """iteration id if node is in iteration"""
+    in_loop_id: Optional[str] = None
+    """loop id if node is in loop"""
    start_at: datetime

    inputs: Optional[Mapping[str, Any]] = None
@@ -549,6 +685,8 @@ class QueueParallelBranchRunStartedEvent(AppQueueEvent):
    """parent parallel start node id if node is in parallel"""
    in_iteration_id: Optional[str] = None
    """iteration id if node is in iteration"""
+    in_loop_id: Optional[str] = None
+    """loop id if node is in loop"""


 class QueueParallelBranchRunSucceededEvent(AppQueueEvent):
@@ -566,6 +704,8 @@ class QueueParallelBranchRunSucceededEvent(AppQueueEvent):
    """parent parallel start node id if node is in parallel"""
    in_iteration_id: Optional[str] = None
    """iteration id if node is in iteration"""
+    in_loop_id: Optional[str] = None
+    """loop id if node is in loop"""


 class QueueParallelBranchRunFailedEvent(AppQueueEvent):
@@ -583,4 +723,6 @@ class QueueParallelBranchRunFailedEvent(AppQueueEvent):
    """parent parallel start node id if node is in parallel"""
    in_iteration_id: Optional[str] = None
    """iteration id if node is in iteration"""
+    in_loop_id: Optional[str] = None
+    """loop id if node is in loop"""
    error: str
@@ -59,6 +59,9 @@ class StreamEvent(Enum):
    ITERATION_STARTED = "iteration_started"
    ITERATION_NEXT = "iteration_next"
    ITERATION_COMPLETED = "iteration_completed"
+    LOOP_STARTED = "loop_started"
+    LOOP_NEXT = "loop_next"
+    LOOP_COMPLETED = "loop_completed"
    TEXT_CHUNK = "text_chunk"
    TEXT_REPLACE = "text_replace"
    AGENT_LOG = "agent_log"
@@ -248,6 +251,7 @@ class NodeStartStreamResponse(StreamResponse):
        parent_parallel_id: Optional[str] = None
        parent_parallel_start_node_id: Optional[str] = None
        iteration_id: Optional[str] = None
+        loop_id: Optional[str] = None
        parallel_run_id: Optional[str] = None
        agent_strategy: Optional[AgentNodeStrategyInit] = None

@@ -275,6 +279,7 @@ class NodeStartStreamResponse(StreamResponse):
                "parent_parallel_id": self.data.parent_parallel_id,
                "parent_parallel_start_node_id": self.data.parent_parallel_start_node_id,
                "iteration_id": self.data.iteration_id,
+                "loop_id": self.data.loop_id,
            },
        }

@@ -310,6 +315,7 @@ class NodeFinishStreamResponse(StreamResponse):
        parent_parallel_id: Optional[str] = None
        parent_parallel_start_node_id: Optional[str] = None
        iteration_id: Optional[str] = None
+        loop_id: Optional[str] = None

    event: StreamEvent = StreamEvent.NODE_FINISHED
    workflow_run_id: str
@@ -342,6 +348,7 @@ class NodeFinishStreamResponse(StreamResponse):
                "parent_parallel_id": self.data.parent_parallel_id,
                "parent_parallel_start_node_id": self.data.parent_parallel_start_node_id,
                "iteration_id": self.data.iteration_id,
+                "loop_id": self.data.loop_id,
            },
        }

@@ -377,6 +384,7 @@ class NodeRetryStreamResponse(StreamResponse):
        parent_parallel_id: Optional[str] = None
        parent_parallel_start_node_id: Optional[str] = None
        iteration_id: Optional[str] = None
+        loop_id: Optional[str] = None
        retry_index: int = 0

    event: StreamEvent = StreamEvent.NODE_RETRY
@@ -410,6 +418,7 @@ class NodeRetryStreamResponse(StreamResponse):
                "parent_parallel_id": self.data.parent_parallel_id,
                "parent_parallel_start_node_id": self.data.parent_parallel_start_node_id,
                "iteration_id": self.data.iteration_id,
+                "loop_id": self.data.loop_id,
                "retry_index": self.data.retry_index,
            },
        }
@@ -430,6 +439,7 @@ class ParallelBranchStartStreamResponse(StreamResponse):
        parent_parallel_id: Optional[str] = None
        parent_parallel_start_node_id: Optional[str] = None
        iteration_id: Optional[str] = None
+        loop_id: Optional[str] = None
        created_at: int

    event: StreamEvent = StreamEvent.PARALLEL_BRANCH_STARTED
@@ -452,6 +462,7 @@ class ParallelBranchFinishedStreamResponse(StreamResponse):
        parent_parallel_id: Optional[str] = None
        parent_parallel_start_node_id: Optional[str] = None
        iteration_id: Optional[str] = None
+        loop_id: Optional[str] = None
        status: str
        error: Optional[str] = None
        created_at: int
@@ -548,6 +559,93 @@ class IterationNodeCompletedStreamResponse(StreamResponse):
    data: Data


+class LoopNodeStartStreamResponse(StreamResponse):
+    """
+    NodeStartStreamResponse entity
+    """
+
+    class Data(BaseModel):
+        """
+        Data entity
+        """
+
+        id: str
+        node_id: str
+        node_type: str
+        title: str
+        created_at: int
+        extras: dict = {}
+        metadata: Mapping = {}
+        inputs: Mapping = {}
+        parallel_id: Optional[str] = None
+        parallel_start_node_id: Optional[str] = None
+
+    event: StreamEvent = StreamEvent.LOOP_STARTED
+    workflow_run_id: str
+    data: Data
+
+
+class LoopNodeNextStreamResponse(StreamResponse):
+    """
+    NodeStartStreamResponse entity
+    """
+
+    class Data(BaseModel):
+        """
+        Data entity
+        """
+
+        id: str
+        node_id: str
+        node_type: str
+        title: str
+        index: int
+        created_at: int
+        pre_loop_output: Optional[Any] = None
+        extras: dict = {}
+        parallel_id: Optional[str] = None
+        parallel_start_node_id: Optional[str] = None
+        parallel_mode_run_id: Optional[str] = None
+        duration: Optional[float] = None
+
+    event: StreamEvent = StreamEvent.LOOP_NEXT
+    workflow_run_id: str
+    data: Data
+
+
+class LoopNodeCompletedStreamResponse(StreamResponse):
+    """
+    NodeCompletedStreamResponse entity
+    """
+
+    class Data(BaseModel):
+        """
+        Data entity
+        """
+
+        id: str
+        node_id: str
+        node_type: str
+        title: str
+        outputs: Optional[Mapping] = None
+        created_at: int
+        extras: Optional[dict] = None
+        inputs: Optional[Mapping] = None
+        status: WorkflowNodeExecutionStatus
+        error: Optional[str] = None
+        elapsed_time: float
+        total_tokens: int
+        execution_metadata: Optional[Mapping] = None
+        finished_at: int
+        steps: int
+        parallel_id: Optional[str] = None
+        parallel_start_node_id: Optional[str] = None
+
+    event: StreamEvent = StreamEvent.LOOP_COMPLETED
+    workflow_run_id: str
+    data: Data
+
+
 class TextChunkStreamResponse(StreamResponse):
    """
    TextChunkStreamResponse entity
@@ -719,6 +817,7 @@ class AgentLogStreamResponse(StreamResponse):
        status: str
        data: Mapping[str, Any]
        metadata: Optional[Mapping[str, Any]] = None
+        node_id: str

    event: StreamEvent = StreamEvent.AGENT_LOG
    data: Data
@@ -14,9 +14,13 @@ from core.app.entities.queue_entities import (
    QueueIterationCompletedEvent,
    QueueIterationNextEvent,
    QueueIterationStartEvent,
+    QueueLoopCompletedEvent,
+    QueueLoopNextEvent,
+    QueueLoopStartEvent,
    QueueNodeExceptionEvent,
    QueueNodeFailedEvent,
    QueueNodeInIterationFailedEvent,
+    QueueNodeInLoopFailedEvent,
    QueueNodeRetryEvent,
    QueueNodeStartedEvent,
    QueueNodeSucceededEvent,
@@ -29,6 +33,9 @@ from core.app.entities.task_entities import (
    IterationNodeCompletedStreamResponse,
    IterationNodeNextStreamResponse,
    IterationNodeStartStreamResponse,
+    LoopNodeCompletedStreamResponse,
+    LoopNodeNextStreamResponse,
+    LoopNodeStartStreamResponse,
    NodeFinishStreamResponse,
    NodeRetryStreamResponse,
    NodeStartStreamResponse,
@@ -304,6 +311,7 @@ class WorkflowCycleManage:
            {
                NodeRunMetadataKey.PARALLEL_MODE_RUN_ID: event.parallel_mode_run_id,
                NodeRunMetadataKey.ITERATION_ID: event.in_iteration_id,
+                NodeRunMetadataKey.LOOP_ID: event.in_loop_id,
            }
        )
        workflow_node_execution.created_at = datetime.now(UTC).replace(tzinfo=None)
@@ -344,7 +352,10 @@ class WorkflowCycleManage:
        self,
        *,
        session: Session,
-        event: QueueNodeFailedEvent | QueueNodeInIterationFailedEvent | QueueNodeExceptionEvent,
+        event: QueueNodeFailedEvent
+        | QueueNodeInIterationFailedEvent
+        | QueueNodeInLoopFailedEvent
+        | QueueNodeExceptionEvent,
    ) -> WorkflowNodeExecution:
        """
        Workflow node execution failed
@@ -396,6 +407,7 @@ class WorkflowCycleManage:
        origin_metadata = {
            NodeRunMetadataKey.ITERATION_ID: event.in_iteration_id,
            NodeRunMetadataKey.PARALLEL_MODE_RUN_ID: event.parallel_mode_run_id,
+            NodeRunMetadataKey.LOOP_ID: event.in_loop_id,
        }
        merged_metadata = (
            {**jsonable_encoder(event.execution_metadata), **origin_metadata}
@@ -540,6 +552,7 @@ class WorkflowCycleManage:
                parent_parallel_id=event.parent_parallel_id,
                parent_parallel_start_node_id=event.parent_parallel_start_node_id,
                iteration_id=event.in_iteration_id,
+                loop_id=event.in_loop_id,
                parallel_run_id=event.parallel_mode_run_id,
                agent_strategy=event.agent_strategy,
            ),
@@ -563,6 +576,7 @@ class WorkflowCycleManage:
        event: QueueNodeSucceededEvent
        | QueueNodeFailedEvent
        | QueueNodeInIterationFailedEvent
+        | QueueNodeInLoopFailedEvent
        | QueueNodeExceptionEvent,
        task_id: str,
        workflow_node_execution: WorkflowNodeExecution,
@@ -601,6 +615,7 @@ class WorkflowCycleManage:
                parent_parallel_id=event.parent_parallel_id,
                parent_parallel_start_node_id=event.parent_parallel_start_node_id,
                iteration_id=event.in_iteration_id,
+                loop_id=event.in_loop_id,
            ),
        )

@@ -646,6 +661,7 @@ class WorkflowCycleManage:
                parent_parallel_id=event.parent_parallel_id,
                parent_parallel_start_node_id=event.parent_parallel_start_node_id,
                iteration_id=event.in_iteration_id,
+                loop_id=event.in_loop_id,
                retry_index=event.retry_index,
            ),
        )
@@ -664,6 +680,7 @@ class WorkflowCycleManage:
                parent_parallel_id=event.parent_parallel_id,
                parent_parallel_start_node_id=event.parent_parallel_start_node_id,
                iteration_id=event.in_iteration_id,
+                loop_id=event.in_loop_id,
                created_at=int(time.time()),
            ),
        )
@@ -687,6 +704,7 @@ class WorkflowCycleManage:
                parent_parallel_id=event.parent_parallel_id,
                parent_parallel_start_node_id=event.parent_parallel_start_node_id,
                iteration_id=event.in_iteration_id,
+                loop_id=event.in_loop_id,
                status="succeeded" if isinstance(event, QueueParallelBranchRunSucceededEvent) else "failed",
                error=event.error if isinstance(event, QueueParallelBranchRunFailedEvent) else None,
                created_at=int(time.time()),
@@ -770,6 +788,83 @@ class WorkflowCycleManage:
            ),
        )

+    def _workflow_loop_start_to_stream_response(
+        self, *, session: Session, task_id: str, workflow_run: WorkflowRun, event: QueueLoopStartEvent
+    ) -> LoopNodeStartStreamResponse:
+        # receive session to make sure the workflow_run won't be expired, need a more elegant way to handle this
+        _ = session
+        return LoopNodeStartStreamResponse(
+            task_id=task_id,
+            workflow_run_id=workflow_run.id,
+            data=LoopNodeStartStreamResponse.Data(
+                id=event.node_id,
+                node_id=event.node_id,
+                node_type=event.node_type.value,
+                title=event.node_data.title,
+                created_at=int(time.time()),
+                extras={},
+                inputs=event.inputs or {},
+                metadata=event.metadata or {},
+                parallel_id=event.parallel_id,
+                parallel_start_node_id=event.parallel_start_node_id,
+            ),
+        )
+
+    def _workflow_loop_next_to_stream_response(
+        self, *, session: Session, task_id: str, workflow_run: WorkflowRun, event: QueueLoopNextEvent
+    ) -> LoopNodeNextStreamResponse:
+        # receive session to make sure the workflow_run won't be expired, need a more elegant way to handle this
+        _ = session
+        return LoopNodeNextStreamResponse(
+            task_id=task_id,
+            workflow_run_id=workflow_run.id,
+            data=LoopNodeNextStreamResponse.Data(
+                id=event.node_id,
+                node_id=event.node_id,
+                node_type=event.node_type.value,
+                title=event.node_data.title,
+                index=event.index,
+                pre_loop_output=event.output,
+                created_at=int(time.time()),
+                extras={},
+                parallel_id=event.parallel_id,
+                parallel_start_node_id=event.parallel_start_node_id,
+                parallel_mode_run_id=event.parallel_mode_run_id,
+                duration=event.duration,
+            ),
+        )
+
+    def _workflow_loop_completed_to_stream_response(
+        self, *, session: Session, task_id: str, workflow_run: WorkflowRun, event: QueueLoopCompletedEvent
+    ) -> LoopNodeCompletedStreamResponse:
+        # receive session to make sure the workflow_run won't be expired, need a more elegant way to handle this
+        _ = session
+        return LoopNodeCompletedStreamResponse(
+            task_id=task_id,
+            workflow_run_id=workflow_run.id,
+            data=LoopNodeCompletedStreamResponse.Data(
+                id=event.node_id,
+                node_id=event.node_id,
+                node_type=event.node_type.value,
+                title=event.node_data.title,
+                outputs=event.outputs,
+                created_at=int(time.time()),
+                extras={},
+                inputs=event.inputs or {},
+                status=WorkflowNodeExecutionStatus.SUCCEEDED
+                if event.error is None
+                else WorkflowNodeExecutionStatus.FAILED,
+                error=None,
+                elapsed_time=(datetime.now(UTC).replace(tzinfo=None) - event.start_at).total_seconds(),
+                total_tokens=event.metadata.get("total_tokens", 0) if event.metadata else 0,
+                execution_metadata=event.metadata,
+                finished_at=int(time.time()),
+                steps=event.steps,
+                parallel_id=event.parallel_id,
+                parallel_start_node_id=event.parallel_start_node_id,
+            ),
+        )
+
    def _fetch_files_from_node_outputs(self, outputs_dict: Mapping[str, Any]) -> Sequence[Mapping[str, Any]]:
        """
        Fetch files from node outputs
@@ -864,5 +959,6 @@ class WorkflowCycleManage:
                status=event.status,
                data=event.data,
                metadata=event.metadata,
+                node_id=event.node_id,
            ),
        )
@@ -1,9 +1,11 @@
 from core.app.apps.base_app_queue_manager import AppQueueManager, PublishFrom
 from core.app.entities.app_invoke_entities import InvokeFrom
 from core.app.entities.queue_entities import QueueRetrieverResourcesEvent
+from core.rag.index_processor.constant.index_type import IndexType
 from core.rag.models.document import Document
 from extensions.ext_database import db
-from models.dataset import DatasetQuery, DocumentSegment
+from models.dataset import ChildChunk, DatasetQuery, DocumentSegment
+from models.dataset import Document as DatasetDocument
 from models.model import DatasetRetrieverResource


@@ -41,15 +43,29 @@ class DatasetIndexToolCallbackHandler:
        """Handle tool end."""
        for document in documents:
            if document.metadata is not None:
-                query = db.session.query(DocumentSegment).filter(
-                    DocumentSegment.index_node_id == document.metadata["doc_id"]
-                )
+                dataset_document = DatasetDocument.query.filter(
+                    DatasetDocument.id == document.metadata["document_id"]
+                ).first()
+                if dataset_document.doc_form == IndexType.PARENT_CHILD_INDEX:
+                    child_chunk = ChildChunk.query.filter(
+                        ChildChunk.index_node_id == document.metadata["doc_id"],
+                        ChildChunk.dataset_id == dataset_document.dataset_id,
+                        ChildChunk.document_id == dataset_document.id,
+                    ).first()
+                    if child_chunk:
+                        segment = DocumentSegment.query.filter(DocumentSegment.id == child_chunk.segment_id).update(
+                            {DocumentSegment.hit_count: DocumentSegment.hit_count + 1}, synchronize_session=False
+                        )
+                else:
+                    query = db.session.query(DocumentSegment).filter(
+                        DocumentSegment.index_node_id == document.metadata["doc_id"]
+                    )

-                if "dataset_id" in document.metadata:
-                    query = query.filter(DocumentSegment.dataset_id == document.metadata["dataset_id"])
+                    if "dataset_id" in document.metadata:
+                        query = query.filter(DocumentSegment.dataset_id == document.metadata["dataset_id"])

-                # add hit count to document segment
-                query.update({DocumentSegment.hit_count: DocumentSegment.hit_count + 1}, synchronize_session=False)
+                    # add hit count to document segment
+                    query.update({DocumentSegment.hit_count: DocumentSegment.hit_count + 1}, synchronize_session=False)

                db.session.commit()

@@ -7,7 +7,6 @@ from json import JSONDecodeError
 from typing import Optional

 from pydantic import BaseModel, ConfigDict, Field
-from sqlalchemy import or_

 from constants import HIDDEN_VALUE
 from core.entities.model_entities import ModelStatus, ModelWithProviderEntity, SimpleModelProviderEntity
@@ -180,25 +179,35 @@ class ProviderConfiguration(BaseModel):
            else [],
        )

+    def _get_custom_provider_credentials(self) -> Provider | None:
+        """
+        Get custom provider credentials.
+        """
+        # get provider
+        model_provider_id = ModelProviderID(self.provider.provider)
+        provider_names = [self.provider.provider]
+        if model_provider_id.is_langgenius():
+            provider_names.append(model_provider_id.provider_name)
+
+        provider_record = (
+            db.session.query(Provider)
+            .filter(
+                Provider.tenant_id == self.tenant_id,
+                Provider.provider_type == ProviderType.CUSTOM.value,
+                Provider.provider_name.in_(provider_names),
+            )
+            .first()
+        )
+
+        return provider_record
+
    def custom_credentials_validate(self, credentials: dict) -> tuple[Provider | None, dict]:
        """
        Validate custom credentials.
        :param credentials: provider credentials
        :return:
        """
-        # get provider
-        provider_record = (
-            db.session.query(Provider)
-            .filter(
-                Provider.tenant_id == self.tenant_id,
-                Provider.provider_type == ProviderType.CUSTOM.value,
-                or_(
-                    Provider.provider_name == ModelProviderID(self.provider.provider).plugin_name,
-                    Provider.provider_name == self.provider.provider,
-                ),
-            )
-            .first()
-        )
+        provider_record = self._get_custom_provider_credentials()

        # Get provider credential secret variables
        provider_credential_secret_variables = self.extract_secret_variables(
@@ -279,18 +288,7 @@ class ProviderConfiguration(BaseModel):
        :return:
        """
        # get provider
-        provider_record = (
-            db.session.query(Provider)
-            .filter(
-                Provider.tenant_id == self.tenant_id,
-                or_(
-                    Provider.provider_name == ModelProviderID(self.provider.provider).plugin_name,
-                    Provider.provider_name == self.provider.provider,
-                ),
-                Provider.provider_type == ProviderType.CUSTOM.value,
-            )
-            .first()
-        )
+        provider_record = self._get_custom_provider_credentials()

        # delete provider
        if provider_record:
@@ -337,6 +335,33 @@ class ProviderConfiguration(BaseModel):

        return None

+    def _get_custom_model_credentials(
+        self,
+        model_type: ModelType,
+        model: str,
+    ) -> ProviderModel | None:
+        """
+        Get custom model credentials.
+        """
+        # get provider model
+        model_provider_id = ModelProviderID(self.provider.provider)
+        provider_names = [self.provider.provider]
+        if model_provider_id.is_langgenius():
+            provider_names.append(model_provider_id.provider_name)
+
+        provider_model_record = (
+            db.session.query(ProviderModel)
+            .filter(
+                ProviderModel.tenant_id == self.tenant_id,
+                ProviderModel.provider_name.in_(provider_names),
+                ProviderModel.model_name == model,
+                ProviderModel.model_type == model_type.to_origin_model_type(),
+            )
+            .first()
+        )
+
+        return provider_model_record
+
    def custom_model_credentials_validate(
        self, model_type: ModelType, model: str, credentials: dict
    ) -> tuple[ProviderModel | None, dict]:
@@ -349,16 +374,7 @@ class ProviderConfiguration(BaseModel):
        :return:
        """
        # get provider model
-        provider_model_record = (
-            db.session.query(ProviderModel)
-            .filter(
-                ProviderModel.tenant_id == self.tenant_id,
-                ProviderModel.provider_name == self.provider.provider,
-                ProviderModel.model_name == model,
-                ProviderModel.model_type == model_type.to_origin_model_type(),
-            )
-            .first()
-        )
+        provider_model_record = self._get_custom_model_credentials(model_type, model)

        # Get provider credential secret variables
        provider_credential_secret_variables = self.extract_secret_variables(
@@ -439,16 +455,7 @@ class ProviderConfiguration(BaseModel):
        :return:
        """
        # get provider model
-        provider_model_record = (
-            db.session.query(ProviderModel)
-            .filter(
-                ProviderModel.tenant_id == self.tenant_id,
-                ProviderModel.provider_name == self.provider.provider,
-                ProviderModel.model_name == model,
-                ProviderModel.model_type == model_type.to_origin_model_type(),
-            )
-            .first()
-        )
+        provider_model_record = self._get_custom_model_credentials(model_type, model)

        # delete provider model
        if provider_model_record:
@@ -463,6 +470,26 @@ class ProviderConfiguration(BaseModel):

            provider_model_credentials_cache.delete()

+    def _get_provider_model_setting(self, model_type: ModelType, model: str) -> ProviderModelSetting | None:
+        """
+        Get provider model setting.
+        """
+        model_provider_id = ModelProviderID(self.provider.provider)
+        provider_names = [self.provider.provider]
+        if model_provider_id.is_langgenius():
+            provider_names.append(model_provider_id.provider_name)
+
+        return (
+            db.session.query(ProviderModelSetting)
+            .filter(
+                ProviderModelSetting.tenant_id == self.tenant_id,
+                ProviderModelSetting.provider_name.in_(provider_names),
+                ProviderModelSetting.model_type == model_type.to_origin_model_type(),
+                ProviderModelSetting.model_name == model,
+            )
+            .first()
+        )
+
    def enable_model(self, model_type: ModelType, model: str) -> ProviderModelSetting:
        """
        Enable model.
@@ -470,16 +497,7 @@ class ProviderConfiguration(BaseModel):
        :param model: model name
        :return:
        """
-        model_setting = (
-            db.session.query(ProviderModelSetting)
-            .filter(
-                ProviderModelSetting.tenant_id == self.tenant_id,
-                ProviderModelSetting.provider_name == self.provider.provider,
-                ProviderModelSetting.model_type == model_type.to_origin_model_type(),
-                ProviderModelSetting.model_name == model,
-            )
-            .first()
-        )
+        model_setting = self._get_provider_model_setting(model_type, model)

        if model_setting:
            model_setting.enabled = True
@@ -504,16 +522,7 @@ class ProviderConfiguration(BaseModel):
        :param model: model name
        :return:
        """
-        model_setting = (
-            db.session.query(ProviderModelSetting)
-            .filter(
-                ProviderModelSetting.tenant_id == self.tenant_id,
-                ProviderModelSetting.provider_name == self.provider.provider,
-                ProviderModelSetting.model_type == model_type.to_origin_model_type(),
-                ProviderModelSetting.model_name == model,
-            )
-            .first()
-        )
+        model_setting = self._get_provider_model_setting(model_type, model)

        if model_setting:
            model_setting.enabled = False
@@ -538,13 +547,24 @@ class ProviderConfiguration(BaseModel):
        :param model: model name
        :return:
        """
+        return self._get_provider_model_setting(model_type, model)
+
+    def _get_load_balancing_config(self, model_type: ModelType, model: str) -> Optional[LoadBalancingModelConfig]:
+        """
+        Get load balancing config.
+        """
+        model_provider_id = ModelProviderID(self.provider.provider)
+        provider_names = [self.provider.provider]
+        if model_provider_id.is_langgenius():
+            provider_names.append(model_provider_id.provider_name)
+
        return (
-            db.session.query(ProviderModelSetting)
+            db.session.query(LoadBalancingModelConfig)
            .filter(
-                ProviderModelSetting.tenant_id == self.tenant_id,
-                ProviderModelSetting.provider_name == self.provider.provider,
-                ProviderModelSetting.model_type == model_type.to_origin_model_type(),
-                ProviderModelSetting.model_name == model,
+                LoadBalancingModelConfig.tenant_id == self.tenant_id,
+                LoadBalancingModelConfig.provider_name.in_(provider_names),
+                LoadBalancingModelConfig.model_type == model_type.to_origin_model_type(),
+                LoadBalancingModelConfig.model_name == model,
            )
            .first()
        )
@@ -556,11 +576,16 @@ class ProviderConfiguration(BaseModel):
        :param model: model name
        :return:
        """
+        model_provider_id = ModelProviderID(self.provider.provider)
+        provider_names = [self.provider.provider]
+        if model_provider_id.is_langgenius():
+            provider_names.append(model_provider_id.provider_name)
+
        load_balancing_config_count = (
            db.session.query(LoadBalancingModelConfig)
            .filter(
                LoadBalancingModelConfig.tenant_id == self.tenant_id,
-                LoadBalancingModelConfig.provider_name == self.provider.provider,
+                LoadBalancingModelConfig.provider_name.in_(provider_names),
                LoadBalancingModelConfig.model_type == model_type.to_origin_model_type(),
                LoadBalancingModelConfig.model_name == model,
            )
@@ -570,16 +595,7 @@ class ProviderConfiguration(BaseModel):
        if load_balancing_config_count <= 1:
            raise ValueError("Model load balancing configuration must be more than 1.")

-        model_setting = (
-            db.session.query(ProviderModelSetting)
-            .filter(
-                ProviderModelSetting.tenant_id == self.tenant_id,
-                ProviderModelSetting.provider_name == self.provider.provider,
-                ProviderModelSetting.model_type == model_type.to_origin_model_type(),
-                ProviderModelSetting.model_name == model,
-            )
-            .first()
-        )
+        model_setting = self._get_provider_model_setting(model_type, model)

        if model_setting:
            model_setting.load_balancing_enabled = True
@@ -604,11 +620,16 @@ class ProviderConfiguration(BaseModel):
        :param model: model name
        :return:
        """
+        model_provider_id = ModelProviderID(self.provider.provider)
+        provider_names = [self.provider.provider]
+        if model_provider_id.is_langgenius():
+            provider_names.append(model_provider_id.provider_name)
+
        model_setting = (
            db.session.query(ProviderModelSetting)
            .filter(
                ProviderModelSetting.tenant_id == self.tenant_id,
-                ProviderModelSetting.provider_name == self.provider.provider,
+                ProviderModelSetting.provider_name.in_(provider_names),
                ProviderModelSetting.model_type == model_type.to_origin_model_type(),
                ProviderModelSetting.model_name == model,
            )
@@ -665,11 +686,16 @@ class ProviderConfiguration(BaseModel):
            return

        # get preferred provider
+        model_provider_id = ModelProviderID(self.provider.provider)
+        provider_names = [self.provider.provider]
+        if model_provider_id.is_langgenius():
+            provider_names.append(model_provider_id.provider_name)
+
        preferred_model_provider = (
            db.session.query(TenantPreferredModelProvider)
            .filter(
                TenantPreferredModelProvider.tenant_id == self.tenant_id,
-                TenantPreferredModelProvider.provider_name == self.provider.provider,
+                TenantPreferredModelProvider.provider_name.in_(provider_names),
            )
            .first()
        )
@@ -63,7 +63,9 @@ class File(BaseModel):
        extension: Optional[str] = None,
        mime_type: Optional[str] = None,
        size: int = -1,
-        storage_key: str,
+        storage_key: Optional[str] = None,
+        dify_model_identity: Optional[str] = FILE_MODEL_IDENTITY,
+        url: Optional[str] = None,
    ):
        super().__init__(
            id=id,
@@ -76,8 +78,10 @@ class File(BaseModel):
            extension=extension,
            mime_type=mime_type,
            size=size,
+            dify_model_identity=dify_model_identity,
+            url=url,
        )
-        self._storage_key = storage_key
+        self._storage_key = str(storage_key)

    def to_dict(self) -> Mapping[str, str | int | None]:
        data = self.model_dump(mode="json")
@@ -97,32 +101,18 @@ class File(BaseModel):
        return text

    def generate_url(self) -> Optional[str]:
-        if self.type == FileType.IMAGE:
-            if self.transfer_method == FileTransferMethod.REMOTE_URL:
-                return self.remote_url
-            elif self.transfer_method == FileTransferMethod.LOCAL_FILE:
-                if self.related_id is None:
-                    raise ValueError("Missing file related_id")
-                return helpers.get_signed_file_url(upload_file_id=self.related_id)
-            elif self.transfer_method == FileTransferMethod.TOOL_FILE:
-                assert self.related_id is not None
-                assert self.extension is not None
-                return ToolFileParser.get_tool_file_manager().sign_file(
-                    tool_file_id=self.related_id, extension=self.extension
-                )
-        else:
-            if self.transfer_method == FileTransferMethod.REMOTE_URL:
-                return self.remote_url
-            elif self.transfer_method == FileTransferMethod.LOCAL_FILE:
-                if self.related_id is None:
-                    raise ValueError("Missing file related_id")
-                return helpers.get_signed_file_url(upload_file_id=self.related_id)
-            elif self.transfer_method == FileTransferMethod.TOOL_FILE:
-                assert self.related_id is not None
-                assert self.extension is not None
-                return ToolFileParser.get_tool_file_manager().sign_file(
-                    tool_file_id=self.related_id, extension=self.extension
-                )
+        if self.transfer_method == FileTransferMethod.REMOTE_URL:
+            return self.remote_url
+        elif self.transfer_method == FileTransferMethod.LOCAL_FILE:
+            if self.related_id is None:
+                raise ValueError("Missing file related_id")
+            return helpers.get_signed_file_url(upload_file_id=self.related_id)
+        elif self.transfer_method == FileTransferMethod.TOOL_FILE:
+            assert self.related_id is not None
+            assert self.extension is not None
+            return ToolFileParser.get_tool_file_manager().sign_file(
+                tool_file_id=self.related_id, extension=self.extension
+            )

    def to_plugin_parameter(self) -> dict[str, Any]:
        return {
@@ -11,6 +11,19 @@ from configs import dify_config

 SSRF_DEFAULT_MAX_RETRIES = dify_config.SSRF_DEFAULT_MAX_RETRIES

+HTTP_REQUEST_NODE_SSL_VERIFY = True  # Default value for HTTP_REQUEST_NODE_SSL_VERIFY is True
+try:
+    HTTP_REQUEST_NODE_SSL_VERIFY = dify_config.HTTP_REQUEST_NODE_SSL_VERIFY
+    http_request_node_ssl_verify_lower = str(HTTP_REQUEST_NODE_SSL_VERIFY).lower()
+    if http_request_node_ssl_verify_lower == "true":
+        HTTP_REQUEST_NODE_SSL_VERIFY = True
+    elif http_request_node_ssl_verify_lower == "false":
+        HTTP_REQUEST_NODE_SSL_VERIFY = False
+    else:
+        raise ValueError("Invalid value. HTTP_REQUEST_NODE_SSL_VERIFY should be 'True' or 'False'")
+except NameError:
+    HTTP_REQUEST_NODE_SSL_VERIFY = True
+
 BACKOFF_FACTOR = 0.5
 STATUS_FORCELIST = [429, 500, 502, 503, 504]

@@ -39,17 +52,17 @@ def make_request(method, url, max_retries=SSRF_DEFAULT_MAX_RETRIES, **kwargs):
    while retries <= max_retries:
        try:
            if dify_config.SSRF_PROXY_ALL_URL:
-                with httpx.Client(proxy=dify_config.SSRF_PROXY_ALL_URL) as client:
+                with httpx.Client(proxy=dify_config.SSRF_PROXY_ALL_URL, verify=HTTP_REQUEST_NODE_SSL_VERIFY) as client:
                    response = client.request(method=method, url=url, **kwargs)
            elif dify_config.SSRF_PROXY_HTTP_URL and dify_config.SSRF_PROXY_HTTPS_URL:
                proxy_mounts = {
                    "http://": httpx.HTTPTransport(proxy=dify_config.SSRF_PROXY_HTTP_URL),
                    "https://": httpx.HTTPTransport(proxy=dify_config.SSRF_PROXY_HTTPS_URL),
                }
-                with httpx.Client(mounts=proxy_mounts) as client:
+                with httpx.Client(mounts=proxy_mounts, verify=HTTP_REQUEST_NODE_SSL_VERIFY) as client:
                    response = client.request(method=method, url=url, **kwargs)
            else:
-                with httpx.Client() as client:
+                with httpx.Client(verify=HTTP_REQUEST_NODE_SSL_VERIFY) as client:
                    response = client.request(method=method, url=url, **kwargs)

            if response.status_code not in STATUS_FORCELIST:
@@ -493,7 +493,7 @@ If inputting a combination of text and images, the images need to be constructed
 The base class for all Role message bodies, used only for parameter declaration and cannot be initialized.

 ```python
-class PromptMessage(ABC, BaseModel):
+class PromptMessage(BaseModel):
    """
    Model class for prompt message.
    """
@@ -533,7 +533,7 @@ class ImagePromptMessageContent(PromptMessageContent):
 所有 Role 消息体的基类，仅作为参数声明用，不可初始化。

 ```python
-class PromptMessage(ABC, BaseModel):
+class PromptMessage(BaseModel):
    """
    Model class for prompt message.
    """
@@ -31,11 +31,9 @@ __all__ = [
    "ModelPropertyKey",
    "MultiModalPromptMessageContent",
    "PromptMessage",
-    "PromptMessage",
    "PromptMessageContent",
    "PromptMessageContentType",
    "PromptMessageRole",
-    "PromptMessageRole",
    "PromptMessageTool",
    "SystemPromptMessage",
    "TextPromptMessageContent",
@@ -1,4 +1,3 @@
-from abc import ABC
 from collections.abc import Sequence
 from enum import Enum, StrEnum
 from typing import Optional
@@ -119,7 +118,7 @@ class DocumentPromptMessageContent(MultiModalPromptMessageContent):
    type: PromptMessageContentType = PromptMessageContentType.DOCUMENT


-class PromptMessage(ABC, BaseModel):
+class PromptMessage(BaseModel):
    """
    Model class for prompt message.
    """
@@ -80,7 +80,7 @@ class AIModel(BaseModel):
                        )
                    )
                elif isinstance(invoke_error, InvokeError):
-                    return invoke_error(description=f"[{self.provider_name}] {invoke_error.description}, {str(error)}")
+                    return InvokeError(description=f"[{self.provider_name}] {invoke_error.description}, {str(error)}")
                else:
                    return error

@@ -214,6 +214,8 @@ class OpsTraceManager:
                provider_config_map[tracing_provider]["trace_instance"],
                provider_config_map[tracing_provider]["config_class"],
            )
+            if not decrypt_trace_config:
+                return None
            tracing_instance = trace_instance(config_class(**decrypt_trace_config))
            return tracing_instance

@@ -3,7 +3,7 @@ from binascii import hexlify, unhexlify
 from collections.abc import Generator

 from core.model_manager import ModelManager
-from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk
+from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta
 from core.model_runtime.entities.message_entities import (
    PromptMessage,
    SystemPromptMessage,
@@ -46,7 +46,7 @@ class PluginModelBackwardsInvocation(BaseBackwardsInvocation):
            model_parameters=payload.completion_params,
            tools=payload.tools,
            stop=payload.stop,
-            stream=payload.stream or True,
+            stream=True if payload.stream is None else payload.stream,
            user=user_id,
        )

@@ -64,7 +64,21 @@ class PluginModelBackwardsInvocation(BaseBackwardsInvocation):
        else:
            if response.usage:
                LLMNode.deduct_llm_quota(tenant_id=tenant.id, model_instance=model_instance, usage=response.usage)
-            return response
+
+            def handle_non_streaming(response: LLMResult) -> Generator[LLMResultChunk, None, None]:
+                yield LLMResultChunk(
+                    model=response.model,
+                    prompt_messages=response.prompt_messages,
+                    system_fingerprint=response.system_fingerprint,
+                    delta=LLMResultChunkDelta(
+                        index=0,
+                        message=response.message,
+                        usage=response.usage,
+                        finish_reason="",
+                    ),
+                )
+
+            return handle_non_streaming(response)

    @classmethod
    def invoke_text_embedding(cls, user_id: str, tenant: Tenant, payload: RequestInvokeTextEmbedding):
@@ -147,7 +147,7 @@ def init_frontend_parameter(rule: PluginParameter, type: enum.StrEnum, value: An
    init frontend parameter by rule
    """
    parameter_value = value
-    if not parameter_value and parameter_value != 0:
+    if not parameter_value and parameter_value != 0 and type != PluginParameterType.TOOLS_SELECTOR:
        # get default value
        parameter_value = rule.default
        if not parameter_value and rule.required:
@@ -5,6 +5,7 @@ from collections.abc import Mapping
 from typing import Any, Optional

 from pydantic import BaseModel, Field, model_validator
+from werkzeug.exceptions import NotFound

 from core.agent.plugin_entities import AgentStrategyProviderEntity
 from core.model_runtime.entities.provider_entities import ProviderEntity
@@ -153,6 +154,8 @@ class GenericProviderID:
        return f"{self.organization}/{self.plugin_name}/{self.provider_name}"

    def __init__(self, value: str, is_hardcoded: bool = False) -> None:
+        if not value:
+            raise NotFound("plugin not found, please add plugin")
        # check if the value is a valid plugin id with format: $organization/$plugin_name/$provider_name
        if not re.match(r"^[a-z0-9_-]+\/[a-z0-9_-]+\/[a-z0-9_-]+$", value):
            # check if matches [a-z0-9_-]+, if yes, append with langgenius/$value/$value
@@ -164,6 +167,9 @@ class GenericProviderID:
        self.organization, self.plugin_name, self.provider_name = value.split("/")
        self.is_hardcoded = is_hardcoded

+    def is_langgenius(self) -> bool:
+        return self.organization == "langgenius"
+
    @property
    def plugin_id(self) -> str:
        return f"{self.organization}/{self.plugin_name}"
@@ -46,6 +46,7 @@ class AdvancedPromptTransform(PromptTransform):
        memory_config: Optional[MemoryConfig],
        memory: Optional[TokenBufferMemory],
        model_config: ModelConfigWithCredentialsEntity,
+        image_detail_config: Optional[ImagePromptMessageContent.DETAIL] = None,
    ) -> list[PromptMessage]:
        prompt_messages = []

@@ -59,6 +60,7 @@ class AdvancedPromptTransform(PromptTransform):
                memory_config=memory_config,
                memory=memory,
                model_config=model_config,
+                image_detail_config=image_detail_config,
            )
        elif isinstance(prompt_template, list) and all(isinstance(item, ChatModelMessage) for item in prompt_template):
            prompt_messages = self._get_chat_model_prompt_messages(
@@ -70,6 +72,7 @@ class AdvancedPromptTransform(PromptTransform):
                memory_config=memory_config,
                memory=memory,
                model_config=model_config,
+                image_detail_config=image_detail_config,
            )

        return prompt_messages
@@ -84,6 +87,7 @@ class AdvancedPromptTransform(PromptTransform):
        memory_config: Optional[MemoryConfig],
        memory: Optional[TokenBufferMemory],
        model_config: ModelConfigWithCredentialsEntity,
+        image_detail_config: Optional[ImagePromptMessageContent.DETAIL] = None,
    ) -> list[PromptMessage]:
        """
        Get completion model prompt messages.
@@ -124,7 +128,9 @@ class AdvancedPromptTransform(PromptTransform):
            prompt_message_contents: list[PromptMessageContent] = []
            prompt_message_contents.append(TextPromptMessageContent(data=prompt))
            for file in files:
-                prompt_message_contents.append(file_manager.to_prompt_message_content(file))
+                prompt_message_contents.append(
+                    file_manager.to_prompt_message_content(file, image_detail_config=image_detail_config)
+                )

            prompt_messages.append(UserPromptMessage(content=prompt_message_contents))
        else:
@@ -142,6 +148,7 @@ class AdvancedPromptTransform(PromptTransform):
        memory_config: Optional[MemoryConfig],
        memory: Optional[TokenBufferMemory],
        model_config: ModelConfigWithCredentialsEntity,
+        image_detail_config: Optional[ImagePromptMessageContent.DETAIL] = None,
    ) -> list[PromptMessage]:
        """
        Get chat model prompt messages.
@@ -197,7 +204,9 @@ class AdvancedPromptTransform(PromptTransform):
                prompt_message_contents: list[PromptMessageContent] = []
                prompt_message_contents.append(TextPromptMessageContent(data=query))
                for file in files:
-                    prompt_message_contents.append(file_manager.to_prompt_message_content(file))
+                    prompt_message_contents.append(
+                        file_manager.to_prompt_message_content(file, image_detail_config=image_detail_config)
+                    )
                prompt_messages.append(UserPromptMessage(content=prompt_message_contents))
            else:
                prompt_messages.append(UserPromptMessage(content=query))
@@ -209,19 +218,25 @@ class AdvancedPromptTransform(PromptTransform):
                    # get last user message content and add files
                    prompt_message_contents = [TextPromptMessageContent(data=cast(str, last_message.content))]
                    for file in files:
-                        prompt_message_contents.append(file_manager.to_prompt_message_content(file))
+                        prompt_message_contents.append(
+                            file_manager.to_prompt_message_content(file, image_detail_config=image_detail_config)
+                        )

                    last_message.content = prompt_message_contents
                else:
                    prompt_message_contents = [TextPromptMessageContent(data="")]  # not for query
                    for file in files:
-                        prompt_message_contents.append(file_manager.to_prompt_message_content(file))
+                        prompt_message_contents.append(
+                            file_manager.to_prompt_message_content(file, image_detail_config=image_detail_config)
+                        )

                    prompt_messages.append(UserPromptMessage(content=prompt_message_contents))
            else:
                prompt_message_contents = [TextPromptMessageContent(data=query)]
                for file in files:
-                    prompt_message_contents.append(file_manager.to_prompt_message_content(file))
+                    prompt_message_contents.append(
+                        file_manager.to_prompt_message_content(file, image_detail_config=image_detail_config)
+                    )

                prompt_messages.append(UserPromptMessage(content=prompt_message_contents))
        elif query:
@@ -9,6 +9,7 @@ from core.app.entities.app_invoke_entities import ModelConfigWithCredentialsEnti
 from core.file import file_manager
 from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_runtime.entities.message_entities import (
+    ImagePromptMessageContent,
    PromptMessage,
    PromptMessageContent,
    SystemPromptMessage,
@@ -60,6 +61,7 @@ class SimplePromptTransform(PromptTransform):
        context: Optional[str],
        memory: Optional[TokenBufferMemory],
        model_config: ModelConfigWithCredentialsEntity,
+        image_detail_config: Optional[ImagePromptMessageContent.DETAIL] = None,
    ) -> tuple[list[PromptMessage], Optional[list[str]]]:
        inputs = {key: str(value) for key, value in inputs.items()}

@@ -74,6 +76,7 @@ class SimplePromptTransform(PromptTransform):
                context=context,
                memory=memory,
                model_config=model_config,
+                image_detail_config=image_detail_config,
            )
        else:
            prompt_messages, stops = self._get_completion_model_prompt_messages(
@@ -85,6 +88,7 @@ class SimplePromptTransform(PromptTransform):
                context=context,
                memory=memory,
                model_config=model_config,
+                image_detail_config=image_detail_config,
            )

        return prompt_messages, stops
@@ -175,6 +179,7 @@ class SimplePromptTransform(PromptTransform):
        files: Sequence["File"],
        memory: Optional[TokenBufferMemory],
        model_config: ModelConfigWithCredentialsEntity,
+        image_detail_config: Optional[ImagePromptMessageContent.DETAIL] = None,
    ) -> tuple[list[PromptMessage], Optional[list[str]]]:
        prompt_messages: list[PromptMessage] = []

@@ -204,9 +209,9 @@ class SimplePromptTransform(PromptTransform):
            )

        if query:
-            prompt_messages.append(self.get_last_user_message(query, files))
+            prompt_messages.append(self.get_last_user_message(query, files, image_detail_config))
        else:
-            prompt_messages.append(self.get_last_user_message(prompt, files))
+            prompt_messages.append(self.get_last_user_message(prompt, files, image_detail_config))

        return prompt_messages, None

@@ -220,6 +225,7 @@ class SimplePromptTransform(PromptTransform):
        files: Sequence["File"],
        memory: Optional[TokenBufferMemory],
        model_config: ModelConfigWithCredentialsEntity,
+        image_detail_config: Optional[ImagePromptMessageContent.DETAIL] = None,
    ) -> tuple[list[PromptMessage], Optional[list[str]]]:
        # get prompt
        prompt, prompt_rules = self.get_prompt_str_and_rules(
@@ -262,14 +268,21 @@ class SimplePromptTransform(PromptTransform):
        if stops is not None and len(stops) == 0:
            stops = None

-        return [self.get_last_user_message(prompt, files)], stops
+        return [self.get_last_user_message(prompt, files, image_detail_config)], stops

-    def get_last_user_message(self, prompt: str, files: Sequence["File"]) -> UserPromptMessage:
+    def get_last_user_message(
+        self,
+        prompt: str,
+        files: Sequence["File"],
+        image_detail_config: Optional[ImagePromptMessageContent.DETAIL] = None,
+    ) -> UserPromptMessage:
        if files:
            prompt_message_contents: list[PromptMessageContent] = []
            prompt_message_contents.append(TextPromptMessageContent(data=prompt))
            for file in files:
-                prompt_message_contents.append(file_manager.to_prompt_message_content(file))
+                prompt_message_contents.append(
+                    file_manager.to_prompt_message_content(file, image_detail_config=image_detail_config)
+                )

            prompt_message = UserPromptMessage(content=prompt_message_contents)
        else:
@@ -149,6 +149,11 @@ class ProviderManager:
            provider_name = provider_entity.provider
            provider_records = provider_name_to_provider_records_dict.get(provider_entity.provider, [])
            provider_model_records = provider_name_to_provider_model_records_dict.get(provider_entity.provider, [])
+            provider_id_entity = ModelProviderID(provider_name)
+            if provider_id_entity.is_langgenius():
+                provider_model_records.extend(
+                    provider_name_to_provider_model_records_dict.get(provider_id_entity.provider_name, [])
+                )

            # Convert to custom configuration
            custom_configuration = self._to_custom_configuration(
@@ -190,6 +195,20 @@ class ProviderManager:
                provider_name
            )

+            provider_id_entity = ModelProviderID(provider_name)
+
+            if provider_id_entity.is_langgenius():
+                if provider_model_settings is not None:
+                    provider_model_settings.extend(
+                        provider_name_to_provider_model_settings_dict.get(provider_id_entity.provider_name, [])
+                    )
+                if provider_load_balancing_configs is not None:
+                    provider_load_balancing_configs.extend(
+                        provider_name_to_provider_load_balancing_model_configs_dict.get(
+                            provider_id_entity.provider_name, []
+                        )
+                    )
+
            # Convert to model settings
            model_settings = self._to_model_settings(
                provider_entity=provider_entity,
@@ -207,7 +226,7 @@ class ProviderManager:
                model_settings=model_settings,
            )

-            provider_configurations[str(ModelProviderID(provider_name))] = provider_configuration
+            provider_configurations[str(provider_id_entity)] = provider_configuration

        # Return the encapsulated object
        return provider_configurations
@@ -88,16 +88,17 @@ class Jieba(BaseKeyword):
        keyword_table = self._get_dataset_keyword_table()

        k = kwargs.get("top_k", 4)
-
+        document_ids_filter = kwargs.get("document_ids_filter")
        sorted_chunk_indices = self._retrieve_ids_by_query(keyword_table or {}, query, k)

        documents = []
        for chunk_index in sorted_chunk_indices:
-            segment = (
-                db.session.query(DocumentSegment)
-                .filter(DocumentSegment.dataset_id == self.dataset.id, DocumentSegment.index_node_id == chunk_index)
-                .first()
+            segment_query = db.session.query(DocumentSegment).filter(
+                DocumentSegment.dataset_id == self.dataset.id, DocumentSegment.index_node_id == chunk_index
            )
+            if document_ids_filter:
+                segment_query = segment_query.filter(DocumentSegment.document_id.in_(document_ids_filter))
+            segment = segment_query.first()

            if segment:
                documents.append(
@@ -1,5 +1,4 @@
 import concurrent.futures
-import json
 from concurrent.futures import ThreadPoolExecutor
 from typing import Optional

@@ -42,6 +41,7 @@ class RetrievalService:
        reranking_model: Optional[dict] = None,
        reranking_mode: str = "reranking_model",
        weights: Optional[dict] = None,
+        document_ids_filter: Optional[list[str]] = None,
    ):
        if not query:
            return []
@@ -65,6 +65,7 @@ class RetrievalService:
                        top_k=top_k,
                        all_documents=all_documents,
                        exceptions=exceptions,
+                        document_ids_filter=document_ids_filter,
                    )
                )
            if RetrievalMethod.is_support_semantic_search(retrieval_method):
@@ -80,6 +81,7 @@ class RetrievalService:
                        all_documents=all_documents,
                        retrieval_method=retrieval_method,
                        exceptions=exceptions,
+                        document_ids_filter=document_ids_filter,
                    )
                )
            if RetrievalMethod.is_support_fulltext_search(retrieval_method):
@@ -131,7 +133,14 @@ class RetrievalService:

    @classmethod
    def keyword_search(
-        cls, flask_app: Flask, dataset_id: str, query: str, top_k: int, all_documents: list, exceptions: list
+        cls,
+        flask_app: Flask,
+        dataset_id: str,
+        query: str,
+        top_k: int,
+        all_documents: list,
+        exceptions: list,
+        document_ids_filter: Optional[list[str]] = None,
    ):
        with flask_app.app_context():
            try:
@@ -140,7 +149,10 @@ class RetrievalService:
                    raise ValueError("dataset not found")

                keyword = Keyword(dataset=dataset)
-                documents = keyword.search(cls.escape_query_for_search(query), top_k=top_k)
+
+                documents = keyword.search(
+                    cls.escape_query_for_search(query), top_k=top_k, document_ids_filter=document_ids_filter
+                )
                all_documents.extend(documents)
            except Exception as e:
                exceptions.append(str(e))
@@ -157,6 +169,7 @@ class RetrievalService:
        all_documents: list,
        retrieval_method: str,
        exceptions: list,
+        document_ids_filter: Optional[list[str]] = None,
    ):
        with flask_app.app_context():
            try:
@@ -171,6 +184,7 @@ class RetrievalService:
                    top_k=top_k,
                    score_threshold=score_threshold,
                    filter={"group_id": [dataset.id]},
+                    document_ids_filter=document_ids_filter,
                )

                if documents:
@@ -243,7 +257,7 @@ class RetrievalService:

    @staticmethod
    def escape_query_for_search(query: str) -> str:
-        return json.dumps(query).strip('"')
+        return query.replace('"', '\\"')

    @classmethod
    def format_retrieval_documents(cls, documents: list[Document]) -> list[RetrievalSegments]:
@@ -277,6 +291,8 @@ class RetrievalService:
                    continue

                dataset_document = dataset_documents[document_id]
+                if not dataset_document:
+                    continue

                if dataset_document.doc_form == IndexType.PARENT_CHILD_INDEX:
                    # Handle parent-child documents
@@ -53,7 +53,7 @@ class AnalyticdbVector(BaseVector):
        self.analyticdb_vector.delete_by_metadata_field(key, value)

    def search_by_vector(self, query_vector: list[float], **kwargs: Any) -> list[Document]:
-        return self.analyticdb_vector.search_by_vector(query_vector)
+        return self.analyticdb_vector.search_by_vector(query_vector, **kwargs)

    def search_by_full_text(self, query: str, **kwargs: Any) -> list[Document]:
        return self.analyticdb_vector.search_by_full_text(query, **kwargs)
@@ -194,6 +194,13 @@ class AnalyticdbVectorBySql:

    def search_by_vector(self, query_vector: list[float], **kwargs: Any) -> list[Document]:
        top_k = kwargs.get("top_k", 4)
+        if not isinstance(top_k, int) or top_k <= 0:
+            raise ValueError("top_k must be a positive integer")
+        document_ids_filter = kwargs.get("document_ids_filter")
+        where_clause = "WHERE 1=1"
+        if document_ids_filter:
+            document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+            where_clause += f"AND metadata_->>'document_id' IN ({document_ids})"
        score_threshold = float(kwargs.get("score_threshold") or 0.0)
        with self._get_cursor() as cur:
            query_vector_str = json.dumps(query_vector)
@@ -202,7 +209,7 @@ class AnalyticdbVectorBySql:
                f"SELECT t.id AS id, t.vector AS vector, (1.0 - t.score) AS score, "
                f"t.page_content as page_content, t.metadata_ AS metadata_ "
                f"FROM (SELECT id, vector, page_content, metadata_, vector <=> %s AS score "
-                f"FROM {self.table_name} ORDER BY score LIMIT {top_k} ) t",
+                f"FROM {self.table_name} {where_clause} ORDER BY score LIMIT {top_k} ) t",
                (query_vector_str,),
            )
            documents = []
@@ -220,12 +227,19 @@ class AnalyticdbVectorBySql:

    def search_by_full_text(self, query: str, **kwargs: Any) -> list[Document]:
        top_k = kwargs.get("top_k", 4)
+        if not isinstance(top_k, int) or top_k <= 0:
+            raise ValueError("top_k must be a positive integer")
+        document_ids_filter = kwargs.get("document_ids_filter")
+        where_clause = ""
+        if document_ids_filter:
+            document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+            where_clause += f"AND metadata_->>'document_id' IN ({document_ids})"
        with self._get_cursor() as cur:
            cur.execute(
                f"""SELECT id, vector, page_content, metadata_, 
                ts_rank(to_tsvector, to_tsquery_from_text(%s, 'zh_cn'), 32) AS score
                FROM {self.table_name}
-                WHERE to_tsvector@@to_tsquery_from_text(%s, 'zh_cn')
+                WHERE to_tsvector@@to_tsquery_from_text(%s, 'zh_cn') {where_clause}
                ORDER BY score DESC
                LIMIT {top_k}""",
                (f"'{query}'", f"'{query}'"),
@@ -123,11 +123,21 @@ class BaiduVector(BaseVector):

    def search_by_vector(self, query_vector: list[float], **kwargs: Any) -> list[Document]:
        query_vector = [float(val) if isinstance(val, np.float64) else val for val in query_vector]
-        anns = AnnSearch(
-            vector_field=self.field_vector,
-            vector_floats=query_vector,
-            params=HNSWSearchParams(ef=kwargs.get("ef", 10), limit=kwargs.get("top_k", 4)),
-        )
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+            anns = AnnSearch(
+                vector_field=self.field_vector,
+                vector_floats=query_vector,
+                params=HNSWSearchParams(ef=kwargs.get("ef", 10), limit=kwargs.get("top_k", 4)),
+                filter=f"document_id IN ({document_ids})",
+            )
+        else:
+            anns = AnnSearch(
+                vector_field=self.field_vector,
+                vector_floats=query_vector,
+                params=HNSWSearchParams(ef=kwargs.get("ef", 10), limit=kwargs.get("top_k", 4)),
+            )
        res = self._db.table(self._collection_name).search(
            anns=anns,
            projections=[self.field_id, self.field_text, self.field_metadata],
@@ -95,7 +95,15 @@ class ChromaVector(BaseVector):

    def search_by_vector(self, query_vector: list[float], **kwargs: Any) -> list[Document]:
        collection = self._client.get_or_create_collection(self._collection_name)
-        results: QueryResult = collection.query(query_embeddings=query_vector, n_results=kwargs.get("top_k", 4))
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            results: QueryResult = collection.query(
+                query_embeddings=query_vector,
+                n_results=kwargs.get("top_k", 4),
+                where={"document_id": {"$in": document_ids_filter}},  # type: ignore
+            )
+        else:
+            results: QueryResult = collection.query(query_embeddings=query_vector, n_results=kwargs.get("top_k", 4))  # type: ignore
        score_threshold = float(kwargs.get("score_threshold") or 0.0)

        # Check if results contain data
@@ -117,6 +117,9 @@ class ElasticSearchVector(BaseVector):
        top_k = kwargs.get("top_k", 4)
        num_candidates = math.ceil(top_k * 1.5)
        knn = {"field": Field.VECTOR.value, "query_vector": query_vector, "k": top_k, "num_candidates": num_candidates}
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            knn["filter"] = {"terms": {"metadata.document_id": document_ids_filter}}

        results = self._client.search(index=self._collection_name, knn=knn, size=top_k)

@@ -145,6 +148,9 @@ class ElasticSearchVector(BaseVector):

    def search_by_full_text(self, query: str, **kwargs: Any) -> list[Document]:
        query_str = {"match": {Field.CONTENT_KEY.value: query}}
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            query_str["filter"] = {"terms": {"metadata.document_id": document_ids_filter}}  # type: ignore
        results = self._client.search(index=self._collection_name, query=query_str, size=kwargs.get("top_k", 4))
        docs = []
        for hit in results["hits"]["hits"]:
@@ -168,7 +168,12 @@ class LindormVectorStore(BaseVector):
            raise ValueError("All elements in query_vector should be floats")

        top_k = kwargs.get("top_k", 10)
-        query = default_vector_search_query(query_vector=query_vector, k=top_k, **kwargs)
+        document_ids_filter = kwargs.get("document_ids_filter")
+        filters = []
+        if document_ids_filter:
+            filters.append({"terms": {"metadata.document_id": document_ids_filter}})
+        query = default_vector_search_query(query_vector=query_vector, k=top_k, filters=filters, **kwargs)
+
        try:
            params = {}
            if self._using_ugc:
@@ -206,7 +211,10 @@ class LindormVectorStore(BaseVector):
        should = kwargs.get("should")
        minimum_should_match = kwargs.get("minimum_should_match", 0)
        top_k = kwargs.get("top_k", 10)
-        filters = kwargs.get("filter")
+        filters = kwargs.get("filter", [])
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            filters.append({"terms": {"metadata.document_id": document_ids_filter}})
        routing = self._routing
        full_text_query = default_text_search_query(
            query_text=query,
@@ -228,12 +228,18 @@ class MilvusVector(BaseVector):
        """
        Search for documents by vector similarity.
        """
+        document_ids_filter = kwargs.get("document_ids_filter")
+        filter = ""
+        if document_ids_filter:
+            document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+            filter = f'metadata["document_id"] in ({document_ids})'
        results = self._client.search(
            collection_name=self._collection_name,
            data=[query_vector],
            anns_field=Field.VECTOR.value,
            limit=kwargs.get("top_k", 4),
            output_fields=[Field.CONTENT_KEY.value, Field.METADATA_KEY.value],
+            filter=filter,
        )

        return self._process_search_results(
@@ -249,6 +255,11 @@ class MilvusVector(BaseVector):
        if not self._hybrid_search_enabled or not self.field_exists(Field.SPARSE_VECTOR.value):
            logger.warning("Full-text search is not supported in current Milvus version (requires >= 2.5.0)")
            return []
+        document_ids_filter = kwargs.get("document_ids_filter")
+        filter = ""
+        if document_ids_filter:
+            document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+            filter = f'metadata["document_id"] in ({document_ids})'

        results = self._client.search(
            collection_name=self._collection_name,
@@ -256,6 +267,7 @@ class MilvusVector(BaseVector):
            anns_field=Field.SPARSE_VECTOR.value,
            limit=kwargs.get("top_k", 4),
            output_fields=[Field.CONTENT_KEY.value, Field.METADATA_KEY.value],
+            filter=filter,
        )

        return self._process_search_results(
@@ -125,12 +125,18 @@ class MyScaleVector(BaseVector):

    def _search(self, dist: str, order: SortOrder, **kwargs: Any) -> list[Document]:
        top_k = kwargs.get("top_k", 4)
+        if not isinstance(top_k, int) or top_k <= 0:
+            raise ValueError("top_k must be a positive integer")
        score_threshold = float(kwargs.get("score_threshold") or 0.0)
        where_str = (
            f"WHERE dist < {1 - score_threshold}"
            if self._metric.upper() == "COSINE" and order == SortOrder.ASC and score_threshold > 0.0
            else ""
        )
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+            where_str = f"{where_str} AND metadata['document_id'] in ({document_ids})"
        sql = f"""
            SELECT text, vector, metadata, {dist} as dist FROM {self._config.database}.{self._collection_name}
            {where_str} ORDER BY dist {order.value} LIMIT {top_k}
@@ -154,6 +154,11 @@ class OceanBaseVector(BaseVector):
        return []

    def search_by_vector(self, query_vector: list[float], **kwargs: Any) -> list[Document]:
+        document_ids_filter = kwargs.get("document_ids_filter")
+        where_clause = None
+        if document_ids_filter:
+            document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+            where_clause = f"metadata->>'$.document_id' in ({document_ids})"
        ef_search = kwargs.get("ef_search", self._hnsw_ef_search)
        if ef_search != self._hnsw_ef_search:
            self._client.set_ob_hnsw_ef_search(ef_search)
@@ -167,6 +172,7 @@ class OceanBaseVector(BaseVector):
            distance_func=func.l2_distance,
            output_column_names=["text", "metadata"],
            with_dist=True,
+            where_clause=where_clause,
        )
        docs = []
        for text, metadata, distance in cur:
@@ -0,0 +1,240 @@
+import json
+import uuid
+from contextlib import contextmanager
+from typing import Any
+
+import psycopg2.extras  # type: ignore
+import psycopg2.pool  # type: ignore
+from pydantic import BaseModel, model_validator
+
+from configs import dify_config
+from core.rag.datasource.vdb.vector_base import BaseVector
+from core.rag.datasource.vdb.vector_factory import AbstractVectorFactory
+from core.rag.datasource.vdb.vector_type import VectorType
+from core.rag.embedding.embedding_base import Embeddings
+from core.rag.models.document import Document
+from extensions.ext_redis import redis_client
+from models.dataset import Dataset
+
+
+class OpenGaussConfig(BaseModel):
+    host: str
+    port: int
+    user: str
+    password: str
+    database: str
+    min_connection: int
+    max_connection: int
+
+    @model_validator(mode="before")
+    @classmethod
+    def validate_config(cls, values: dict) -> dict:
+        if not values["host"]:
+            raise ValueError("config OPENGAUSS_HOST is required")
+        if not values["port"]:
+            raise ValueError("config OPENGAUSS_PORT is required")
+        if not values["user"]:
+            raise ValueError("config OPENGAUSS_USER is required")
+        if not values["password"]:
+            raise ValueError("config OPENGAUSS_PASSWORD is required")
+        if not values["database"]:
+            raise ValueError("config OPENGAUSS_DATABASE is required")
+        if not values["min_connection"]:
+            raise ValueError("config OPENGAUSS_MIN_CONNECTION is required")
+        if not values["max_connection"]:
+            raise ValueError("config OPENGAUSS_MAX_CONNECTION is required")
+        if values["min_connection"] > values["max_connection"]:
+            raise ValueError("config OPENGAUSS_MIN_CONNECTION should less than OPENGAUSS_MAX_CONNECTION")
+        return values
+
+
+SQL_CREATE_TABLE = """
+CREATE TABLE IF NOT EXISTS {table_name} (
+    id UUID PRIMARY KEY,
+    text TEXT NOT NULL,
+    meta JSONB NOT NULL,
+    embedding vector({dimension}) NOT NULL
+);
+"""
+
+SQL_CREATE_INDEX = """
+CREATE INDEX IF NOT EXISTS embedding_cosine_{table_name}_idx ON {table_name} 
+USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);
+"""
+
+
+class OpenGauss(BaseVector):
+    def __init__(self, collection_name: str, config: OpenGaussConfig):
+        super().__init__(collection_name)
+        self.pool = self._create_connection_pool(config)
+        self.table_name = f"embedding_{collection_name}"
+
+    def get_type(self) -> str:
+        return VectorType.OPENGAUSS
+
+    def _create_connection_pool(self, config: OpenGaussConfig):
+        return psycopg2.pool.SimpleConnectionPool(
+            config.min_connection,
+            config.max_connection,
+            host=config.host,
+            port=config.port,
+            user=config.user,
+            password=config.password,
+            database=config.database,
+        )
+
+    @contextmanager
+    def _get_cursor(self):
+        conn = self.pool.getconn()
+        cur = conn.cursor()
+        try:
+            yield cur
+        finally:
+            cur.close()
+            conn.commit()
+            self.pool.putconn(conn)
+
+    def create(self, texts: list[Document], embeddings: list[list[float]], **kwargs):
+        dimension = len(embeddings[0])
+        self._create_collection(dimension)
+        return self.add_texts(texts, embeddings)
+
+    def add_texts(self, documents: list[Document], embeddings: list[list[float]], **kwargs):
+        values = []
+        pks = []
+        for i, doc in enumerate(documents):
+            if doc.metadata is not None:
+                doc_id = doc.metadata.get("doc_id", str(uuid.uuid4()))
+                pks.append(doc_id)
+                values.append(
+                    (
+                        doc_id,
+                        doc.page_content,
+                        json.dumps(doc.metadata),
+                        embeddings[i],
+                    )
+                )
+        with self._get_cursor() as cur:
+            psycopg2.extras.execute_values(
+                cur, f"INSERT INTO {self.table_name} (id, text, meta, embedding) VALUES %s", values
+            )
+        return pks
+
+    def text_exists(self, id: str) -> bool:
+        with self._get_cursor() as cur:
+            cur.execute(f"SELECT id FROM {self.table_name} WHERE id = %s", (id,))
+            return cur.fetchone() is not None
+
+    def get_by_ids(self, ids: list[str]) -> list[Document]:
+        with self._get_cursor() as cur:
+            cur.execute(f"SELECT meta, text FROM {self.table_name} WHERE id IN %s", (tuple(ids),))
+            docs = []
+            for record in cur:
+                docs.append(Document(page_content=record[1], metadata=record[0]))
+        return docs
+
+    def delete_by_ids(self, ids: list[str]) -> None:
+        # Avoiding crashes caused by performing delete operations on empty lists in certain scenarios
+        # Scenario 1: extract a document fails, resulting in a table not being created.
+        # Then clicking the retry button triggers a delete operation on an empty list.
+        if not ids:
+            return
+        with self._get_cursor() as cur:
+            cur.execute(f"DELETE FROM {self.table_name} WHERE id IN %s", (tuple(ids),))
+
+    def delete_by_metadata_field(self, key: str, value: str) -> None:
+        with self._get_cursor() as cur:
+            cur.execute(f"DELETE FROM {self.table_name} WHERE meta->>%s = %s", (key, value))
+
+    def search_by_vector(self, query_vector: list[float], **kwargs: Any) -> list[Document]:
+        """
+        Search the nearest neighbors to a vector.
+
+        :param query_vector: The input vector to search for similar items.
+        :param top_k: The number of nearest neighbors to return, default is 5.
+        :return: List of Documents that are nearest to the query vector.
+        """
+        top_k = kwargs.get("top_k", 4)
+        if not isinstance(top_k, int) or top_k <= 0:
+            raise ValueError("top_k must be a positive integer")
+        with self._get_cursor() as cur:
+            cur.execute(
+                f"SELECT meta, text, embedding <=> %s AS distance FROM {self.table_name}"
+                f" ORDER BY distance LIMIT {top_k}",
+                (json.dumps(query_vector),),
+            )
+            docs = []
+            score_threshold = float(kwargs.get("score_threshold") or 0.0)
+            for record in cur:
+                metadata, text, distance = record
+                score = 1 - distance
+                metadata["score"] = score
+                if score > score_threshold:
+                    docs.append(Document(page_content=text, metadata=metadata))
+        return docs
+
+    def search_by_full_text(self, query: str, **kwargs: Any) -> list[Document]:
+        top_k = kwargs.get("top_k", 5)
+        if not isinstance(top_k, int) or top_k <= 0:
+            raise ValueError("top_k must be a positive integer")
+        with self._get_cursor() as cur:
+            cur.execute(
+                f"""SELECT meta, text, ts_rank(to_tsvector(coalesce(text, '')), plainto_tsquery(%s)) AS score
+                FROM {self.table_name}
+                WHERE to_tsvector(text) @@ plainto_tsquery(%s)
+                ORDER BY score DESC
+                LIMIT {top_k}""",
+                # f"'{query}'" is required in order to account for whitespace in query
+                (f"'{query}'", f"'{query}'"),
+            )
+
+            docs = []
+
+            for record in cur:
+                metadata, text, score = record
+                metadata["score"] = score
+                docs.append(Document(page_content=text, metadata=metadata))
+
+        return docs
+
+    def delete(self) -> None:
+        with self._get_cursor() as cur:
+            cur.execute(f"DROP TABLE IF EXISTS {self.table_name}")
+
+    def _create_collection(self, dimension: int):
+        cache_key = f"vector_indexing_{self._collection_name}"
+        lock_name = f"{cache_key}_lock"
+        with redis_client.lock(lock_name, timeout=20):
+            collection_exist_cache_key = f"vector_indexing_{self._collection_name}"
+            if redis_client.get(collection_exist_cache_key):
+                return
+
+            with self._get_cursor() as cur:
+                cur.execute(SQL_CREATE_TABLE.format(table_name=self.table_name, dimension=dimension))
+                if dimension <= 2000:
+                    cur.execute(SQL_CREATE_INDEX.format(table_name=self.table_name))
+            redis_client.set(collection_exist_cache_key, 1, ex=3600)
+
+
+class OpenGaussFactory(AbstractVectorFactory):
+    def init_vector(self, dataset: Dataset, attributes: list, embeddings: Embeddings) -> OpenGauss:
+        if dataset.index_struct_dict:
+            class_prefix: str = dataset.index_struct_dict["vector_store"]["class_prefix"]
+            collection_name = class_prefix
+        else:
+            dataset_id = dataset.id
+            collection_name = Dataset.gen_collection_name_by_id(dataset_id)
+            dataset.index_struct = json.dumps(self.gen_index_struct_dict(VectorType.OPENGAUSS, collection_name))
+
+        return OpenGauss(
+            collection_name=collection_name,
+            config=OpenGaussConfig(
+                host=dify_config.OPENGAUSS_HOST or "localhost",
+                port=dify_config.OPENGAUSS_PORT,
+                user=dify_config.OPENGAUSS_USER or "postgres",
+                password=dify_config.OPENGAUSS_PASSWORD or "",
+                database=dify_config.OPENGAUSS_DATABASE or "dify",
+                min_connection=dify_config.OPENGAUSS_MIN_CONNECTION,
+                max_connection=dify_config.OPENGAUSS_MAX_CONNECTION,
+            ),
+        )
@@ -154,6 +154,9 @@ class OpenSearchVector(BaseVector):
            "size": kwargs.get("top_k", 4),
            "query": {"knn": {Field.VECTOR.value: {Field.VECTOR.value: query_vector, "k": kwargs.get("top_k", 4)}}},
        }
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            query["query"] = {"terms": {"metadata.document_id": document_ids_filter}}

        try:
            response = self._client.search(index=self._collection_name.lower(), body=query)
@@ -179,6 +182,9 @@ class OpenSearchVector(BaseVector):

    def search_by_full_text(self, query: str, **kwargs: Any) -> list[Document]:
        full_text_query = {"query": {"match": {Field.CONTENT_KEY.value: query}}}
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            full_text_query["query"]["terms"] = {"metadata.document_id": document_ids_filter}

        response = self._client.search(index=self._collection_name.lower(), body=full_text_query)

@@ -61,7 +61,7 @@ CREATE TABLE IF NOT EXISTS {table_name} (
 SQL_CREATE_INDEX = """
 CREATE INDEX IF NOT EXISTS idx_docs_{table_name} ON {table_name}(text) 
 INDEXTYPE IS CTXSYS.CONTEXT PARAMETERS 
-('FILTER CTXSYS.NULL_FILTER SECTION GROUP CTXSYS.HTML_SECTION_GROUP LEXER multilingual_lexer')
+('FILTER CTXSYS.NULL_FILTER SECTION GROUP CTXSYS.HTML_SECTION_GROUP LEXER world_lexer')
 """


@@ -201,10 +201,15 @@ class OracleVector(BaseVector):
        :return: List of Documents that are nearest to the query vector.
        """
        top_k = kwargs.get("top_k", 4)
+        document_ids_filter = kwargs.get("document_ids_filter")
+        where_clause = ""
+        if document_ids_filter:
+            document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+            where_clause = f"WHERE metadata->>'document_id' in ({document_ids})"
        with self._get_cursor() as cur:
            cur.execute(
                f"SELECT meta, text, vector_distance(embedding,:1) AS distance FROM {self.table_name}"
-                f" ORDER BY distance fetch first {top_k} rows only",
+                f" {where_clause} ORDER BY distance fetch first {top_k} rows only",
                [numpy.array(query_vector)],
            )
            docs = []
@@ -257,9 +262,15 @@ class OracleVector(BaseVector):
                    if token not in stop_words:
                        entities.append(token)
            with self._get_cursor() as cur:
+                document_ids_filter = kwargs.get("document_ids_filter")
+                where_clause = ""
+                if document_ids_filter:
+                    document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+                    where_clause = f" AND metadata->>'document_id' in ({document_ids}) "
                cur.execute(
                    f"select meta, text, embedding FROM {self.table_name}"
-                    f" WHERE CONTAINS(text, :1, 1) > 0 order by score(1) desc fetch first {top_k} rows only",
+                    f"WHERE CONTAINS(text, :1, 1) > 0 {where_clause} "
+                    f"order by score(1) desc fetch first {top_k} rows only",
                    [" ACCUM ".join(entities)],
                )
                docs = []
@@ -189,6 +189,9 @@ class PGVectoRS(BaseVector):
                .limit(kwargs.get("top_k", 4))
                .order_by("distance")
            )
+            document_ids_filter = kwargs.get("document_ids_filter")
+            if document_ids_filter:
+                stmt = stmt.where(self._table.meta["document_id"].in_(document_ids_filter))
            res = session.execute(stmt)
            results = [(row[0], row[1]) for row in res]

@@ -1,8 +1,10 @@
 import json
+import logging
 import uuid
 from contextlib import contextmanager
 from typing import Any

+import psycopg2.errors
 import psycopg2.extras  # type: ignore
 import psycopg2.pool  # type: ignore
 from pydantic import BaseModel, model_validator
@@ -25,6 +27,7 @@ class PGVectorConfig(BaseModel):
    database: str
    min_connection: int
    max_connection: int
+    pg_bigm: bool = False

    @model_validator(mode="before")
    @classmethod
@@ -62,12 +65,18 @@ CREATE INDEX IF NOT EXISTS embedding_cosine_v1_idx ON {table_name}
 USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);
 """

+SQL_CREATE_INDEX_PG_BIGM = """
+CREATE INDEX IF NOT EXISTS bigm_idx ON {table_name}
+USING gin (text gin_bigm_ops);
+"""
+

 class PGVector(BaseVector):
    def __init__(self, collection_name: str, config: PGVectorConfig):
        super().__init__(collection_name)
        self.pool = self._create_connection_pool(config)
        self.table_name = f"embedding_{collection_name}"
+        self.pg_bigm = config.pg_bigm

    def get_type(self) -> str:
        return VectorType.PGVECTOR
@@ -140,7 +149,14 @@ class PGVector(BaseVector):
        if not ids:
            return
        with self._get_cursor() as cur:
-            cur.execute(f"DELETE FROM {self.table_name} WHERE id IN %s", (tuple(ids),))
+            try:
+                cur.execute(f"DELETE FROM {self.table_name} WHERE id IN %s", (tuple(ids),))
+            except psycopg2.errors.UndefinedTable:
+                # table not exists
+                logging.warning(f"Table {self.table_name} not found, skipping delete operation.")
+                return
+            except Exception as e:
+                raise e

    def delete_by_metadata_field(self, key: str, value: str) -> None:
        with self._get_cursor() as cur:
@@ -155,10 +171,18 @@ class PGVector(BaseVector):
        :return: List of Documents that are nearest to the query vector.
        """
        top_k = kwargs.get("top_k", 4)
+        if not isinstance(top_k, int) or top_k <= 0:
+            raise ValueError("top_k must be a positive integer")
+        document_ids_filter = kwargs.get("document_ids_filter")
+        where_clause = ""
+        if document_ids_filter:
+            document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+            where_clause = f" WHERE metadata->>'document_id' in ({document_ids}) "

        with self._get_cursor() as cur:
            cur.execute(
                f"SELECT meta, text, embedding <=> %s AS distance FROM {self.table_name}"
+                f" {where_clause}"
                f" ORDER BY distance LIMIT {top_k}",
                (json.dumps(query_vector),),
            )
@@ -174,17 +198,37 @@ class PGVector(BaseVector):

    def search_by_full_text(self, query: str, **kwargs: Any) -> list[Document]:
        top_k = kwargs.get("top_k", 5)
-
+        if not isinstance(top_k, int) or top_k <= 0:
+            raise ValueError("top_k must be a positive integer")
        with self._get_cursor() as cur:
-            cur.execute(
-                f"""SELECT meta, text, ts_rank(to_tsvector(coalesce(text, '')), plainto_tsquery(%s)) AS score
-                FROM {self.table_name}
-                WHERE to_tsvector(text) @@ plainto_tsquery(%s)
-                ORDER BY score DESC
-                LIMIT {top_k}""",
-                # f"'{query}'" is required in order to account for whitespace in query
-                (f"'{query}'", f"'{query}'"),
-            )
+            document_ids_filter = kwargs.get("document_ids_filter")
+            where_clause = ""
+            if document_ids_filter:
+                document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+                where_clause = f" AND metadata->>'document_id' in ({document_ids}) "
+            if self.pg_bigm:
+                cur.execute("SET pg_bigm.similarity_limit TO 0.000001")
+                cur.execute(
+                    f"""SELECT meta, text, bigm_similarity(unistr(%s), coalesce(text, '')) AS score
+                    FROM {self.table_name}
+                    WHERE text =%% unistr(%s)
+                    {where_clause}
+                    ORDER BY score DESC
+                    LIMIT {top_k}""",
+                    # f"'{query}'" is required in order to account for whitespace in query
+                    (f"'{query}'", f"'{query}'"),
+                )
+            else:
+                cur.execute(
+                    f"""SELECT meta, text, ts_rank(to_tsvector(coalesce(text, '')), plainto_tsquery(%s)) AS score
+                    FROM {self.table_name}
+                    WHERE to_tsvector(text) @@ plainto_tsquery(%s)
+                    {where_clause}
+                    ORDER BY score DESC
+                    LIMIT {top_k}""",
+                    # f"'{query}'" is required in order to account for whitespace in query
+                    (f"'{query}'", f"'{query}'"),
+                )

            docs = []

@@ -214,6 +258,9 @@ class PGVector(BaseVector):
                # ref: https://github.com/pgvector/pgvector?tab=readme-ov-file#indexing
                if dimension <= 2000:
                    cur.execute(SQL_CREATE_INDEX.format(table_name=self.table_name))
+                if self.pg_bigm:
+                    cur.execute("CREATE EXTENSION IF NOT EXISTS pg_bigm")
+                    cur.execute(SQL_CREATE_INDEX_PG_BIGM.format(table_name=self.table_name))
            redis_client.set(collection_exist_cache_key, 1, ex=3600)


@@ -237,5 +284,6 @@ class PGVectorFactory(AbstractVectorFactory):
                database=dify_config.PGVECTOR_DATABASE or "postgres",
                min_connection=dify_config.PGVECTOR_MIN_CONNECTION,
                max_connection=dify_config.PGVECTOR_MAX_CONNECTION,
+                pg_bigm=dify_config.PGVECTOR_PG_BIGM,
            ),
        )
@@ -286,27 +286,26 @@ class QdrantVector(BaseVector):
        from qdrant_client.http import models
        from qdrant_client.http.exceptions import UnexpectedResponse

-        for node_id in ids:
-            try:
-                filter = models.Filter(
-                    must=[
-                        models.FieldCondition(
-                            key="metadata.doc_id",
-                            match=models.MatchValue(value=node_id),
-                        ),
-                    ],
-                )
-                self._client.delete(
-                    collection_name=self._collection_name,
-                    points_selector=FilterSelector(filter=filter),
-                )
-            except UnexpectedResponse as e:
-                # Collection does not exist, so return
-                if e.status_code == 404:
-                    return
-                # Some other error occurred, so re-raise the exception
-                else:
-                    raise e
+        try:
+            filter = models.Filter(
+                must=[
+                    models.FieldCondition(
+                        key="metadata.doc_id",
+                        match=models.MatchAny(any=ids),
+                    ),
+                ],
+            )
+            self._client.delete(
+                collection_name=self._collection_name,
+                points_selector=FilterSelector(filter=filter),
+            )
+        except UnexpectedResponse as e:
+            # Collection does not exist, so return
+            if e.status_code == 404:
+                return
+            # Some other error occurred, so re-raise the exception
+            else:
+                raise e

    def text_exists(self, id: str) -> bool:
        all_collection_name = []
@@ -331,6 +330,15 @@ class QdrantVector(BaseVector):
                ),
            ],
        )
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            if filter.must:
+                filter.must.append(
+                    models.FieldCondition(
+                        key="metadata.document_id",
+                        match=models.MatchAny(any=document_ids_filter),
+                    )
+                )
        results = self._client.search(
            collection_name=self._collection_name,
            query_vector=query_vector,
@@ -377,6 +385,15 @@ class QdrantVector(BaseVector):
                ),
            ]
        )
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            if scroll_filter.must:
+                scroll_filter.must.append(
+                    models.FieldCondition(
+                        key="metadata.document_id",
+                        match=models.MatchAny(any=document_ids_filter),
+                    )
+                )
        response = self._client.scroll(
            collection_name=self._collection_name,
            scroll_filter=scroll_filter,
@@ -223,8 +223,12 @@ class RelytVector(BaseVector):
        return len(result) > 0

    def search_by_vector(self, query_vector: list[float], **kwargs: Any) -> list[Document]:
+        document_ids_filter = kwargs.get("document_ids_filter")
+        filter = kwargs.get("filter", {})
+        if document_ids_filter:
+            filter["document_id"] = document_ids_filter
        results = self.similarity_search_with_score_by_vector(
-            k=int(kwargs.get("top_k", 4)), embedding=query_vector, filter=kwargs.get("filter")
+            k=int(kwargs.get("top_k", 4)), embedding=query_vector, filter=filter
        )

        # Organize results.
@@ -246,9 +250,9 @@ class RelytVector(BaseVector):
        filter_condition = ""
        if filter is not None:
            conditions = [
-                f"metadata->>{key!r} in ({', '.join(map(repr, value))})"
+                f"metadata->>'{key!r}' in ({', '.join(map(repr, value))})"
                if len(value) > 1
-                else f"metadata->>{key!r} = {value[0]!r}"
+                else f"metadata->>'{key!r}' = {value[0]!r}"
                for key, value in filter.items()
            ]
            filter_condition = f"WHERE {' AND '.join(conditions)}"
@@ -145,11 +145,16 @@ class TencentVector(BaseVector):
        self._db.collection(self._collection_name).delete(document_ids=ids)

    def delete_by_metadata_field(self, key: str, value: str) -> None:
-        self._db.collection(self._collection_name).delete(filter=Filter(Filter.In(key, [value])))
+        self._db.collection(self._collection_name).delete(filter=Filter(Filter.In(f"metadata.{key}", [value])))

    def search_by_vector(self, query_vector: list[float], **kwargs: Any) -> list[Document]:
+        document_ids_filter = kwargs.get("document_ids_filter")
+        filter = None
+        if document_ids_filter:
+            filter = Filter(Filter.In("metadata.document_id", document_ids_filter))
        res = self._db.collection(self._collection_name).search(
            vectors=[query_vector],
+            filter=filter,
            params=document.HNSWSearchParams(ef=kwargs.get("ef", 10)),
            retrieve_vector=False,
            limit=kwargs.get("top_k", 4),
@@ -326,6 +326,18 @@ class TidbOnQdrantVector(BaseVector):
                ),
            ],
        )
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            should_conditions = []
+            for document_id_filter in document_ids_filter:
+                should_conditions.append(
+                    models.FieldCondition(
+                        key="metadata.document_id",
+                        match=models.MatchValue(value=document_id_filter),
+                    )
+                )
+            if should_conditions:
+                filter.should = should_conditions  # type: ignore
        results = self._client.search(
            collection_name=self._collection_name,
            query_vector=query_vector,
@@ -368,6 +380,18 @@ class TidbOnQdrantVector(BaseVector):
                )
            ]
        )
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            should_conditions = []
+            for document_id_filter in document_ids_filter:
+                should_conditions.append(
+                    models.FieldCondition(
+                        key="metadata.document_id",
+                        match=models.MatchValue(value=document_id_filter),
+                    )
+                )
+            if should_conditions:
+                scroll_filter.should = should_conditions  # type: ignore
        response = self._client.scroll(
            collection_name=self._collection_name,
            scroll_filter=scroll_filter,
@@ -196,6 +196,11 @@ class TiDBVector(BaseVector):

        docs = []
        tidb_dist_func = self._get_distance_func()
+        document_ids_filter = kwargs.get("document_ids_filter")
+        where_clause = ""
+        if document_ids_filter:
+            document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+            where_clause = f" WHERE meta->>'$.document_id' in ({document_ids}) "

        with Session(self._engine) as session:
            select_statement = sql_text(f"""
@@ -206,6 +211,7 @@ class TiDBVector(BaseVector):
                    text,
                    {tidb_dist_func}(vector, :query_vector_str) AS distance
                  FROM {self._collection_name}
+                  {where_clause}
                  ORDER BY distance ASC
                  LIMIT :top_k
                ) t
@@ -88,7 +88,20 @@ class UpstashVector(BaseVector):

    def search_by_vector(self, query_vector: list[float], **kwargs: Any) -> list[Document]:
        top_k = kwargs.get("top_k", 4)
-        result = self.index.query(vector=query_vector, top_k=top_k, include_metadata=True, include_data=True)
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            document_ids = ", ".join(f"'{id}'" for id in document_ids_filter)
+            filter = f"document_id in ({document_ids})"
+        else:
+            filter = ""
+        result = self.index.query(
+            vector=query_vector,
+            top_k=top_k,
+            include_metadata=True,
+            include_data=True,
+            include_vectors=False,
+            filter=filter,
+        )
        docs = []
        score_threshold = float(kwargs.get("score_threshold") or 0.0)
        for record in result:
@@ -148,6 +148,10 @@ class Vector:
                from core.rag.datasource.vdb.oceanbase.oceanbase_vector import OceanBaseVectorFactory

                return OceanBaseVectorFactory
+            case VectorType.OPENGAUSS:
+                from core.rag.datasource.vdb.opengauss.opengauss import OpenGaussFactory
+
+                return OpenGaussFactory
            case _:
                raise ValueError(f"Vector store {vector_type} is not supported.")

@@ -24,3 +24,4 @@ class VectorType(StrEnum):
    UPSTASH = "upstash"
    TIDB_ON_QDRANT = "tidb_on_qdrant"
    OCEANBASE = "oceanbase"
+    OPENGAUSS = "opengauss"
@@ -177,7 +177,11 @@ class VikingDBVector(BaseVector):
            query_vector, limit=kwargs.get("top_k", 4)
        )
        score_threshold = float(kwargs.get("score_threshold") or 0.0)
-        return self._get_search_res(results, score_threshold)
+        docs = self._get_search_res(results, score_threshold)
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            docs = [doc for doc in docs if doc.metadata.get("document_id") in document_ids_filter]
+        return docs

    def _get_search_res(self, results, score_threshold) -> list[Document]:
        if len(results) == 0:
@@ -187,8 +187,10 @@ class WeaviateVector(BaseVector):
        query_obj = self._client.query.get(collection_name, properties)

        vector = {"vector": query_vector}
-        if kwargs.get("where_filter"):
-            query_obj = query_obj.with_where(kwargs.get("where_filter"))
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            where_filter = {"operator": "ContainsAny", "path": ["document_id"], "valueTextArray": document_ids_filter}
+            query_obj = query_obj.with_where(where_filter)
        result = (
            query_obj.with_near_vector(vector)
            .with_limit(kwargs.get("top_k", 4))
@@ -233,8 +235,10 @@ class WeaviateVector(BaseVector):
        if kwargs.get("search_distance"):
            content["certainty"] = kwargs.get("search_distance")
        query_obj = self._client.query.get(collection_name, properties)
-        if kwargs.get("where_filter"):
-            query_obj = query_obj.with_where(kwargs.get("where_filter"))
+        document_ids_filter = kwargs.get("document_ids_filter")
+        if document_ids_filter:
+            where_filter = {"operator": "ContainsAny", "path": ["document_id"], "valueTextArray": document_ids_filter}
+            query_obj = query_obj.with_where(where_filter)
        query_obj = query_obj.with_additional(["vector"])
        properties = ["text"]
        result = query_obj.with_bm25(query=query, properties=properties).with_limit(kwargs.get("top_k", 4)).do()
@@ -0,0 +1,45 @@
+from collections.abc import Sequence
+from typing import Literal, Optional
+
+from pydantic import BaseModel, Field
+
+SupportedComparisonOperator = Literal[
+    # for string or array
+    "contains",
+    "not contains",
+    "start with",
+    "end with",
+    "is",
+    "is not",
+    "empty",
+    "not empty",
+    # for number
+    "=",
+    "≠",
+    ">",
+    "<",
+    "≥",
+    "≤",
+    # for time
+    "before",
+    "after",
+]
+
+
+class Condition(BaseModel):
+    """
+    Conditon detail
+    """
+
+    name: str
+    comparison_operator: SupportedComparisonOperator
+    value: str | Sequence[str] | None | int | float = None
+
+
+class MetadataCondition(BaseModel):
+    """
+    Metadata Condition.
+    """
+
+    logical_operator: Optional[Literal["and", "or"]] = "and"
+    conditions: Optional[list[Condition]] = Field(default=None, deprecated=True)
@@ -0,0 +1,15 @@
+from enum import Enum
+
+
+class BuiltInField(str, Enum):
+    document_name = "document_name"
+    uploader = "uploader"
+    upload_date = "upload_date"
+    last_update_date = "last_update_date"
+    source = "source"
+
+
+class MetadataDataSource(Enum):
+    upload_file = "file_upload"
+    website_crawl = "website"
+    notion_import = "notion"
@@ -1,34 +1,61 @@
+import json
 import math
+import re
 import threading
-from collections import Counter
-from typing import Any, Optional, cast
+from collections import Counter, defaultdict
+from collections.abc import Generator, Mapping
+from typing import Any, Optional, Union, cast

 from flask import Flask, current_app
+from sqlalchemy import Integer, and_, or_, text
+from sqlalchemy import cast as sqlalchemy_cast

-from core.app.app_config.entities import DatasetEntity, DatasetRetrieveConfigEntity
+from core.app.app_config.entities import (
+    DatasetEntity,
+    DatasetRetrieveConfigEntity,
+    MetadataFilteringCondition,
+    ModelConfig,
+)
 from core.app.entities.app_invoke_entities import InvokeFrom, ModelConfigWithCredentialsEntity
 from core.callback_handler.index_tool_callback_handler import DatasetIndexToolCallbackHandler
 from core.entities.agent_entities import PlanningStrategy
+from core.entities.model_entities import ModelStatus
 from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_manager import ModelInstance, ModelManager
-from core.model_runtime.entities.message_entities import PromptMessageTool
+from core.model_runtime.entities.llm_entities import LLMResult, LLMUsage
+from core.model_runtime.entities.message_entities import PromptMessage, PromptMessageRole, PromptMessageTool
 from core.model_runtime.entities.model_entities import ModelFeature, ModelType
 from core.model_runtime.model_providers.__base.large_language_model import LargeLanguageModel
 from core.ops.entities.trace_entity import TraceTaskName
 from core.ops.ops_trace_manager import TraceQueueManager, TraceTask
 from core.ops.utils import measure_time
+from core.prompt.advanced_prompt_transform import AdvancedPromptTransform
+from core.prompt.entities.advanced_prompt_entities import ChatModelMessage, CompletionModelPromptTemplate
+from core.prompt.simple_prompt_transform import ModelMode
 from core.rag.data_post_processor.data_post_processor import DataPostProcessor
 from core.rag.datasource.keyword.jieba.jieba_keyword_table_handler import JiebaKeywordTableHandler
 from core.rag.datasource.retrieval_service import RetrievalService
 from core.rag.entities.context_entities import DocumentContext
+from core.rag.entities.metadata_entities import Condition, MetadataCondition
+from core.rag.index_processor.constant.index_type import IndexType
 from core.rag.models.document import Document
 from core.rag.rerank.rerank_type import RerankMode
 from core.rag.retrieval.retrieval_methods import RetrievalMethod
 from core.rag.retrieval.router.multi_dataset_function_call_router import FunctionCallMultiDatasetRouter
 from core.rag.retrieval.router.multi_dataset_react_route import ReactMultiDatasetRouter
+from core.rag.retrieval.template_prompts import (
+    METADATA_FILTER_ASSISTANT_PROMPT_1,
+    METADATA_FILTER_ASSISTANT_PROMPT_2,
+    METADATA_FILTER_COMPLETION_PROMPT,
+    METADATA_FILTER_SYSTEM_PROMPT,
+    METADATA_FILTER_USER_PROMPT_1,
+    METADATA_FILTER_USER_PROMPT_2,
+    METADATA_FILTER_USER_PROMPT_3,
+)
 from core.tools.utils.dataset_retriever.dataset_retriever_base_tool import DatasetRetrieverBaseTool
 from extensions.ext_database import db
-from models.dataset import Dataset, DatasetQuery, DocumentSegment
+from libs.json_in_md_parser import parse_and_check_json_markdown
+from models.dataset import ChildChunk, Dataset, DatasetMetadata, DatasetQuery, DocumentSegment
 from models.dataset import Document as DatasetDocument
 from services.external_knowledge_service import ExternalDatasetService

@@ -58,6 +85,7 @@ class DatasetRetrieval:
        hit_callback: DatasetIndexToolCallbackHandler,
        message_id: str,
        memory: Optional[TokenBufferMemory] = None,
+        inputs: Optional[Mapping[str, Any]] = None,
    ) -> Optional[str]:
        """
        Retrieve dataset.
@@ -115,6 +143,22 @@ class DatasetRetrieval:
                continue

            available_datasets.append(dataset)
+        if inputs:
+            inputs = {key: str(value) for key, value in inputs.items()}
+        else:
+            inputs = {}
+        available_datasets_ids = [dataset.id for dataset in available_datasets]
+        metadata_filter_document_ids, metadata_condition = self._get_metadata_filter_condition(
+            available_datasets_ids,
+            query,
+            tenant_id,
+            user_id,
+            retrieve_config.metadata_filtering_mode,  # type: ignore
+            retrieve_config.metadata_model_config,  # type: ignore
+            retrieve_config.metadata_filtering_conditions,
+            inputs,
+        )
+
        all_documents = []
        user_from = "account" if invoke_from in {InvokeFrom.EXPLORE, InvokeFrom.DEBUGGER} else "end_user"
        if retrieve_config.retrieve_strategy == DatasetRetrieveConfigEntity.RetrieveStrategy.SINGLE:
@@ -129,6 +173,8 @@ class DatasetRetrieval:
                model_config,
                planning_strategy,
                message_id,
+                metadata_filter_document_ids,
+                metadata_condition,
            )
        elif retrieve_config.retrieve_strategy == DatasetRetrieveConfigEntity.RetrieveStrategy.MULTIPLE:
            all_documents = self.multiple_retrieve(
@@ -145,6 +191,8 @@ class DatasetRetrieval:
                retrieve_config.weights,
                retrieve_config.reranking_enabled or True,
                message_id,
+                metadata_filter_document_ids,
+                metadata_condition,
            )

        dify_documents = [item for item in all_documents if item.provider == "dify"]
@@ -238,6 +286,8 @@ class DatasetRetrieval:
        model_config: ModelConfigWithCredentialsEntity,
        planning_strategy: PlanningStrategy,
        message_id: Optional[str] = None,
+        metadata_filter_document_ids: Optional[dict[str, list[str]]] = None,
+        metadata_condition: Optional[MetadataCondition] = None,
    ):
        tools = []
        for dataset in available_datasets:
@@ -278,6 +328,7 @@ class DatasetRetrieval:
                        dataset_id=dataset_id,
                        query=query,
                        external_retrieval_parameters=dataset.retrieval_model,
+                        metadata_condition=metadata_condition,
                    )
                    for external_document in external_documents:
                        document = Document(
@@ -292,6 +343,15 @@ class DatasetRetrieval:
                            document.metadata["dataset_name"] = dataset.name
                        results.append(document)
                else:
+                    if metadata_condition and not metadata_filter_document_ids:
+                        return []
+                    document_ids_filter = None
+                    if metadata_filter_document_ids:
+                        document_ids = metadata_filter_document_ids.get(dataset.id, [])
+                        if document_ids:
+                            document_ids_filter = document_ids
+                        else:
+                            return []
                    retrieval_model_config = dataset.retrieval_model or default_retrieval_model

                    # get top k
@@ -323,6 +383,7 @@ class DatasetRetrieval:
                            reranking_model=reranking_model,
                            reranking_mode=retrieval_model_config.get("reranking_mode", "reranking_model"),
                            weights=retrieval_model_config.get("weights", None),
+                            document_ids_filter=document_ids_filter,
                        )
                self._on_query(query, [dataset_id], app_id, user_from, user_id)

@@ -347,6 +408,8 @@ class DatasetRetrieval:
        weights: Optional[dict[str, Any]] = None,
        reranking_enable: bool = True,
        message_id: Optional[str] = None,
+        metadata_filter_document_ids: Optional[dict[str, list[str]]] = None,
+        metadata_condition: Optional[MetadataCondition] = None,
    ):
        if not available_datasets:
            return []
@@ -386,6 +449,16 @@ class DatasetRetrieval:

        for dataset in available_datasets:
            index_type = dataset.indexing_technique
+            document_ids_filter = None
+            if dataset.provider != "external":
+                if metadata_condition and not metadata_filter_document_ids:
+                    continue
+                if metadata_filter_document_ids:
+                    document_ids = metadata_filter_document_ids.get(dataset.id, [])
+                    if document_ids:
+                        document_ids_filter = document_ids
+                    else:
+                        continue
            retrieval_thread = threading.Thread(
                target=self._retriever,
                kwargs={
@@ -394,6 +467,8 @@ class DatasetRetrieval:
                    "query": query,
                    "top_k": top_k,
                    "all_documents": all_documents,
+                    "document_ids_filter": document_ids_filter,
+                    "metadata_condition": metadata_condition,
                },
            )
            threads.append(retrieval_thread)
@@ -429,18 +504,36 @@ class DatasetRetrieval:
        dify_documents = [document for document in documents if document.provider == "dify"]
        for document in dify_documents:
            if document.metadata is not None:
-                query = db.session.query(DocumentSegment).filter(
-                    DocumentSegment.index_node_id == document.metadata["doc_id"]
-                )
+                dataset_document = DatasetDocument.query.filter(
+                    DatasetDocument.id == document.metadata["document_id"]
+                ).first()
+                if dataset_document:
+                    if dataset_document.doc_form == IndexType.PARENT_CHILD_INDEX:
+                        child_chunk = ChildChunk.query.filter(
+                            ChildChunk.index_node_id == document.metadata["doc_id"],
+                            ChildChunk.dataset_id == dataset_document.dataset_id,
+                            ChildChunk.document_id == dataset_document.id,
+                        ).first()
+                        if child_chunk:
+                            segment = DocumentSegment.query.filter(DocumentSegment.id == child_chunk.segment_id).update(
+                                {DocumentSegment.hit_count: DocumentSegment.hit_count + 1}, synchronize_session=False
+                            )
+                            db.session.commit()
+                    else:
+                        query = db.session.query(DocumentSegment).filter(
+                            DocumentSegment.index_node_id == document.metadata["doc_id"]
+                        )

-                # if 'dataset_id' in document.metadata:
-                if "dataset_id" in document.metadata:
-                    query = query.filter(DocumentSegment.dataset_id == document.metadata["dataset_id"])
+                        # if 'dataset_id' in document.metadata:
+                        if "dataset_id" in document.metadata:
+                            query = query.filter(DocumentSegment.dataset_id == document.metadata["dataset_id"])

-                # add hit count to document segment
-                query.update({DocumentSegment.hit_count: DocumentSegment.hit_count + 1}, synchronize_session=False)
+                        # add hit count to document segment
+                        query.update(
+                            {DocumentSegment.hit_count: DocumentSegment.hit_count + 1}, synchronize_session=False
+                        )

-                db.session.commit()
+                    db.session.commit()

        # get tracing instance
        trace_manager: TraceQueueManager | None = (
@@ -474,7 +567,16 @@ class DatasetRetrieval:
            db.session.add_all(dataset_queries)
        db.session.commit()

-    def _retriever(self, flask_app: Flask, dataset_id: str, query: str, top_k: int, all_documents: list):
+    def _retriever(
+        self,
+        flask_app: Flask,
+        dataset_id: str,
+        query: str,
+        top_k: int,
+        all_documents: list,
+        document_ids_filter: Optional[list[str]] = None,
+        metadata_condition: Optional[MetadataCondition] = None,
+    ):
        with flask_app.app_context():
            dataset = db.session.query(Dataset).filter(Dataset.id == dataset_id).first()

@@ -487,6 +589,7 @@ class DatasetRetrieval:
                    dataset_id=dataset_id,
                    query=query,
                    external_retrieval_parameters=dataset.retrieval_model,
+                    metadata_condition=metadata_condition,
                )
                for external_document in external_documents:
                    document = Document(
@@ -527,6 +630,7 @@ class DatasetRetrieval:
                            else None,
                            reranking_mode=retrieval_model.get("reranking_mode") or "reranking_model",
                            weights=retrieval_model.get("weights", None),
+                            document_ids_filter=document_ids_filter,
                        )

                        all_documents.extend(documents)
@@ -714,3 +818,340 @@ class DatasetRetrieval:
            filter_documents, key=lambda x: x.metadata.get("score", 0) if x.metadata else 0, reverse=True
        )
        return filter_documents[:top_k] if top_k else filter_documents
+
+    def _get_metadata_filter_condition(
+        self,
+        dataset_ids: list,
+        query: str,
+        tenant_id: str,
+        user_id: str,
+        metadata_filtering_mode: str,
+        metadata_model_config: ModelConfig,
+        metadata_filtering_conditions: Optional[MetadataFilteringCondition],
+        inputs: dict,
+    ) -> tuple[Optional[dict[str, list[str]]], Optional[MetadataCondition]]:
+        document_query = db.session.query(DatasetDocument).filter(
+            DatasetDocument.dataset_id.in_(dataset_ids),
+            DatasetDocument.indexing_status == "completed",
+            DatasetDocument.enabled == True,
+            DatasetDocument.archived == False,
+        )
+        filters = []  # type: ignore
+        metadata_condition = None
+        if metadata_filtering_mode == "disabled":
+            return None, None
+        elif metadata_filtering_mode == "automatic":
+            automatic_metadata_filters = self._automatic_metadata_filter_func(
+                dataset_ids, query, tenant_id, user_id, metadata_model_config
+            )
+            if automatic_metadata_filters:
+                conditions = []
+                for filter in automatic_metadata_filters:
+                    self._process_metadata_filter_func(
+                        filter.get("condition"),  # type: ignore
+                        filter.get("metadata_name"),  # type: ignore
+                        filter.get("value"),
+                        filters,  # type: ignore
+                    )
+                    conditions.append(
+                        Condition(
+                            name=filter.get("metadata_name"),  # type: ignore
+                            comparison_operator=filter.get("condition"),  # type: ignore
+                            value=filter.get("value"),
+                        )
+                    )
+                metadata_condition = MetadataCondition(
+                    logical_operator=metadata_filtering_conditions.logical_operator,  # type: ignore
+                    conditions=conditions,
+                )
+        elif metadata_filtering_mode == "manual":
+            if metadata_filtering_conditions:
+                metadata_condition = MetadataCondition(**metadata_filtering_conditions.model_dump())
+                for condition in metadata_filtering_conditions.conditions:  # type: ignore
+                    metadata_name = condition.name
+                    expected_value = condition.value
+                    if expected_value or condition.comparison_operator in ("empty", "not empty"):
+                        if isinstance(expected_value, str):
+                            expected_value = self._replace_metadata_filter_value(expected_value, inputs)
+                        filters = self._process_metadata_filter_func(
+                            condition.comparison_operator, metadata_name, expected_value, filters
+                        )
+        else:
+            raise ValueError("Invalid metadata filtering mode")
+        if filters:
+            if metadata_filtering_conditions.logical_operator == "or":  # type: ignore
+                document_query = document_query.filter(or_(*filters))
+            else:
+                document_query = document_query.filter(and_(*filters))
+        documents = document_query.all()
+        # group by dataset_id
+        metadata_filter_document_ids = defaultdict(list) if documents else None  # type: ignore
+        for document in documents:
+            metadata_filter_document_ids[document.dataset_id].append(document.id)  # type: ignore
+        return metadata_filter_document_ids, metadata_condition
+
+    def _replace_metadata_filter_value(self, text: str, inputs: dict) -> str:
+        def replacer(match):
+            key = match.group(1)
+            return str(inputs.get(key, f"{{{{{key}}}}}"))
+
+        pattern = re.compile(r"\{\{(\w+)\}\}")
+        return pattern.sub(replacer, text)
+
+    def _automatic_metadata_filter_func(
+        self, dataset_ids: list, query: str, tenant_id: str, user_id: str, metadata_model_config: ModelConfig
+    ) -> Optional[list[dict[str, Any]]]:
+        # get all metadata field
+        metadata_fields = db.session.query(DatasetMetadata).filter(DatasetMetadata.dataset_id.in_(dataset_ids)).all()
+        all_metadata_fields = [metadata_field.name for metadata_field in metadata_fields]
+        # get metadata model config
+        if metadata_model_config is None:
+            raise ValueError("metadata_model_config is required")
+        # get metadata model instance
+        # fetch model config
+        model_instance, model_config = self._fetch_model_config(tenant_id, metadata_model_config)
+
+        # fetch prompt messages
+        prompt_messages, stop = self._get_prompt_template(
+            model_config=model_config,
+            mode=metadata_model_config.mode,
+            metadata_fields=all_metadata_fields,
+            query=query or "",
+        )
+
+        result_text = ""
+        try:
+            # handle invoke result
+            invoke_result = cast(
+                Generator[LLMResult, None, None],
+                model_instance.invoke_llm(
+                    prompt_messages=prompt_messages,
+                    model_parameters=model_config.parameters,
+                    stop=stop,
+                    stream=True,
+                    user=user_id,
+                ),
+            )
+
+            # handle invoke result
+            result_text, usage = self._handle_invoke_result(invoke_result=invoke_result)
+
+            result_text_json = parse_and_check_json_markdown(result_text, [])
+            automatic_metadata_filters = []
+            if "metadata_map" in result_text_json:
+                metadata_map = result_text_json["metadata_map"]
+                for item in metadata_map:
+                    if item.get("metadata_field_name") in all_metadata_fields:
+                        automatic_metadata_filters.append(
+                            {
+                                "metadata_name": item.get("metadata_field_name"),
+                                "value": item.get("metadata_field_value"),
+                                "condition": item.get("comparison_operator"),
+                            }
+                        )
+        except Exception as e:
+            return None
+        return automatic_metadata_filters
+
+    def _process_metadata_filter_func(self, condition: str, metadata_name: str, value: Optional[Any], filters: list):
+        match condition:
+            case "contains":
+                filters.append(
+                    (text("documents.doc_metadata ->> :key LIKE :value")).params(key=metadata_name, value=f"%{value}%")
+                )
+            case "not contains":
+                filters.append(
+                    (text("documents.doc_metadata ->> :key NOT LIKE :value")).params(
+                        key=metadata_name, value=f"%{value}%"
+                    )
+                )
+            case "start with":
+                filters.append(
+                    (text("documents.doc_metadata ->> :key LIKE :value")).params(key=metadata_name, value=f"{value}%")
+                )
+
+            case "end with":
+                filters.append(
+                    (text("documents.doc_metadata ->> :key LIKE :value")).params(key=metadata_name, value=f"%{value}")
+                )
+            case "is" | "=":
+                if isinstance(value, str):
+                    filters.append(DatasetDocument.doc_metadata[metadata_name] == f'"{value}"')
+                else:
+                    filters.append(
+                        sqlalchemy_cast(DatasetDocument.doc_metadata[metadata_name].astext, Integer) == value
+                    )
+            case "is not" | "≠":
+                if isinstance(value, str):
+                    filters.append(DatasetDocument.doc_metadata[metadata_name] != f'"{value}"')
+                else:
+                    filters.append(
+                        sqlalchemy_cast(DatasetDocument.doc_metadata[metadata_name].astext, Integer) != value
+                    )
+            case "empty":
+                filters.append(DatasetDocument.doc_metadata[metadata_name].is_(None))
+            case "not empty":
+                filters.append(DatasetDocument.doc_metadata[metadata_name].isnot(None))
+            case "before" | "<":
+                filters.append(sqlalchemy_cast(DatasetDocument.doc_metadata[metadata_name].astext, Integer) < value)
+            case "after" | ">":
+                filters.append(sqlalchemy_cast(DatasetDocument.doc_metadata[metadata_name].astext, Integer) > value)
+            case "≤" | ">=":
+                filters.append(sqlalchemy_cast(DatasetDocument.doc_metadata[metadata_name].astext, Integer) <= value)
+            case "≥" | ">=":
+                filters.append(sqlalchemy_cast(DatasetDocument.doc_metadata[metadata_name].astext, Integer) >= value)
+            case _:
+                pass
+        return filters
+
+    def _fetch_model_config(
+        self, tenant_id: str, model: ModelConfig
+    ) -> tuple[ModelInstance, ModelConfigWithCredentialsEntity]:
+        """
+        Fetch model config
+        :param node_data: node data
+        :return:
+        """
+        if model is None:
+            raise ValueError("single_retrieval_config is required")
+        model_name = model.name
+        provider_name = model.provider
+
+        model_manager = ModelManager()
+        model_instance = model_manager.get_model_instance(
+            tenant_id=tenant_id, model_type=ModelType.LLM, provider=provider_name, model=model_name
+        )
+
+        provider_model_bundle = model_instance.provider_model_bundle
+        model_type_instance = model_instance.model_type_instance
+        model_type_instance = cast(LargeLanguageModel, model_type_instance)
+
+        model_credentials = model_instance.credentials
+
+        # check model
+        provider_model = provider_model_bundle.configuration.get_provider_model(
+            model=model_name, model_type=ModelType.LLM
+        )
+
+        if provider_model is None:
+            raise ValueError(f"Model {model_name} not exist.")
+
+        if provider_model.status == ModelStatus.NO_CONFIGURE:
+            raise ValueError(f"Model {model_name} credentials is not initialized.")
+        elif provider_model.status == ModelStatus.NO_PERMISSION:
+            raise ValueError(f"Dify Hosted OpenAI {model_name} currently not support.")
+        elif provider_model.status == ModelStatus.QUOTA_EXCEEDED:
+            raise ValueError(f"Model provider {provider_name} quota exceeded.")
+
+        # model config
+        completion_params = model.completion_params
+        stop = []
+        if "stop" in completion_params:
+            stop = completion_params["stop"]
+            del completion_params["stop"]
+
+        # get model mode
+        model_mode = model.mode
+        if not model_mode:
+            raise ValueError("LLM mode is required.")
+
+        model_schema = model_type_instance.get_model_schema(model_name, model_credentials)
+
+        if not model_schema:
+            raise ValueError(f"Model {model_name} not exist.")
+
+        return model_instance, ModelConfigWithCredentialsEntity(
+            provider=provider_name,
+            model=model_name,
+            model_schema=model_schema,
+            mode=model_mode,
+            provider_model_bundle=provider_model_bundle,
+            credentials=model_credentials,
+            parameters=completion_params,
+            stop=stop,
+        )
+
+    def _get_prompt_template(
+        self, model_config: ModelConfigWithCredentialsEntity, mode: str, metadata_fields: list, query: str
+    ):
+        model_mode = ModelMode.value_of(mode)
+        input_text = query
+
+        prompt_template: Union[CompletionModelPromptTemplate, list[ChatModelMessage]]
+        if model_mode == ModelMode.CHAT:
+            prompt_template = []
+            system_prompt_messages = ChatModelMessage(role=PromptMessageRole.SYSTEM, text=METADATA_FILTER_SYSTEM_PROMPT)
+            prompt_template.append(system_prompt_messages)
+            user_prompt_message_1 = ChatModelMessage(role=PromptMessageRole.USER, text=METADATA_FILTER_USER_PROMPT_1)
+            prompt_template.append(user_prompt_message_1)
+            assistant_prompt_message_1 = ChatModelMessage(
+                role=PromptMessageRole.ASSISTANT, text=METADATA_FILTER_ASSISTANT_PROMPT_1
+            )
+            prompt_template.append(assistant_prompt_message_1)
+            user_prompt_message_2 = ChatModelMessage(role=PromptMessageRole.USER, text=METADATA_FILTER_USER_PROMPT_2)
+            prompt_template.append(user_prompt_message_2)
+            assistant_prompt_message_2 = ChatModelMessage(
+                role=PromptMessageRole.ASSISTANT, text=METADATA_FILTER_ASSISTANT_PROMPT_2
+            )
+            prompt_template.append(assistant_prompt_message_2)
+            user_prompt_message_3 = ChatModelMessage(
+                role=PromptMessageRole.USER,
+                text=METADATA_FILTER_USER_PROMPT_3.format(
+                    input_text=input_text,
+                    metadata_fields=json.dumps(metadata_fields, ensure_ascii=False),
+                ),
+            )
+            prompt_template.append(user_prompt_message_3)
+        elif model_mode == ModelMode.COMPLETION:
+            prompt_template = CompletionModelPromptTemplate(
+                text=METADATA_FILTER_COMPLETION_PROMPT.format(
+                    input_text=input_text,
+                    metadata_fields=json.dumps(metadata_fields, ensure_ascii=False),
+                )
+            )
+
+        else:
+            raise ValueError(f"Model mode {model_mode} not support.")
+
+        prompt_transform = AdvancedPromptTransform()
+        prompt_messages = prompt_transform.get_prompt(
+            prompt_template=prompt_template,
+            inputs={},
+            query=query or "",
+            files=[],
+            context=None,
+            memory_config=None,
+            memory=None,
+            model_config=model_config,
+        )
+        stop = model_config.stop
+
+        return prompt_messages, stop
+
+    def _handle_invoke_result(self, invoke_result: Generator) -> tuple[str, LLMUsage]:
+        """
+        Handle invoke result
+        :param invoke_result: invoke result
+        :return:
+        """
+        model = None
+        prompt_messages: list[PromptMessage] = []
+        full_text = ""
+        usage = None
+        for result in invoke_result:
+            text = result.delta.message.content
+            full_text += text
+
+            if not model:
+                model = result.model
+
+            if not prompt_messages:
+                prompt_messages = result.prompt_messages
+
+            if not usage and result.delta.usage:
+                usage = result.delta.usage
+
+        if not usage:
+            usage = LLMUsage.empty_usage()
+
+        return full_text, usage
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
-LAN-andGitHub	ac80c04bd3	chore: bump version to 1.1.0 (#16128 ) Signed-off-by: -LAN- <laipz8200@outlook.com>	2025-03-18 20:01:24 +08:00
JyongandGitHub	fa9b767bf2	fix chatflow metadata field name (#16130 )	2025-03-18 19:40:42 +08:00
JyongandGitHub	abeaea4f79	Support knowledge metadata filter (#15982 )	2025-03-18 16:42:19 +08:00
JyongandGitHub	b65f2eb55f	fix embedding model name translate issue (#16111 )	2025-03-18 16:41:35 +08:00
KVOJJJinandGitHub	7d620ffd5e	Feat：app list dark mode (#16110 )	2025-03-18 16:21:53 +08:00
YeuolyandGitHub	6f6ba2f025	fix(api): enhance provider model records handling for missing langgenius providers (#16089 )	2025-03-18 15:07:53 +08:00
JyongandGitHub	33ba7e659b	fix vector db sql injection (#16096 )	2025-03-18 15:07:29 +08:00
yihongandGitHub	750ec55646	doc: auto correct the doc using autocorrect close #16091 (#16092 ) Signed-off-by: yihong0618 <zouzou0208@gmail.com>	2025-03-18 14:57:14 +08:00
kurokoboandGitHub	86d3fff666	fix: respect resolution settings for vision for basic chatbot, text generator, and parameter extractor node (#16041 )	2025-03-18 14:37:07 +08:00
Naoki KOBAYASHI GitHub crazywoola	e91531fc23	fix: error in migrate_annotation_vector_database when exec vdb-migrate (#15937 ) Co-authored-by: crazywoola <427733928@qq.com>	2025-03-18 14:15:48 +08:00
StoneFancyX GitHub StoneFancyX crazywoola	2524f16525	support config filename in meta for create_blob_message (#15605 ) Co-authored-by: StoneFancyX <kindbin@qq.com> Co-authored-by: crazywoola <427733928@qq.com>	2025-03-18 13:59:00 +08:00
-LAN-GitHub twwu	cefec44070	feat: add app_mode field to app import and model definitions (#15729 ) Signed-off-by: -LAN- <laipz8200@outlook.com> Co-authored-by: twwu <twwu@dify.ai>	2025-03-18 11:12:25 +08:00
zxhlyh GitHub Joel NFish	20376ca951	feat: upgrade knowledge metadata (#16063 ) Support filter knowledge by metadata. Co-authored-by: Joel <iamjoel007@gmail.com> Co-authored-by: NFish <douxc512@gmail.com>	2025-03-18 11:01:06 +08:00
Gen SatoandGitHub	475b8d731e	Fix HTTP Request node to give priority to file extension of content-disposition (#12653 )	2025-03-18 11:00:20 +08:00
Yongtao HuangandGitHub	963b6f628a	Chore: PromptMessage is not an abstract base class (#15965 )	2025-03-18 10:57:52 +08:00
luckylhb90 GitHub hobo.l	63ea6f1ecf	Fixed: Run failed: Failed to invoke tool: File.__init__() got an unexpected keyword argument (#14073 ) Co-authored-by: hobo.l <hobo.l@binance.com>	2025-03-18 10:55:58 +08:00
诗浓andGitHub	947c9f70fb	fix: improve InputNumber component step behavior and disabled state (#16044 )	2025-03-18 10:42:29 +08:00
XiaoBa GitHub Xiaoba Yu	5e52d4d6b3	feat: add Maximum number of Parallelism branches to env (#15964 ) Co-authored-by: Xiaoba Yu <xb1823725853@gmail.com>	2025-03-18 09:32:47 +08:00
Kalo ChinandGitHub	939dcb4c0a	chore: enhance ListWrapper and PluginPage components with stable scro… (#16048 )	2025-03-18 09:12:49 +08:00
LittleFish-15andGitHub	223ab5a38f	feat: support openGauss vector database (#15865 )	2025-03-17 19:42:54 +08:00
GuanMuandGitHub	db7a37a111	fix: adjust position of table of contents in Doc component (#15996 )	2025-03-17 19:37:21 +08:00
Novice GitHub Novice Lee	fe0d932f50	fix: fail-branch stream output error (#13401 ) Co-authored-by: Novice Lee <novicelee@NoviPro.local>	2025-03-17 19:35:37 +08:00
QuantumGhostandGitHub	69fb0a4a28	chore: use POSIX shell syntax in pre-commit script (#16025 )	2025-03-17 19:28:25 +08:00
NoviceandGitHub	04a0ae3aa9	feat: add llm blocking invoke (#15732 )	2025-03-17 16:47:10 +08:00
QuantumGhostandGitHub	e5d6047fb4	chore(api): Disable preview rules of Ruff while running pre-commit hook (#15999 )	2025-03-17 16:40:27 +08:00
Bowen LiangandGitHub	9e782d4c1e	chore: bump ruff to 0.11.0 and fix linting violations (#15953 )	2025-03-17 16:13:11 +08:00
L8ngandGitHub	98a4b3e78b	fix: typo when assign doc_metadata when non-empty (#15975 )	2025-03-17 14:14:07 +08:00
QuantumGhostandGitHub	2b4d1cf1db	fix(api): fix fail branch functionality for `WorkflowTool` (#15966 )	2025-03-17 11:53:32 +08:00
傻笑zzandGitHub	fe76dfe1f8	When decrypt_trace_config is empty, it should be skipped directly (#15870 )	2025-03-17 11:29:20 +08:00
csurongandGitHub	c3774bef7e	fix: api error of get all workspaces (#15880 )	2025-03-17 11:22:27 +08:00
huangzhuo1949 GitHub huangzhuo	695a7400a9	fix:delete empty table bug (#15517 ) Co-authored-by: huangzhuo <huangzhuo1@xiaomi.com>	2025-03-17 10:53:26 +08:00
ArcanerandGitHub	e6a8800f66	fix: validation for upload methods of non-image files within the work… (#15932 )	2025-03-17 09:50:21 +08:00
LiuBodongandGitHub	cee8731393	fix:Nginx template not replace env correctly (#15651 )	2025-03-16 11:19:09 +08:00
Yongtao HuangandGitHub	4ae94dc027	Chore: fix wrong annotations (#15871 )	2025-03-16 11:16:28 +08:00
BenjaminandGitHub	3a69a6a452	Fix/enable marketplace bug (#15895 )	2025-03-16 11:14:12 +08:00
JoelandGitHub	f8f21ef7c0	fix: node use vision model may caused page crash (#15921 )	2025-03-16 08:54:18 +08:00
ShadowJobs GitHub LinYing	0587eb4956	FIX:microsoft word text copy and paste error (#14905 ) Co-authored-by: LinYing <linying@momenta.ai>	2025-03-14 18:31:20 +08:00
Yongtao HuangandGitHub	433374abea	Chore: remove unused fields (#15764 )	2025-03-14 18:13:25 +08:00
QuantumGhostandGitHub	23ed3a520b	chore(api): improve type hints for BaseNode and its subclasses (#15826 )	2025-03-14 18:09:11 +08:00
jiangbo721 GitHub 刘江波	5646442931	fix: iteration total tokens calculate error (#15813 ) Co-authored-by: 刘江波 <jiangbo721@163.com>	2025-03-14 17:44:24 +08:00
Yi FengandGitHub	1a6298b6ea	fix: Remove any extra Spaces in the title (#15841 )	2025-03-14 17:12:29 +08:00
非法操作andGitHub	bf9b572bc3	fix tool selector with empty tools raise error (#15829 )	2025-03-14 16:47:52 +08:00
非法操作andGitHub	cf72e53a10	chore: remove useless doc and font (#15838 )	2025-03-14 16:47:42 +08:00
過世秋風andGitHub	98bd79f548	fix: update Knowledge Api doc: 【Update a Chunk in a Document】 (#15823 )	2025-03-14 16:45:20 +08:00
JyongandGitHub	84a866028a	fix document could be None (#15818 )	2025-03-14 16:40:01 +08:00
KVOJJJinandGitHub	10bd03611c	Fix style of opening statement (#15821 )	2025-03-14 15:50:28 +08:00
sho-takano-devandGitHub	7c27d4b202	feat: add Http Request Node to skip ssl verify function #15177 (#15664 )	2025-03-14 10:05:37 +08:00
RookieAgentandGitHub	8165d0b469	fix: http_request node form-data support array[file] (#15731 )	2025-03-14 09:58:18 +08:00
诗浓andGitHub	e796937d02	feat: add keyboard shortcuts support for dialog confirmation (#15752 )	2025-03-13 21:42:53 +08:00
-LAN-andGitHub	49c952a631	fix: streamline file upload configuration handling in manager.py (#15714 ) Signed-off-by: -LAN- <laipz8200@outlook.com>	2025-03-13 16:32:49 +08:00
Yuichiro UtsumiandGitHub	5f9d236d22	Feat: Add pg_bigm for keyword search in pgvector (#13876 ) Signed-off-by: Yuichiro Utsumi <utsumi.yuichiro@fujitsu.com>	2025-03-13 16:32:34 +08:00
zhangyuhang GitHub yuhang2.zhang	59f5a82261	fix: Resolve errors in SQL queries caused by SELECT fields not appearing in the GROUP BY clause. (#15659 ) Co-authored-by: yuhang2.zhang <yuhang2.zhang@ly.com>	2025-03-13 16:06:42 +08:00
XiaoBa GitHub Xiaoba Yu	f22a1adb8b	fix: Integration langfuse, front-end error( #15695 ) (#15709 ) Co-authored-by: Xiaoba Yu <xb1823725853@gmail.com>	2025-03-13 15:43:41 +08:00
JyongandGitHub	a8e8c37fdd	improve text split (#15719 )	2025-03-13 15:29:33 +08:00
NFishandGitHub	37486a9cc6	fix: update default github star count value (#15708 )	2025-03-13 14:39:26 +08:00
KVOJJJinandGitHub	efebbffe96	Fix：webapp UI issues (#15601 )	2025-03-13 14:23:41 +08:00
Xiyuan ChenandGitHub	5e035a4209	Ci/deploy enterprise (#15699 )	2025-03-13 02:22:21 -04:00
ArcanerandGitHub	12fa517297	fix: if-else-node handles missing optional file variables (#15693 )	2025-03-13 13:11:49 +08:00
Fei HeandGitHub	36ae0e5476	fix: set score_threshold only when score_threshold_enabled is true. (#14221 )	2025-03-12 20:55:57 +08:00
codingjaguar GitHub ChengZi	74f66d3119	Update .env.example to fix MILVUS_URI default value (#13140 ) Signed-off-by: ChengZi <chen.zhang@zilliz.com> Co-authored-by: ChengZi <chen.zhang@zilliz.com>	2025-03-12 20:31:45 +08:00
LamandGitHub	adfaee7ab5	fix: prevent AppIconPicker click event from propagating (#15575 ) (#15647 )	2025-03-12 20:03:09 +08:00
JyongandGitHub	d37490adc3	fix dataset reranking mode miss (#15643 )	2025-03-12 18:44:10 +08:00
kenwoodjwandGitHub	087bb60b31	fix: preserve Unicode characters in keyword search queries (#15522 ) Signed-off-by: kenwoodjw <blackxin55+@gmail.com>	2025-03-12 18:34:42 +08:00
非法操作andGitHub	5019547d33	fix: can not test custom tool (#15606 )	2025-03-12 16:34:56 +08:00
JoeandGitHub	58f012f3de	fix: no attribute error (#15597 )	2025-03-12 15:27:42 +08:00
JoelandGitHub	b938c9b7f6	fix: trace return null cause page crash (#15588 )	2025-03-12 14:40:43 +08:00
YeuolyandGitHub	2b1facc7a6	fix: set marketplace feature to false in feature_service.py (#15578 )	2025-03-12 14:13:41 +08:00
Rafael Carvalho GitHub crazywoola	1d5ea80a2b	feat: env MAX_TOOLS_NUM (#15431 ) Co-authored-by: crazywoola <427733928@qq.com>	2025-03-12 12:57:05 +08:00
jiangbo721 GitHub 刘江波	0415cc209d	chore: use TenantAccountRole instead of TenantAccountJoinRole (#15514 ) Co-authored-by: 刘江波 <jiangbo721@163.com>	2025-03-12 12:56:30 +08:00
JoeandGitHub	545e5cbcd6	fix: dataset editor (#15218 )	2025-03-12 12:51:00 +08:00
Mars GitHub mars	1fab02c25a	fix:message api doc (#15568 ) Co-authored-by: mars <linjx2@by-health.com>	2025-03-12 12:38:23 +08:00
crazywoolaandGitHub	258736f505	chore: remove unused parameter (#15558 )	2025-03-12 12:09:39 +08:00
LamandGitHub	0bc4da38fc	feat: add debounced enter key submission to install form (#15445 ) (#15542 )	2025-03-12 11:25:54 +08:00
jiangbo721 GitHub 刘江波	037f200527	fix: invoke_error is not callable (#15555 ) Co-authored-by: 刘江波 <jiangbo721@163.com>	2025-03-12 10:58:44 +08:00
zxhlyhandGitHub	b541792465	fix: workflow loop node break conditions (#15549 )	2025-03-12 10:10:51 +08:00
NFishandGitHub	eb9b256ee8	fix: remove size prop in PlanBadge component because UpgradeBtn size … (#15544 )	2025-03-12 09:49:15 +08:00
Kalo ChinandGitHub	5d8b32a249	feat: add click-away and mounting logic to agent setting component (#15521 )	2025-03-11 22:23:06 +08:00
yihong GitHub -LAN-	c960b364c9	chore: update opendal version (#14343 ) Signed-off-by: yihong0618 <zouzou0208@gmail.com> Signed-off-by: -LAN- <laipz8200@outlook.com> Co-authored-by: -LAN- <laipz8200@outlook.com>	2025-03-11 20:44:09 +08:00
NoviceandGitHub	b817036343	fix: nesting of conditional branches causing streaming output error (#14065 )	2025-03-11 20:30:03 +08:00
-LAN-andGitHub	46036e6ce6	fix: update version to 1.0.1 in configuration and Docker files (#15478 ) Signed-off-by: -LAN- <laipz8200@outlook.com>	2025-03-11 18:50:42 +08:00
JyongandGitHub	1ffda0dd34	fix notion page display (#15508 )	2025-03-11 18:40:02 +08:00
JyongandGitHub	da01b460fe	support workspace billing info (#15510 )	2025-03-11 18:38:23 +08:00
Wu TianweiandGitHub	90a1508b87	fix: update placeholders in version info modal to indicate optional field (#15499 )	2025-03-11 18:30:47 +08:00
DDDDD12138andGitHub	b07016113c	fix: add animation to workflow process loader icon (#15497 )	2025-03-11 18:04:58 +08:00
Wu TianweiandGitHub	d8317fcf81	fix: remove unnecessary modal (#15493 )	2025-03-11 17:18:23 +08:00
YeuolyandGitHub	a6bc642721	refactor: optimize provider configuration queries with provider name … (#15491 )	2025-03-11 17:09:51 +08:00
NFishandGitHub	b730f243dc	fix: displan badge based on workspace plan (#15489 )	2025-03-11 17:01:17 +08:00
Yuichiro UtsumiandGitHub	71a57275ab	fix: improve selection of variable in workflow (#15484 ) Signed-off-by: Yuichiro Utsumi <utsumi.yuichiro@fujitsu.com>	2025-03-11 16:57:45 +08:00
Lick-liu GitHub crazywoola	41bf8d925f	fix:To fix the issue of missing reference to body parameter (#15443 ) Co-authored-by: crazywoola <427733928@qq.com>	2025-03-11 16:16:53 +08:00
Yu Chun Chang GitHub Kyle Chang	6d172498d1	Update the provider_id validation to fix the error message displayed … (#15466 ) Co-authored-by: Kyle Chang <kylechang@91app.com>	2025-03-11 16:11:24 +08:00
-LAN-andGitHub	cad58658c2	fix: simplify S3 client configuration by removing redundant checksum settings (#15474 ) Signed-off-by: -LAN- <laipz8200@outlook.com>	2025-03-11 14:50:03 +08:00
heysztandGitHub	a58b990855	fix agent_execution_metadata (#15444 )	2025-03-11 14:35:08 +08:00
Wu TianweiandGitHub	b6b1903a37	fix: fix chatbot publish and restore handling (#15462 )	2025-03-11 13:36:45 +08:00
Jacky WuandGitHub	ed5596a8f4	fix: avoid llm node result var not init issue while do retry. (#14286 )	2025-03-11 12:43:24 +08:00
Hao ChengandGitHub	49d0acd188	fix: replace old-style <br> tags to fix Mermaid rendering issues (#13792 )	2025-03-11 12:40:55 +08:00
NoviceandGitHub	58a74fe1fb	chore: add comment to the PLUGIN_DIFY_INNER_API_KEY key (#15381 )	2025-03-11 00:25:11 +08:00
JyongandGitHub	a1ab4aec3d	fix db migration (#15422 )	2025-03-11 00:24:57 +08:00
JyongandGitHub	f77f7e1437	fix text split (#15426 )	2025-03-11 00:24:27 +08:00
kenwoodjwandGitHub	adda049265	fix kb permission (#15199 ) Signed-off-by: kenwoodjw <blackxin55@gmail.com> Signed-off-by: kenwoodjw <blackxin55+@gmail.com>	2025-03-10 23:47:45 +08:00
JyongandGitHub	9b2a9260ef	Feat/new saas billing (#14996 )	2025-03-10 19:50:11 +08:00
JoeandGitHub	c8cc31af88	fix: app trace permission (#15397 )	2025-03-10 18:45:25 +08:00
-LAN-andGitHub	d333de274f	chore(.github): add a new tracker template (#15391 ) Signed-off-by: -LAN- <laipz8200@outlook.com>	2025-03-10 18:39:35 +08:00
KVOJJJinandGitHub	9e220d5d30	Feat: configure dark mode legacy (#15394 )	2025-03-10 16:41:06 +08:00
Wu TianweiandGitHub	2cf0cb471f	fix: fix document list overlap and optimize document list fetching (#15377 )	2025-03-10 15:34:40 +08:00
kurokoboandGitHub	269ba6add9	fix: remove port expose on db (#15286 )	2025-03-10 15:01:34 +08:00
KVOJJJin GitHub -LAN-	78d460a6d1	Feat: time period filter for workflow logs (#14271 ) Signed-off-by: -LAN- <laipz8200@outlook.com> Co-authored-by: -LAN- <laipz8200@outlook.com>	2025-03-10 14:02:58 +08:00
-LAN-andGitHub	3254018ddb	feat(workflow_service): workflow version control api. (#14860 ) Signed-off-by: -LAN- <laipz8200@outlook.com>	2025-03-10 13:34:31 +08:00
kurokoboandGitHub	f2b7df94d7	fix: return absolute path as the icon url if CONSOLE_API_URL is empty (#15279 )	2025-03-10 13:15:06 +08:00
YeuolyandGitHub	59fd3aad31	feat: add PIP_MIRROR_URL environment variable support (#15353 )	2025-03-10 12:59:31 +08:00
zxhlyhandGitHub	a3d18d43ed	fix: tool name in agent (#15344 )	2025-03-10 11:21:46 +08:00
engchina GitHub engchina	20cbebeef1	modify OCI_ENDPOINT example value Fixes #15336 (#15337 ) Co-authored-by: engchina <atjapan2015@gmail.com>	2025-03-10 10:47:39 +08:00
engchina GitHub engchina	2968482199	downgrade boto3 to use s3 compatible storage. Fixes #15225 (#15261 ) Co-authored-by: engchina <atjapan2015@gmail.com>	2025-03-10 09:56:38 +08:00
znnandGitHub	f8ac382072	raising error if plugin not initialized (#15319 )	2025-03-10 09:54:51 +08:00
Will GitHub crazywoola	aef43910b1	fix: octet/stream => application/octet-stream (#15329 ) Co-authored-by: crazywoola <427733928@qq.com>	2025-03-10 09:49:27 +08:00
albcunhaandGitHub	87efd4ab84	Update login.py (#15320 )	2025-03-10 09:49:14 +08:00
heysztandGitHub	a8b600845e	fix Unicode Escape Characters (#15318 )	2025-03-10 09:22:41 +08:00
Wu TianweiandGitHub	fcd9fd8513	fix: update image gallery styles (#15301 )	2025-03-09 15:32:03 +08:00
kurokoboandGitHub	ffe73f0124	feat: add docker-compose.override.yaml to .gitignore (#15289 )	2025-03-09 10:51:55 +08:00
kurokoboandGitHub	0c57250d87	feat: expose PYTHON_ENV_INIT_TIMEOUT and PLUGIN_MAX_EXECUTION_TIMEOUT (#15283 )	2025-03-09 10:45:19 +08:00
Hantaek Lim GitHub crazywoola	f7e012d216	Fix: reranker OFF logic to preserve user setting (#15235 ) Co-authored-by: crazywoola <427733928@qq.com>	2025-03-08 19:08:48 +08:00
RhysandGitHub	c9e3c8b38d	fix: dataset pagination state keeps resetting when filters changed (#15268 )	2025-03-08 17:38:07 +08:00
crazywoolaandGitHub	908a7b6c3d	fix: tool icons are missing (#15241 )	2025-03-08 11:04:53 +08:00
Che KunandGitHub	cfd7e8a829	fix: missing action value to `tools.includeToolNum` lang for custom t… (#15239 )	2025-03-08 10:55:13 +08:00
Bo-Yi WuandGitHub	804b818c6b	docs(readme): add a Traditional Chinese badge for README (#15258 ) Signed-off-by: appleboy <appleboy.tw@gmail.com>	2025-03-08 10:48:16 +08:00
Xiyuan ChenandGitHub	9b9d14c2c4	Feat/compliance (#14982 )	2025-03-07 14:56:38 -05:00
yosuke0715andGitHub	38fc8eeaba	typo チュンク to チャンク (#15240 )	2025-03-07 20:55:07 +08:00
jiangbo721 GitHub 刘江波	e70221a9f1	fix: website remote url display error (#15217 ) Co-authored-by: 刘江波 <jiangbo721@163.com>	2025-03-07 20:32:29 +08:00
MarsandGitHub	126202648f	fix message sort (#15231 )	2025-03-07 19:36:44 +08:00
NFishandGitHub	dc8475995f	fix: web style check task throw error (#15226 )	2025-03-07 19:23:06 +08:00
Wu TianweiandGitHub	3ca1373274	feat: version tag (#14949 )	2025-03-07 18:10:40 +08:00
NFishandGitHub	4aaf07d62a	fix: update the link of contact sales in billing page (#15219 )	2025-03-07 16:53:01 +08:00
Likename HaojieandGitHub	ff10a4603f	bugfix:cant correct display latex (#14910 )	2025-03-07 14:06:14 +08:00
mr-chenguangandGitHub	53eb56bb1e	Fix: psycopg2.errors.StringDataRightTruncation value too long for type character varying(40) (#15179 )	2025-03-07 12:15:52 +08:00
Marc KlingenandGitHub	c6209d76eb	chore: update langfuse description (#15136 )	2025-03-07 12:15:38 +08:00
非法操作andGitHub	99dc8c7871	fix: http node request detect text/xml as file (#15174 )	2025-03-07 12:12:06 +08:00
KVOJJJinandGitHub	f588ccff72	Feat: settings dark mode (#15184 )	2025-03-07 11:56:20 +08:00
Moch. Ainun NajibandGitHub	69746f2f0b	add: allowed_domains marketplace.dify.ai (#15139 )	2025-03-07 10:55:08 +08:00
YeuolyandGitHub	65da9425df	Fix: only retrieval plugin-compatible providers when provider_name starts with `langgenius` (#15133 )	2025-03-07 00:41:56 +08:00
Kalo ChinandGitHub	b7583e95a5	fix: adjust scroll detection threshold in chat component (#14640 )	2025-03-06 22:36:59 +08:00
Bo-Yi WuandGitHub	9437a1a844	docs: add comprehensive Traditional Chinese contribution guide (#14816 ) Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>	2025-03-06 22:36:05 +08:00
JyongandGitHub	435564f0f2	fix parent-child retrival count (#15119 )	2025-03-06 22:32:38 +08:00
JimmiaxieandGitHub	2a6e522a87	Fixed incorrect use of key in the page /plugins?category=discover #15126 (#15127 )	2025-03-06 20:14:39 +08:00
engchina GitHub engchina	9c1db7dca7	modify oracle lexer name Fixes #15106 (#15108 ) Co-authored-by: engchina <atjapan2015@gmail.com>	2025-03-06 18:58:51 +08:00
JimmiaxieandGitHub	cd7cb19aee	hotfix: Fixed tags not updating in real time in the label management of apps #15113 (#15110 )	2025-03-06 18:55:25 +08:00
非法操作andGitHub	d84fa4d154	fix: with file conversation second chat raise error (#15097 )	2025-03-06 18:21:40 +08:00
w4-jinhyeonkimandGitHub	d574706600	Update ko-KR/plugin.ts (#15103 )	2025-03-06 17:40:37 +08:00
crazywoolaandGitHub	8369e59b4d	Feat/14573 support more types in form (#15093 )	2025-03-06 17:38:50 +08:00
Wu TianweiandGitHub	5be8fbab56	feat: refactor date-and-time-picker to use custom dayjs utility and add timezone support (#15101 )	2025-03-06 16:24:03 +08:00
Pascal M GitHub crazywoola	6101733232	fix(docker): plugin daemon lacks database dependency (#15038 ) Co-authored-by: crazywoola <427733928@qq.com>	2025-03-06 13:18:59 +08:00
NoviceandGitHub	778861f461	fix: agent node can't use in parallel (#15047 )	2025-03-06 13:13:24 +08:00
Wu TianweiandGitHub	6c9d6a4d57	fix: add custom disallowed elements to Markdown component and restore the default disallowed elements (#15057 )	2025-03-06 10:57:49 +08:00
NFishandGitHub	9962118dbd	Fix: new upgrade page (#12417 )	2025-03-06 10:27:13 +08:00
NFishandGitHub	a4b2c10fb8	Feat/compliance report download (#14477 )	2025-03-06 10:25:18 +08:00
Wood GitHub arkunzz	2c17bb2c36	Feature/newnew workflow loop node (#14863 ) Co-authored-by: arkunzz <4873204@qq.com>	2025-03-05 17:41:15 +08:00
Junjie.MandGitHub	da91217bc9	fix docker-compose.yaml and docker-compose.middleware.yaml plugin_daemon environment parameter values (#14992 )	2025-03-05 16:27:55 +08:00