add "add block" shortcut ui

Co-authored-by: 莫岳恒 <[email protected]>

.gitignore

@@ -178,3 +178,4 @@ pyrightconfig.json
 api/.vscode
 
 .idea/
+.vscode

CONTRIBUTING_VI.md

@@ -0,0 +1,156 @@
+Thật tuyệt vời khi bạn muốn đóng góp cho Dify! Chúng tôi rất mong chờ được thấy những gì bạn sẽ làm. Là một startup với nguồn nhân lực và tài chính hạn chế, chúng tôi có tham vọng lớn là thiết kế quy trình trực quan nhất để xây dựng và quản lý các ứng dụng LLM. Mọi sự giúp đỡ từ cộng đồng đều rất quý giá đối với chúng tôi.
+
+Chúng tôi cần linh hoạt và làm việc nhanh chóng, nhưng đồng thời cũng muốn đảm bảo các cộng tác viên như bạn có trải nghiệm đóng góp thuận lợi nhất có thể. Chúng tôi đã tạo ra hướng dẫn đóng góp này nhằm giúp bạn làm quen với codebase và cách chúng tôi làm việc với các cộng tác viên, để bạn có thể nhanh chóng bắt tay vào phần thú vị.
+
+Hướng dẫn này, cũng như bản thân Dify, đang trong quá trình cải tiến liên tục. Chúng tôi rất cảm kích sự thông cảm của bạn nếu đôi khi nó không theo kịp dự án thực tế, và chúng tôi luôn hoan nghênh mọi phản hồi để cải thiện.
+
+Về vấn đề cấp phép, xin vui lòng dành chút thời gian đọc qua [Thỏa thuận Cấp phép và Đóng góp](./LICENSE) ngắn gọn của chúng tôi. Cộng đồng cũng tuân thủ [quy tắc ứng xử](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md).
+
+## Trước khi bắt đầu
+
+[Tìm kiếm](https://github.com/langgenius/dify/issues?q=is:issue+is:closed) một vấn đề hiện có, hoặc [tạo mới](https://github.com/langgenius/dify/issues/new/choose) một vấn đề. Chúng tôi phân loại các vấn đề thành 2 loại:
+
+### Yêu cầu tính năng:
+
+* Nếu bạn đang tạo một yêu cầu tính năng mới, chúng tôi muốn bạn giải thích tính năng đề xuất sẽ đạt được điều gì và cung cấp càng nhiều thông tin chi tiết càng tốt. [@perzeusss](https://github.com/perzeuss) đã tạo một [Trợ lý Yêu cầu Tính năng](https://udify.app/chat/MK2kVSnw1gakVwMX) rất hữu ích để giúp bạn soạn thảo nhu cầu của mình. Hãy thử dùng nó nhé.
+
+* Nếu bạn muốn chọn một vấn đề từ danh sách hiện có, chỉ cần để lại bình luận dưới vấn đề đó nói rằng bạn sẽ làm.
+
+  Một thành viên trong nhóm làm việc trong lĩnh vực liên quan sẽ được thông báo. Nếu mọi thứ ổn, họ sẽ cho phép bạn bắt đầu code. Chúng tôi yêu cầu bạn chờ đợi cho đến lúc đó trước khi bắt tay vào làm tính năng, để không lãng phí công sức của bạn nếu chúng tôi đề xuất thay đổi.
+
+  Tùy thuộc vào lĩnh vực mà tính năng đề xuất thuộc về, bạn có thể nói chuyện với các thành viên khác nhau trong nhóm. Dưới đây là danh sách các lĩnh vực mà các thành viên trong nhóm chúng tôi đang làm việc hiện tại:
+
+  | Thành viên                                                   | Phạm vi                                              |
+  | ------------------------------------------------------------ | ---------------------------------------------------- |
+  | [@yeuoly](https://github.com/Yeuoly)                         | Thiết kế kiến trúc Agents                            |
+  | [@jyong](https://github.com/JohnJyong)                       | Thiết kế quy trình RAG                               |
+  | [@GarfieldDai](https://github.com/GarfieldDai)               | Xây dựng quy trình làm việc                          |
+  | [@iamjoel](https://github.com/iamjoel) & [@zxhlyh](https://github.com/zxhlyh) | Làm cho giao diện người dùng dễ sử dụng              |
+  | [@guchenhe](https://github.com/guchenhe) & [@crazywoola](https://github.com/crazywoola) | Trải nghiệm nhà phát triển, đầu mối liên hệ cho mọi vấn đề |
+  | [@takatost](https://github.com/takatost)                     | Định hướng và kiến trúc tổng thể sản phẩm            |
+
+  Cách chúng tôi ưu tiên:
+
+  | Loại tính năng                                               | Mức độ ưu tiên |
+  | ------------------------------------------------------------ | -------------- |
+  | Tính năng ưu tiên cao được gắn nhãn bởi thành viên trong nhóm | Ưu tiên cao    |
+  | Yêu cầu tính năng phổ biến từ [bảng phản hồi cộng đồng](https://github.com/langgenius/dify/discussions/categories/feedbacks) của chúng tôi | Ưu tiên trung bình |
+  | Tính năng không quan trọng và cải tiến nhỏ                   | Ưu tiên thấp   |
+  | Có giá trị nhưng không cấp bách                              | Tính năng tương lai |
+
+### Những vấn đề khác (ví dụ: báo cáo lỗi, tối ưu hiệu suất, sửa lỗi chính tả):
+
+* Bắt đầu code ngay lập tức.
+
+  Cách chúng tôi ưu tiên:
+
+  | Loại vấn đề                                                  | Mức độ ưu tiên |
+  | ------------------------------------------------------------ | -------------- |
+  | Lỗi trong các chức năng chính (không thể đăng nhập, ứng dụng không hoạt động, lỗ hổng bảo mật) | Nghiêm trọng   |
+  | Lỗi không quan trọng, cải thiện hiệu suất                    | Ưu tiên trung bình |
+  | Sửa lỗi nhỏ (lỗi chính tả, giao diện người dùng gây nhầm lẫn nhưng vẫn hoạt động) | Ưu tiên thấp   |
+
+
+## Cài đặt
+
+Dưới đây là các bước để thiết lập Dify cho việc phát triển:
+
+### 1. Fork repository này
+
+### 2. Clone repository
+
+ Clone repository đã fork từ terminal của bạn:
+
+```
+git clone [email protected]:<tên_người_dùng_github>/dify.git
+```
+
+### 3. Kiểm tra các phụ thuộc
+
+Dify yêu cầu các phụ thuộc sau để build, hãy đảm bảo chúng đã được cài đặt trên hệ thống của bạn:
+
+- [Docker](https://www.docker.com/)
+- [Docker Compose](https://docs.docker.com/compose/install/)
+- [Node.js v18.x (LTS)](http://nodejs.org)
+- [npm](https://www.npmjs.com/) phiên bản 8.x.x hoặc [Yarn](https://yarnpkg.com/)
+- [Python](https://www.python.org/) phiên bản 3.10.x
+
+### 4. Cài đặt
+
+Dify bao gồm một backend và một frontend. Đi đến thư mục backend bằng lệnh `cd api/`, sau đó làm theo hướng dẫn trong [README của Backend](api/README.md) để cài đặt. Trong một terminal khác, đi đến thư mục frontend bằng lệnh `cd web/`, sau đó làm theo hướng dẫn trong [README của Frontend](web/README.md) để cài đặt.
+
+Kiểm tra [FAQ về cài đặt](https://docs.dify.ai/learn-more/faq/self-host-faq) để xem danh sách các vấn đề thường gặp và các bước khắc phục.
+
+### 5. Truy cập Dify trong trình duyệt của bạn
+
+Để xác nhận cài đặt của bạn, hãy truy cập [http://localhost:3000](http://localhost:3000) (địa chỉ mặc định, hoặc URL và cổng bạn đã cấu hình) trong trình duyệt. Bạn sẽ thấy Dify đang chạy.
+
+## Phát triển
+
+Nếu bạn đang thêm một nhà cung cấp mô hình, [hướng dẫn này](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/README.md) dành cho bạn.
+
+Nếu bạn đang thêm một nhà cung cấp công cụ cho Agent hoặc Workflow, [hướng dẫn này](./api/core/tools/README.md) dành cho bạn.
+
+Để giúp bạn nhanh chóng định hướng phần đóng góp của mình, dưới đây là một bản phác thảo ngắn gọn về cấu trúc backend & frontend của Dify:
+
+### Backend
+
+Backend của Dify được viết bằng Python sử dụng [Flask](https://flask.palletsprojects.com/en/3.0.x/). Nó sử dụng [SQLAlchemy](https://www.sqlalchemy.org/) cho ORM và [Celery](https://docs.celeryq.dev/en/stable/getting-started/introduction.html) cho hàng đợi tác vụ. Logic xác thực được thực hiện thông qua Flask-login.
+
+```
+[api/]
+├── constants             // Các cài đặt hằng số được sử dụng trong toàn bộ codebase.
+├── controllers           // Định nghĩa các route API và logic xử lý yêu cầu.           
+├── core                  // Điều phối ứng dụng cốt lõi, tích hợp mô hình và công cụ.
+├── docker                // Cấu hình liên quan đến Docker & containerization.
+├── events                // Xử lý và xử lý sự kiện
+├── extensions            // Mở rộng với các framework/nền tảng bên thứ 3.
+├── fields                // Định nghĩa trường cho serialization/marshalling.
+├── libs                  // Thư viện và tiện ích có thể tái sử dụng.
+├── migrations            // Script cho việc di chuyển cơ sở dữ liệu.
+├── models                // Mô hình cơ sở dữ liệu & định nghĩa schema.
+├── services              // Xác định logic nghiệp vụ.
+├── storage               // Lưu trữ khóa riêng tư.      
+├── tasks                 // Xử lý các tác vụ bất đồng bộ và công việc nền.
+└── tests
+```
+
+### Frontend
+
+Website được khởi tạo trên boilerplate [Next.js](https://nextjs.org/) bằng Typescript và sử dụng [Tailwind CSS](https://tailwindcss.com/) cho styling. [React-i18next](https://react.i18next.com/) được sử dụng cho việc quốc tế hóa.
+
+```
+[web/]
+├── app                   // layouts, pages và components
+│   ├── (commonLayout)    // layout chung được sử dụng trong toàn bộ ứng dụng
+│   ├── (shareLayout)     // layouts được chia sẻ cụ thể cho các phiên dựa trên token 
+│   ├── activate          // trang kích hoạt
+│   ├── components        // được chia sẻ bởi các trang và layouts
+│   ├── install           // trang cài đặt
+│   ├── signin            // trang đăng nhập
+│   └── styles            // styles được chia sẻ toàn cục
+├── assets                // Tài nguyên tĩnh
+├── bin                   // scripts chạy ở bước build
+├── config                // cài đặt và tùy chọn có thể điều chỉnh 
+├── context               // contexts được chia sẻ bởi các phần khác nhau của ứng dụng
+├── dictionaries          // File dịch cho từng ngôn ngữ 
+├── docker                // cấu hình container
+├── hooks                 // Hooks có thể tái sử dụng
+├── i18n                  // Cấu hình quốc tế hóa
+├── models                // mô tả các mô hình dữ liệu & hình dạng của phản hồi API
+├── public                // tài nguyên meta như favicon
+├── service               // xác định hình dạng của các hành động API
+├── test                  
+├── types                 // mô tả các tham số hàm và giá trị trả về
+└── utils                 // Các hàm tiện ích được chia sẻ
+```
+
+## Gửi PR của bạn
+
+Cuối cùng, đã đến lúc mở một pull request (PR) đến repository của chúng tôi. Đối với các tính năng lớn, chúng tôi sẽ merge chúng vào nhánh `deploy/dev` để kiểm tra trước khi đưa vào nhánh `main`. Nếu bạn gặp vấn đề như xung đột merge hoặc không biết cách mở pull request, hãy xem [hướng dẫn về pull request của GitHub](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests).
+
+Và thế là xong! Khi PR của bạn được merge, bạn sẽ được giới thiệu là một người đóng góp trong [README](https://github.com/langgenius/dify/blob/main/README.md) của chúng tôi.
+
+## Nhận trợ giúp
+
+Nếu bạn gặp khó khăn hoặc có câu hỏi cấp bách trong quá trình đóng góp, hãy đặt câu hỏi của bạn trong vấn đề GitHub liên quan, hoặc tham gia [Discord](https://discord.gg/8Tpq4AcN9c) của chúng tôi để trò chuyện nhanh chóng.

api/.env.example

@@ -247,8 +247,8 @@ API_TOOL_DEFAULT_READ_TIMEOUT=60
 HTTP_REQUEST_MAX_CONNECT_TIMEOUT=300
 HTTP_REQUEST_MAX_READ_TIMEOUT=600
 HTTP_REQUEST_MAX_WRITE_TIMEOUT=600
-HTTP_REQUEST_NODE_MAX_BINARY_SIZE=10485760 # 10MB
-HTTP_REQUEST_NODE_MAX_TEXT_SIZE=1048576 # 1MB
+HTTP_REQUEST_NODE_MAX_BINARY_SIZE=10485760
+HTTP_REQUEST_NODE_MAX_TEXT_SIZE=1048576
 
 # Log file path
 LOG_FILE=
@@ -267,4 +267,13 @@ APP_MAX_ACTIVE_REQUESTS=0
 
 
 # Celery beat configuration
-CELERY_BEAT_SCHEDULER_TIME=1
+CELERY_BEAT_SCHEDULER_TIME=1
+
+# Position configuration
+POSITION_TOOL_PINS=
+POSITION_TOOL_INCLUDES=
+POSITION_TOOL_EXCLUDES=
+
+POSITION_PROVIDER_PINS=
+POSITION_PROVIDER_INCLUDES=
+POSITION_PROVIDER_EXCLUDES=

api/.idea/vcs.xml

@@ -13,4 +13,4 @@
   <component name="VcsDirectoryMappings">
     <mapping directory="" vcs="Git" />
   </component>
-</project>
+</project>

api/.vscode/launch.json.example

@@ -5,8 +5,8 @@
             "name": "Python: Flask",
             "type": "debugpy",
             "request": "launch",
-            "python": "${workspaceFolder}/api/.venv/bin/python",
-            "cwd": "${workspaceFolder}/api",
+            "python": "${workspaceFolder}/.venv/bin/python",
+            "cwd": "${workspaceFolder}",
             "envFile": ".env",
             "module": "flask",
             "justMyCode": true,
@@ -18,15 +18,15 @@
             "args": [
                 "run",
                 "--host=0.0.0.0",
-                "--port=5001",
+                "--port=5001"
             ]
         },
         {
             "name": "Python: Celery",
             "type": "debugpy",
             "request": "launch",
-            "python": "${workspaceFolder}/api/.venv/bin/python",
-            "cwd": "${workspaceFolder}/api",
+            "python": "${workspaceFolder}/.venv/bin/python",
+            "cwd": "${workspaceFolder}",
             "module": "celery",
             "justMyCode": true,
             "envFile": ".env",

api/configs/app_config.py

@@ -37,6 +37,8 @@ class DifyConfig(
 
     CODE_MAX_NUMBER: int = 9223372036854775807
     CODE_MIN_NUMBER: int = -9223372036854775808
+    CODE_MAX_DEPTH: int = 5
+    CODE_MAX_PRECISION: int = 20
     CODE_MAX_STRING_LENGTH: int = 80000
     CODE_MAX_STRING_ARRAY_LENGTH: int = 30
     CODE_MAX_OBJECT_ARRAY_LENGTH: int = 30

api/configs/feature/__init__.py

@@ -406,6 +406,7 @@ class DataSetConfig(BaseSettings):
         default=False,
     )
 
+
 class WorkspaceConfig(BaseSettings):
     """
     Workspace configs
@@ -442,6 +443,63 @@ class CeleryBeatConfig(BaseSettings):
     )
 
 
+class PositionConfig(BaseSettings):
+
+    POSITION_PROVIDER_PINS: str = Field(
+        description='The heads of model providers',
+        default='',
+    )
+
+    POSITION_PROVIDER_INCLUDES: str = Field(
+        description='The included model providers',
+        default='',
+    )
+
+    POSITION_PROVIDER_EXCLUDES: str = Field(
+        description='The excluded model providers',
+        default='',
+    )
+
+    POSITION_TOOL_PINS: str = Field(
+        description='The heads of tools',
+        default='',
+    )
+
+    POSITION_TOOL_INCLUDES: str = Field(
+        description='The included tools',
+        default='',
+    )
+
+    POSITION_TOOL_EXCLUDES: str = Field(
+        description='The excluded tools',
+        default='',
+    )
+
+    @computed_field
+    def POSITION_PROVIDER_PINS_LIST(self) -> list[str]:
+        return [item.strip() for item in self.POSITION_PROVIDER_PINS.split(',') if item.strip() != '']
+
+    @computed_field
+    def POSITION_PROVIDER_INCLUDES_SET(self) -> set[str]:
+        return {item.strip() for item in self.POSITION_PROVIDER_INCLUDES.split(',') if item.strip() != ''}
+
+    @computed_field
+    def POSITION_PROVIDER_EXCLUDES_SET(self) -> set[str]:
+        return {item.strip() for item in self.POSITION_PROVIDER_EXCLUDES.split(',') if item.strip() != ''}
+
+    @computed_field
+    def POSITION_TOOL_PINS_LIST(self) -> list[str]:
+        return [item.strip() for item in self.POSITION_TOOL_PINS.split(',') if item.strip() != '']
+
+    @computed_field
+    def POSITION_TOOL_INCLUDES_SET(self) -> set[str]:
+        return {item.strip() for item in self.POSITION_TOOL_INCLUDES.split(',') if item.strip() != ''}
+
+    @computed_field
+    def POSITION_TOOL_EXCLUDES_SET(self) -> set[str]:
+        return {item.strip() for item in self.POSITION_TOOL_EXCLUDES.split(',') if item.strip() != ''}
+
+
 class FeatureConfig(
     # place the configs in alphabet order
     AppExecutionConfig,
@@ -466,6 +524,7 @@ class FeatureConfig(
     UpdateConfig,
     WorkflowConfig,
     WorkspaceConfig,
+    PositionConfig,
 
     # hosted services config
     HostedServiceConfig,

api/configs/packaging/__init__.py

@@ -9,7 +9,7 @@ class PackagingInfo(BaseSettings):
 
     CURRENT_VERSION: str = Field(
         description='Dify version',
-        default='0.7.0',
+        default='0.7.1',
     )
 
     COMMIT_SHA: str = Field(

api/controllers/console/app/conversation.py

@@ -154,6 +154,8 @@ class ChatConversationApi(Resource):
         parser.add_argument('message_count_gte', type=int_range(1, 99999), required=False, location='args')
         parser.add_argument('page', type=int_range(1, 99999), required=False, default=1, location='args')
         parser.add_argument('limit', type=int_range(1, 100), required=False, default=20, location='args')
+        parser.add_argument('sort_by', type=str, choices=['created_at', '-created_at', 'updated_at', '-updated_at'],
+                            required=False, default='-updated_at', location='args')
         args = parser.parse_args()
 
         subquery = (
@@ -225,7 +227,17 @@ class ChatConversationApi(Resource):
         if app_model.mode == AppMode.ADVANCED_CHAT.value:
             query = query.where(Conversation.invoke_from != InvokeFrom.DEBUGGER.value)
 
-        query = query.order_by(Conversation.created_at.desc())
+        match args['sort_by']:
+            case 'created_at':
+                query = query.order_by(Conversation.created_at.asc())
+            case '-created_at':
+                query = query.order_by(Conversation.created_at.desc())
+            case 'updated_at':
+                query = query.order_by(Conversation.updated_at.asc())
+            case '-updated_at':
+                query = query.order_by(Conversation.updated_at.desc())
+            case _:
+                query = query.order_by(Conversation.created_at.desc())
 
         conversations = db.paginate(
             query,

api/controllers/console/datasets/datasets.py

@@ -24,7 +24,7 @@ from fields.app_fields import related_app_list
 from fields.dataset_fields import dataset_detail_fields, dataset_query_detail_fields
 from fields.document_fields import document_status_fields
 from libs.login import login_required
-from models.dataset import Dataset, Document, DocumentSegment
+from models.dataset import Dataset, DatasetPermissionEnum, Document, DocumentSegment
 from models.model import ApiToken, UploadFile
 from services.dataset_service import DatasetPermissionService, DatasetService, DocumentService
 
@@ -202,7 +202,7 @@ class DatasetApi(Resource):
                             nullable=True,
                             help='Invalid indexing technique.')
         parser.add_argument('permission', type=str, location='json', choices=(
-            'only_me', 'all_team_members', 'partial_members'), help='Invalid permission.'
+            DatasetPermissionEnum.ONLY_ME, DatasetPermissionEnum.ALL_TEAM, DatasetPermissionEnum.PARTIAL_TEAM), help='Invalid permission.'
                             )
         parser.add_argument('embedding_model', type=str,
                             location='json', help='Invalid embedding model.')
@@ -239,7 +239,7 @@ class DatasetApi(Resource):
                 tenant_id, dataset_id_str, data.get('partial_member_list')
             )
         # clear partial member list when permission is only_me or all_team_members
-        elif data.get('permission') == 'only_me' or data.get('permission') == 'all_team_members':
+        elif data.get('permission') == DatasetPermissionEnum.ONLY_ME or data.get('permission') == DatasetPermissionEnum.ALL_TEAM:
             DatasetPermissionService.clear_partial_member_list(dataset_id_str)
 
         partial_member_list = DatasetPermissionService.get_dataset_partial_member_list(dataset_id_str)
@@ -573,13 +573,13 @@ class DatasetRetrievalSettingMockApi(Resource):
     @account_initialization_required
     def get(self, vector_type):
         match vector_type:
-            case VectorType.MILVUS | VectorType.RELYT | VectorType.PGVECTOR | VectorType.TIDB_VECTOR | VectorType.CHROMA | VectorType.TENCENT:
+            case VectorType.MILVUS | VectorType.RELYT | VectorType.TIDB_VECTOR | VectorType.CHROMA | VectorType.TENCENT | VectorType.PGVECTO_RS:
                 return {
                     'retrieval_method': [
                         RetrievalMethod.SEMANTIC_SEARCH.value
                     ]
                 }
-            case VectorType.QDRANT | VectorType.WEAVIATE | VectorType.OPENSEARCH| VectorType.ANALYTICDB | VectorType.MYSCALE | VectorType.ORACLE | VectorType.ELASTICSEARCH:
+            case VectorType.QDRANT | VectorType.WEAVIATE | VectorType.OPENSEARCH | VectorType.ANALYTICDB | VectorType.MYSCALE | VectorType.ORACLE | VectorType.ELASTICSEARCH | VectorType.PGVECTOR:
                 return {
                     'retrieval_method': [
                         RetrievalMethod.SEMANTIC_SEARCH.value,

api/controllers/service_api/app/conversation.py

@@ -25,6 +25,8 @@ class ConversationApi(Resource):
         parser = reqparse.RequestParser()
         parser.add_argument('last_id', type=uuid_value, location='args')
         parser.add_argument('limit', type=int_range(1, 100), required=False, default=20, location='args')
+        parser.add_argument('sort_by', type=str, choices=['created_at', '-created_at', 'updated_at', '-updated_at'],
+                            required=False, default='-updated_at', location='args')
         args = parser.parse_args()
 
         try:
@@ -33,7 +35,8 @@ class ConversationApi(Resource):
                 user=end_user,
                 last_id=args['last_id'],
                 limit=args['limit'],
-                invoke_from=InvokeFrom.SERVICE_API
+                invoke_from=InvokeFrom.SERVICE_API,
+                sort_by=args['sort_by']
             )
         except services.errors.conversation.LastConversationNotExistsError:
             raise NotFound("Last Conversation Not Exists.")

api/controllers/service_api/dataset/dataset.py

@@ -10,7 +10,7 @@ from core.model_runtime.entities.model_entities import ModelType
 from core.provider_manager import ProviderManager
 from fields.dataset_fields import dataset_detail_fields
 from libs.login import current_user
-from models.dataset import Dataset
+from models.dataset import Dataset, DatasetPermissionEnum
 from services.dataset_service import DatasetService
 
 
@@ -78,6 +78,8 @@ class DatasetListApi(DatasetApiResource):
         parser.add_argument('indexing_technique', type=str, location='json',
                             choices=Dataset.INDEXING_TECHNIQUE_LIST,
                             help='Invalid indexing technique.')
+        parser.add_argument('permission', type=str, location='json', choices=(
+            DatasetPermissionEnum.ONLY_ME, DatasetPermissionEnum.ALL_TEAM, DatasetPermissionEnum.PARTIAL_TEAM), help='Invalid permission.', required=False, nullable=False)
         args = parser.parse_args()
 
         try:
@@ -85,7 +87,8 @@ class DatasetListApi(DatasetApiResource):
                 tenant_id=tenant_id,
                 name=args['name'],
                 indexing_technique=args['indexing_technique'],
-                account=current_user
+                account=current_user,
+                permission=args['permission']
             )
         except services.errors.dataset.DatasetNameDuplicateError:
             raise DatasetNameDuplicateError()

api/controllers/web/conversation.py

@@ -26,6 +26,8 @@ class ConversationListApi(WebApiResource):
         parser.add_argument('last_id', type=uuid_value, location='args')
         parser.add_argument('limit', type=int_range(1, 100), required=False, default=20, location='args')
         parser.add_argument('pinned', type=str, choices=['true', 'false', None], location='args')
+        parser.add_argument('sort_by', type=str, choices=['created_at', '-created_at', 'updated_at', '-updated_at'],
+                            required=False, default='-updated_at', location='args')
         args = parser.parse_args()
 
         pinned = None
@@ -40,6 +42,7 @@ class ConversationListApi(WebApiResource):
                 limit=args['limit'],
                 invoke_from=InvokeFrom.WEB_APP,
                 pinned=pinned,
+                sort_by=args['sort_by']
             )
         except LastConversationNotExistsError:
             raise NotFound("Last Conversation Not Exists.")

api/core/app/app_config/easy_ui_based_app/dataset/manager.py

@@ -93,7 +93,7 @@ class DatasetConfigManager:
                     reranking_model=dataset_configs.get('reranking_model'),
                     weights=dataset_configs.get('weights'),
                     reranking_enabled=dataset_configs.get('reranking_enabled', True),
-                    rerank_mode=dataset_configs["reranking_mode"],
+                    rerank_mode=dataset_configs.get('rerank_mode', 'reranking_model'),
                 )
             )
 

api/core/app/app_config/easy_ui_based_app/variables/manager.py

@@ -1,6 +1,6 @@
 import re
 
-from core.app.app_config.entities import ExternalDataVariableEntity, VariableEntity
+from core.app.app_config.entities import ExternalDataVariableEntity, VariableEntity, VariableEntityType
 from core.external_data_tool.factory import ExternalDataToolFactory
 
 
@@ -13,7 +13,7 @@ class BasicVariablesConfigManager:
         :param config: model config args
         """
         external_data_variables = []
-        variables = []
+        variable_entities = []
 
         # old external_data_tools
         external_data_tools = config.get('external_data_tools', [])
@@ -30,50 +30,41 @@ class BasicVariablesConfigManager:
             )
 
         # variables and external_data_tools
-        for variable in config.get('user_input_form', []):
-            typ = list(variable.keys())[0]
-            if typ == 'external_data_tool':
-                val = variable[typ]
-                if 'config' not in val:
+        for variables in config.get('user_input_form', []):
+            variable_type = list(variables.keys())[0]
+            if variable_type == VariableEntityType.EXTERNAL_DATA_TOOL:
+                variable = variables[variable_type]
+                if 'config' not in variable:
                     continue
 
                 external_data_variables.append(
                     ExternalDataVariableEntity(
-                        variable=val['variable'],
-                        type=val['type'],
-                        config=val['config']
+                        variable=variable['variable'],
+                        type=variable['type'],
+                        config=variable['config']
                     )
                 )
-            elif typ in [
-                VariableEntity.Type.TEXT_INPUT.value,
-                VariableEntity.Type.PARAGRAPH.value,
-                VariableEntity.Type.NUMBER.value,
+            elif variable_type in [
+                VariableEntityType.TEXT_INPUT,
+                VariableEntityType.PARAGRAPH,
+                VariableEntityType.NUMBER,
+                VariableEntityType.SELECT,
             ]:
-                variables.append(
+                variable = variables[variable_type]
+                variable_entities.append(
                     VariableEntity(
-                        type=VariableEntity.Type.value_of(typ),
-                        variable=variable[typ].get('variable'),
-                        description=variable[typ].get('description'),
-                        label=variable[typ].get('label'),
-                        required=variable[typ].get('required', False),
-                        max_length=variable[typ].get('max_length'),
-                        default=variable[typ].get('default'),
-                    )
-                )
-            elif typ == VariableEntity.Type.SELECT.value:
-                variables.append(
-                    VariableEntity(
-                        type=VariableEntity.Type.SELECT,
-                        variable=variable[typ].get('variable'),
-                        description=variable[typ].get('description'),
-                        label=variable[typ].get('label'),
-                        required=variable[typ].get('required', False),
-                        options=variable[typ].get('options'),
-                        default=variable[typ].get('default'),
+                        type=variable_type,
+                        variable=variable.get('variable'),
+                        description=variable.get('description'),
+                        label=variable.get('label'),
+                        required=variable.get('required', False),
+                        max_length=variable.get('max_length'),
+                        options=variable.get('options'),
+                        default=variable.get('default'),
                     )
                 )
 
-        return variables, external_data_variables
+        return variable_entities, external_data_variables
 
     @classmethod
     def validate_and_set_defaults(cls, tenant_id: str, config: dict) -> tuple[dict, list[str]]:
@@ -183,4 +174,4 @@ class BasicVariablesConfigManager:
                 config=config
             )
 
-        return config, ["external_data_tools"]
+        return config, ["external_data_tools"]

api/core/app/app_config/entities.py

@@ -82,43 +82,29 @@ class PromptTemplateEntity(BaseModel):
     advanced_completion_prompt_template: Optional[AdvancedCompletionPromptTemplateEntity] = None
 
 
+class VariableEntityType(str, Enum):
+    TEXT_INPUT = "text-input"
+    SELECT = "select"
+    PARAGRAPH = "paragraph"
+    NUMBER = "number"
+    EXTERNAL_DATA_TOOL = "external-data-tool"
+
+
 class VariableEntity(BaseModel):
     """
     Variable Entity.
     """
-    class Type(Enum):
-        TEXT_INPUT = 'text-input'
-        SELECT = 'select'
-        PARAGRAPH = 'paragraph'
-        NUMBER = 'number'
-
-        @classmethod
-        def value_of(cls, value: str) -> 'VariableEntity.Type':
-            """
-            Get value of given mode.
-
-            :param value: mode value
-            :return: mode
-            """
-            for mode in cls:
-                if mode.value == value:
-                    return mode
-            raise ValueError(f'invalid variable type value {value}')
 
     variable: str
     label: str
     description: Optional[str] = None
-    type: Type
+    type: VariableEntityType
     required: bool = False
     max_length: Optional[int] = None
     options: Optional[list[str]] = None
     default: Optional[str] = None
     hint: Optional[str] = None
 
-    @property
-    def name(self) -> str:
-        return self.variable
-
 
 class ExternalDataVariableEntity(BaseModel):
     """
@@ -252,4 +238,4 @@ class WorkflowUIBasedAppConfig(AppConfig):
     """
     Workflow UI Based App Config Entity.
     """
-    workflow_id: str
+    workflow_id: str

api/core/app/apps/advanced_chat/app_generator.py

@@ -29,7 +29,7 @@ from core.file.message_file_parser import MessageFileParser
 from core.model_runtime.errors.invoke import InvokeAuthorizationError, InvokeError
 from core.ops.ops_trace_manager import TraceQueueManager
 from core.workflow.entities.variable_pool import VariablePool
-from core.workflow.enums import SystemVariable
+from core.workflow.enums import SystemVariableKey
 from extensions.ext_database import db
 from models.account import Account
 from models.model import App, Conversation, EndUser, Message
@@ -46,7 +46,7 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
         args: dict,
         invoke_from: InvokeFrom,
         stream: bool = True,
-    ) -> Union[dict, Generator[dict, None, None]]:
+    ):
         """
         Generate App response.
 
@@ -73,8 +73,9 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
 
         # get conversation
         conversation = None
-        if args.get('conversation_id'):
-            conversation = self._get_conversation_by_user(app_model, args.get('conversation_id'), user)
+        conversation_id = args.get('conversation_id')
+        if conversation_id:
+            conversation = self._get_conversation_by_user(app_model=app_model, conversation_id=conversation_id, user=user)
 
         # parse files
         files = args['files'] if args.get('files') else []
@@ -133,8 +134,7 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
                                   node_id: str,
                                   user: Account,
                                   args: dict,
-                                  stream: bool = True) \
-            -> Union[dict, Generator[dict, None, None]]:
+                                  stream: bool = True):
         """
         Generate App response.
 
@@ -157,8 +157,9 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
 
         # get conversation
         conversation = None
-        if args.get('conversation_id'):
-            conversation = self._get_conversation_by_user(app_model, args.get('conversation_id'), user)
+        conversation_id = args.get('conversation_id')
+        if conversation_id:
+            conversation = self._get_conversation_by_user(app_model=app_model, conversation_id=conversation_id, user=user)
 
         # convert to app config
         app_config = AdvancedChatAppConfigManager.get_app_config(
@@ -200,8 +201,7 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
                  invoke_from: InvokeFrom,
                  application_generate_entity: AdvancedChatAppGenerateEntity,
                  conversation: Conversation | None = None,
-                 stream: bool = True) \
-            -> Union[dict, Generator[dict, None, None]]:
+                 stream: bool = True):
         is_first_conversation = False
         if not conversation:
             is_first_conversation = True
@@ -270,11 +270,11 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
 
         # Create a variable pool.
         system_inputs = {
-            SystemVariable.QUERY: query,
-            SystemVariable.FILES: files,
-            SystemVariable.CONVERSATION_ID: conversation_id,
-            SystemVariable.USER_ID: user_id,
-            SystemVariable.DIALOGUE_COUNT: conversation_dialogue_count,
+            SystemVariableKey.QUERY: query,
+            SystemVariableKey.FILES: files,
+            SystemVariableKey.CONVERSATION_ID: conversation_id,
+            SystemVariableKey.USER_ID: user_id,
+            SystemVariableKey.DIALOGUE_COUNT: conversation_dialogue_count,
         }
         variable_pool = VariablePool(
             system_variables=system_inputs,
@@ -362,7 +362,7 @@ class AdvancedChatAppGenerator(MessageBasedAppGenerator):
                 logger.exception("Validation Error when generating")
                 queue_manager.publish_error(e, PublishFrom.APPLICATION_MANAGER)
             except (ValueError, InvokeError) as e:
-                if os.environ.get("DEBUG") and os.environ.get("DEBUG").lower() == 'true':
+                if os.environ.get("DEBUG", "false").lower() == 'true':
                     logger.exception("Error when generating")
                 queue_manager.publish_error(e, PublishFrom.APPLICATION_MANAGER)
             except Exception as e:

api/core/app/apps/advanced_chat/generate_task_pipeline.py

@@ -49,7 +49,7 @@ from core.model_runtime.entities.llm_entities import LLMUsage
 from core.model_runtime.utils.encoders import jsonable_encoder
 from core.ops.ops_trace_manager import TraceQueueManager
 from core.workflow.entities.node_entities import NodeType
-from core.workflow.enums import SystemVariable
+from core.workflow.enums import SystemVariableKey
 from core.workflow.nodes.answer.answer_node import AnswerNode
 from core.workflow.nodes.answer.entities import TextGenerateRouteChunk, VarGenerateRouteChunk
 from events.message_event import message_was_created
@@ -74,7 +74,7 @@ class AdvancedChatAppGenerateTaskPipeline(BasedGenerateTaskPipeline, WorkflowCyc
     _workflow: Workflow
     _user: Union[Account, EndUser]
     # Deprecated
-    _workflow_system_variables: dict[SystemVariable, Any]
+    _workflow_system_variables: dict[SystemVariableKey, Any]
     _iteration_nested_relations: dict[str, list[str]]
 
     def __init__(
@@ -108,10 +108,10 @@ class AdvancedChatAppGenerateTaskPipeline(BasedGenerateTaskPipeline, WorkflowCyc
         self._message = message
         # Deprecated
         self._workflow_system_variables = {
-            SystemVariable.QUERY: message.query,
-            SystemVariable.FILES: application_generate_entity.files,
-            SystemVariable.CONVERSATION_ID: conversation.id,
-            SystemVariable.USER_ID: user_id,
+            SystemVariableKey.QUERY: message.query,
+            SystemVariableKey.FILES: application_generate_entity.files,
+            SystemVariableKey.CONVERSATION_ID: conversation.id,
+            SystemVariableKey.USER_ID: user_id,
         }
 
         self._task_state = AdvancedChatTaskState(

api/core/app/apps/base_app_generator.py

@@ -1,7 +1,7 @@
 from collections.abc import Mapping
 from typing import Any, Optional
 
-from core.app.app_config.entities import AppConfig, VariableEntity
+from core.app.app_config.entities import AppConfig, VariableEntity, VariableEntityType
 
 
 class BaseAppGenerator:
@@ -9,29 +9,29 @@ class BaseAppGenerator:
         user_inputs = user_inputs or {}
         # Filter input variables from form configuration, handle required fields, default values, and option values
         variables = app_config.variables
-        filtered_inputs = {var.name: self._validate_input(inputs=user_inputs, var=var) for var in variables}
+        filtered_inputs = {var.variable: self._validate_input(inputs=user_inputs, var=var) for var in variables}
         filtered_inputs = {k: self._sanitize_value(v) for k, v in filtered_inputs.items()}
         return filtered_inputs
 
     def _validate_input(self, *, inputs: Mapping[str, Any], var: VariableEntity):
-        user_input_value = inputs.get(var.name)
+        user_input_value = inputs.get(var.variable)
         if var.required and not user_input_value:
-            raise ValueError(f'{var.name} is required in input form')
+            raise ValueError(f'{var.variable} is required in input form')
         if not var.required and not user_input_value:
             # TODO: should we return None here if the default value is None?
             return var.default or ''
         if (
             var.type
             in (
-                VariableEntity.Type.TEXT_INPUT,
-                VariableEntity.Type.SELECT,
-                VariableEntity.Type.PARAGRAPH,
+                VariableEntityType.TEXT_INPUT,
+                VariableEntityType.SELECT,
+                VariableEntityType.PARAGRAPH,
             )
             and user_input_value
             and not isinstance(user_input_value, str)
         ):
-            raise ValueError(f"(type '{var.type}') {var.name} in input form must be a string")
-        if var.type == VariableEntity.Type.NUMBER and isinstance(user_input_value, str):
+            raise ValueError(f"(type '{var.type}') {var.variable} in input form must be a string")
+        if var.type == VariableEntityType.NUMBER and isinstance(user_input_value, str):
             # may raise ValueError if user_input_value is not a valid number
             try:
                 if '.' in user_input_value:
@@ -39,14 +39,14 @@ class BaseAppGenerator:
                 else:
                     return int(user_input_value)
             except ValueError:
-                raise ValueError(f"{var.name} in input form must be a valid number")
-        if var.type == VariableEntity.Type.SELECT:
+                raise ValueError(f"{var.variable} in input form must be a valid number")
+        if var.type == VariableEntityType.SELECT:
             options = var.options or []
             if user_input_value not in options:
-                raise ValueError(f'{var.name} in input form must be one of the following: {options}')
-        elif var.type in (VariableEntity.Type.TEXT_INPUT, VariableEntity.Type.PARAGRAPH):
+                raise ValueError(f'{var.variable} in input form must be one of the following: {options}')
+        elif var.type in (VariableEntityType.TEXT_INPUT, VariableEntityType.PARAGRAPH):
             if var.max_length and user_input_value and len(user_input_value) > var.max_length:
-                raise ValueError(f'{var.name} in input form must be less than {var.max_length} characters')
+                raise ValueError(f'{var.variable} in input form must be less than {var.max_length} characters')
 
         return user_input_value
 

api/core/app/apps/message_based_app_generator.py

@@ -1,6 +1,7 @@
 import json
 import logging
 from collections.abc import Generator
+from datetime import datetime, timezone
 from typing import Optional, Union
 
 from sqlalchemy import and_
@@ -36,17 +37,17 @@ logger = logging.getLogger(__name__)
 class MessageBasedAppGenerator(BaseAppGenerator):
 
     def _handle_response(
-        self, application_generate_entity: Union[
-            ChatAppGenerateEntity,
-            CompletionAppGenerateEntity,
-            AgentChatAppGenerateEntity,
-            AdvancedChatAppGenerateEntity
-        ],
-        queue_manager: AppQueueManager,
-        conversation: Conversation,
-        message: Message,
-        user: Union[Account, EndUser],
-        stream: bool = False,
+            self, application_generate_entity: Union[
+                ChatAppGenerateEntity,
+                CompletionAppGenerateEntity,
+                AgentChatAppGenerateEntity,
+                AdvancedChatAppGenerateEntity
+            ],
+            queue_manager: AppQueueManager,
+            conversation: Conversation,
+            message: Message,
+            user: Union[Account, EndUser],
+            stream: bool = False,
     ) -> Union[
         ChatbotAppBlockingResponse,
         CompletionAppBlockingResponse,
@@ -193,6 +194,9 @@ class MessageBasedAppGenerator(BaseAppGenerator):
             db.session.add(conversation)
             db.session.commit()
             db.session.refresh(conversation)
+        else:
+            conversation.updated_at = datetime.now(timezone.utc).replace(tzinfo=None)
+            db.session.commit()
 
         message = Message(
             app_id=app_config.app_id,

api/core/app/apps/workflow/app_runner.py

@@ -12,7 +12,7 @@ from core.app.entities.app_invoke_entities import (
 )
 from core.workflow.callbacks.base_workflow_callback import WorkflowCallback
 from core.workflow.entities.variable_pool import VariablePool
-from core.workflow.enums import SystemVariable
+from core.workflow.enums import SystemVariableKey
 from core.workflow.nodes.base_node import UserFrom
 from core.workflow.workflow_engine_manager import WorkflowEngineManager
 from extensions.ext_database import db
@@ -67,8 +67,8 @@ class WorkflowAppRunner:
 
         # Create a variable pool.
         system_inputs = {
-            SystemVariable.FILES: files,
-            SystemVariable.USER_ID: user_id,
+            SystemVariableKey.FILES: files,
+            SystemVariableKey.USER_ID: user_id,
         }
         variable_pool = VariablePool(
             system_variables=system_inputs,

api/core/app/apps/workflow/generate_task_pipeline.py

@@ -43,7 +43,7 @@ from core.app.task_pipeline.based_generate_task_pipeline import BasedGenerateTas
 from core.app.task_pipeline.workflow_cycle_manage import WorkflowCycleManage
 from core.ops.ops_trace_manager import TraceQueueManager
 from core.workflow.entities.node_entities import NodeType
-from core.workflow.enums import SystemVariable
+from core.workflow.enums import SystemVariableKey
 from core.workflow.nodes.end.end_node import EndNode
 from extensions.ext_database import db
 from models.account import Account
@@ -67,7 +67,7 @@ class WorkflowAppGenerateTaskPipeline(BasedGenerateTaskPipeline, WorkflowCycleMa
     _user: Union[Account, EndUser]
     _task_state: WorkflowTaskState
     _application_generate_entity: WorkflowAppGenerateEntity
-    _workflow_system_variables: dict[SystemVariable, Any]
+    _workflow_system_variables: dict[SystemVariableKey, Any]
     _iteration_nested_relations: dict[str, list[str]]
 
     def __init__(self, application_generate_entity: WorkflowAppGenerateEntity,
@@ -92,8 +92,8 @@ class WorkflowAppGenerateTaskPipeline(BasedGenerateTaskPipeline, WorkflowCycleMa
 
         self._workflow = workflow
         self._workflow_system_variables = {
-            SystemVariable.FILES: application_generate_entity.files,
-            SystemVariable.USER_ID: user_id
+            SystemVariableKey.FILES: application_generate_entity.files,
+            SystemVariableKey.USER_ID: user_id
         }
 
         self._task_state = WorkflowTaskState(

api/core/app/task_pipeline/workflow_cycle_state_manager.py

@@ -2,7 +2,7 @@ from typing import Any, Union
 
 from core.app.entities.app_invoke_entities import AdvancedChatAppGenerateEntity, WorkflowAppGenerateEntity
 from core.app.entities.task_entities import AdvancedChatTaskState, WorkflowTaskState
-from core.workflow.enums import SystemVariable
+from core.workflow.enums import SystemVariableKey
 from models.account import Account
 from models.model import EndUser
 from models.workflow import Workflow
@@ -13,4 +13,4 @@ class WorkflowCycleStateManager:
     _workflow: Workflow
     _user: Union[Account, EndUser]
     _task_state: Union[AdvancedChatTaskState, WorkflowTaskState]
-    _workflow_system_variables: dict[SystemVariable, Any]
+    _workflow_system_variables: dict[SystemVariableKey, Any]

api/core/helper/code_executor/code_executor.py

@@ -1,15 +1,13 @@
 import logging
-import time
 from enum import Enum
 from threading import Lock
-from typing import Literal, Optional
+from typing import Optional
 
-from httpx import get, post
+from httpx import Timeout, post
 from pydantic import BaseModel
 from yarl import URL
 
 from configs import dify_config
-from core.helper.code_executor.entities import CodeDependency
 from core.helper.code_executor.javascript.javascript_transformer import NodeJsTemplateTransformer
 from core.helper.code_executor.jinja2.jinja2_transformer import Jinja2TemplateTransformer
 from core.helper.code_executor.python3.python3_transformer import Python3TemplateTransformer
@@ -21,7 +19,7 @@ logger = logging.getLogger(__name__)
 CODE_EXECUTION_ENDPOINT = dify_config.CODE_EXECUTION_ENDPOINT
 CODE_EXECUTION_API_KEY = dify_config.CODE_EXECUTION_API_KEY
 
-CODE_EXECUTION_TIMEOUT = (10, 60)
+CODE_EXECUTION_TIMEOUT = Timeout(connect=10, write=10, read=60, pool=None)
 
 class CodeExecutionException(Exception):
     pass
@@ -66,8 +64,7 @@ class CodeExecutor:
     def execute_code(cls, 
                      language: CodeLanguage, 
                      preload: str, 
-                     code: str, 
-                     dependencies: Optional[list[CodeDependency]] = None) -> str:
+                     code: str) -> str:
         """
         Execute code
         :param language: code language
@@ -87,9 +84,6 @@ class CodeExecutor:
             'enable_network': True
         }
 
-        if dependencies:
-            data['dependencies'] = [dependency.model_dump() for dependency in dependencies]
-
         try:
             response = post(str(url), json=data, headers=headers, timeout=CODE_EXECUTION_TIMEOUT)
             if response.status_code == 503:
@@ -116,10 +110,10 @@ class CodeExecutor:
         if response.data.error:
             raise CodeExecutionException(response.data.error)
         
-        return response.data.stdout
+        return response.data.stdout or ''
 
     @classmethod
-    def execute_workflow_code_template(cls, language: CodeLanguage, code: str, inputs: dict, dependencies: Optional[list[CodeDependency]] = None) -> dict:
+    def execute_workflow_code_template(cls, language: CodeLanguage, code: str, inputs: dict) -> dict:
         """
         Execute code
         :param language: code language
@@ -131,67 +125,12 @@ class CodeExecutor:
         if not template_transformer:
             raise CodeExecutionException(f'Unsupported language {language}')
 
-        runner, preload, dependencies = template_transformer.transform_caller(code, inputs, dependencies)
+        runner, preload = template_transformer.transform_caller(code, inputs)
 
         try:
-            response = cls.execute_code(language, preload, runner, dependencies)
+            response = cls.execute_code(language, preload, runner)
         except CodeExecutionException as e:
             raise e
 
         return template_transformer.transform_response(response)
-    
-    @classmethod
-    def list_dependencies(cls, language: str) -> list[CodeDependency]:
-        if language not in cls.supported_dependencies_languages:
-            return []
-
-        with cls.dependencies_cache_lock:
-            if language in cls.dependencies_cache:
-                # check expiration
-                dependencies = cls.dependencies_cache[language]
-                if dependencies['expiration'] > time.time():
-                    return dependencies['data']
-                # remove expired cache
-                del cls.dependencies_cache[language]
-        
-        dependencies = cls._get_dependencies(language)
-        with cls.dependencies_cache_lock:
-            cls.dependencies_cache[language] = {
-                'data': dependencies,
-                'expiration': time.time() + 60
-            }
-        
-        return dependencies
-        
-    @classmethod
-    def _get_dependencies(cls, language: Literal['python3']) -> list[CodeDependency]:
-        """
-        List dependencies
-        """
-        url = URL(CODE_EXECUTION_ENDPOINT) / 'v1' / 'sandbox' / 'dependencies'
-
-        headers = {
-            'X-Api-Key': CODE_EXECUTION_API_KEY
-        }
-
-        running_language = cls.code_language_to_running_language.get(language)
-        if isinstance(running_language, Enum):
-            running_language = running_language.value
-
-        data = {
-            'language': running_language,
-        }
-
-        try:
-            response = get(str(url), params=data, headers=headers, timeout=CODE_EXECUTION_TIMEOUT)
-            if response.status_code != 200:
-                raise Exception(f'Failed to list dependencies, got status code {response.status_code}, please check if the sandbox service is running')
-            response = response.json()
-            dependencies = response.get('data', {}).get('dependencies', [])
-            return [
-                CodeDependency(**dependency) for dependency in dependencies
-                if dependency.get('name') not in Python3TemplateTransformer.get_standard_packages()
-            ]
-        except Exception as e:
-            logger.exception(f'Failed to list dependencies: {e}')
-            return []
+    

api/core/helper/code_executor/code_node_provider.py

@@ -2,8 +2,6 @@ from abc import abstractmethod
 
 from pydantic import BaseModel
 
-from core.helper.code_executor.code_executor import CodeExecutor
-
 
 class CodeNodeProvider(BaseModel):
     @staticmethod
@@ -23,10 +21,6 @@ class CodeNodeProvider(BaseModel):
         """
         pass
 
-    @classmethod
-    def get_default_available_packages(cls) -> list[dict]:
-        return [p.model_dump() for p in CodeExecutor.list_dependencies(cls.get_language())]
-
     @classmethod
     def get_default_config(cls) -> dict:
         return {
@@ -50,6 +44,5 @@ class CodeNodeProvider(BaseModel):
                         "children": None
                     }
                 }
-            },
-            "available_dependencies": cls.get_default_available_packages(),
+            }
         }

api/core/helper/code_executor/entities.py

@@ -1,6 +0,0 @@
-from pydantic import BaseModel
-
-
-class CodeDependency(BaseModel):
-    name: str
-    version: str

api/core/helper/code_executor/jinja2/jinja2_formatter.py

@@ -3,7 +3,7 @@ from core.helper.code_executor.code_executor import CodeExecutor, CodeLanguage
 
 class Jinja2Formatter:
     @classmethod
-    def format(cls, template: str, inputs: str) -> str:
+    def format(cls, template: str, inputs: dict) -> str:
         """
         Format template
         :param template: template

api/core/helper/code_executor/jinja2/jinja2_transformer.py

@@ -1,14 +1,9 @@
 from textwrap import dedent
 
-from core.helper.code_executor.python3.python3_transformer import Python3TemplateTransformer
 from core.helper.code_executor.template_transformer import TemplateTransformer
 
 
 class Jinja2TemplateTransformer(TemplateTransformer):
-    @classmethod
-    def get_standard_packages(cls) -> set[str]:
-        return {'jinja2'} | Python3TemplateTransformer.get_standard_packages()
-
     @classmethod
     def transform_response(cls, response: str) -> dict:
         """

api/core/helper/code_executor/python3/python3_code_provider.py

@@ -13,7 +13,7 @@ class Python3CodeProvider(CodeNodeProvider):
     def get_default_code(cls) -> str:
         return dedent(
             """
-            def main(arg1: int, arg2: int) -> dict:
+            def main(arg1: str, arg2: str) -> dict:
                 return {
                     "result": arg1 + arg2,
                 }

api/core/helper/code_executor/python3/python3_transformer.py

@@ -4,30 +4,6 @@ from core.helper.code_executor.template_transformer import TemplateTransformer
 
 
 class Python3TemplateTransformer(TemplateTransformer):
-    @classmethod
-    def get_standard_packages(cls) -> set[str]:
-        return {
-            'base64',
-            'binascii',
-            'collections',
-            'datetime',
-            'functools',
-            'hashlib',
-            'hmac',
-            'itertools',
-            'json',
-            'math',
-            'operator',
-            'os',
-            'random',
-            're',
-            'string',
-            'sys',
-            'time',
-            'traceback',
-            'uuid',
-        }
-
     @classmethod
     def get_runner_script(cls) -> str:
         runner_script = dedent(f"""

api/core/helper/code_executor/template_transformer.py

@@ -2,9 +2,6 @@ import json
 import re
 from abc import ABC, abstractmethod
 from base64 import b64encode
-from typing import Optional
-
-from core.helper.code_executor.entities import CodeDependency
 
 
 class TemplateTransformer(ABC):
@@ -13,12 +10,7 @@ class TemplateTransformer(ABC):
     _result_tag: str = '<<RESULT>>'
 
     @classmethod
-    def get_standard_packages(cls) -> set[str]:
-        return set()
-
-    @classmethod
-    def transform_caller(cls, code: str, inputs: dict,
-                         dependencies: Optional[list[CodeDependency]] = None) -> tuple[str, str, list[CodeDependency]]:
+    def transform_caller(cls, code: str, inputs: dict) -> tuple[str, str]:
         """
         Transform code to python runner
         :param code: code
@@ -28,14 +20,7 @@ class TemplateTransformer(ABC):
         runner_script = cls.assemble_runner_script(code, inputs)
         preload_script = cls.get_preload_script()
 
-        packages = dependencies or []
-        standard_packages = cls.get_standard_packages()
-        for package in standard_packages:
-            if package not in packages:
-                packages.append(CodeDependency(name=package, version=''))
-        packages = list({dep.name: dep for dep in packages if dep.name}.values())
-
-        return runner_script, preload_script, packages
+        return runner_script, preload_script
 
     @classmethod
     def extract_result_str_from_response(cls, response: str) -> str:

api/core/helper/position_helper.py

@@ -3,6 +3,7 @@ from collections import OrderedDict
 from collections.abc import Callable
 from typing import Any
 
+from configs import dify_config
 from core.tools.utils.yaml_utils import load_yaml_file
 
 
@@ -19,6 +20,87 @@ def get_position_map(folder_path: str, *, file_name: str = "_position.yaml") ->
     return {name: index for index, name in enumerate(positions)}
 
 
+def get_tool_position_map(folder_path: str, file_name: str = "_position.yaml") -> dict[str, int]:
+    """
+    Get the mapping for tools from name to index from a YAML file.
+    :param folder_path:
+    :param file_name: the YAML file name, default to '_position.yaml'
+    :return: a dict with name as key and index as value
+    """
+    position_map = get_position_map(folder_path, file_name=file_name)
+
+    return pin_position_map(
+        position_map,
+        pin_list=dify_config.POSITION_TOOL_PINS_LIST,
+    )
+
+
+def get_provider_position_map(folder_path: str, file_name: str = "_position.yaml") -> dict[str, int]:
+    """
+    Get the mapping for providers from name to index from a YAML file.
+    :param folder_path:
+    :param file_name: the YAML file name, default to '_position.yaml'
+    :return: a dict with name as key and index as value
+    """
+    position_map = get_position_map(folder_path, file_name=file_name)
+    return pin_position_map(
+        position_map,
+        pin_list=dify_config.POSITION_PROVIDER_PINS_LIST,
+    )
+
+
+def pin_position_map(original_position_map: dict[str, int], pin_list: list[str]) -> dict[str, int]:
+    """
+    Pin the items in the pin list to the beginning of the position map.
+    Overall logic: exclude > include > pin
+    :param position_map: the position map to be sorted and filtered
+    :param pin_list: the list of pins to be put at the beginning
+    :return: the sorted position map
+    """
+    positions = sorted(original_position_map.keys(), key=lambda x: original_position_map[x])
+
+    # Add pins to position map
+    position_map = {name: idx for idx, name in enumerate(pin_list)}
+
+    # Add remaining positions to position map
+    start_idx = len(position_map)
+    for name in positions:
+        if name not in position_map:
+            position_map[name] = start_idx
+            start_idx += 1
+
+    return position_map
+
+
+def is_filtered(
+        include_set: set[str],
+        exclude_set: set[str],
+        data: Any,
+        name_func: Callable[[Any], str],
+) -> bool:
+    """
+    Chcek if the object should be filtered out.
+    Overall logic: exclude > include > pin
+    :param include_set: the set of names to be included
+    :param exclude_set: the set of names to be excluded
+    :param name_func: the function to get the name of the object
+    :param data: the data to be filtered
+    :return: True if the object should be filtered out, False otherwise
+    """
+    if not data:
+        return False
+    if not include_set and not exclude_set:
+        return False
+
+    name = name_func(data)
+
+    if name in exclude_set:  # exclude_set is prioritized
+        return True
+    if include_set and name not in include_set:  # filter out only if include_set is not empty
+        return True
+    return False
+
+
 def sort_by_position_map(
         position_map: dict[str, int],
         data: list[Any],

api/core/indexing_runner.py

@@ -700,6 +700,7 @@ class IndexingRunner:
                 DatasetDocument.tokens: tokens,
                 DatasetDocument.completed_at: datetime.datetime.now(datetime.timezone.utc).replace(tzinfo=None),
                 DatasetDocument.indexing_latency: indexing_end_at - indexing_start_at,
+                DatasetDocument.error: None,
             }
         )
 

api/core/model_manager.py

@@ -368,6 +368,15 @@ class ModelManager:
 
         return ModelInstance(provider_model_bundle, model)
 
+    def get_default_provider_model_name(self, tenant_id: str, model_type: ModelType) -> tuple[str, str]:
+        """
+        Return first provider and the first model in the provider
+        :param tenant_id: tenant id
+        :param model_type: model type
+        :return: provider name, model name
+        """
+        return self._provider_manager.get_first_provider_first_model(tenant_id, model_type)
+
     def get_default_model_instance(self, tenant_id: str, model_type: ModelType) -> ModelInstance:
         """
         Get default model instance
@@ -502,7 +511,6 @@ class LBModelManager:
             config.id
         )
 
-
         res = redis_client.exists(cooldown_cache_key)
         res = cast(bool, res)
         return res

api/core/model_runtime/model_providers/__base/ai_model.py

@@ -151,9 +151,9 @@ class AIModel(ABC):
             os.path.join(provider_model_type_path, model_schema_yaml)
             for model_schema_yaml in os.listdir(provider_model_type_path)
             if not model_schema_yaml.startswith('__')
-               and not model_schema_yaml.startswith('_')
-               and os.path.isfile(os.path.join(provider_model_type_path, model_schema_yaml))
-               and model_schema_yaml.endswith('.yaml')
+            and not model_schema_yaml.startswith('_')
+            and os.path.isfile(os.path.join(provider_model_type_path, model_schema_yaml))
+            and model_schema_yaml.endswith('.yaml')
         ]
 
         # get _position.yaml file path

api/core/model_runtime/model_providers/__base/large_language_model.py

@@ -185,7 +185,7 @@ if you are not sure about the structure.
                 stream=stream,
                 user=user
             )
-        
+
         model_parameters.pop("response_format")
         stop = stop or []
         stop.extend(["\n```", "```\n"])
@@ -249,10 +249,10 @@ if you are not sure about the structure.
                     prompt_messages=prompt_messages,
                     input_generator=new_generator()
                 )
-            
+
         return response
 
-    def _code_block_mode_stream_processor(self, model: str, prompt_messages: list[PromptMessage], 
+    def _code_block_mode_stream_processor(self, model: str, prompt_messages: list[PromptMessage],
                                           input_generator: Generator[LLMResultChunk, None, None]
                                         ) -> Generator[LLMResultChunk, None, None]:
         """
@@ -310,7 +310,7 @@ if you are not sure about the structure.
                     )
                 )
 
-    def _code_block_mode_stream_processor_with_backtick(self, model: str, prompt_messages: list, 
+    def _code_block_mode_stream_processor_with_backtick(self, model: str, prompt_messages: list,
                                         input_generator:  Generator[LLMResultChunk, None, None]) \
                                     ->  Generator[LLMResultChunk, None, None]:
         """
@@ -470,7 +470,7 @@ if you are not sure about the structure.
         :return: full response or stream response chunk generator result
         """
         raise NotImplementedError
-    
+
     @abstractmethod
     def get_num_tokens(self, model: str, credentials: dict, prompt_messages: list[PromptMessage],
                        tools: Optional[list[PromptMessageTool]] = None) -> int:
@@ -792,6 +792,13 @@ if you are not sure about the structure.
                 if not isinstance(parameter_value, str):
                     raise ValueError(f"Model Parameter {parameter_name} should be string.")
 
+                # validate options
+                if parameter_rule.options and parameter_value not in parameter_rule.options:
+                    raise ValueError(f"Model Parameter {parameter_name} should be one of {parameter_rule.options}.")
+            elif parameter_rule.type == ParameterType.TEXT:
+                if not isinstance(parameter_value, str):
+                    raise ValueError(f"Model Parameter {parameter_name} should be text.")
+
                 # validate options
                 if parameter_rule.options and parameter_value not in parameter_rule.options:
                     raise ValueError(f"Model Parameter {parameter_name} should be one of {parameter_rule.options}.")

api/core/model_runtime/model_providers/azure_openai/tts/tts.py

@@ -70,7 +70,7 @@ class AzureOpenAIText2SpeechModel(_CommonAzureOpenAI, TTSModel):
             # doc: https://platform.openai.com/docs/guides/text-to-speech
             credentials_kwargs = self._to_credential_kwargs(credentials)
             client = AzureOpenAI(**credentials_kwargs)
-            # max font is 4096,there is 3500 limit for each request
+            # max length is 4096 characters, there is 3500 limit for each request
             max_length = 3500
             if len(content_text) > max_length:
                 sentences = self._split_text_into_sentences(content_text, max_length=max_length)

api/core/model_runtime/model_providers/model_provider_factory.py

@@ -6,7 +6,7 @@ from typing import Optional
 from pydantic import BaseModel, ConfigDict
 
 from core.helper.module_import_helper import load_single_subclass_from_source
-from core.helper.position_helper import get_position_map, sort_to_dict_by_position_map
+from core.helper.position_helper import get_provider_position_map, sort_to_dict_by_position_map
 from core.model_runtime.entities.model_entities import ModelType
 from core.model_runtime.entities.provider_entities import ProviderConfig, ProviderEntity, SimpleProviderEntity
 from core.model_runtime.model_providers.__base.model_provider import ModelProvider
@@ -234,7 +234,7 @@ class ModelProviderFactory:
         ]
 
         # get _position.yaml file path
-        position_map = get_position_map(model_providers_path)
+        position_map = get_provider_position_map(model_providers_path)
 
         # traverse all model_provider_dir_paths
         model_providers: list[ModelProviderExtension] = []

api/core/model_runtime/model_providers/openai/llm/gpt-4o-mini-2024-07-18.yaml

@@ -37,6 +37,9 @@ parameter_rules:
     options:
       - text
       - json_object
+      - json_schema
+  - name: json_schema
+    use_template: json_schema
 pricing:
   input: '0.15'
   output: '0.60'

api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py

@@ -428,7 +428,7 @@ class OAIAPICompatLargeLanguageModel(_CommonOAI_API_Compat, LargeLanguageModel):
                 if new_tool_call.function.arguments:
                     tool_call.function.arguments += new_tool_call.function.arguments
 
-        finish_reason = 'Unknown'
+        finish_reason = None  # The default value of finish_reason is None
 
         for chunk in response.iter_lines(decode_unicode=True, delimiter=delimiter):
             chunk = chunk.strip()
@@ -437,6 +437,8 @@ class OAIAPICompatLargeLanguageModel(_CommonOAI_API_Compat, LargeLanguageModel):
                 if chunk.startswith(':'):
                     continue
                 decoded_chunk = chunk.strip().lstrip('data: ').lstrip()
+                if decoded_chunk == '[DONE]':  # Some provider returns "data: [DONE]"
+                    continue
 
                 try:
                     chunk_json = json.loads(decoded_chunk)

api/core/model_runtime/model_providers/openrouter/llm/gpt-4o-2024-08-06.yaml

@@ -0,0 +1,44 @@
+model: gpt-4o-2024-08-06
+label:
+  zh_Hans: gpt-4o-2024-08-06
+  en_US: gpt-4o-2024-08-06
+model_type: llm
+features:
+  - multi-tool-call
+  - agent-thought
+  - stream-tool-call
+  - vision
+model_properties:
+  mode: chat
+  context_size: 128000
+parameter_rules:
+  - name: temperature
+    use_template: temperature
+  - name: top_p
+    use_template: top_p
+  - name: presence_penalty
+    use_template: presence_penalty
+  - name: frequency_penalty
+    use_template: frequency_penalty
+  - name: max_tokens
+    use_template: max_tokens
+    default: 512
+    min: 1
+    max: 16384
+  - name: response_format
+    label:
+      zh_Hans: 回复格式
+      en_US: response_format
+    type: string
+    help:
+      zh_Hans: 指定模型必须输出的格式
+      en_US: specifying the format that the model must output
+    required: false
+    options:
+      - text
+      - json_object
+pricing:
+  input: '2.50'
+  output: '10.00'
+  unit: '0.000001'
+  currency: USD

api/core/model_runtime/model_providers/tongyi/text_embedding/text-embedding-v3.yaml

@@ -0,0 +1,9 @@
+model: text-embedding-v3
+model_type: text-embedding
+model_properties:
+  context_size: 8192
+  max_chunks: 25
+pricing:
+  input: "0.0007"
+  unit: "0.001"
+  currency: RMB

api/core/model_runtime/model_providers/wenxin/_common.py

@@ -118,6 +118,9 @@ class _CommonWenxin:
         'ernie-4.0-turbo-8k-preview': 'https://aip.baidubce.com/rpc/2.0/ai_custom/v1/wenxinworkshop/chat/ernie-4.0-turbo-8k-preview',
         'yi_34b_chat': 'https://aip.baidubce.com/rpc/2.0/ai_custom/v1/wenxinworkshop/chat/yi_34b_chat',
         'embedding-v1': 'https://aip.baidubce.com/rpc/2.0/ai_custom/v1/wenxinworkshop/embeddings/embedding-v1',
+        'bge-large-en': 'https://aip.baidubce.com/rpc/2.0/ai_custom/v1/wenxinworkshop/embeddings/bge_large_en',
+        'bge-large-zh': 'https://aip.baidubce.com/rpc/2.0/ai_custom/v1/wenxinworkshop/embeddings/bge_large_zh',
+        'tao-8k': 'https://aip.baidubce.com/rpc/2.0/ai_custom/v1/wenxinworkshop/embeddings/tao_8k',
     }
 
     function_calling_supports = [

api/core/model_runtime/model_providers/wenxin/text_embedding/bge-large-en.yaml

@@ -0,0 +1,9 @@
+model: bge-large-en
+model_type: text-embedding
+model_properties:
+  context_size: 512
+  max_chunks: 16
+pricing:
+  input: '0.0005'
+  unit: '0.001'
+  currency: RMB

api/core/model_runtime/model_providers/wenxin/text_embedding/bge-large-zh.yaml

@@ -0,0 +1,9 @@
+model: bge-large-zh
+model_type: text-embedding
+model_properties:
+  context_size: 512
+  max_chunks: 16
+pricing:
+  input: '0.0005'
+  unit: '0.001'
+  currency: RMB

api/core/model_runtime/model_providers/wenxin/text_embedding/tao-8k.yaml

@@ -0,0 +1,9 @@
+model: tao-8k
+model_type: text-embedding
+model_properties:
+  context_size: 8192
+  max_chunks: 1
+pricing:
+  input: '0.0005'
+  unit: '0.001'
+  currency: RMB

api/core/model_runtime/model_providers/zhipuai/llm/llm.py

@@ -444,6 +444,7 @@ class ZhipuAILargeLanguageModel(_CommonZhipuaiAI, LargeLanguageModel):
                     delta=LLMResultChunkDelta(
                         index=delta.index,
                         message=assistant_prompt_message,
+                        finish_reason=delta.finish_reason
                     )
                 )
 

api/core/ops/entities/config_entity.py

@@ -21,6 +21,7 @@ class LangfuseConfig(BaseTracingConfig):
     """
     public_key: str
     secret_key: str
+    project_key: str
     host: str = 'https://api.langfuse.com'
 
     @field_validator("host")

api/core/ops/langfuse_trace/langfuse_trace.py

@@ -419,3 +419,11 @@ class LangFuseDataTrace(BaseTraceInstance):
         except Exception as e:
             logger.debug(f"LangFuse API check failed: {str(e)}")
             raise ValueError(f"LangFuse API check failed: {str(e)}")
+
+    def get_project_key(self):
+        try:
+            projects = self.langfuse_client.client.projects.get()
+            return projects.data[0].id
+        except Exception as e:
+            logger.debug(f"LangFuse get project key failed: {str(e)}")
+            raise ValueError(f"LangFuse get project key failed: {str(e)}")

api/core/ops/ops_trace_manager.py

@@ -38,7 +38,7 @@ provider_config_map = {
     TracingProviderEnum.LANGFUSE.value: {
         'config_class': LangfuseConfig,
         'secret_keys': ['public_key', 'secret_key'],
-        'other_keys': ['host'],
+        'other_keys': ['host', 'project_key'],
         'trace_instance': LangFuseDataTrace
     },
     TracingProviderEnum.LANGSMITH.value: {
@@ -123,7 +123,6 @@ class OpsTraceManager:
 
         for key in other_keys:
             new_config[key] = decrypt_tracing_config.get(key, "")
-
         return config_class(**new_config).model_dump()
 
     @classmethod
@@ -252,6 +251,19 @@ class OpsTraceManager:
         tracing_config = config_type(**tracing_config)
         return trace_instance(tracing_config).api_check()
 
+    @staticmethod
+    def get_trace_config_project_key(tracing_config: dict, tracing_provider: str):
+        """
+        get trace config is project key
+        :param tracing_config: tracing config
+        :param tracing_provider: tracing provider
+        :return:
+        """
+        config_type, trace_instance = provider_config_map[tracing_provider]['config_class'], \
+            provider_config_map[tracing_provider]['trace_instance']
+        tracing_config = config_type(**tracing_config)
+        return trace_instance(tracing_config).get_project_key()
+
 
 class TraceTask:
     def __init__(

api/core/provider_manager.py

@@ -5,6 +5,7 @@ from typing import Optional
 
 from sqlalchemy.exc import IntegrityError
 
+from configs import dify_config
 from core.entities.model_entities import DefaultModelEntity, DefaultModelProviderEntity
 from core.entities.provider_configuration import ProviderConfiguration, ProviderConfigurations, ProviderModelBundle
 from core.entities.provider_entities import (
@@ -18,12 +19,9 @@ from core.entities.provider_entities import (
 )
 from core.helper import encrypter
 from core.helper.model_provider_cache import ProviderCredentialsCache, ProviderCredentialsCacheType
+from core.helper.position_helper import is_filtered
 from core.model_runtime.entities.model_entities import ModelType
-from core.model_runtime.entities.provider_entities import (
-    CredentialFormSchema,
-    FormType,
-    ProviderEntity,
-)
+from core.model_runtime.entities.provider_entities import CredentialFormSchema, FormType, ProviderEntity
 from core.model_runtime.model_providers import model_provider_factory
 from extensions import ext_hosting_provider
 from extensions.ext_database import db
@@ -45,6 +43,7 @@ class ProviderManager:
     """
     ProviderManager is a class that manages the model providers includes Hosting and Customize Model Providers.
     """
+
     def __init__(self) -> None:
         self.decoding_rsa_key = None
         self.decoding_cipher_rsa = None
@@ -117,6 +116,16 @@ class ProviderManager:
 
         # Construct ProviderConfiguration objects for each provider
         for provider_entity in provider_entities:
+
+            # handle include, exclude
+            if is_filtered(
+                    include_set=dify_config.POSITION_PROVIDER_INCLUDES_SET,
+                    exclude_set=dify_config.POSITION_PROVIDER_EXCLUDES_SET,
+                    data=provider_entity,
+                    name_func=lambda x: x.provider,
+            ):
+                continue
+
             provider_name = provider_entity.provider
             provider_records = provider_name_to_provider_records_dict.get(provider_entity.provider, [])
             provider_model_records = provider_name_to_provider_model_records_dict.get(provider_entity.provider, [])
@@ -271,6 +280,24 @@ class ProviderManager:
             )
         )
 
+    def get_first_provider_first_model(self, tenant_id: str, model_type: ModelType) -> tuple[str, str]:
+        """
+        Get names of first model and its provider
+
+        :param tenant_id: workspace id
+        :param model_type: model type
+        :return: provider name, model name
+        """
+        provider_configurations = self.get_configurations(tenant_id)
+
+        # get available models from provider_configurations
+        all_models = provider_configurations.get_models(
+            model_type=model_type,
+            only_active=False
+        )
+
+        return all_models[0].provider.provider, all_models[0].model
+
     def update_default_model_record(self, tenant_id: str, model_type: ModelType, provider: str, model: str) \
             -> TenantDefaultModel:
         """

api/core/rag/datasource/vdb/pgvector/pgvector.py

@@ -152,8 +152,27 @@ class PGVector(BaseVector):
         return docs
 
     def search_by_full_text(self, query: str, **kwargs: Any) -> list[Document]:
-        # do not support bm25 search
-        return []
+        top_k = kwargs.get("top_k", 5)
+
+        with self._get_cursor() as cur:
+            cur.execute(
+                f"""SELECT meta, text, ts_rank(to_tsvector(coalesce(text, '')), to_tsquery(%s)) AS score
+                FROM {self.table_name}
+                WHERE to_tsvector(text) @@ plainto_tsquery(%s)
+                ORDER BY score DESC
+                LIMIT {top_k}""",
+                # f"'{query}'" is required in order to account for whitespace in query
+                (f"'{query}'", f"'{query}'"),
+            )
+
+            docs = []
+
+            for record in cur:
+                metadata, text, score = record
+                metadata["score"] = score
+                docs.append(Document(page_content=text, metadata=metadata))
+
+        return docs
 
     def delete(self) -> None:
         with self._get_cursor() as cur:

api/core/tools/provider/builtin/_positions.py

@@ -1,6 +1,6 @@
 import os.path
 
-from core.helper.position_helper import get_position_map, sort_by_position_map
+from core.helper.position_helper import get_tool_position_map, sort_by_position_map
 from core.tools.entities.api_entities import UserToolProvider
 
 
@@ -10,11 +10,11 @@ class BuiltinToolProviderSort:
     @classmethod
     def sort(cls, providers: list[UserToolProvider]) -> list[UserToolProvider]:
         if not cls._position:
-            cls._position = get_position_map(os.path.join(os.path.dirname(__file__), '..'))
+            cls._position = get_tool_position_map(os.path.join(os.path.dirname(__file__), '..'))
 
         def name_func(provider: UserToolProvider) -> str:
             return provider.name
 
         sorted_providers = sort_by_position_map(cls._position, providers, name_func)
 
-        return sorted_providers
+        return sorted_providers

api/core/tools/provider/builtin/azuredalle/tools/dalle3.yaml

@@ -25,7 +25,7 @@ parameters:
       pt_BR: Prompt
     human_description:
       en_US: Image prompt, you can check the official documentation of DallE 3
-      zh_Hans: 图像提示词，您可以查看DallE 3 的官方文档
+      zh_Hans: 图像提示词，您可以查看 DallE 3 的官方文档
       pt_BR: Imagem prompt, você pode verificar a documentação oficial do DallE 3
     llm_description: Image prompt of DallE 3, you should describe the image you want to generate as a list of words as possible as detailed
     form: llm

api/core/tools/provider/builtin/cogview/tools/cogview3.yaml

@@ -25,7 +25,7 @@ parameters:
       pt_BR: Prompt
     human_description:
       en_US: Image prompt, you can check the official documentation of CogView 3
-      zh_Hans: 图像提示词，您可以查看CogView 3 的官方文档
+      zh_Hans: 图像提示词，您可以查看 CogView 3 的官方文档
       pt_BR: Image prompt, you can check the official documentation of CogView 3
     llm_description: Image prompt of CogView 3, you should describe the image you want to generate as a list of words as possible as detailed
     form: llm

api/core/tools/provider/builtin/crossref/_assets/icon.svg

@@ -0,0 +1,49 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 19.2.1, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+	 viewBox="0 0 200 130.2" style="enable-background:new 0 0 200 130.2;" xml:space="preserve">
+<style type="text/css">
+	.st0{fill:#3EB1C8;}
+	.st1{fill:#D8D2C4;}
+	.st2{fill:#4F5858;}
+	.st3{fill:#FFC72C;}
+	.st4{fill:#EF3340;}
+</style>
+<g>
+	<polygon class="st0" points="111.8,95.5 111.8,66.8 135.4,59 177.2,73.3 	"/>
+	<polygon class="st1" points="153.6,36.8 111.8,51.2 135.4,59 177.2,44.6 	"/>
+	<polygon class="st2" points="135.4,59 177.2,44.6 177.2,73.3 	"/>
+	<polygon class="st3" points="177.2,0.3 177.2,29 153.6,36.8 111.8,22.5 	"/>
+	<polygon class="st4" points="153.6,36.8 111.8,51.2 111.8,22.5 	"/>
+	<g>
+		<g>
+			<g>
+				<g>
+					<path class="st2" d="M26.3,104.8c-0.5-3.7-4.1-6.5-8.1-6.5c-7.3,0-10.1,6.2-10.1,12.7c0,6.2,2.8,12.4,10.1,12.4
+						c5,0,7.8-3.4,8.4-8.3h7.9c-0.8,9.2-7.2,15.2-16.3,15.2C6.8,130.2,0,121.7,0,111c0-11,6.8-19.6,18.2-19.6c8.2,0,15,4.8,16,13.3
+						H26.3z"/>
+					<path class="st2" d="M37.4,102.5h7v5h0.1c1.4-3.4,5-5.7,8.6-5.7c0.5,0,1.1,0.1,1.6,0.3v6.9c-0.7-0.2-1.8-0.3-2.6-0.3
+						c-5.4,0-7.3,3.9-7.3,8.6v12.1h-7.4V102.5z"/>
+					<path class="st2" d="M68.7,101.8c8.5,0,13.9,5.6,13.9,14.2c0,8.5-5.5,14.1-13.9,14.1c-8.4,0-13.9-5.6-13.9-14.1
+						C54.9,107.4,60.3,101.8,68.7,101.8z M68.7,124.5c5,0,6.5-4.3,6.5-8.6c0-4.3-1.5-8.6-6.5-8.6c-5,0-6.5,4.3-6.5,8.6
+						C62.2,120.2,63.8,124.5,68.7,124.5z"/>
+					<path class="st2" d="M91.2,120.6c0.1,3.2,2.8,4.5,5.7,4.5c2.1,0,4.8-0.8,4.8-3.4c0-2.2-3.1-3-8.4-4.2c-4.3-0.9-8.5-2.4-8.5-7.2
+						c0-6.9,5.9-8.6,11.7-8.6c5.9,0,11.3,2,11.8,8.6h-7c-0.2-2.9-2.4-3.6-5-3.6c-1.7,0-4.1,0.3-4.1,2.5c0,2.6,4.2,3,8.4,4
+						c4.3,1,8.5,2.5,8.5,7.5c0,7.1-6.1,9.3-12.3,9.3c-6.2,0-12.3-2.3-12.6-9.5H91.2z"/>
+					<path class="st2" d="M118.1,120.6c0.1,3.2,2.8,4.5,5.7,4.5c2.1,0,4.8-0.8,4.8-3.4c0-2.2-3.1-3-8.4-4.2
+						c-4.3-0.9-8.5-2.4-8.5-7.2c0-6.9,5.9-8.6,11.7-8.6c5.9,0,11.3,2,11.8,8.6h-7c-0.2-2.9-2.4-3.6-5-3.6c-1.7,0-4.1,0.3-4.1,2.5
+						c0,2.6,4.2,3,8.4,4c4.3,1,8.5,2.5,8.5,7.5c0,7.1-6.1,9.3-12.3,9.3c-6.2,0-12.3-2.3-12.6-9.5H118.1z"/>
+					<path class="st2" d="M138.4,102.5h7v5h0.1c1.4-3.4,5-5.7,8.6-5.7c0.5,0,1.1,0.1,1.6,0.3v6.9c-0.7-0.2-1.8-0.3-2.6-0.3
+						c-5.4,0-7.3,3.9-7.3,8.6v12.1h-7.4V102.5z"/>
+					<path class="st2" d="M163.7,117.7c0.2,4.7,2.5,6.8,6.6,6.8c3,0,5.3-1.8,5.8-3.5h6.5c-2.1,6.3-6.5,9-12.6,9
+						c-8.5,0-13.7-5.8-13.7-14.1c0-8,5.6-14.2,13.7-14.2c9.1,0,13.6,7.7,13,15.9H163.7z M175.7,113.1c-0.7-3.7-2.3-5.7-5.9-5.7
+						c-4.7,0-6,3.6-6.1,5.7H175.7z"/>
+					<path class="st2" d="M187.2,107.5h-4.4v-4.9h4.4v-2.1c0-4.7,3-8.2,9-8.2c1.3,0,2.6,0.2,3.9,0.2V98c-0.9-0.1-1.8-0.2-2.7-0.2
+						c-2,0-2.8,0.8-2.8,3.1v1.6h5.1v4.9h-5.1v21.9h-7.4V107.5z"/>
+				</g>
+			</g>
+		</g>
+	</g>
+</g>
+</svg>

api/core/tools/provider/builtin/crossref/crossref.py

@@ -0,0 +1,20 @@
+from core.tools.errors import ToolProviderCredentialValidationError
+from core.tools.provider.builtin.crossref.tools.query_doi import CrossRefQueryDOITool
+from core.tools.provider.builtin_tool_provider import BuiltinToolProviderController
+
+
+class CrossRefProvider(BuiltinToolProviderController):
+    def _validate_credentials(self, credentials: dict) -> None:
+        try:
+            CrossRefQueryDOITool().fork_tool_runtime(
+                runtime={
+                    "credentials": credentials,
+                }
+            ).invoke(
+                user_id='',
+                tool_parameters={
+                    "doi": '10.1007/s00894-022-05373-8',
+                },
+            )
+        except Exception as e:
+            raise ToolProviderCredentialValidationError(str(e))

api/core/tools/provider/builtin/crossref/crossref.yaml

@@ -0,0 +1,29 @@
+identity:
+  author: Sakura4036
+  name: crossref
+  label:
+    en_US: CrossRef
+    zh_Hans: CrossRef
+  description:
+    en_US: Crossref is a cross-publisher reference linking registration query system using DOI technology created in 2000. Crossref establishes cross-database links between the reference list and citation full text of papers, making it very convenient for readers to access the full text of papers.
+    zh_Hans: Crossref是于2000年创建的使用DOI技术的跨出版商参考文献链接注册查询系统。Crossref建立了在论文的参考文献列表和引文全文之间的跨数据库链接，使得读者能够非常便捷地获取文献全文。
+  icon: icon.svg
+  tags:
+    - search
+credentials_for_provider:
+  mailto:
+    type: text-input
+    required: true
+    label:
+      en_US: email address
+      zh_Hans: email地址
+      pt_BR: email address
+    placeholder:
+      en_US: Please input your email address
+      zh_Hans: 请输入你的email地址
+      pt_BR: Please input your email address
+    help:
+      en_US: According to the requirements of Crossref, an email address is required
+      zh_Hans: 根据Crossref的要求，需要提供一个邮箱地址
+      pt_BR: According to the requirements of Crossref, an email address is required
+    url: https://api.crossref.org/swagger-ui/index.html

api/core/tools/provider/builtin/crossref/tools/query_doi.py

@@ -0,0 +1,25 @@
+from typing import Any, Union
+
+import requests
+
+from core.tools.entities.tool_entities import ToolInvokeMessage
+from core.tools.errors import ToolParameterValidationError
+from core.tools.tool.builtin_tool import BuiltinTool
+
+
+class CrossRefQueryDOITool(BuiltinTool):
+    """
+    Tool for querying the metadata of a publication using its DOI.
+    """
+    def _invoke(self, user_id: str, tool_parameters: dict[str, Any]) -> Union[ToolInvokeMessage, list[ToolInvokeMessage]]:
+        doi = tool_parameters.get('doi')
+        if not doi:
+            raise ToolParameterValidationError('doi is required.')
+        # doc: https://github.com/CrossRef/rest-api-doc
+        url = f"https://api.crossref.org/works/{doi}"
+        response = requests.get(url)
+        response.raise_for_status()
+        response = response.json()
+        message = response.get('message', {})
+
+        return self.create_json_message(message)

api/core/tools/provider/builtin/crossref/tools/query_doi.yaml

@@ -0,0 +1,23 @@
+identity:
+  name: crossref_query_doi
+  author: Sakura4036
+  label:
+    en_US: CrossRef Query DOI
+    zh_Hans: CrossRef DOI 查询
+    pt_BR: CrossRef Query DOI
+description:
+  human:
+    en_US: A tool for searching literature information using CrossRef by DOI.
+    zh_Hans: 一个使用CrossRef通过DOI获取文献信息的工具。
+    pt_BR: A tool for searching literature information using CrossRef by DOI.
+  llm: A tool for searching literature information using CrossRef by DOI.
+parameters:
+  - name: doi
+    type: string
+    required: true
+    label:
+      en_US: DOI
+      zh_Hans: DOI
+      pt_BR: DOI
+    llm_description: DOI for searching in CrossRef
+    form: llm

api/core/tools/provider/builtin/crossref/tools/query_title.py

@@ -0,0 +1,120 @@
+import time
+from typing import Any, Union
+
+import requests
+
+from core.tools.entities.tool_entities import ToolInvokeMessage
+from core.tools.tool.builtin_tool import BuiltinTool
+
+
+def convert_time_str_to_seconds(time_str: str) -> int:
+    """
+    Convert a time string to seconds.
+    example: 1s -> 1,  1m30s -> 90, 1h30m -> 5400, 1h30m30s -> 5430
+    """
+    time_str = time_str.lower().strip().replace(' ', '')
+    seconds = 0
+    if 'h' in time_str:
+        hours, time_str = time_str.split('h')
+        seconds += int(hours) * 3600
+    if 'm' in time_str:
+        minutes, time_str = time_str.split('m')
+        seconds += int(minutes) * 60
+    if 's' in time_str:
+        seconds += int(time_str.replace('s', ''))
+    return seconds
+
+
+class CrossRefQueryTitleAPI:
+    """
+    Tool for querying the metadata of a publication using its title.
+    Crossref API doc: https://github.com/CrossRef/rest-api-doc
+    """
+    query_url_template: str = "https://api.crossref.org/works?query.bibliographic={query}&rows={rows}&offset={offset}&sort={sort}&order={order}&mailto={mailto}"
+    rate_limit: int = 50
+    rate_interval: float = 1
+    max_limit: int = 1000
+
+    def __init__(self, mailto: str):
+        self.mailto = mailto
+
+    def _query(self, query: str, rows: int = 5, offset: int = 0, sort: str = 'relevance', order: str = 'desc', fuzzy_query: bool = False) -> list[dict]:
+        """
+        Query the metadata of a publication using its title.
+        :param query: the title of the publication
+        :param rows: the number of results to return
+        :param sort: the sort field
+        :param order: the sort order
+        :param fuzzy_query: whether to return all items that match the query
+        """
+        url = self.query_url_template.format(query=query, rows=rows, offset=offset, sort=sort, order=order, mailto=self.mailto)
+        response = requests.get(url)
+        response.raise_for_status()
+        rate_limit = int(response.headers['x-ratelimit-limit'])
+        # convert time string to seconds
+        rate_interval = convert_time_str_to_seconds(response.headers['x-ratelimit-interval'])
+
+        self.rate_limit = rate_limit
+        self.rate_interval = rate_interval
+
+        response = response.json()
+        if response['status'] != 'ok':
+            return []
+
+        message = response['message']
+        if fuzzy_query:
+            # fuzzy query return all items
+            return message['items']
+        else:
+            for paper in message['items']:
+                title = paper['title'][0]
+                if title.lower() != query.lower():
+                    continue
+                return [paper]
+        return []
+
+    def query(self, query: str, rows: int = 5, sort: str = 'relevance', order: str = 'desc', fuzzy_query: bool = False) -> list[dict]:
+        """
+        Query the metadata of a publication using its title.
+        :param query: the title of the publication
+        :param rows: the number of results to return
+        :param sort: the sort field
+        :param order: the sort order
+        :param fuzzy_query: whether to return all items that match the query
+        """
+        rows = min(rows, self.max_limit)
+        if rows > self.rate_limit:
+            # query multiple times
+            query_times = rows // self.rate_limit + 1
+            results = []
+
+            for i in range(query_times):
+                result = self._query(query, rows=self.rate_limit, offset=i * self.rate_limit, sort=sort, order=order, fuzzy_query=fuzzy_query)
+                if fuzzy_query:
+                    results.extend(result)
+                else:
+                    # fuzzy_query=False, only one result
+                    if result:
+                        return result
+                time.sleep(self.rate_interval)
+            return results
+        else:
+            # query once
+            return self._query(query, rows, sort=sort, order=order, fuzzy_query=fuzzy_query)
+
+
+class CrossRefQueryTitleTool(BuiltinTool):
+    """
+    Tool for querying the metadata of a publication using its title.
+    """
+    def _invoke(self, user_id: str, tool_parameters: dict[str, Any]) -> Union[ToolInvokeMessage, list[ToolInvokeMessage]]:
+        query = tool_parameters.get('query')
+        fuzzy_query = tool_parameters.get('fuzzy_query', False)
+        rows = tool_parameters.get('rows', 3)
+        sort = tool_parameters.get('sort', 'relevance')
+        order = tool_parameters.get('order', 'desc')
+        mailto = self.runtime.credentials['mailto']
+
+        result = CrossRefQueryTitleAPI(mailto).query(query, rows, sort, order, fuzzy_query)
+
+        return [self.create_json_message(r) for r in result]

api/core/tools/provider/builtin/crossref/tools/query_title.yaml

@@ -0,0 +1,105 @@
+identity:
+  name: crossref_query_title
+  author: Sakura4036
+  label:
+    en_US: CrossRef Title Query
+    zh_Hans: CrossRef 标题查询
+    pt_BR: CrossRef Title Query
+description:
+  human:
+    en_US: A tool for querying literature information using CrossRef by title.
+    zh_Hans: 一个使用CrossRef通过标题搜索文献信息的工具。
+    pt_BR: A tool for querying literature information using CrossRef by title.
+  llm: A tool for querying literature information using CrossRef by title.
+parameters:
+  - name: query
+    type: string
+    required: true
+    label:
+      en_US: 标题
+      zh_Hans: 查询语句
+      pt_BR: 标题
+    human_description:
+      en_US: Query bibliographic information, useful for citation look up. Includes titles, authors, ISSNs and publication years
+      zh_Hans: 用于搜索文献信息，有助于查找引用。包括标题，作者，ISSN和出版年份
+      pt_BR: Query bibliographic information, useful for citation look up. Includes titles, authors, ISSNs and publication years
+    llm_description: key words for querying in Web of Science
+    form: llm
+  - name: fuzzy_query
+    type: boolean
+    default: false
+    label:
+      en_US: Whether to fuzzy search
+      zh_Hans: 是否模糊搜索
+      pt_BR: Whether to fuzzy search
+    human_description:
+      en_US: used for selecting the query type, fuzzy query returns more results, precise query returns 1 or none
+      zh_Hans: 用于选择搜索类型，模糊搜索返回更多结果，精确搜索返回1条结果或无
+      pt_BR: used for selecting the query type, fuzzy query returns more results, precise query returns 1 or none
+    form: form
+  - name: limit
+    type: number
+    required: false
+    label:
+      en_US: max query number
+      zh_Hans: 最大搜索数
+      pt_BR: max query number
+    human_description:
+      en_US: max query number(fuzzy search returns the maximum number of results or precise search the maximum number of matches)
+      zh_Hans: 最大搜索数(模糊搜索返回的最大结果数或精确搜索最大匹配数)
+      pt_BR: max query number(fuzzy search returns the maximum number of results or precise search the maximum number of matches)
+    form: llm
+    default: 50
+  - name: sort
+    type: select
+    required: true
+    options:
+      - value: relevance
+        label:
+          en_US: relevance
+          zh_Hans: 相关性
+          pt_BR: relevance
+      - value: published
+        label:
+          en_US: publication date
+          zh_Hans: 出版日期
+          pt_BR: publication date
+      - value: references-count
+        label:
+          en_US: references-count
+          zh_Hans: 引用次数
+          pt_BR: references-count
+    default: relevance
+    label:
+      en_US: sorting field
+      zh_Hans: 排序字段
+      pt_BR: sorting field
+    human_description:
+      en_US: Sorting of query results
+      zh_Hans: 检索结果的排序字段
+      pt_BR: Sorting of query results
+    form: form
+  - name: order
+    type: select
+    required: true
+    options:
+      - value: desc
+        label:
+          en_US: descending
+          zh_Hans: 降序
+          pt_BR: descending
+      - value: asc
+        label:
+          en_US: ascending
+          zh_Hans: 升序
+          pt_BR: ascending
+    default: desc
+    label:
+      en_US: Order
+      zh_Hans: 排序
+      pt_BR: Order
+    human_description:
+      en_US: Order of query results
+      zh_Hans: 检索结果的排序方式
+      pt_BR: Order of query results
+    form: form

api/core/tools/provider/builtin/dalle/tools/dalle2.yaml

@@ -24,7 +24,7 @@ parameters:
       pt_BR: Prompt
     human_description:
       en_US: Image prompt, you can check the official documentation of DallE 2
-      zh_Hans: 图像提示词，您可以查看DallE 2 的官方文档
+      zh_Hans: 图像提示词，您可以查看 DallE 2 的官方文档
       pt_BR: Image prompt, you can check the official documentation of DallE 2
     llm_description: Image prompt of DallE 2, you should describe the image you want to generate as a list of words as possible as detailed
     form: llm

api/core/tools/provider/builtin/dalle/tools/dalle3.yaml

@@ -25,7 +25,7 @@ parameters:
       pt_BR: Prompt
     human_description:
       en_US: Image prompt, you can check the official documentation of DallE 3
-      zh_Hans: 图像提示词，您可以查看DallE 3 的官方文档
+      zh_Hans: 图像提示词，您可以查看 DallE 3 的官方文档
       pt_BR: Image prompt, you can check the official documentation of DallE 3
     llm_description: Image prompt of DallE 3, you should describe the image you want to generate as a list of words as possible as detailed
     form: llm

api/core/tools/provider/builtin/jina/tools/jina_tokenizer.yaml

@@ -60,5 +60,11 @@ parameters:
     label:
       en_US: Tokenizer
     human_description:
-      en_US: cl100k_base - gpt-4,gpt-3.5-turbo,gpt-3.5;  o200k_base - gpt-4o,gpt-4o-mini; p50k_base - text-davinci-003,text-davinci-002
+      en_US: |
+        · cl100k_base --- gpt-4, gpt-3.5-turbo, gpt-3.5
+        · o200k_base  --- gpt-4o, gpt-4o-mini
+        · p50k_base   --- text-davinci-003, text-davinci-002
+        · r50k_base   --- text-davinci-001, text-curie-001
+        · p50k_edit   --- text-davinci-edit-001, code-davinci-edit-001
+        · gpt2        --- gpt-2
     form: form

api/core/tools/provider/builtin/novitaai/_novita_tool_base.py

@@ -0,0 +1,73 @@
+from novita_client import (
+    Txt2ImgV3Embedding,
+    Txt2ImgV3HiresFix,
+    Txt2ImgV3LoRA,
+    Txt2ImgV3Refiner,
+    V3TaskImage,
+)
+
+
+class NovitaAiToolBase:
+    def _extract_loras(self, loras_str: str):
+        if not loras_str:
+            return []
+
+        loras_ori_list = lora_str.strip().split(';')
+        result_list = []
+        for lora_str in loras_ori_list:
+            lora_info = lora_str.strip().split(',')
+            lora = Txt2ImgV3LoRA(
+                model_name=lora_info[0].strip(),
+                strength=float(lora_info[1]),
+            )
+            result_list.append(lora)
+
+        return result_list
+
+    def _extract_embeddings(self, embeddings_str: str):
+        if not embeddings_str:
+            return []
+
+        embeddings_ori_list = embeddings_str.strip().split(';')
+        result_list = []
+        for embedding_str in embeddings_ori_list:
+            embedding = Txt2ImgV3Embedding(
+                model_name=embedding_str.strip()
+            )
+            result_list.append(embedding)
+
+        return result_list
+
+    def _extract_hires_fix(self, hires_fix_str: str):
+        hires_fix_info = hires_fix_str.strip().split(',')
+        if 'upscaler' in hires_fix_info:
+            hires_fix = Txt2ImgV3HiresFix(
+                target_width=int(hires_fix_info[0]),
+                target_height=int(hires_fix_info[1]),
+                strength=float(hires_fix_info[2]),
+                upscaler=hires_fix_info[3].strip()
+            )
+        else:
+            hires_fix = Txt2ImgV3HiresFix(
+                target_width=int(hires_fix_info[0]),
+                target_height=int(hires_fix_info[1]),
+                strength=float(hires_fix_info[2])
+            )
+
+        return hires_fix
+
+    def _extract_refiner(self, switch_at: str):
+        refiner = Txt2ImgV3Refiner(
+            switch_at=float(switch_at)
+        )
+        return refiner
+
+    def _is_hit_nsfw_detection(self, image: V3TaskImage, confidence_threshold: float) -> bool:
+        """
+            is hit nsfw
+        """
+        if image.nsfw_detection_result is None:
+            return False
+        if image.nsfw_detection_result.valid and image.nsfw_detection_result.confidence >= confidence_threshold:
+            return True
+        return False

api/core/tools/provider/builtin/novitaai/tools/novitaai_txt2img.py

@@ -4,19 +4,15 @@ from typing import Any, Union
 
 from novita_client import (
     NovitaClient,
-    Txt2ImgV3Embedding,
-    Txt2ImgV3HiresFix,
-    Txt2ImgV3LoRA,
-    Txt2ImgV3Refiner,
-    V3TaskImage,
 )
 
 from core.tools.entities.tool_entities import ToolInvokeMessage
 from core.tools.errors import ToolProviderCredentialValidationError
+from core.tools.provider.builtin.novitaai._novita_tool_base import NovitaAiToolBase
 from core.tools.tool.builtin_tool import BuiltinTool
 
 
-class NovitaAiTxt2ImgTool(BuiltinTool):
+class NovitaAiTxt2ImgTool(BuiltinTool, NovitaAiToolBase):
     def _invoke(self,
                 user_id: str,
                 tool_parameters: dict[str, Any],
@@ -73,65 +69,19 @@ class NovitaAiTxt2ImgTool(BuiltinTool):
 
         # process loras
         if 'loras' in res_parameters:
-            loras_ori_list = res_parameters.get('loras').strip().split(';')
-            locals_list = []
-            for lora_str in loras_ori_list:
-                lora_info = lora_str.strip().split(',')
-                lora = Txt2ImgV3LoRA(
-                    model_name=lora_info[0].strip(),
-                    strength=float(lora_info[1]),
-                )
-                locals_list.append(lora)
-
-            res_parameters['loras'] = locals_list
+            res_parameters['loras'] = self._extract_loras(res_parameters.get('loras'))
 
         # process embeddings
         if 'embeddings' in res_parameters:
-            embeddings_ori_list = res_parameters.get('embeddings').strip().split(';')
-            locals_list = []
-            for embedding_str in embeddings_ori_list:
-                embedding = Txt2ImgV3Embedding(
-                    model_name=embedding_str.strip()
-                )
-                locals_list.append(embedding)
-
-            res_parameters['embeddings'] = locals_list
+            res_parameters['embeddings'] = self._extract_embeddings(res_parameters.get('embeddings'))
 
         # process hires_fix
         if 'hires_fix' in res_parameters:
-            hires_fix_ori = res_parameters.get('hires_fix')
-            hires_fix_info = hires_fix_ori.strip().split(',')
-            if 'upscaler' in hires_fix_info:
-                hires_fix = Txt2ImgV3HiresFix(
-                    target_width=int(hires_fix_info[0]),
-                    target_height=int(hires_fix_info[1]),
-                    strength=float(hires_fix_info[2]),
-                    upscaler=hires_fix_info[3].strip()
-                )
-            else:
-                hires_fix = Txt2ImgV3HiresFix(
-                    target_width=int(hires_fix_info[0]),
-                    target_height=int(hires_fix_info[1]),
-                    strength=float(hires_fix_info[2])
-                )
+            res_parameters['hires_fix'] = self._extract_hires_fix(res_parameters.get('hires_fix'))
 
-            res_parameters['hires_fix'] = hires_fix
-
-            if 'refiner_switch_at' in res_parameters:
-                refiner = Txt2ImgV3Refiner(
-                    switch_at=float(res_parameters.get('refiner_switch_at'))
-                )
-                del res_parameters['refiner_switch_at']
-                res_parameters['refiner'] = refiner
+        # process refiner
+        if 'refiner_switch_at' in res_parameters:
+            res_parameters['refiner'] = self._extract_refiner(res_parameters.get('refiner_switch_at'))
+            del res_parameters['refiner_switch_at']
 
         return res_parameters
-
-    def _is_hit_nsfw_detection(self, image: V3TaskImage, confidence_threshold: float) -> bool:
-        """
-            is hit nsfw
-        """
-        if image.nsfw_detection_result is None:
-            return False
-        if image.nsfw_detection_result.valid and image.nsfw_detection_result.confidence >= confidence_threshold:
-            return True
-        return False

api/core/tools/provider/builtin/stepfun/tools/image.yaml

@@ -25,7 +25,7 @@ parameters:
       pt_BR: Prompt
     human_description:
       en_US: Image prompt, you can check the official documentation of step-1x
-      zh_Hans: 图像提示词，您可以查看step-1x 的官方文档
+      zh_Hans: 图像提示词，您可以查看 step-1x 的官方文档
       pt_BR: Image prompt, you can check the official documentation of step-1x
     llm_description: Image prompt of step-1x you should describe the image you want to generate as a list of words as possible as detailed
     form: llm

api/core/tools/provider/workflow_tool_provider.py

@@ -1,6 +1,6 @@
 from typing import Optional
 
-from core.app.app_config.entities import VariableEntity
+from core.app.app_config.entities import VariableEntity, VariableEntityType
 from core.app.apps.workflow.app_config_manager import WorkflowAppConfigManager
 from core.tools.entities.common_entities import I18nObject
 from core.tools.entities.tool_entities import (
@@ -18,6 +18,13 @@ from models.model import App, AppMode
 from models.tools import WorkflowToolProvider
 from models.workflow import Workflow
 
+VARIABLE_TO_PARAMETER_TYPE_MAPPING = {
+    VariableEntityType.TEXT_INPUT: ToolParameter.ToolParameterType.STRING,
+    VariableEntityType.PARAGRAPH: ToolParameter.ToolParameterType.STRING,
+    VariableEntityType.SELECT: ToolParameter.ToolParameterType.SELECT,
+    VariableEntityType.NUMBER: ToolParameter.ToolParameterType.NUMBER,
+}
+
 
 class WorkflowToolProviderController(ToolProviderController):
     provider_id: str
@@ -28,7 +35,7 @@ class WorkflowToolProviderController(ToolProviderController):
 
         if not app:
             raise ValueError('app not found')
-        
+
         controller = WorkflowToolProviderController(**{
             'identity': {
                 'author': db_provider.user.name if db_provider.user_id and db_provider.user else '',
@@ -46,7 +53,7 @@ class WorkflowToolProviderController(ToolProviderController):
             'credentials_schema': {},
             'provider_id': db_provider.id or '',
         })
-        
+
         # init tools
 
         controller.tools = [controller._get_db_provider_tool(db_provider, app)]
@@ -56,7 +63,7 @@ class WorkflowToolProviderController(ToolProviderController):
     @property
     def provider_type(self) -> ToolProviderType:
         return ToolProviderType.WORKFLOW
-    
+
     def _get_db_provider_tool(self, db_provider: WorkflowToolProvider, app: App) -> WorkflowTool:
         """
             get db provider tool
@@ -93,23 +100,11 @@ class WorkflowToolProviderController(ToolProviderController):
             if variable:
                 parameter_type = None
                 options = None
-                if variable.type in [
-                    VariableEntity.Type.TEXT_INPUT, 
-                    VariableEntity.Type.PARAGRAPH, 
-                ]:
-                    parameter_type = ToolParameter.ToolParameterType.STRING
-                elif variable.type in [
-                    VariableEntity.Type.SELECT
-                ]:
-                    parameter_type = ToolParameter.ToolParameterType.SELECT
-                elif variable.type in [
-                    VariableEntity.Type.NUMBER
-                ]:
-                    parameter_type = ToolParameter.ToolParameterType.NUMBER
-                else:
+                if variable.type not in VARIABLE_TO_PARAMETER_TYPE_MAPPING:
                     raise ValueError(f'unsupported variable type {variable.type}')
-                
-                if variable.type == VariableEntity.Type.SELECT and variable.options:
+                parameter_type = VARIABLE_TO_PARAMETER_TYPE_MAPPING[variable.type]
+
+                if variable.type == VariableEntityType.SELECT and variable.options:
                     options = [
                         ToolParameterOption(
                             value=option,
@@ -200,7 +195,7 @@ class WorkflowToolProviderController(ToolProviderController):
         """
         if self.tools is not None:
             return self.tools
-        
+
         db_providers: WorkflowToolProvider = db.session.query(WorkflowToolProvider).filter(
             WorkflowToolProvider.tenant_id == tenant_id,
             WorkflowToolProvider.app_id == self.provider_id,
@@ -208,11 +203,11 @@ class WorkflowToolProviderController(ToolProviderController):
 
         if not db_providers:
             return []
-        
+
         self.tools = [self._get_db_provider_tool(db_providers, db_providers.app)]
 
         return self.tools
-    
+
     def get_tool(self, tool_name: str) -> Optional[WorkflowTool]:
         """
             get tool by name
@@ -226,5 +221,5 @@ class WorkflowToolProviderController(ToolProviderController):
         for tool in self.tools:
             if tool.identity.name == tool_name:
                 return tool
-        
+
         return None

api/core/tools/tool_manager.py

@@ -10,14 +10,11 @@ from configs import dify_config
 from core.agent.entities import AgentToolEntity
 from core.app.entities.app_invoke_entities import InvokeFrom
 from core.helper.module_import_helper import load_single_subclass_from_source
+from core.helper.position_helper import is_filtered
 from core.model_runtime.utils.encoders import jsonable_encoder
 from core.tools.entities.api_entities import UserToolProvider, UserToolProviderTypeLiteral
 from core.tools.entities.common_entities import I18nObject
-from core.tools.entities.tool_entities import (
-    ApiProviderAuthType,
-    ToolInvokeFrom,
-    ToolParameter,
-)
+from core.tools.entities.tool_entities import ApiProviderAuthType, ToolInvokeFrom, ToolParameter
 from core.tools.errors import ToolProviderNotFoundError
 from core.tools.provider.api_tool_provider import ApiToolProviderController
 from core.tools.provider.builtin._positions import BuiltinToolProviderSort
@@ -26,10 +23,7 @@ from core.tools.tool.api_tool import ApiTool
 from core.tools.tool.builtin_tool import BuiltinTool
 from core.tools.tool.tool import Tool
 from core.tools.tool_label_manager import ToolLabelManager
-from core.tools.utils.configuration import (
-    ToolConfigurationManager,
-    ToolParameterConfigurationManager,
-)
+from core.tools.utils.configuration import ToolConfigurationManager, ToolParameterConfigurationManager
 from core.tools.utils.tool_parameter_converter import ToolParameterConverter
 from core.workflow.nodes.tool.entities import ToolEntity
 from extensions.ext_database import db
@@ -38,6 +32,7 @@ from services.tools.tools_transform_service import ToolTransformService
 
 logger = logging.getLogger(__name__)
 
+
 class ToolManager:
     _builtin_provider_lock = Lock()
     _builtin_providers = {}
@@ -107,7 +102,7 @@ class ToolManager:
                          tenant_id: str,
                          invoke_from: InvokeFrom = InvokeFrom.DEBUGGER,
                          tool_invoke_from: ToolInvokeFrom = ToolInvokeFrom.AGENT) \
-        -> Union[BuiltinTool, ApiTool]:
+            -> Union[BuiltinTool, ApiTool]:
         """
             get the tool runtime
 
@@ -346,7 +341,7 @@ class ToolManager:
                     provider_class = load_single_subclass_from_source(
                         module_name=f'core.tools.provider.builtin.{provider}.{provider}',
                         script_path=path.join(path.dirname(path.realpath(__file__)),
-                                            'provider', 'builtin', provider, f'{provider}.py'),
+                                              'provider', 'builtin', provider, f'{provider}.py'),
                         parent_type=BuiltinToolProviderController)
                     provider: BuiltinToolProviderController = provider_class()
                     cls._builtin_providers[provider.identity.name] = provider
@@ -414,6 +409,15 @@ class ToolManager:
 
             # append builtin providers
             for provider in builtin_providers:
+                # handle include, exclude
+                if is_filtered(
+                        include_set=dify_config.POSITION_TOOL_INCLUDES_SET,
+                        exclude_set=dify_config.POSITION_TOOL_EXCLUDES_SET,
+                        data=provider,
+                        name_func=lambda x: x.identity.name
+                ):
+                    continue
+
                 user_provider = ToolTransformService.builtin_provider_to_user_provider(
                     provider_controller=provider,
                     db_provider=find_db_builtin_provider(provider.identity.name),
@@ -473,7 +477,7 @@ class ToolManager:
 
     @classmethod
     def get_api_provider_controller(cls, tenant_id: str, provider_id: str) -> tuple[
-        ApiToolProviderController, dict[str, Any]]:
+            ApiToolProviderController, dict[str, Any]]:
         """
             get the api provider
 
@@ -593,4 +597,5 @@ class ToolManager:
         else:
             raise ValueError(f"provider type {provider_type} not found")
 
+
 ToolManager.load_builtin_providers_cache()

api/core/workflow/entities/variable_pool.py

@@ -6,20 +6,20 @@ from typing_extensions import deprecated
 
 from core.app.segments import Segment, Variable, factory
 from core.file.file_obj import FileVar
-from core.workflow.enums import SystemVariable
+from core.workflow.enums import SystemVariableKey
 
 VariableValue = Union[str, int, float, dict, list, FileVar]
 
 
-SYSTEM_VARIABLE_NODE_ID = 'sys'
-ENVIRONMENT_VARIABLE_NODE_ID = 'env'
-CONVERSATION_VARIABLE_NODE_ID = 'conversation'
+SYSTEM_VARIABLE_NODE_ID = "sys"
+ENVIRONMENT_VARIABLE_NODE_ID = "env"
+CONVERSATION_VARIABLE_NODE_ID = "conversation"
 
 
 class VariablePool:
     def __init__(
         self,
-        system_variables: Mapping[SystemVariable, Any],
+        system_variables: Mapping[SystemVariableKey, Any],
         user_inputs: Mapping[str, Any],
         environment_variables: Sequence[Variable],
         conversation_variables: Sequence[Variable] | None = None,
@@ -68,7 +68,7 @@ class VariablePool:
             None
         """
         if len(selector) < 2:
-            raise ValueError('Invalid selector')
+            raise ValueError("Invalid selector")
 
         if value is None:
             return
@@ -95,13 +95,13 @@ class VariablePool:
             ValueError: If the selector is invalid.
         """
         if len(selector) < 2:
-            raise ValueError('Invalid selector')
+            raise ValueError("Invalid selector")
         hash_key = hash(tuple(selector[1:]))
         value = self._variable_dictionary[selector[0]].get(hash_key)
 
         return value
 
-    @deprecated('This method is deprecated, use `get` instead.')
+    @deprecated("This method is deprecated, use `get` instead.")
     def get_any(self, selector: Sequence[str], /) -> Any | None:
         """
         Retrieves the value from the variable pool based on the given selector.
@@ -116,7 +116,7 @@ class VariablePool:
             ValueError: If the selector is invalid.
         """
         if len(selector) < 2:
-            raise ValueError('Invalid selector')
+            raise ValueError("Invalid selector")
         hash_key = hash(tuple(selector[1:]))
         value = self._variable_dictionary[selector[0]].get(hash_key)
         return value.to_object() if value else None

api/core/workflow/enums.py

@@ -1,25 +1,13 @@
 from enum import Enum
 
 
-class SystemVariable(str, Enum):
+class SystemVariableKey(str, Enum):
     """
     System Variables.
     """
-    QUERY = 'query'
-    FILES = 'files'
-    CONVERSATION_ID = 'conversation_id'
-    USER_ID = 'user_id'
-    DIALOGUE_COUNT = 'dialogue_count'
 
-    @classmethod
-    def value_of(cls, value: str):
-        """
-        Get value of given system variable.
-
-        :param value: system variable value
-        :return: system variable
-        """
-        for system_variable in cls:
-            if system_variable.value == value:
-                return system_variable
-        raise ValueError(f'invalid system variable value {value}')
+    QUERY = "query"
+    FILES = "files"
+    CONVERSATION_ID = "conversation_id"
+    USER_ID = "user_id"
+    DIALOGUE_COUNT = "dialogue_count"

api/core/workflow/nodes/code/code_node.py

@@ -13,8 +13,8 @@ from models.workflow import WorkflowNodeExecutionStatus
 
 MAX_NUMBER = dify_config.CODE_MAX_NUMBER
 MIN_NUMBER = dify_config.CODE_MIN_NUMBER
-MAX_PRECISION = 20
-MAX_DEPTH = 5
+MAX_PRECISION = dify_config.CODE_MAX_PRECISION
+MAX_DEPTH = dify_config.CODE_MAX_DEPTH
 MAX_STRING_LENGTH = dify_config.CODE_MAX_STRING_LENGTH
 MAX_STRING_ARRAY_LENGTH = dify_config.CODE_MAX_STRING_ARRAY_LENGTH
 MAX_OBJECT_ARRAY_LENGTH = dify_config.CODE_MAX_OBJECT_ARRAY_LENGTH
@@ -23,7 +23,7 @@ MAX_NUMBER_ARRAY_LENGTH = dify_config.CODE_MAX_NUMBER_ARRAY_LENGTH
 
 class CodeNode(BaseNode):
     _node_data_cls = CodeNodeData
-    node_type = NodeType.CODE
+    _node_type = NodeType.CODE
 
     @classmethod
     def get_default_config(cls, filters: Optional[dict] = None) -> dict:
@@ -48,8 +48,7 @@ class CodeNode(BaseNode):
         :param variable_pool: variable pool
         :return:
         """
-        node_data = self.node_data
-        node_data: CodeNodeData = cast(self._node_data_cls, node_data)
+        node_data = cast(CodeNodeData, self.node_data)
 
         # Get code language
         code_language = node_data.code_language
@@ -68,7 +67,6 @@ class CodeNode(BaseNode):
                 language=code_language,
                 code=code,
                 inputs=variables,
-                dependencies=node_data.dependencies
             )
 
             # Transform result

api/core/workflow/nodes/code/entities.py

@@ -3,7 +3,6 @@ from typing import Literal, Optional
 from pydantic import BaseModel
 
 from core.helper.code_executor.code_executor import CodeLanguage
-from core.helper.code_executor.entities import CodeDependency
 from core.workflow.entities.base_node_data_entities import BaseNodeData
 from core.workflow.entities.variable_entities import VariableSelector
 
@@ -16,8 +15,12 @@ class CodeNodeData(BaseNodeData):
         type: Literal['string', 'number', 'object', 'array[string]', 'array[number]', 'array[object]']
         children: Optional[dict[str, 'Output']] = None
 
+    class Dependency(BaseModel):
+        name: str
+        version: str
+
     variables: list[VariableSelector]
     code_language: Literal[CodeLanguage.PYTHON3, CodeLanguage.JAVASCRIPT]
     code: str
     outputs: dict[str, Output]
-    dependencies: Optional[list[CodeDependency]] = None
+    dependencies: Optional[list[Dependency]] = None

api/core/workflow/nodes/llm/llm_node.py

@@ -24,7 +24,7 @@ from core.prompt.entities.advanced_prompt_entities import CompletionModelPromptT
 from core.prompt.utils.prompt_message_util import PromptMessageUtil
 from core.workflow.entities.node_entities import NodeRunMetadataKey, NodeRunResult, NodeType
 from core.workflow.entities.variable_pool import VariablePool
-from core.workflow.enums import SystemVariable
+from core.workflow.enums import SystemVariableKey
 from core.workflow.nodes.base_node import BaseNode
 from core.workflow.nodes.llm.entities import (
     LLMNodeChatModelMessage,
@@ -94,7 +94,7 @@ class LLMNode(BaseNode):
             # fetch prompt messages
             prompt_messages, stop = self._fetch_prompt_messages(
                 node_data=node_data,
-                query=variable_pool.get_any(['sys', SystemVariable.QUERY.value])
+                query=variable_pool.get_any(['sys', SystemVariableKey.QUERY.value])
                 if node_data.memory else None,
                 query_prompt_template=node_data.memory.query_prompt_template if node_data.memory else None,
                 inputs=inputs,
@@ -113,7 +113,7 @@ class LLMNode(BaseNode):
             }
 
             # handle invoke result
-            result_text, usage = self._invoke_llm(
+            result_text, usage, finish_reason = self._invoke_llm(
                 node_data_model=node_data.model,
                 model_instance=model_instance,
                 prompt_messages=prompt_messages,
@@ -129,7 +129,8 @@ class LLMNode(BaseNode):
 
         outputs = {
             'text': result_text,
-            'usage': jsonable_encoder(usage)
+            'usage': jsonable_encoder(usage),
+            'finish_reason': finish_reason
         }
 
         return NodeRunResult(
@@ -167,14 +168,14 @@ class LLMNode(BaseNode):
         )
 
         # handle invoke result
-        text, usage = self._handle_invoke_result(
+        text, usage, finish_reason = self._handle_invoke_result(
             invoke_result=invoke_result
         )
 
         # deduct quota
         self.deduct_llm_quota(tenant_id=self.tenant_id, model_instance=model_instance, usage=usage)
 
-        return text, usage
+        return text, usage, finish_reason
 
     def _handle_invoke_result(self, invoke_result: Generator) -> tuple[str, LLMUsage]:
         """
@@ -186,6 +187,7 @@ class LLMNode(BaseNode):
         prompt_messages = []
         full_text = ''
         usage = None
+        finish_reason = None
         for result in invoke_result:
             text = result.delta.message.content
             full_text += text
@@ -201,10 +203,13 @@ class LLMNode(BaseNode):
             if not usage and result.delta.usage:
                 usage = result.delta.usage
 
+            if not finish_reason and result.delta.finish_reason:
+                finish_reason = result.delta.finish_reason
+
         if not usage:
             usage = LLMUsage.empty_usage()
 
-        return full_text, usage
+        return full_text, usage, finish_reason
 
     def _transform_chat_messages(self,
         messages: list[LLMNodeChatModelMessage] | LLMNodeCompletionModelPromptTemplate
@@ -335,7 +340,7 @@ class LLMNode(BaseNode):
         if not node_data.vision.enabled:
             return []
 
-        files = variable_pool.get_any(['sys', SystemVariable.FILES.value])
+        files = variable_pool.get_any(['sys', SystemVariableKey.FILES.value])
         if not files:
             return []
 
@@ -500,7 +505,7 @@ class LLMNode(BaseNode):
             return None
 
         # get conversation id
-        conversation_id = variable_pool.get_any(['sys', SystemVariable.CONVERSATION_ID.value])
+        conversation_id = variable_pool.get_any(['sys', SystemVariableKey.CONVERSATION_ID.value])
         if conversation_id is None:
             return None
 
@@ -672,10 +677,10 @@ class LLMNode(BaseNode):
             variable_mapping['#context#'] = node_data.context.variable_selector
 
         if node_data.vision.enabled:
-            variable_mapping['#files#'] = ['sys', SystemVariable.FILES.value]
+            variable_mapping['#files#'] = ['sys', SystemVariableKey.FILES.value]
 
         if node_data.memory:
-            variable_mapping['#sys.query#'] = ['sys', SystemVariable.QUERY.value]
+            variable_mapping['#sys.query#'] = ['sys', SystemVariableKey.QUERY.value]
 
         if node_data.prompt_config:
             enable_jinja = False

api/core/workflow/nodes/question_classifier/question_classifier_node.py

@@ -63,7 +63,7 @@ class QuestionClassifierNode(LLMNode):
         )
 
         # handle invoke result
-        result_text, usage = self._invoke_llm(
+        result_text, usage, finish_reason = self._invoke_llm(
             node_data_model=node_data.model,
             model_instance=model_instance,
             prompt_messages=prompt_messages,
@@ -93,6 +93,7 @@ class QuestionClassifierNode(LLMNode):
                     prompt_messages=prompt_messages
                 ),
                 'usage': jsonable_encoder(usage),
+                'finish_reason': finish_reason
             }
             outputs = {
                 'class_name': category_name

api/core/workflow/nodes/start/entities.py

@@ -1,3 +1,7 @@
+from collections.abc import Sequence
+
+from pydantic import Field
+
 from core.app.app_config.entities import VariableEntity
 from core.workflow.entities.base_node_data_entities import BaseNodeData
 
@@ -6,4 +10,4 @@ class StartNodeData(BaseNodeData):
     """
     Start Node Data
     """
-    variables: list[VariableEntity] = []
+    variables: Sequence[VariableEntity] = Field(default_factory=list)

api/core/workflow/nodes/start/start_node.py

@@ -1,7 +1,7 @@
 
 from core.workflow.entities.base_node_data_entities import BaseNodeData
 from core.workflow.entities.node_entities import NodeRunResult, NodeType
-from core.workflow.entities.variable_pool import VariablePool
+from core.workflow.entities.variable_pool import SYSTEM_VARIABLE_NODE_ID, VariablePool
 from core.workflow.nodes.base_node import BaseNode
 from core.workflow.nodes.start.entities import StartNodeData
 from models.workflow import WorkflowNodeExecutionStatus
@@ -17,16 +17,16 @@ class StartNode(BaseNode):
         :param variable_pool: variable pool
         :return:
         """
-        # Get cleaned inputs
-        cleaned_inputs = dict(variable_pool.user_inputs)
+        node_inputs = dict(variable_pool.user_inputs)
+        system_inputs = variable_pool.system_variables
 
-        for var in variable_pool.system_variables:
-            cleaned_inputs['sys.' + var.value] = variable_pool.system_variables[var]
+        for var in system_inputs:
+            node_inputs[SYSTEM_VARIABLE_NODE_ID + '.' + var] = system_inputs[var]
 
         return NodeRunResult(
             status=WorkflowNodeExecutionStatus.SUCCEEDED,
-            inputs=cleaned_inputs,
-            outputs=cleaned_inputs
+            inputs=node_inputs,
+            outputs=node_inputs
         )
 
     @classmethod

api/core/workflow/nodes/tool/tool_node.py

@@ -2,7 +2,7 @@ from collections.abc import Mapping, Sequence
 from os import path
 from typing import Any, cast
 
-from core.app.segments import ArrayAnyVariable, parser
+from core.app.segments import ArrayAnySegment, ArrayAnyVariable, parser
 from core.callback_handler.workflow_tool_callback_handler import DifyWorkflowCallbackHandler
 from core.file.file_obj import FileTransferMethod, FileType, FileVar
 from core.tools.entities.tool_entities import ToolInvokeMessage, ToolParameter
@@ -11,7 +11,7 @@ from core.tools.tool_manager import ToolManager
 from core.tools.utils.message_transformer import ToolFileMessageTransformer
 from core.workflow.entities.node_entities import NodeRunMetadataKey, NodeRunResult, NodeType
 from core.workflow.entities.variable_pool import VariablePool
-from core.workflow.enums import SystemVariable
+from core.workflow.enums import SystemVariableKey
 from core.workflow.nodes.base_node import BaseNode
 from core.workflow.nodes.tool.entities import ToolNodeData
 from core.workflow.utils.variable_template_parser import VariableTemplateParser
@@ -141,8 +141,8 @@ class ToolNode(BaseNode):
         return result
 
     def _fetch_files(self, variable_pool: VariablePool) -> list[FileVar]:
-        variable = variable_pool.get(['sys', SystemVariable.FILES.value])
-        assert isinstance(variable, ArrayAnyVariable)
+        variable = variable_pool.get(['sys', SystemVariableKey.FILES.value])
+        assert isinstance(variable, ArrayAnyVariable | ArrayAnySegment)
         return list(variable.value) if variable else []
 
     def _convert_tool_messages(self, messages: list[ToolInvokeMessage]):

api/core/workflow/nodes/variable_assigner/__init__.py

@@ -1,109 +1,8 @@
-from collections.abc import Sequence
-from enum import Enum
-from typing import Optional, cast
+from .node import VariableAssignerNode
+from .node_data import VariableAssignerData, WriteMode
 
-from sqlalchemy import select
-from sqlalchemy.orm import Session
-
-from core.app.segments import SegmentType, Variable, factory
-from core.workflow.entities.base_node_data_entities import BaseNodeData
-from core.workflow.entities.node_entities import NodeRunResult, NodeType
-from core.workflow.entities.variable_pool import VariablePool
-from core.workflow.nodes.base_node import BaseNode
-from extensions.ext_database import db
-from models import ConversationVariable, WorkflowNodeExecutionStatus
-
-
-class VariableAssignerNodeError(Exception):
-    pass
-
-
-class WriteMode(str, Enum):
-    OVER_WRITE = 'over-write'
-    APPEND = 'append'
-    CLEAR = 'clear'
-
-
-class VariableAssignerData(BaseNodeData):
-    title: str = 'Variable Assigner'
-    desc: Optional[str] = 'Assign a value to a variable'
-    assigned_variable_selector: Sequence[str]
-    write_mode: WriteMode
-    input_variable_selector: Sequence[str]
-
-
-class VariableAssignerNode(BaseNode):
-    _node_data_cls: type[BaseNodeData] = VariableAssignerData
-    _node_type: NodeType = NodeType.CONVERSATION_VARIABLE_ASSIGNER
-
-    def _run(self, variable_pool: VariablePool) -> NodeRunResult:
-        data = cast(VariableAssignerData, self.node_data)
-
-        # Should be String, Number, Object, ArrayString, ArrayNumber, ArrayObject
-        original_variable = variable_pool.get(data.assigned_variable_selector)
-        if not isinstance(original_variable, Variable):
-            raise VariableAssignerNodeError('assigned variable not found')
-
-        match data.write_mode:
-            case WriteMode.OVER_WRITE:
-                income_value = variable_pool.get(data.input_variable_selector)
-                if not income_value:
-                    raise VariableAssignerNodeError('input value not found')
-                updated_variable = original_variable.model_copy(update={'value': income_value.value})
-
-            case WriteMode.APPEND:
-                income_value = variable_pool.get(data.input_variable_selector)
-                if not income_value:
-                    raise VariableAssignerNodeError('input value not found')
-                updated_value = original_variable.value + [income_value.value]
-                updated_variable = original_variable.model_copy(update={'value': updated_value})
-
-            case WriteMode.CLEAR:
-                income_value = get_zero_value(original_variable.value_type)
-                updated_variable = original_variable.model_copy(update={'value': income_value.to_object()})
-
-            case _:
-                raise VariableAssignerNodeError(f'unsupported write mode: {data.write_mode}')
-
-        # Over write the variable.
-        variable_pool.add(data.assigned_variable_selector, updated_variable)
-
-        # Update conversation variable.
-        # TODO: Find a better way to use the database.
-        conversation_id = variable_pool.get(['sys', 'conversation_id'])
-        if not conversation_id:
-            raise VariableAssignerNodeError('conversation_id not found')
-        update_conversation_variable(conversation_id=conversation_id.text, variable=updated_variable)
-
-        return NodeRunResult(
-            status=WorkflowNodeExecutionStatus.SUCCEEDED,
-            inputs={
-                'value': income_value.to_object(),
-            },
-        )
-
-
-def update_conversation_variable(conversation_id: str, variable: Variable):
-    stmt = select(ConversationVariable).where(
-        ConversationVariable.id == variable.id, ConversationVariable.conversation_id == conversation_id
-    )
-    with Session(db.engine) as session:
-        row = session.scalar(stmt)
-        if not row:
-            raise VariableAssignerNodeError('conversation variable not found in the database')
-        row.data = variable.model_dump_json()
-        session.commit()
-
-
-def get_zero_value(t: SegmentType):
-    match t:
-        case SegmentType.ARRAY_OBJECT | SegmentType.ARRAY_STRING | SegmentType.ARRAY_NUMBER:
-            return factory.build_segment([])
-        case SegmentType.OBJECT:
-            return factory.build_segment({})
-        case SegmentType.STRING:
-            return factory.build_segment('')
-        case SegmentType.NUMBER:
-            return factory.build_segment(0)
-        case _:
-            raise VariableAssignerNodeError(f'unsupported variable type: {t}')
+__all__ = [
+    'VariableAssignerNode',
+    'VariableAssignerData',
+    'WriteMode',
+]

api/core/workflow/nodes/variable_assigner/exc.py

@@ -0,0 +1,2 @@
+class VariableAssignerNodeError(Exception):
+    pass

api/core/workflow/nodes/variable_assigner/node.py

@@ -0,0 +1,92 @@
+from typing import cast
+
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from core.app.segments import SegmentType, Variable, factory
+from core.workflow.entities.base_node_data_entities import BaseNodeData
+from core.workflow.entities.node_entities import NodeRunResult, NodeType
+from core.workflow.entities.variable_pool import VariablePool
+from core.workflow.nodes.base_node import BaseNode
+from extensions.ext_database import db
+from models import ConversationVariable, WorkflowNodeExecutionStatus
+
+from .exc import VariableAssignerNodeError
+from .node_data import VariableAssignerData, WriteMode
+
+
+class VariableAssignerNode(BaseNode):
+    _node_data_cls: type[BaseNodeData] = VariableAssignerData
+    _node_type: NodeType = NodeType.CONVERSATION_VARIABLE_ASSIGNER
+
+    def _run(self, variable_pool: VariablePool) -> NodeRunResult:
+        data = cast(VariableAssignerData, self.node_data)
+
+        # Should be String, Number, Object, ArrayString, ArrayNumber, ArrayObject
+        original_variable = variable_pool.get(data.assigned_variable_selector)
+        if not isinstance(original_variable, Variable):
+            raise VariableAssignerNodeError('assigned variable not found')
+
+        match data.write_mode:
+            case WriteMode.OVER_WRITE:
+                income_value = variable_pool.get(data.input_variable_selector)
+                if not income_value:
+                    raise VariableAssignerNodeError('input value not found')
+                updated_variable = original_variable.model_copy(update={'value': income_value.value})
+
+            case WriteMode.APPEND:
+                income_value = variable_pool.get(data.input_variable_selector)
+                if not income_value:
+                    raise VariableAssignerNodeError('input value not found')
+                updated_value = original_variable.value + [income_value.value]
+                updated_variable = original_variable.model_copy(update={'value': updated_value})
+
+            case WriteMode.CLEAR:
+                income_value = get_zero_value(original_variable.value_type)
+                updated_variable = original_variable.model_copy(update={'value': income_value.to_object()})
+
+            case _:
+                raise VariableAssignerNodeError(f'unsupported write mode: {data.write_mode}')
+
+        # Over write the variable.
+        variable_pool.add(data.assigned_variable_selector, updated_variable)
+
+        # TODO: Move database operation to the pipeline.
+        # Update conversation variable.
+        conversation_id = variable_pool.get(['sys', 'conversation_id'])
+        if not conversation_id:
+            raise VariableAssignerNodeError('conversation_id not found')
+        update_conversation_variable(conversation_id=conversation_id.text, variable=updated_variable)
+
+        return NodeRunResult(
+            status=WorkflowNodeExecutionStatus.SUCCEEDED,
+            inputs={
+                'value': income_value.to_object(),
+            },
+        )
+
+
+def update_conversation_variable(conversation_id: str, variable: Variable):
+    stmt = select(ConversationVariable).where(
+        ConversationVariable.id == variable.id, ConversationVariable.conversation_id == conversation_id
+    )
+    with Session(db.engine) as session:
+        row = session.scalar(stmt)
+        if not row:
+            raise VariableAssignerNodeError('conversation variable not found in the database')
+        row.data = variable.model_dump_json()
+        session.commit()
+
+
+def get_zero_value(t: SegmentType):
+    match t:
+        case SegmentType.ARRAY_OBJECT | SegmentType.ARRAY_STRING | SegmentType.ARRAY_NUMBER:
+            return factory.build_segment([])
+        case SegmentType.OBJECT:
+            return factory.build_segment({})
+        case SegmentType.STRING:
+            return factory.build_segment('')
+        case SegmentType.NUMBER:
+            return factory.build_segment(0)
+        case _:
+            raise VariableAssignerNodeError(f'unsupported variable type: {t}')

api/core/workflow/nodes/variable_assigner/node_data.py

@@ -0,0 +1,19 @@
+from collections.abc import Sequence
+from enum import Enum
+from typing import Optional
+
+from core.workflow.entities.base_node_data_entities import BaseNodeData
+
+
+class WriteMode(str, Enum):
+    OVER_WRITE = 'over-write'
+    APPEND = 'append'
+    CLEAR = 'clear'
+
+
+class VariableAssignerData(BaseNodeData):
+    title: str = 'Variable Assigner'
+    desc: Optional[str] = 'Assign a value to a variable'
+    assigned_variable_selector: Sequence[str]
+    write_mode: WriteMode
+    input_variable_selector: Sequence[str]

api/extensions/storage/s3_storage.py

@@ -28,6 +28,16 @@ class S3Storage(BaseStorage):
                 region_name=app_config.get("S3_REGION"),
                 config=Config(s3={"addressing_style": app_config.get("S3_ADDRESS_STYLE")}),
             )
+        # create bucket
+        try:
+            self.client.head_bucket(Bucket=self.bucket_name)
+        except ClientError as e:
+            # if bucket not exists, create it
+            if e.response["Error"]["Code"] == "404":
+                self.client.create_bucket(Bucket=self.bucket_name)
+            else:
+                # other error, raise exception
+                raise
 
     def save(self, filename, data):
         self.client.put_object(Bucket=self.bucket_name, Key=filename, Body=data)

api/fields/conversation_fields.py

@@ -150,6 +150,7 @@ conversation_with_summary_fields = {
     "summary": fields.String(attribute="summary_or_query"),
     "read_at": TimestampField,
     "created_at": TimestampField,
+    "updated_at": TimestampField,
     "annotated": fields.Boolean,
     "model_config": fields.Nested(simple_model_config_fields),
     "message_count": fields.Integer,

api/migrations/versions/2024_08_20_0455-2dbe42621d96_rename_workflow__conversation_variables_.py

@@ -0,0 +1,28 @@
+"""rename workflow__conversation_variables to workflow_conversation_variables
+
+Revision ID: 2dbe42621d96
+Revises: a6be81136580
+Create Date: 2024-08-20 04:55:38.160010
+
+"""
+from alembic import op
+
+import models as models
+
+# revision identifiers, used by Alembic.
+revision = '2dbe42621d96'
+down_revision = 'a6be81136580'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.rename_table('workflow__conversation_variables', 'workflow_conversation_variables')
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.rename_table('workflow_conversation_variables', 'workflow__conversation_variables')
+    # ### end Alembic commands ###

api/models/dataset.py

@@ -1,4 +1,5 @@
 import base64
+import enum
 import hashlib
 import hmac
 import json
@@ -22,6 +23,11 @@ from .model import App, Tag, TagBinding, UploadFile
 from .types import StringUUID
 
 
+class DatasetPermissionEnum(str, enum.Enum):
+    ONLY_ME = 'only_me'
+    ALL_TEAM = 'all_team_members'
+    PARTIAL_TEAM = 'partial_members'
+
 class Dataset(db.Model):
     __tablename__ = 'datasets'
     __table_args__ = (

api/models/workflow.py

@@ -1,5 +1,6 @@
 import json
 from collections.abc import Mapping, Sequence
+from datetime import datetime
 from enum import Enum
 from typing import Any, Optional, Union
 
@@ -110,19 +111,32 @@ class Workflow(db.Model):
         db.Index('workflow_version_idx', 'tenant_id', 'app_id', 'version'),
     )
 
-    id = db.Column(StringUUID, server_default=db.text('uuid_generate_v4()'))
-    tenant_id = db.Column(StringUUID, nullable=False)
-    app_id = db.Column(StringUUID, nullable=False)
-    type = db.Column(db.String(255), nullable=False)
-    version = db.Column(db.String(255), nullable=False)
-    graph = db.Column(db.Text)
-    features = db.Column(db.Text)
-    created_by = db.Column(StringUUID, nullable=False)
-    created_at = db.Column(db.DateTime, nullable=False, server_default=db.text('CURRENT_TIMESTAMP(0)'))
-    updated_by = db.Column(StringUUID)
-    updated_at = db.Column(db.DateTime)
-    _environment_variables = db.Column('environment_variables', db.Text, nullable=False, server_default='{}')
-    _conversation_variables = db.Column('conversation_variables', db.Text, nullable=False, server_default='{}')
+    id: Mapped[str] = db.Column(StringUUID, server_default=db.text('uuid_generate_v4()'))
+    tenant_id: Mapped[str] = db.Column(StringUUID, nullable=False)
+    app_id: Mapped[str] = db.Column(StringUUID, nullable=False)
+    type: Mapped[str] = db.Column(db.String(255), nullable=False)
+    version: Mapped[str] = db.Column(db.String(255), nullable=False)
+    graph: Mapped[str] = db.Column(db.Text)
+    features: Mapped[str] = db.Column(db.Text)
+    created_by: Mapped[str] = db.Column(StringUUID, nullable=False)
+    created_at: Mapped[datetime] = db.Column(db.DateTime, nullable=False, server_default=db.text('CURRENT_TIMESTAMP(0)'))
+    updated_by: Mapped[str] = db.Column(StringUUID)
+    updated_at: Mapped[datetime] = db.Column(db.DateTime)
+    _environment_variables: Mapped[str] = db.Column('environment_variables', db.Text, nullable=False, server_default='{}')
+    _conversation_variables: Mapped[str] = db.Column('conversation_variables', db.Text, nullable=False, server_default='{}')
+
+    def __init__(self, *, tenant_id: str, app_id: str, type: str, version: str, graph: str,
+                 features: str, created_by: str, environment_variables: Sequence[Variable],
+                 conversation_variables: Sequence[Variable]):
+        self.tenant_id = tenant_id
+        self.app_id = app_id
+        self.type = type
+        self.version = version
+        self.graph = graph
+        self.features = features
+        self.created_by = created_by
+        self.environment_variables = environment_variables or []
+        self.conversation_variables = conversation_variables or []
 
     @property
     def created_by_account(self):
@@ -724,7 +738,7 @@ class WorkflowAppLog(db.Model):
 
 
 class ConversationVariable(db.Model):
-    __tablename__ = 'workflow__conversation_variables'
+    __tablename__ = 'workflow_conversation_variables'
 
     id: Mapped[str] = db.Column(StringUUID, primary_key=True)
     conversation_id: Mapped[str] = db.Column(StringUUID, nullable=False, primary_key=True)

api/pyproject.toml

@@ -153,7 +153,7 @@ langfuse = "^2.36.1"
 langsmith = "^0.1.77"
 mailchimp-transactional = "~1.0.50"
 markdown = "~3.5.1"
-novita-client = "^0.5.6"
+novita-client = "^0.5.7"
 numpy = "~1.26.4"
 openai = "~1.29.0"
 openpyxl = "~3.1.5"
@@ -216,7 +216,7 @@ twilio = "~9.0.4"
 vanna = { version = "0.5.5", extras = ["postgres", "mysql", "clickhouse", "duckdb"] }
 wikipedia = "1.4.0"
 yfinance = "~0.2.40"
-
+nltk = "3.8.1"
 ############################################################
 # VDB dependencies required by vector store clients
 ############################################################

api/services/app_service.py

@@ -111,6 +111,12 @@ class AppService:
                         'completion_params': {}
                     }
             else:
+                provider, model = model_manager.get_default_provider_model_name(
+                    tenant_id=account.current_tenant_id,
+                    model_type=ModelType.LLM
+                )
+                default_model_config['model']['provider'] = provider
+                default_model_config['model']['name'] = model
                 default_model_dict = default_model_config['model']
 
             default_model_config['model'] = json.dumps(default_model_dict)
@@ -190,13 +196,14 @@ class AppService:
                 """
                 Modified App class
                 """
+
                 def __init__(self, app):
                     self.__dict__.update(app.__dict__)
 
                 @property
                 def app_model_config(self):
                     return model_config
-                
+
             app = ModifiedApp(app)
 
         return app

api/services/conversation_service.py

@@ -1,6 +1,7 @@
+from datetime import datetime, timezone
 from typing import Optional, Union
 
-from sqlalchemy import or_
+from sqlalchemy import asc, desc, or_
 
 from core.app.entities.app_invoke_entities import InvokeFrom
 from core.llm_generator.llm_generator import LLMGenerator
@@ -18,7 +19,8 @@ class ConversationService:
                               last_id: Optional[str], limit: int,
                               invoke_from: InvokeFrom,
                               include_ids: Optional[list] = None,
-                              exclude_ids: Optional[list] = None) -> InfiniteScrollPagination:
+                              exclude_ids: Optional[list] = None,
+                              sort_by: str = '-updated_at') -> InfiniteScrollPagination:
         if not user:
             return InfiniteScrollPagination(data=[], limit=limit, has_more=False)
 
@@ -37,28 +39,28 @@ class ConversationService:
         if exclude_ids is not None:
             base_query = base_query.filter(~Conversation.id.in_(exclude_ids))
 
-        if last_id:
-            last_conversation = base_query.filter(
-                Conversation.id == last_id,
-            ).first()
+        # define sort fields and directions
+        sort_field, sort_direction = cls._get_sort_params(sort_by)
 
+        if last_id:
+            last_conversation = base_query.filter(Conversation.id == last_id).first()
             if not last_conversation:
                 raise LastConversationNotExistsError()
 
-            conversations = base_query.filter(
-                Conversation.created_at < last_conversation.created_at,
-                Conversation.id != last_conversation.id
-            ).order_by(Conversation.created_at.desc()).limit(limit).all()
-        else:
-            conversations = base_query.order_by(Conversation.created_at.desc()).limit(limit).all()
+            # build filters based on sorting
+            filter_condition = cls._build_filter_condition(sort_field, sort_direction, last_conversation)
+            base_query = base_query.filter(filter_condition)
+
+        base_query = base_query.order_by(sort_direction(getattr(Conversation, sort_field)))
+
+        conversations = base_query.limit(limit).all()
 
         has_more = False
         if len(conversations) == limit:
-            current_page_first_conversation = conversations[-1]
-            rest_count = base_query.filter(
-                Conversation.created_at < current_page_first_conversation.created_at,
-                Conversation.id != current_page_first_conversation.id
-            ).count()
+            current_page_last_conversation = conversations[-1]
+            rest_filter_condition = cls._build_filter_condition(sort_field, sort_direction,
+                                                                current_page_last_conversation, is_next_page=True)
+            rest_count = base_query.filter(rest_filter_condition).count()
 
             if rest_count > 0:
                 has_more = True
@@ -69,6 +71,21 @@ class ConversationService:
             has_more=has_more
         )
 
+    @classmethod
+    def _get_sort_params(cls, sort_by: str) -> tuple[str, callable]:
+        if sort_by.startswith('-'):
+            return sort_by[1:], desc
+        return sort_by, asc
+
+    @classmethod
+    def _build_filter_condition(cls, sort_field: str, sort_direction: callable, reference_conversation: Conversation,
+                                is_next_page: bool = False):
+        field_value = getattr(reference_conversation, sort_field)
+        if (sort_direction == desc and not is_next_page) or (sort_direction == asc and is_next_page):
+            return getattr(Conversation, sort_field) < field_value
+        else:
+            return getattr(Conversation, sort_field) > field_value
+
     @classmethod
     def rename(cls, app_model: App, conversation_id: str,
                user: Optional[Union[Account, EndUser]], name: str, auto_generate: bool):
@@ -78,6 +95,7 @@ class ConversationService:
             return cls.auto_generate_name(app_model, conversation)
         else:
             conversation.name = name
+            conversation.updated_at = datetime.now(timezone.utc).replace(tzinfo=None)
             db.session.commit()
 
         return conversation
@@ -87,9 +105,9 @@ class ConversationService:
         # get conversation first message
         message = db.session.query(Message) \
             .filter(
-                Message.app_id == app_model.id,
-                Message.conversation_id == conversation.id
-            ).order_by(Message.created_at.asc()).first()
+            Message.app_id == app_model.id,
+            Message.conversation_id == conversation.id
+        ).order_by(Message.created_at.asc()).first()
 
         if not message:
             raise MessageNotExistsError()

api/services/dataset_service.py

@@ -27,6 +27,7 @@ from models.dataset import (
     Dataset,
     DatasetCollectionBinding,
     DatasetPermission,
+    DatasetPermissionEnum,
     DatasetProcessRule,
     DatasetQuery,
     Document,
@@ -80,21 +81,21 @@ class DatasetService:
                 if permitted_dataset_ids:
                     query = query.filter(
                         db.or_(
-                            Dataset.permission == 'all_team_members',
-                            db.and_(Dataset.permission == 'only_me', Dataset.created_by == user.id),
-                            db.and_(Dataset.permission == 'partial_members', Dataset.id.in_(permitted_dataset_ids))
+                            Dataset.permission == DatasetPermissionEnum.ALL_TEAM,
+                            db.and_(Dataset.permission == DatasetPermissionEnum.ONLY_ME, Dataset.created_by == user.id),
+                            db.and_(Dataset.permission == DatasetPermissionEnum.PARTIAL_TEAM, Dataset.id.in_(permitted_dataset_ids))
                         )
                     )
                 else:
                     query = query.filter(
                         db.or_(
-                            Dataset.permission == 'all_team_members',
-                            db.and_(Dataset.permission == 'only_me', Dataset.created_by == user.id)
+                            Dataset.permission == DatasetPermissionEnum.ALL_TEAM,
+                            db.and_(Dataset.permission == DatasetPermissionEnum.ONLY_ME, Dataset.created_by == user.id)
                         )
                     )
         else:
             # if no user, only show datasets that are shared with all team members
-            query = query.filter(Dataset.permission == 'all_team_members')
+            query = query.filter(Dataset.permission == DatasetPermissionEnum.ALL_TEAM)
 
         if search:
             query = query.filter(Dataset.name.ilike(f'%{search}%'))
@@ -330,7 +331,7 @@ class DatasetService:
             raise NoPermissionError(
                 'You do not have permission to access this dataset.'
             )
-        if dataset.permission == 'only_me' and dataset.created_by != user.id:
+        if dataset.permission == DatasetPermissionEnum.ONLY_ME and dataset.created_by != user.id:
             logging.debug(
                 f'User {user.id} does not have permission to access dataset {dataset.id}'
             )
@@ -351,11 +352,11 @@ class DatasetService:
 
     @staticmethod
     def check_dataset_operator_permission(user: Account = None, dataset: Dataset = None):
-        if dataset.permission == 'only_me':
+        if dataset.permission == DatasetPermissionEnum.ONLY_ME:
             if dataset.created_by != user.id:
                 raise NoPermissionError('You do not have permission to access this dataset.')
 
-        elif dataset.permission == 'partial_members':
+        elif dataset.permission == DatasetPermissionEnum.PARTIAL_TEAM:
             if not any(
                 dp.dataset_id == dataset.id for dp in DatasetPermission.query.filter_by(account_id=user.id).all()
             ):

api/services/model_provider_service.py

@@ -30,6 +30,7 @@ class ModelProviderService:
     """
     Model Provider Service
     """
+
     def __init__(self) -> None:
         self.provider_manager = ProviderManager()
 
@@ -387,18 +388,21 @@ class ModelProviderService:
             tenant_id=tenant_id,
             model_type=model_type_enum
         )
-
-        return DefaultModelResponse(
-            model=result.model,
-            model_type=result.model_type,
-            provider=SimpleProviderEntityResponse(
-                provider=result.provider.provider,
-                label=result.provider.label,
-                icon_small=result.provider.icon_small,
-                icon_large=result.provider.icon_large,
-                supported_model_types=result.provider.supported_model_types
-            )
-        ) if result else None
+        try:
+            return DefaultModelResponse(
+                model=result.model,
+                model_type=result.model_type,
+                provider=SimpleProviderEntityResponse(
+                    provider=result.provider.provider,
+                    label=result.provider.label,
+                    icon_small=result.provider.icon_small,
+                    icon_large=result.provider.icon_large,
+                    supported_model_types=result.provider.supported_model_types
+                )
+            ) if result else None
+        except Exception as e:
+            logger.info(f"get_default_model_of_model_type error: {e}")
+            return None
 
     def update_default_model_of_model_type(self, tenant_id: str, model_type: str, provider: str, model: str) -> None:
         """

api/services/ops_service.py

@@ -22,6 +22,10 @@ class OpsService:
         # decrypt_token and obfuscated_token
         tenant_id = db.session.query(App).filter(App.id == app_id).first().tenant_id
         decrypt_tracing_config = OpsTraceManager.decrypt_tracing_config(tenant_id, tracing_provider, trace_config_data.tracing_config)
+        if tracing_provider == 'langfuse' and ('project_key' not in decrypt_tracing_config or not decrypt_tracing_config.get('project_key')):
+            project_key = OpsTraceManager.get_trace_config_project_key(decrypt_tracing_config, tracing_provider)
+            decrypt_tracing_config['project_key'] = project_key
+
         decrypt_tracing_config = OpsTraceManager.obfuscated_decrypt_token(tracing_provider, decrypt_tracing_config)
 
         trace_config_data.tracing_config = decrypt_tracing_config
@@ -37,7 +41,7 @@ class OpsService:
         :param tracing_config: tracing config
         :return:
         """
-        if tracing_provider not in provider_config_map.keys() and tracing_provider != None:
+        if tracing_provider not in provider_config_map.keys() and tracing_provider:
             return {"error": f"Invalid tracing provider: {tracing_provider}"}
 
         config_class, other_keys = provider_config_map[tracing_provider]['config_class'], \
@@ -51,6 +55,9 @@ class OpsService:
         if not OpsTraceManager.check_trace_config_is_effective(tracing_config, tracing_provider):
             return {"error": "Invalid Credentials"}
 
+        # get project key
+        project_key = OpsTraceManager.get_trace_config_project_key(tracing_config, tracing_provider)
+
         # check if trace config already exists
         trace_config_data: TraceAppConfig = db.session.query(TraceAppConfig).filter(
             TraceAppConfig.app_id == app_id, TraceAppConfig.tracing_provider == tracing_provider
@@ -62,6 +69,8 @@ class OpsService:
         # get tenant id
         tenant_id = db.session.query(App).filter(App.id == app_id).first().tenant_id
         tracing_config = OpsTraceManager.encrypt_tracing_config(tenant_id, tracing_provider, tracing_config)
+        if tracing_provider == 'langfuse':
+            tracing_config['project_key'] = project_key
         trace_config_data = TraceAppConfig(
             app_id=app_id,
             tracing_provider=tracing_provider,

api/services/tools/builtin_tools_manage_service.py

@@ -1,6 +1,8 @@
 import json
 import logging
 
+from configs import dify_config
+from core.helper.position_helper import is_filtered
 from core.model_runtime.utils.encoders import jsonable_encoder
 from core.tools.entities.api_entities import UserTool, UserToolProvider
 from core.tools.errors import ToolNotFoundError, ToolProviderCredentialValidationError, ToolProviderNotFoundError
@@ -43,14 +45,14 @@ class BuiltinToolManageService:
         result = []
         for tool in tools:
             result.append(ToolTransformService.tool_to_user_tool(
-                tool=tool, 
-                credentials=credentials, 
+                tool=tool,
+                credentials=credentials,
                 tenant_id=tenant_id,
                 labels=ToolLabelManager.get_tool_labels(provider_controller)
             ))
 
         return result
-    
+
     @staticmethod
     def list_builtin_provider_credentials_schema(
         provider_name
@@ -78,7 +80,7 @@ class BuiltinToolManageService:
             BuiltinToolProvider.provider == provider_name,
         ).first()
 
-        try: 
+        try:
             # get provider
             provider_controller = ToolManager.get_builtin_provider(provider_name)
             if not provider_controller.need_credentials:
@@ -119,8 +121,8 @@ class BuiltinToolManageService:
             # delete cache
             tool_configuration.delete_tool_credentials_cache()
 
-        return { 'result': 'success' }
-    
+        return {'result': 'success'}
+
     @staticmethod
     def get_builtin_tool_provider_credentials(
         user_id: str, tenant_id: str, provider: str
@@ -135,7 +137,7 @@ class BuiltinToolManageService:
 
         if provider is None:
             return {}
-        
+
         provider_controller = ToolManager.get_builtin_provider(provider.provider)
         tool_configuration = ToolConfigurationManager(tenant_id=tenant_id, provider_controller=provider_controller)
         credentials = tool_configuration.decrypt_tool_credentials(provider.credentials)
@@ -156,7 +158,7 @@ class BuiltinToolManageService:
 
         if provider is None:
             raise ValueError(f'you have not added provider {provider_name}')
-        
+
         db.session.delete(provider)
         db.session.commit()
 
@@ -165,8 +167,8 @@ class BuiltinToolManageService:
         tool_configuration = ToolConfigurationManager(tenant_id=tenant_id, provider_controller=provider_controller)
         tool_configuration.delete_tool_credentials_cache()
 
-        return { 'result': 'success' }
-    
+        return {'result': 'success'}
+
     @staticmethod
     def get_builtin_tool_provider_icon(
         provider: str
@@ -179,7 +181,7 @@ class BuiltinToolManageService:
             icon_bytes = f.read()
 
         return icon_bytes, mime_type
-    
+
     @staticmethod
     def list_builtin_tools(
         user_id: str, tenant_id: str
@@ -202,6 +204,15 @@ class BuiltinToolManageService:
 
         for provider_controller in provider_controllers:
             try:
+                # handle include, exclude
+                if is_filtered(
+                    include_set=dify_config.POSITION_TOOL_INCLUDES_SET,
+                    exclude_set=dify_config.POSITION_TOOL_EXCLUDES_SET,
+                    data=provider_controller,
+                    name_func=lambda x: x.identity.name
+                ):
+                    continue
+
                 # convert provider controller to user provider
                 user_builtin_provider = ToolTransformService.builtin_provider_to_user_provider(
                     provider_controller=provider_controller,
@@ -226,4 +237,3 @@ class BuiltinToolManageService:
                 raise e
 
         return BuiltinToolProviderSort.sort(result)
-    

api/services/web_conversation_service.py

@@ -13,7 +13,8 @@ class WebConversationService:
     @classmethod
     def pagination_by_last_id(cls, app_model: App, user: Optional[Union[Account, EndUser]],
                               last_id: Optional[str], limit: int, invoke_from: InvokeFrom,
-                              pinned: Optional[bool] = None) -> InfiniteScrollPagination:
+                              pinned: Optional[bool] = None,
+                              sort_by='-updated_at') -> InfiniteScrollPagination:
         include_ids = None
         exclude_ids = None
         if pinned is not None:
@@ -36,6 +37,7 @@ class WebConversationService:
             invoke_from=invoke_from,
             include_ids=include_ids,
             exclude_ids=exclude_ids,
+            sort_by=sort_by
         )
 
     @classmethod

api/services/workflow/workflow_converter.py

@@ -32,12 +32,9 @@ class WorkflowConverter:
     App Convert to Workflow Mode
     """
 
-    def convert_to_workflow(self, app_model: App,
-                            account: Account,
-                            name: str,
-                            icon_type: str,
-                            icon: str,
-                            icon_background: str) -> App:
+    def convert_to_workflow(
+        self, app_model: App, account: Account, name: str, icon_type: str, icon: str, icon_background: str
+    ):
         """
         Convert app to workflow
 
@@ -56,18 +53,18 @@ class WorkflowConverter:
         :return: new App instance
         """
         # convert app model config
+        if not app_model.app_model_config:
+            raise ValueError("App model config is required")
+
         workflow = self.convert_app_model_config_to_workflow(
-            app_model=app_model,
-            app_model_config=app_model.app_model_config,
-            account_id=account.id
+            app_model=app_model, app_model_config=app_model.app_model_config, account_id=account.id
         )
 
         # create new app
         new_app = App()
         new_app.tenant_id = app_model.tenant_id
-        new_app.name = name if name else app_model.name + '(workflow)'
-        new_app.mode = AppMode.ADVANCED_CHAT.value \
-            if app_model.mode == AppMode.CHAT.value else AppMode.WORKFLOW.value
+        new_app.name = name if name else app_model.name + "(workflow)"
+        new_app.mode = AppMode.ADVANCED_CHAT.value if app_model.mode == AppMode.CHAT.value else AppMode.WORKFLOW.value
         new_app.icon_type = icon_type if icon_type else app_model.icon_type
         new_app.icon = icon if icon else app_model.icon
         new_app.icon_background = icon_background if icon_background else app_model.icon_background
@@ -88,30 +85,21 @@ class WorkflowConverter:
 
         return new_app
 
-    def convert_app_model_config_to_workflow(self, app_model: App,
-                                             app_model_config: AppModelConfig,
-                                             account_id: str) -> Workflow:
+    def convert_app_model_config_to_workflow(self, app_model: App, app_model_config: AppModelConfig, account_id: str):
         """
         Convert app model config to workflow mode
         :param app_model: App instance
         :param app_model_config: AppModelConfig instance
         :param account_id: Account ID
-        :return:
         """
         # get new app mode
         new_app_mode = self._get_new_app_mode(app_model)
 
         # convert app model config
-        app_config = self._convert_to_app_config(
-            app_model=app_model,
-            app_model_config=app_model_config
-        )
+        app_config = self._convert_to_app_config(app_model=app_model, app_model_config=app_model_config)
 
         # init workflow graph
-        graph = {
-            "nodes": [],
-            "edges": []
-        }
+        graph = {"nodes": [], "edges": []}
 
         # Convert list:
         # - variables -> start
@@ -123,11 +111,9 @@ class WorkflowConverter:
         # - show_retrieve_source -> knowledge-retrieval
 
         # convert to start node
-        start_node = self._convert_to_start_node(
-            variables=app_config.variables
-        )
+        start_node = self._convert_to_start_node(variables=app_config.variables)
 
-        graph['nodes'].append(start_node)
+        graph["nodes"].append(start_node)
 
         # convert to http request node
         external_data_variable_node_mapping = {}
@@ -135,7 +121,7 @@ class WorkflowConverter:
             http_request_nodes, external_data_variable_node_mapping = self._convert_to_http_request_node(
                 app_model=app_model,
                 variables=app_config.variables,
-                external_data_variables=app_config.external_data_variables
+                external_data_variables=app_config.external_data_variables,
             )
 
             for http_request_node in http_request_nodes:
@@ -144,9 +130,7 @@ class WorkflowConverter:
         # convert to knowledge retrieval node
         if app_config.dataset:
             knowledge_retrieval_node = self._convert_to_knowledge_retrieval_node(
-                new_app_mode=new_app_mode,
-                dataset_config=app_config.dataset,
-                model_config=app_config.model
+                new_app_mode=new_app_mode, dataset_config=app_config.dataset, model_config=app_config.model
             )
 
             if knowledge_retrieval_node:
@@ -160,7 +144,7 @@ class WorkflowConverter:
             model_config=app_config.model,
             prompt_template=app_config.prompt_template,
             file_upload=app_config.additional_features.file_upload,
-            external_data_variable_node_mapping=external_data_variable_node_mapping
+            external_data_variable_node_mapping=external_data_variable_node_mapping,
         )
 
         graph = self._append_node(graph, llm_node)
@@ -199,7 +183,7 @@ class WorkflowConverter:
             tenant_id=app_model.tenant_id,
             app_id=app_model.id,
             type=WorkflowType.from_app_mode(new_app_mode).value,
-            version='draft',
+            version="draft",
             graph=json.dumps(graph),
             features=json.dumps(features),
             created_by=account_id,
@@ -212,24 +196,18 @@ class WorkflowConverter:
 
         return workflow
 
-    def _convert_to_app_config(self, app_model: App,
-                               app_model_config: AppModelConfig) -> EasyUIBasedAppConfig:
+    def _convert_to_app_config(self, app_model: App, app_model_config: AppModelConfig) -> EasyUIBasedAppConfig:
         app_mode = AppMode.value_of(app_model.mode)
         if app_mode == AppMode.AGENT_CHAT or app_model.is_agent:
             app_model.mode = AppMode.AGENT_CHAT.value
             app_config = AgentChatAppConfigManager.get_app_config(
-                app_model=app_model,
-                app_model_config=app_model_config
+                app_model=app_model, app_model_config=app_model_config
             )
         elif app_mode == AppMode.CHAT:
-            app_config = ChatAppConfigManager.get_app_config(
-                app_model=app_model,
-                app_model_config=app_model_config
-            )
+            app_config = ChatAppConfigManager.get_app_config(app_model=app_model, app_model_config=app_model_config)
         elif app_mode == AppMode.COMPLETION:
             app_config = CompletionAppConfigManager.get_app_config(
-                app_model=app_model,
-                app_model_config=app_model_config
+                app_model=app_model, app_model_config=app_model_config
             )
         else:
             raise ValueError("Invalid app mode")
@@ -248,14 +226,13 @@ class WorkflowConverter:
             "data": {
                 "title": "START",
                 "type": NodeType.START.value,
-                "variables": [jsonable_encoder(v) for v in variables]
-            }
+                "variables": [jsonable_encoder(v) for v in variables],
+            },
         }
 
-    def _convert_to_http_request_node(self, app_model: App,
-                                      variables: list[VariableEntity],
-                                      external_data_variables: list[ExternalDataVariableEntity]) \
-            -> tuple[list[dict], dict[str, str]]:
+    def _convert_to_http_request_node(
+        self, app_model: App, variables: list[VariableEntity], external_data_variables: list[ExternalDataVariableEntity]
+    ) -> tuple[list[dict], dict[str, str]]:
         """
         Convert API Based Extension to HTTP Request Node
         :param app_model: App instance
@@ -277,40 +254,33 @@ class WorkflowConverter:
 
             # get params from config
             api_based_extension_id = tool_config.get("api_based_extension_id")
+            if not api_based_extension_id:
+                continue
 
             # get api_based_extension
             api_based_extension = self._get_api_based_extension(
-                tenant_id=tenant_id,
-                api_based_extension_id=api_based_extension_id
+                tenant_id=tenant_id, api_based_extension_id=api_based_extension_id
             )
 
-            if not api_based_extension:
-                raise ValueError("[External data tool] API query failed, variable: {}, "
-                                 "error: api_based_extension_id is invalid"
-                                 .format(tool_variable))
-
             # decrypt api_key
-            api_key = encrypter.decrypt_token(
-                tenant_id=tenant_id,
-                token=api_based_extension.api_key
-            )
+            api_key = encrypter.decrypt_token(tenant_id=tenant_id, token=api_based_extension.api_key)
 
             inputs = {}
             for v in variables:
-                inputs[v.variable] = '{{#start.' + v.variable + '#}}'
+                inputs[v.variable] = "{{#start." + v.variable + "#}}"
 
             request_body = {
-                'point': APIBasedExtensionPoint.APP_EXTERNAL_DATA_TOOL_QUERY.value,
-                'params': {
-                    'app_id': app_model.id,
-                    'tool_variable': tool_variable,
-                    'inputs': inputs,
-                    'query': '{{#sys.query#}}' if app_model.mode == AppMode.CHAT.value else ''
-                }
+                "point": APIBasedExtensionPoint.APP_EXTERNAL_DATA_TOOL_QUERY.value,
+                "params": {
+                    "app_id": app_model.id,
+                    "tool_variable": tool_variable,
+                    "inputs": inputs,
+                    "query": "{{#sys.query#}}" if app_model.mode == AppMode.CHAT.value else "",
+                },
             }
 
             request_body_json = json.dumps(request_body)
-            request_body_json = request_body_json.replace(r'\{\{', '{{').replace(r'\}\}', '}}')
+            request_body_json = request_body_json.replace(r"\{\{", "{{").replace(r"\}\}", "}}")
 
             http_request_node = {
                 "id": f"http_request_{index}",
@@ -320,20 +290,11 @@ class WorkflowConverter:
                     "type": NodeType.HTTP_REQUEST.value,
                     "method": "post",
                     "url": api_based_extension.api_endpoint,
-                    "authorization": {
-                        "type": "api-key",
-                        "config": {
-                            "type": "bearer",
-                            "api_key": api_key
-                        }
-                    },
+                    "authorization": {"type": "api-key", "config": {"type": "bearer", "api_key": api_key}},
                     "headers": "",
                     "params": "",
-                    "body": {
-                        "type": "json",
-                        "data": request_body_json
-                    }
-                }
+                    "body": {"type": "json", "data": request_body_json},
+                },
             }
 
             nodes.append(http_request_node)
@@ -345,32 +306,24 @@ class WorkflowConverter:
                 "data": {
                     "title": f"Parse {api_based_extension.name} Response",
                     "type": NodeType.CODE.value,
-                    "variables": [{
-                        "variable": "response_json",
-                        "value_selector": [http_request_node['id'], "body"]
-                    }],
+                    "variables": [{"variable": "response_json", "value_selector": [http_request_node["id"], "body"]}],
                     "code_language": "python3",
                     "code": "import json\n\ndef main(response_json: str) -> str:\n    response_body = json.loads("
-                            "response_json)\n    return {\n        \"result\": response_body[\"result\"]\n    }",
-                    "outputs": {
-                        "result": {
-                            "type": "string"
-                        }
-                    }
-                }
+                    'response_json)\n    return {\n        "result": response_body["result"]\n    }',
+                    "outputs": {"result": {"type": "string"}},
+                },
             }
 
             nodes.append(code_node)
 
-            external_data_variable_node_mapping[external_data_variable.variable] = code_node['id']
+            external_data_variable_node_mapping[external_data_variable.variable] = code_node["id"]
             index += 1
 
         return nodes, external_data_variable_node_mapping
 
-    def _convert_to_knowledge_retrieval_node(self, new_app_mode: AppMode,
-                                             dataset_config: DatasetEntity,
-                                             model_config: ModelConfigEntity) \
-            -> Optional[dict]:
+    def _convert_to_knowledge_retrieval_node(
+        self, new_app_mode: AppMode, dataset_config: DatasetEntity, model_config: ModelConfigEntity
+    ) -> Optional[dict]:
         """
         Convert datasets to Knowledge Retrieval Node
         :param new_app_mode: new app mode
@@ -404,7 +357,7 @@ class WorkflowConverter:
                         "completion_params": {
                             **model_config.parameters,
                             "stop": model_config.stop,
-                        }
+                        },
                     }
                 }
                 if retrieve_config.retrieve_strategy == DatasetRetrieveConfigEntity.RetrieveStrategy.SINGLE
@@ -412,20 +365,23 @@ class WorkflowConverter:
                 "multiple_retrieval_config": {
                     "top_k": retrieve_config.top_k,
                     "score_threshold": retrieve_config.score_threshold,
-                    "reranking_model": retrieve_config.reranking_model
+                    "reranking_model": retrieve_config.reranking_model,
                 }
                 if retrieve_config.retrieve_strategy == DatasetRetrieveConfigEntity.RetrieveStrategy.MULTIPLE
                 else None,
-            }
+            },
         }
 
-    def _convert_to_llm_node(self, original_app_mode: AppMode,
-                             new_app_mode: AppMode,
-                             graph: dict,
-                             model_config: ModelConfigEntity,
-                             prompt_template: PromptTemplateEntity,
-                             file_upload: Optional[FileExtraConfig] = None,
-                             external_data_variable_node_mapping: dict[str, str] = None) -> dict:
+    def _convert_to_llm_node(
+        self,
+        original_app_mode: AppMode,
+        new_app_mode: AppMode,
+        graph: dict,
+        model_config: ModelConfigEntity,
+        prompt_template: PromptTemplateEntity,
+        file_upload: Optional[FileExtraConfig] = None,
+        external_data_variable_node_mapping: dict[str, str] | None = None,
+    ) -> dict:
         """
         Convert to LLM Node
         :param original_app_mode: original app mode
@@ -437,17 +393,18 @@ class WorkflowConverter:
         :param external_data_variable_node_mapping: external data variable node mapping
         """
         # fetch start and knowledge retrieval node
-        start_node = next(filter(lambda n: n['data']['type'] == NodeType.START.value, graph['nodes']))
-        knowledge_retrieval_node = next(filter(
-            lambda n: n['data']['type'] == NodeType.KNOWLEDGE_RETRIEVAL.value,
-            graph['nodes']
-        ), None)
+        start_node = next(filter(lambda n: n["data"]["type"] == NodeType.START.value, graph["nodes"]))
+        knowledge_retrieval_node = next(
+            filter(lambda n: n["data"]["type"] == NodeType.KNOWLEDGE_RETRIEVAL.value, graph["nodes"]), None
+        )
 
         role_prefix = None
 
         # Chat Model
         if model_config.mode == LLMMode.CHAT.value:
             if prompt_template.prompt_type == PromptTemplateEntity.PromptType.SIMPLE:
+                if not prompt_template.simple_prompt_template:
+                    raise ValueError("Simple prompt template is required")
                 # get prompt template
                 prompt_transform = SimplePromptTransform()
                 prompt_template_config = prompt_transform.get_prompt_template(
@@ -456,45 +413,35 @@ class WorkflowConverter:
                     model=model_config.model,
                     pre_prompt=prompt_template.simple_prompt_template,
                     has_context=knowledge_retrieval_node is not None,
-                    query_in_prompt=False
+                    query_in_prompt=False,
                 )
 
-                template = prompt_template_config['prompt_template'].template
+                template = prompt_template_config["prompt_template"].template
                 if not template:
                     prompts = []
                 else:
                     template = self._replace_template_variables(
-                        template,
-                        start_node['data']['variables'],
-                        external_data_variable_node_mapping
+                        template, start_node["data"]["variables"], external_data_variable_node_mapping
                     )
 
-                    prompts = [
-                        {
-                            "role": 'user',
-                            "text": template
-                        }
-                    ]
+                    prompts = [{"role": "user", "text": template}]
             else:
                 advanced_chat_prompt_template = prompt_template.advanced_chat_prompt_template
 
                 prompts = []
-                for m in advanced_chat_prompt_template.messages:
-                    if advanced_chat_prompt_template:
+                if advanced_chat_prompt_template:
+                    for m in advanced_chat_prompt_template.messages:
                         text = m.text
                         text = self._replace_template_variables(
-                            text,
-                            start_node['data']['variables'],
-                            external_data_variable_node_mapping
+                            text, start_node["data"]["variables"], external_data_variable_node_mapping
                         )
 
-                        prompts.append({
-                            "role": m.role.value,
-                            "text": text
-                        })
+                        prompts.append({"role": m.role.value, "text": text})
         # Completion Model
         else:
             if prompt_template.prompt_type == PromptTemplateEntity.PromptType.SIMPLE:
+                if not prompt_template.simple_prompt_template:
+                    raise ValueError("Simple prompt template is required")
                 # get prompt template
                 prompt_transform = SimplePromptTransform()
                 prompt_template_config = prompt_transform.get_prompt_template(
@@ -503,57 +450,50 @@ class WorkflowConverter:
                     model=model_config.model,
                     pre_prompt=prompt_template.simple_prompt_template,
                     has_context=knowledge_retrieval_node is not None,
-                    query_in_prompt=False
+                    query_in_prompt=False,
                 )
 
-                template = prompt_template_config['prompt_template'].template
+                template = prompt_template_config["prompt_template"].template
                 template = self._replace_template_variables(
-                    template,
-                    start_node['data']['variables'],
-                    external_data_variable_node_mapping
+                    template=template,
+                    variables=start_node["data"]["variables"],
+                    external_data_variable_node_mapping=external_data_variable_node_mapping,
                 )
 
-                prompts = {
-                    "text": template
-                }
+                prompts = {"text": template}
 
-                prompt_rules = prompt_template_config['prompt_rules']
+                prompt_rules = prompt_template_config["prompt_rules"]
                 role_prefix = {
-                    "user": prompt_rules.get('human_prefix', 'Human'),
-                    "assistant": prompt_rules.get('assistant_prefix', 'Assistant')
+                    "user": prompt_rules.get("human_prefix", "Human"),
+                    "assistant": prompt_rules.get("assistant_prefix", "Assistant"),
                 }
             else:
                 advanced_completion_prompt_template = prompt_template.advanced_completion_prompt_template
                 if advanced_completion_prompt_template:
                     text = advanced_completion_prompt_template.prompt
                     text = self._replace_template_variables(
-                        text,
-                        start_node['data']['variables'],
-                        external_data_variable_node_mapping
+                        template=text,
+                        variables=start_node["data"]["variables"],
+                        external_data_variable_node_mapping=external_data_variable_node_mapping,
                     )
                 else:
                     text = ""
 
-                text = text.replace('{{#query#}}', '{{#sys.query#}}')
+                text = text.replace("{{#query#}}", "{{#sys.query#}}")
 
                 prompts = {
                     "text": text,
                 }
 
-                if advanced_completion_prompt_template.role_prefix:
+                if advanced_completion_prompt_template and advanced_completion_prompt_template.role_prefix:
                     role_prefix = {
                         "user": advanced_completion_prompt_template.role_prefix.user,
-                        "assistant": advanced_completion_prompt_template.role_prefix.assistant
+                        "assistant": advanced_completion_prompt_template.role_prefix.assistant,
                     }
 
         memory = None
         if new_app_mode == AppMode.ADVANCED_CHAT:
-            memory = {
-                "role_prefix": role_prefix,
-                "window": {
-                    "enabled": False
-                }
-            }
+            memory = {"role_prefix": role_prefix, "window": {"enabled": False}}
 
         completion_params = model_config.parameters
         completion_params.update({"stop": model_config.stop})
@@ -567,28 +507,29 @@ class WorkflowConverter:
                     "provider": model_config.provider,
                     "name": model_config.model,
                     "mode": model_config.mode,
-                    "completion_params": completion_params
+                    "completion_params": completion_params,
                 },
                 "prompt_template": prompts,
                 "memory": memory,
                 "context": {
                     "enabled": knowledge_retrieval_node is not None,
                     "variable_selector": ["knowledge_retrieval", "result"]
-                    if knowledge_retrieval_node is not None else None
+                    if knowledge_retrieval_node is not None
+                    else None,
                 },
                 "vision": {
                     "enabled": file_upload is not None,
                     "variable_selector": ["sys", "files"] if file_upload is not None else None,
-                    "configs": {
-                        "detail": file_upload.image_config['detail']
-                    } if file_upload is not None else None
-                }
-            }
+                    "configs": {"detail": file_upload.image_config["detail"]}
+                    if file_upload is not None and file_upload.image_config is not None
+                    else None,
+                },
+            },
         }
 
-    def _replace_template_variables(self, template: str,
-                                    variables: list[dict],
-                                    external_data_variable_node_mapping: dict[str, str] = None) -> str:
+    def _replace_template_variables(
+        self, template: str, variables: list[dict], external_data_variable_node_mapping: dict[str, str] | None = None
+    ) -> str:
         """
         Replace Template Variables
         :param template: template
@@ -597,12 +538,11 @@ class WorkflowConverter:
         :return:
         """
         for v in variables:
-            template = template.replace('{{' + v['variable'] + '}}', '{{#start.' + v['variable'] + '#}}')
+            template = template.replace("{{" + v["variable"] + "}}", "{{#start." + v["variable"] + "#}}")
 
         if external_data_variable_node_mapping:
             for variable, code_node_id in external_data_variable_node_mapping.items():
-                template = template.replace('{{' + variable + '}}',
-                                            '{{#' + code_node_id + '.result#}}')
+                template = template.replace("{{" + variable + "}}", "{{#" + code_node_id + ".result#}}")
 
         return template
 
@@ -618,11 +558,8 @@ class WorkflowConverter:
             "data": {
                 "title": "END",
                 "type": NodeType.END.value,
-                "outputs": [{
-                    "variable": "result",
-                    "value_selector": ["llm", "text"]
-                }]
-            }
+                "outputs": [{"variable": "result", "value_selector": ["llm", "text"]}],
+            },
         }
 
     def _convert_to_answer_node(self) -> dict:
@@ -634,11 +571,7 @@ class WorkflowConverter:
         return {
             "id": "answer",
             "position": None,
-            "data": {
-                "title": "ANSWER",
-                "type": NodeType.ANSWER.value,
-                "answer": "{{#llm.text#}}"
-            }
+            "data": {"title": "ANSWER", "type": NodeType.ANSWER.value, "answer": "{{#llm.text#}}"},
         }
 
     def _create_edge(self, source: str, target: str) -> dict:
@@ -648,11 +581,7 @@ class WorkflowConverter:
         :param target: target node id
         :return:
         """
-        return {
-            "id": f"{source}-{target}",
-            "source": source,
-            "target": target
-        }
+        return {"id": f"{source}-{target}", "source": source, "target": target}
 
     def _append_node(self, graph: dict, node: dict) -> dict:
         """
@@ -662,9 +591,9 @@ class WorkflowConverter:
         :param node: Node to append
         :return:
         """
-        previous_node = graph['nodes'][-1]
-        graph['nodes'].append(node)
-        graph['edges'].append(self._create_edge(previous_node['id'], node['id']))
+        previous_node = graph["nodes"][-1]
+        graph["nodes"].append(node)
+        graph["edges"].append(self._create_edge(previous_node["id"], node["id"]))
         return graph
 
     def _get_new_app_mode(self, app_model: App) -> AppMode:
@@ -678,14 +607,20 @@ class WorkflowConverter:
         else:
             return AppMode.ADVANCED_CHAT
 
-    def _get_api_based_extension(self, tenant_id: str, api_based_extension_id: str) -> APIBasedExtension:
+    def _get_api_based_extension(self, tenant_id: str, api_based_extension_id: str):
         """
         Get API Based Extension
         :param tenant_id: tenant id
         :param api_based_extension_id: api based extension id
         :return:
         """
-        return db.session.query(APIBasedExtension).filter(
-            APIBasedExtension.tenant_id == tenant_id,
-            APIBasedExtension.id == api_based_extension_id
-        ).first()
+        api_based_extension = (
+            db.session.query(APIBasedExtension)
+            .filter(APIBasedExtension.tenant_id == tenant_id, APIBasedExtension.id == api_based_extension_id)
+            .first()
+        )
+
+        if not api_based_extension:
+            raise ValueError(f"API Based Extension not found, id: {api_based_extension_id}")
+
+        return api_based_extension

api/tests/integration_tests/model_runtime/wenxin/test_embedding.py

@@ -5,7 +5,7 @@ from core.model_runtime.entities.text_embedding_entities import TextEmbeddingRes
 from core.model_runtime.model_providers.wenxin.text_embedding.text_embedding import WenxinTextEmbeddingModel
 
 
-def test_invoke_embedding_model():
+def test_invoke_embedding_v1():
     sleep(3)
     model = WenxinTextEmbeddingModel()
 
@@ -21,4 +21,61 @@ def test_invoke_embedding_model():
 
     assert isinstance(response, TextEmbeddingResult)
     assert len(response.embeddings) == 3
-    assert isinstance(response.embeddings[0], list)
+    assert isinstance(response.embeddings[0], list)
+
+
+def test_invoke_embedding_bge_large_en():
+    sleep(3)
+    model = WenxinTextEmbeddingModel()
+
+    response = model.invoke(
+        model='bge-large-en',
+        credentials={
+            'api_key': os.environ.get('WENXIN_API_KEY'),
+            'secret_key': os.environ.get('WENXIN_SECRET_KEY')
+        },
+        texts=['hello', '你好', 'xxxxx'],
+        user="abc-123"
+    )
+
+    assert isinstance(response, TextEmbeddingResult)
+    assert len(response.embeddings) == 3
+    assert isinstance(response.embeddings[0], list)
+
+
+def test_invoke_embedding_bge_large_zh():
+    sleep(3)
+    model = WenxinTextEmbeddingModel()
+
+    response = model.invoke(
+        model='bge-large-zh',
+        credentials={
+            'api_key': os.environ.get('WENXIN_API_KEY'),
+            'secret_key': os.environ.get('WENXIN_SECRET_KEY')
+        },
+        texts=['hello', '你好', 'xxxxx'],
+        user="abc-123"
+    )
+
+    assert isinstance(response, TextEmbeddingResult)
+    assert len(response.embeddings) == 3
+    assert isinstance(response.embeddings[0], list)
+
+
+def test_invoke_embedding_tao_8k():
+    sleep(3)
+    model = WenxinTextEmbeddingModel()
+
+    response = model.invoke(
+        model='tao-8k',
+        credentials={
+            'api_key': os.environ.get('WENXIN_API_KEY'),
+            'secret_key': os.environ.get('WENXIN_SECRET_KEY')
+        },
+        texts=['hello', '你好', 'xxxxx'],
+        user="abc-123"
+    )
+
+    assert isinstance(response, TextEmbeddingResult)
+    assert len(response.embeddings) == 3
+    assert isinstance(response.embeddings[0], list)