Update | Description | Documentation |
Data Integration | Addition: [Real-time] Full database capability aligned with single table: Supports searching tasks by table name. Upon submission, checks whether a synchronization task for the same table already exists to avoid duplicate access and data conflicts. [Offline] Doris data source supports more partition configurations: When writing to Doris, you can select all partitions, specified partitions, or overwrite by partition range. Optimization: [Real-time] Startup version consistent with running version: Real-time synchronization tasks keep the startup version consistent with the running version to avoid exceptions caused by version drift. [Real-time] Optimized display of read/write quantity statistics: Optimized the statistics of read/write quantities in "Link Details" under statistical metrics. [Offline] Strict mode for offline task scripts: When this mode is enabled, the connection string, database name, and username configured in the data source override the corresponding settings in the script, adapting to general migration scenarios. [Offline] Atomic Doris partition overwrite: Data is first written to a temporary partition and then atomically replaced with the formal partition, avoiding data incompleteness during partition overwrite. [Offline] Error message and guidance optimization: Optimize error messages and supplement frontend configuration guidance and instructions. Performance and stability: [Real-time] Fix for mapping match preview performance issue: Resolves the issue where the preview loads for a long time with a spinner or displays a prompt. [Real-time] Service stability and metadata performance improvement: Improves the stability of real-time integration services and the overall performance of metadata services. | |
Studio/SQL Exploration | Addition: Notebook cells support sequence numbers, remarks, and automatic directory generation: Improves the readability and navigation efficiency of long Notebooks. Studio and Git configuration feature modules are fully open: No allowlist application is required for trial use. Optimization: Notebook environment management supports setting a default resource group: Reduces the operational cost of manually selecting a resource group for each run. Notebook + EMR usage path optimization: Supports automatically allowing network security group configurations and automatically enabling CPU resource access, without manual operations. | |
Workflow Orchestration/Scheduling | Addition: [Orchestration] Test run supports viewing history records and logs: Facilitates tracing back the debugging process. [Orchestration] WeData dbt node support: Adds the dbt node type and incorporates it into unified scheduling. [Orchestration] Workflow permissions support the run permission: Permission granularity is refined, and only execution permission can be granted. [Scheduling] Minute-level scheduling granularity enhanced: Standard minute-level scheduling tasks support scheduling offset (for example, starting from minute 02 and running every 5 minutes: 02 / 07 / 12 ...). [Orchestration] Supports moving folders in workflow orchestration. Optimization: [Scheduling] International site run log time supports the default scheduling time zone in project management: Run log time can be displayed according to the default scheduling time zone configured in project management, making it easier for cross-time-zone teams to troubleshoot issues. [Scheduling] Increase the foreach loop quantity limit to 1000: Remove the foreach node loop quantity restriction to support larger batch loop scenarios. [Scheduling] Scheduling resource utilization monitoring granularity reaches 1 minute: Resource utilization monitoring precision is improved to the minute level, facilitating refined capacity observation. [Scheduling] WeData supports integration with Tencent Cloud EMR 3.7. [Orchestration] Spark-jar tasks support resource package snapshot versions: In workflow scheduling mode, you can specify a resource package snapshot to ensure version traceability and reproducibility. | |
Data Science | Addition: Model service supports restricting access to private network / VPC: Access can be limited to only the private network or specified VPCs, enhancing access security. Optimization: Optimized detail display after physical feature table deletion: Optimize the detail page display and prompts. Fix the issue where aliases cannot be set for registered MLflow models: Resolve the issue where registered models cannot set aliases due to incompatibility between jupyter-server kernel injection Tag validation and TCLake catalog model tag storage. Feature management client permission optimization: The client no longer uses secretId / key and is bound to the current user's permissions by default, covering DLC and EMR. This resolves the temporary key issue where Studio runs normally but workflow scheduling reports errors. | |
Data Governance | Addition: Metadata collection supports TCHouse SR: Added metadata collection capability for the TCHouse SR data source type. Data lineage: Supports batch query of INPUT / OUTPUT field-level lineage for tables. Quality alarm notifications support custom emails: Supports custom email content and recipient configuration. Supports delivering security audit logs to CLS: Facilitates long-term retention by customers and analysis through Open API. Data service resource group version supports silent upgrade: Supports silent upgrade, reducing manual intervention. Optimization: Asset transfer supports displaying transfer details: Details can be viewed during asset transfer, and the transfer process is traceable. | |
Basic Platform | Addition: Supports alarms for the new version of Teams groups: The alarm notification channel is upgraded to adapt to the new Teams group bot, ensuring the alarm delivery link remains available. Tag feature optimization: Supports tag parameters in resource group creation and query APIs, adds an OpenAPI for modifying tags, and fixes the issue of tags disappearing after resource group operations. Optimization: Approval process optimization: Supports configuring whether approvers can approve their own submitted requests and whether to auto-approve, and release packages support disabling the approval process. Fixes the issue of tags disappearing after resource group configuration adjustment: Resolves the issue of existing tags disappearing after WeData resource groups are scaled in or out. | - |
Update | Description | Documentation |
Data Integration | Addition: TRUNCATE TABLE Synchronization: Real-time synchronization supports synchronizing TRUNCATE TABLE operations from the source end. DDL Encrypted Field Synchronization: This feature supports synchronizing fields encrypted with projection, such as phone numbers, during DDL changes. Multi-Source End Alarm Metric Unification: This feature aligns business latency and transmission latency metrics. Table Creation Logic Unification: The logic for creating tables before and during runtime has been unified to avoid inconsistencies across the two paths. Non-blocking Table Addition During Runtime: Adding tables to a running real-time synchronization job no longer blocks the synchronization performance of existing tables. Iceberg Primary Key Deduplication: This feature supports primary key deduplication during offline integration when data is written to Iceberg tables using the upsert operation. Redis JSON Write: This feature supports JSON format output during offline integration when data is written to Redis. MySQL session Parameter Customization: User customization is supported for session-level parameters in offline integration tasks, such as net_write_timeout=600. TCHouse-P Data Reconciliation Disabled: In the offline data reconciliation scenario, the TCHouse-P option is hidden/disabled on the page to prevent misuse. EMR Kerberos Information Return: The EMR data source supports returning Kerberos authentication information. JDBC Source Parameter Passthrough: JDBC data sources support parameter passthrough. Engine Source Parameter Completion: Engine data sources support additional parameter configuration. Advanced Parameter Self-Service: Instance-class parameters can be self-configured using advanced parameters. Optimization: TCHouse-D Write Core Pain Point Resolution: End-to-end optimization has been performed to address core pain points on the TCHouse-D write side, such as high concurrency, timeouts, and connection reuse. MySQL Stability Optimization: The overall stability of the real-time synchronization from the MySQL source end has been enhanced, covering aspects such as connection loss retry and binlog parsing fault tolerance. Database Sharding Availability Improvement: This feature provides overall availability optimization for real-time database sharding scenarios. Kafka Empty String Handling: The logic for handling empty strings in the Kafka Debezium serialization format during real-time synchronization has been optimized. Hive Data Source Parameter Optimization: The configuration rules and prompts for Hive data source parameters in JDBC mode have been optimized. Both methods in HDFS mode are supported. Resource Group Scale-in Validation: A pre-validation step is added during real-time resource group scale-in to prevent task exceptions caused by the scale-in operation. Incremental-Only Read Resource Configuration: In incremental-only read mode, integrated resource configuration is no longer allocated by synchronization stage, simplifying the resource profile. Table Name Swap DDL: DDL takes effect immediately after a table name swap through real-time synchronization. | - |
Data Development | Addition: Data Integration Automatic Lineage Parsing: Data Integration tasks support automatic parsing of output table registration and upstream/downstream dependencies, eliminating the need for manual lineage maintenance. Code Search Coverage for Data Integration: The code search capability has been extended to support Data Integration tasks. Timeout Without Successful Alarm: This feature supports configuring a time limit for periodic tasks. When an alarm is not triggered successfully after the time limit is exceeded, it is applicable to tasks of all scheduling cycle types. Alarm on Skipping Upstream Tasks: When upstream task execution is skipped, you can configure the system to send an alarm notification to the relevant owner. Ops Center AI Diagnostic Feature for Failed Tasks: The Ops center supports AI diagnosis for the last failed task execution, enabling efficient problem identification and resolution. Data Integration Task Supports Output Registration and Upstream/Downstream Dependencies: Data Integration tasks support automatic parsing of output registration and upstream/downstream dependencies. For-each Node Loop Parameter Enhancement: In workflow scheduling mode, the For-each node loop parameters support defining array constants. Global Spark and Kyuubi Parameter Configuration: This feature supports the configuration of global Spark and Kyuubi parameters. Missing Instance Risk Detection: When a task is submitted, the system detects and alerts for potential risks of missing instances caused by scheduling changes, and supports handling operations (set to success, backfill). Data Integration Time Parameter Configuration: The system content time parameters can be configured in the Data Integration task form fields. OpenAPI Enhancement: An API for querying task execution status by task name/ID + scheduled time has been added. CI/CD Forced deploy: The CI/CD pipeline supports forced deployment. CI/CD Adaptation for Studio: The CI/CD adaptation for Studio includes a submission feature that automatically performs Studio file submission after a deploy operation is executed. DLC Task Import/Export Supports Engine and Resource Group Mapping: During the import or export of DLC-type tasks, the selection of engine and resource group mappings is supported. DLC Task Cross-Project Cloning Supports Engine and Resource Group Mapping: During cross-project cloning of DLC-type tasks, the selection of engine and resource group mappings is supported. Workflow Scheduling Supports Concurrency Configuration: The workflow scheduling mode supports configuring the concurrency number for workflow execution and the queuing policy. Only a specified number of workflows are allowed to run concurrently at the same time. Workflow runs that exceed the concurrency limit can be queued or set to wait. Notebook Task Supports Custom Spark Parameters: In workflow scheduling mode, when a Notebook task uses the DLC engine, users can customize Spark parameters (such as the number of executors, memory, and so on) to meet the resource configuration requirements of different data processing scenarios. Orchestration Space Task Parameter Preview: In workflow scheduling mode, tasks in the orchestration space support a parameter preview feature. Users can view the actual parameter values of a task before submitting it for execution, facilitating confirmation and troubleshooting. Workflow Scheduling Supports Task-Level Success Setting: In workflow scheduling mode, the system supports performing a "set to success" operation on individual tasks. This allows Ops personnel to manually mark a task as successful when it encounters an exception, enabling downstream tasks to continue execution. Workflow Scheduling Supports Data Quality Nodes and CI/CD: In workflow scheduling mode, support for data quality nodes has been added, and the CI/CD release process for data quality tasks is also supported. Workflow Scheduling Supplement for Remaining Task Types: This feature completes the coverage of task types in workflow scheduling mode, ensuring that all task types can be orchestrated and executed via the workflow scheduling mode. Optimization: Batch Upload: The orchestration space resource management supports batch upload. Workflow Batch Filtering: This feature supports batch operations for filtering folders within workflows. Cross-Cycle Backfill Sorting: For backfill data with cross-cycle dependencies (such as day tasks depending on hour tasks), the order is no longer disrupted when partial hour periods are selected. Workflow Scheduling Mode: For-Each Loop Parameter Set to Success When Empty: This fix resolves the issue where tasks were not correctly set to a successful state when the For-Each node loop parameters were empty. System Built-in Time Parameter Fix: This fix resolves the issue where system built-in time parameters (including time zone parameters) configured in tasks or workflows did not take effect correctly in workflow scheduling mode. Workflow Scheduling Mode: Concurrency Queuing Executes in FIFO Order: This optimizes the workflow scheduling concurrency execution mechanism. When the concurrency limit is reached, queued tasks are executed strictly in the order of submission (First In, First Out) to prevent out-of-order issues. | - |
Data Analysis (Studio/SQL Exploration) | Addition: Multiple GitLab Repository Binding: A project supports binding to multiple GitLab repositories. Code from different business modules within the same project can be hosted independently. Notebook Kernel and Environment Management: Notebook tasks support kernel and runtime environment management. Notebook Automatic Advanced Parameter Import: When a Notebook file is uploaded, the system automatically recognizes and imports advanced parameters and their values. File Permission Granularity: Project file access control now includes three new permission types: edit / run / view, replacing the previous coarse-grained permission model. Optimization: SQL Exploration Supports Automatic Addition of LIMIT 1000: Script execution supports the option to obtain the full dataset or to obtain the first 1000 rows of data. Data Query Result Table: Supports formatted display of JSON data. Supports adaptive column width for data results. When cell field content is too long, the system supports collapsing and expanding the display. The system displays the data types of result columns and supports sorting and filtering based on these data types. Supports the search feature for data results. Upon completion of a data query, the system supports sending message notifications via the browser. ESC Disables Association: The system supports using the ESC key to disable metadata association. SQL Exploration Log Optimization: The SQL exploration log information has been optimized. SQL Exploration Fields: The query display exception for SQL exploration fields containing a period (.) has been fixed. SQL Exploration Scheduling Resource Group and Shared Resource Group Feature Alignment: This update fixes the functional discrepancy between scheduling resource groups and shared resource groups in SQL exploration, ensuring consistent behavior of both resource groups during task execution. | - |
Data Governance | Addition: Approval Configuration Metadata: Multi-level approval workflows are supported for medium- and high-sensitivity data. Permissions for database tables support batch authorization by role. Metric Authorization Support: The system supports Owners in authorizing metrics to specified users. Authorized users can then query the metric calculation results. Data Lineage: The system supports Kafka lineage reporting and optimizations for Spark SQL and Hive SQL lineage parsing. Governance Center: The asset catalog has been added to the governance factors. Issues pending governance support export. Data Access Logs: The system now supports log collection and auditing for data access logs from the TCHouse-D and EMR engines. Metadata: The system supports real-time collection of DLC metadata. Catalog: The permission verification prompt messages in Catalog have been optimized. Data Service: The service supports the TCHouse-D data source. The Frankfurt site of the International Station supports the Data Service. | - |
Update | Description | Documentation |
Data Development | Addition: Adding a "No Downstream Dependencies" Item to Task Scheduling Configuration: You can set "No Downstream Dependencies" in the task configuration. Once enabled, this setting blocks downstream tasks from configuring dependencies on this task. It is suitable for scenarios such as temporary or test tasks that you do not want to be referenced by downstream tasks. Adding Approval Records and Task-Specific Approver Support to the Approval Process: Approvers can view historical approval records at the time of submission. The system also supports enabling the "Task-Specific Approver" mode, where the developer designates a specific approver when submitting a task, which improves the efficiency of the approval workflow. Environment Isolation for Integration/Scheduling Resource Groups in Standard Mode: In Standard Mode, you can configure environment isolation for integration resource groups and scheduling resource groups, separating them into development and production environments. Adding the "Last Periodic Run End Time" of Upstream Tasks to Scheduling Configuration: The "Last Periodic Run End Time" field is added to the upstream dependency task list. This helps users reasonably define the planned scheduling time of the current task based on the completion time of upstream tasks. Adding a Default Value Setting for the Target Scheduling Resource Group to the Clone Feature: This feature supports predefining the mapping relationship between the source end and target end scheduling resource groups. During cloning, the mapping is selected by default, reducing the likelihood of users selecting the wrong resource group. Optimization for Handling Duplicate Script Names in Clone Tasks: When a script with the same name exists on the target end, it is automatically renamed to avoid conflicts caused by a single script being referenced by multiple tasks. Adding an Approval Process to Clone Releases: An approval process is added to clone releases. It supports initiating an approval after a simulated clone. The clone is formally released to the production environment only after the approval is passed, enhancing the security and control capabilities of clone releases. DLC Task Backfill Supports Re-specifying Engine Resource Groups: DLC tasks support re-specifying resource groups during data backfill execution. This distinguishes resource allocation between normal scheduling and backfill tasks, preventing backfill operations from affecting normally scheduled tasks. Comprehensive Optimization of the Intelligent Baseline Feature: This feature supports viewing the DAG link of guarantee tasks on the baseline details page and instance details page. It also supports configuring baselines for hourly tasks. Baselines can be configured and alarms can be enabled even when there is no historical execution record. Additionally, baseline event monitoring and an event details page have been added. Optimization: Project-Level Spark Parameters Take Effect in Debug Runs: This fix resolves the issue where project-level configured Spark parameters did not take effect in orchestration space test runs. It ensures that Spark SQL, PySpark, and Spark tasks can correctly recognize project-level parameters during both debug runs and periodic runs. | - |
Data Integration | Addition: Support for Synchronizing Local Files to EMR Hive and DLC: This feature supports uploading and parsing local files such as txt/excel/csv, and then synchronizing them to EMR Hive tables (system source) and DLC tables (supporting both system source and custom source). Adding System Metadata Field References to Batch Synchronization: Batch synchronization supports synchronizing the source end table name as a system field to the downstream table. This facilitates tracing data sources in multi-source aggregation scenarios. Adding the TDSQL-B Data Source: This feature supports the integration of the TDSQL-B data source. It also supports managing real-time synchronization and batch synchronization tasks for this data source. Optimization: Optimizing the Resource Group Selection Policy for DLC Offline Tasks: The resource group options for DLC offline tasks are now limited to only the "SQL Analysis" type. This reduces repeated modifications caused by user misoperations. Integrated Concurrency Reading Supports Sharded Clusters: The Data Integration multi-concurrency reading feature supports sharded clusters. This resolves the issue of being unable to read from the data source in MongoDB sharded collection scenarios. Data Source Compatibility with Hive 2.1: This feature supports creating data sources compatible with Hive 2.1, meeting the access requirements of customers using older versions of the Hive engine. | - |
Data Governance | Addition: Quality Tasks Support Batch Adjustment of Engines and Resource Groups: This feature supports performing batch operations on multiple quality tasks to uniformly adjust their execution engines and resource group configurations. Batch Termination of Quality Tasks: This feature supports performing batch termination operations on running or pending quality tasks, facilitating rapid loss containment in abnormal scenarios. Metadata Support for Kafka: Metadata collection now supports the Kafka data source. This allows you to create streaming data class metadata collection tasks. Optimization: Running Quality Task Instances Support Log Viewing: Execution logs for running quality task instances can be viewed in real time. This facilitates timely troubleshooting during task execution. Optimizing DLC Partition Information: This update adds a display for the upper limit of partition record counts and optimizes the prompt information for partition creation time. These enhancements improve the information completeness on the data assets details page. Others: Asset Scoring Feature Deprecated: The asset scoring feature has been deprecated on the table details page. | - |
Update | Description | Documentation |
Data Development | Addition: Studio Data Development IDE: The newly launched Studio Data Development IDE delivers a unified development experience for SQL and Notebook. It supports Notebook development, multi-syntax switching, and cell-based SQL execution. It also enables scheduling and running Notebook tasks in the orchestration space. Additionally, it provides capabilities for debugging SQL, code reuse, Git integration, and file versioning. Orchestration Space Adds DLC Spark Real-time Computing Capability: This update introduces the DLC Spark Streaming task type, providing a unified management capability for real-time task development, orchestration, and monitoring. Orchestration Space Adds Data Quality Node: This feature supports data quality task orchestration and Ops. Orchestration Space Workflow Adds Overall Scheduling Capability: The scheduling mode has been upgraded in two ways. It now supports both task-level and workflow-level scheduling to meet scheduling management requirements at different granularities. New WeData Bundle Engineering Delivery Capability: The CLI supports command-line operations and automated integration. The Bundle describes the development resources of workflows and tasks as source files to integrate into enterprise software engineering systems. It can be combined with CI/CD tools like GitLab Pipeline to achieve automated cross-environment release migration. Additionally, it supports online preview and quick generation of workflow and task YAML files, enhancing configuration efficiency and maintainability. Optimization: Ops Center Manual Tasks Support Monitoring and Alarms: The rules support configuring alarm rules for manual tasks, enabling monitoring and alarms for them. Ops Center Instance Details Support Displaying Quality Task Details: The instance details page supports displaying the overall operation detection status and the operation detection list for quality tasks. | |
Basic Platform | Addition: Time Zone Adaptation for Basic Modules: The system uniformly uses UTC as the base time. Timestamps and event records are stored in UTC time with time zone offsets attached. They are automatically converted and displayed as local time based on the user's time zone. Displaying Data Source Associated Tasks: Data source management supports displaying the associated tasks that use the data source. Data Permissions New Capabilities: New capabilities have been added for querying data source authorization results and revoking data source authorizations. | - |
Data Integration | Addition: Real-time Integration Link Adds Data Reconciliation Feature: This adds a data reconciliation feature. It monitors data discrepancies between source and target tables to promptly identify data consistency issues. Real-time Integration Full Database Migration Capability Extended: The full-database feature extends single-table configuration capabilities. Full-database tasks support viewing the mapping relationships between source and target tables. You can configure field mapping, data filtering, and other settings at the single-table granularity. Full-database tasks have been changed from a fixed-link to a star-link architecture. Additionally, 30+ synchronization links have been added. Real-time Integration Full Database/Database Sharding Tasks Support Transformation Functions: During data synchronization, it supports lightweight ETL transformation processing. Existing data can be transformed and then synchronized to the target table. Optimization: Real-time Integration Alarm Capability Optimized: This adds a "Data Write Count" monitoring and alarm metric. It provides alarm subscription support for core critical events, optimizes the logic of the cumulative restart count alarm metric, and adds default alarm rules. Real-time Integration Full Database Tasks Support Consuming Multiple Kafka Topics: Full database tasks support consuming multiple Kafka Topics simultaneously and specifying the databases and tables to be synchronized under each Topic. | |
Data Science | Addition: You can jump to view data, feature, and model lineage. Features, models, and experiments support user-level permission authorization based on entities. Supports model service quality monitoring and triggering retraining. Feature management has been fully upgraded to support DLC and EMR as offline features and Redis as online features, as well as offline-online feature synchronization. Online inference supports custom images and load balancing. Newly supports GPU-based deep learning training, experiment management records, deep learning model management, and deep learning model service deployment. Newly supports no-code AutoML experiments for classification, regression, and time series forecasting scenarios. | |
Data Governance | Addition: Metrics Modeling Feature Extended: It supports metric acceleration, the creation of metric views, and the creation of acceleration tasks for metric views. During metric queries, it can automatically route to the accelerated result set. Dimensions are decoupled from models. After a dimension is created based on table fields, it can be reused across multiple models without redefinition, improving dimension management efficiency. For metric modeling, MCP/JDBC and Restful API metric services are exposed. The metric modeling also exposes MCP services, enabling agents to accurately obtain metric metadata and metric calculation results. JDBC services are provided to enable direct integration between the metric platform and BI tools for obtaining metric results and configuring visual reports. RESTful API interfaces are also supported, allowing other business systems to directly obtain metric calculation results by specifying metrics, dimensions, and other information. Catalog Features Added: For scenarios where DLC is integrated and TCLake is enabled, a Catalog module has been added to display unified metadata from TCLake. The unified Catalog is presented in a three-tier structure: Catalog-Schema-Table/View/Model/Volume/Function. It centrally manages structured data, models, and unstructured data within the Catalog, displaying metadata basic information, table lineage, model lineage, change history, data quality, access logs, usage instructions, and more. It also supports unified access control. Inference table quality monitoring has been added. It supports configuring a table Dashboard under the default Catalog and viewing key metrics for inference tasks. Enrich the display information on the Catalog table details page to view metadata. It supports displaying the last change information of a table, which is used to view and analyze metadata change operations on the table. Table Details Feature Added: Usage instructions have been added to the Asset Details page. Usage instructions have also been added at the field level. Snapshot/time-series table quality monitoring has been added. It supports configuring a Dashboard in the EMR Hive table details and viewing key metrics for inference/snapshot/time-series tasks. Enrich the display information on the Catalog table details page to view metadata. It supports displaying the last change information of a table, which is used to view and analyze metadata change operations on the table. Optimization: Table Details Feature Optimized: It supports configuring change monitoring notifications and allows you to configure notification information for common changes. The DLC table details page is now available for viewing DLC metadata. | |
Basic Platform | Optimization: Refactored OpenAPI of WeData core links: Refactored the main APIs of WeData platform foundation, data development, task Ops, and data assets. | - |
Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback