This guide is intended for first-time DataBuddy users and covers the complete workflow from account registration, product activation, and environment configuration to data ingestion, data engineering, data governance, and data analysis, helping you get started quickly and fully leverage the platform's capabilities.
1. Prerequisites
1.1 Account Registration and Identity Verification
Before using DataBuddy, you need to complete the following preparations:
2. Complete enterprise identity verification: A Tencent Cloud root account must complete enterprise identity verification before DataBuddy can be activated and used. Note that DataBuddy activation is currently not supported for accounts with individual identity verification. For detailed steps, see the Tencent Cloud Verification Guide. 1.2 Account Recharge
Ensure that your Tencent Cloud account has sufficient available balance (a minimum of CNY 100 is recommended). When you create compute resources, a certain amount is frozen, and the frozen amount depends on the compute resource specifications (for example, 16CU requires approximately CNY 5.6 to be frozen). This amount is only frozen and will not be actually deducted.
1.3 Free Trial Resource Description
DataBuddy offers a 30-day free trial to new users, during which the following free resources are available:
|
Compute resources (CU) | 600 CU-hours | Covers job clusters, interactive clusters, platform clusters, and real-time clusters. All clusters share the total quota. |
Vouchers | CNY 100 | Used to offset storage fees. Services remain available after the voucher is used up, and billing automatically switches to pay-as-you-go. |
Agent seats | 2 | Seats for using the DataBuddy Agent feature |
Agent points | 5000 points | Used for Agent intelligent Q&A and analysis. After the points are used up, only Agent becomes unavailable. |
Attention:
The trial will stop when it expires or when the 600 CU-hours compute resources are exhausted. The exhaustion of vouchers and Agent credits will not trigger service suspension. To continue using the service after the trial ends, upgrade to the official version.
2. Activating Product Trial
2.1 Activating Product Trial with a Root Account
The operation steps are as follows:
1. Access the DataBuddy product page: Search for "DataBuddy" on the Tencent Cloud official website to open the product homepage, or directly visit DataBuddy. 2. After you enter DataBuddy, a trial activation dialog box will appear. Click "OK" to enter the trial activation process.
The system will automatically complete the following initialization operations:
Activate the DataBuddy trial and allocate free resource quotas.
Create a workspace with the configured name and region.
Enable Multimodal Data Lake in the region corresponding to the workspace.
Create default compute resources in the workspace.
3. After the product is successfully activated, the system will automatically redirect to the Workspace page, where you can start using the product.
2.2 Activating Product Trial with a Sub-Account
If you have already completed activation using the root account, you can skip this section.
Before a sub-account is activated, the root account must grant it the necessary CAM permissions:
Step 1: Grant service role permissions
In CAM - Users - Permissions, attach the policy QcloudCamSubaccountsAuthorizeRoleFullAccess to the sub-account. This policy is used to create a product service role during activation.
Step 2: Grant cluster creation permissions
Create a custom policy in CAM (recommended name: DataBuddyInitializationPolicy) that includes the following three permissions, and then attach the policy to the sub-account:
finance:trade (transaction permission)
cam:PassRole (role pass permission)
dlc:CreateDataEngine (permission to create a data engine)
The JSON content of the custom policy is as follows:
{
"statement": [
{
"action": [
"finance:trade",
"cam:PassRole",
"dlc:CreateDataEngine"
],
"effect": "allow",
"resource": [
"*"
]
}
],
"version": "2.0"
}
After completing permission configuration, the sub-account can activate the product trial by following the steps in 2.1. 3. Product Usage Configuration
3.1 Workspace Management
DataBuddy adopts a two-tier structure consisting of the console and workspaces:
Console: manages workspaces across multiple regions, centrally controls the organizational structure and global configurations, and has no regional attributes.
Workspace: the smallest collection of users, resources, and tasks. It can be divided by business department or project team, or by development, testing, and production environments. Workspaces are completely isolated from each other. You can authorize your workspace to other workspaces to enable cross-workspace viewing or reading and writing of catalogs and tables.
After the product trial is activated, the system automatically creates a default workspace. You can use the default workspace to explore the product, or create a new workspace (which requires the console administrator role).
3.2 User Management
The person who activates the product becomes a DataBuddy console administrator by default. Root accounts and sub-accounts with the AdministratorAccess permission also automatically become console administrators after entering DataBuddy.
DataBuddy users are divided into two levels: console users and workspace users. A user must first become a console user before becoming a workspace user.
Role permission description:
|
Console | Console administrator (system role) | Has full permissions on all console features, including creating and managing workspaces, user/user group management, and platform settings. |
Console | Console member (system role) | Has read-only access to the console. |
Workspace | Workspace administrator (system role) | Has full read and write access to all functional modules within the workspace. |
Workspace | Workspace member (system role) | Has full read and write access to all features within the workspace except management features. |
Workspace | Custom role | Custom roles can be created for fine-grained feature permission control. |
Add a console user: First create a sub-account in CAM, and then add the sub-account in the DataBuddy console.
Add Agent users: Add and manage DataBuddy Agent users in "Service Subscription - Agent Seats and Quota".
Add a workspace user: First add the user to the DataBuddy console, then add the user to the corresponding workspace and assign the appropriate role.
3.3 Configuring Permissions for Sub-Accounts
Users who use DataBuddy must have sufficient permissions. The steps are as follows:
1. Search for and click "CAM" on the Tencent Cloud official website, click "Policy - Create Custom Policy - Create by Policy Generator - JSON", paste the following content, and create the policy:
{
"statement": [
{
"action": [
"cam:PassRole"
],
"effect": "allow",
"resource": [
"*"
]
}
],
"version": "2.0"
}
2. Click "User - User List - Create User - Quick Create", create a sub-account, and grant the above policy to the sub-account.
3.4 Compute Resource Management
Before creating and running tasks, you need to create compute resources first. DataBuddy provides different types of compute resources for different load scenarios. Create the appropriate type based on your actual needs:
Compute resource type | Engine description | Applicable task types |
Job cluster | Includes Spark standard engine and Ray engine for data processing and machine learning tasks. | Notebook tasks and Python standalone tasks |
Interactive cluster | Spark standard engine for interactive SQL queries. | SQL exploration, Dashboard, ChatBI, file upload, batch table creation |
Platform cluster | Supports offline data ingestion, custom scripts, and platform-level servitization. | Offline data ingestion tasks |
Real-time cluster | Based on Flink engine for real-time data processing. | Real-time data ingestion tasks |
Attention:
Creating job clusters and interactive clusters requires the current user to have the following three CAM permissions: finance:trade, cam:PassRole, and dlc:CreateDataEngine.
3.5 Network Settings for Compute Resources
Data ingestion scenarios require compute resources to access data sources for data reading, so the compute resources must be connected to the network where the data sources reside. Configure the network on the platform cluster or real-time cluster.
Public data source: The VPC configured for the compute resource must be connected to a public IP address (the VPC must have a public NAT configured), and the public NAT address of the VPC must be added to the database allowlist.
Private network data source: The VPC configured for the compute resource must be connected to the network where the data source resides, and the CIDR block of the VPC must be added to the database allowlist.
Attention:
After completing the network configuration, make sure to grant workspace members the permission to use the cluster so that their query requests can be submitted through the cluster to the data source for execution.
3.6 Data Source Management
To ingest data from existing systems into DataBuddy, you need to create a data source first. The data source types currently supported by the platform include:
MySQL, Hive, TDSQL-C MySQL, Tencent MySQL, TDSQL MySQL, PostgreSQL, Tencent PostgreSQL, SQL Server, Oracle, ClickHouse, TCHOUSE-D/P/X/C, Doris, StarRocks, HDFS, HBase, FTP, SFTP, COS, Redis, Kafka, CKafka, File System, MongoDB, Tencent MongoDB, Elasticsearch, Tencent ES, IBM DB2, DM, TiDB, OceanBase, Kudu, Greenplum, GaussDB, Iceberg, BigQuery, Hudi, TDengine, GBase, InfluxDB, REST API, and more. For details, see Supported Data Sources and Read Capabilities. Note the following when creating a data source:
When a connectivity test is performed, the selected compute resource currently supports only platform clusters and real-time clusters.
You must first complete the network configuration for the compute resource (see 3.5). Otherwise, the data source may fail to connect.
On the Permissions page of the data source details page, grant usage permissions to members who need to use the data source.
4. Data Ingestion
DataBuddy data ingestion provides a unified entry point for importing data from external systems into the data lakehouse. It covers three scenarios: offline sync, real-time sync, and file upload, and offers the following five ingestion solutions. You can choose an appropriate solution based on the data source type, data scale, and timeliness requirements:
Access solution | Typical Scenario | Entries | Related documentation |
Offline data synchronization | Periodically synchronize data from relational databases, big data storage, and message queues to the data lakehouse in batches. | Data access → Task management → Create sync task → Offline single-table sync / Offline batch sync | |
Real-time data synchronization | Synchronize relational database CDC and message queue data in real time, with support for DDL change synchronization. | Data access → Task management → Create sync task → Real-time whole-database multi-table sync / Real-time sharded database and table sync | |
File upload to table | Parse local or COS files and load them into structured tables. | Data access → File upload → File to table / COS file to table | |
File upload to volume | Upload files to Volume in their original format and preserve the original file structure. | Data access → File upload → File to volume | |
Batch file synchronization to volume | In an offline sync task, write structured data to Volume in the specified format, or synchronize unstructured files to Volume in batches. | Data access → Task management → Create sync task (select a Volume type Catalog as the destination) | |
If you only want to quickly import data into the data lake for product evaluation, follow the file upload steps below.
4.1 Uploading a File to a Table (Single File)
Operation steps:
1. Click the left menu bar and go to Data Integration > File Upload to Table.
2. Select the file to upload.
3. Wait for the file preview. If the cluster is not started, the wait time may be long.
4. Confirm the table and field information, and then click Create Table.
5. After the table is created, the system automatically redirects to the Catalog directory.
Attention:
If you upload multiple files at a time, their structures must be consistent. Files with inconsistent structures will also be stored in the same table, which may result in missing fields.
4.2 Uploading Files to a Volume (Batch Files)
This applies to scenarios where unstructured files are uploaded in batches. Operation path: left menu bar > Data Integration > File Upload to Volume. After the upload, the files are stored in DataBuddy volume storage and can be processed later through Agent or SQL.
4.3 Analyzing by Directly Connecting to Existing Data Sources
If you already have an existing database, you can quickly build semantic models and perform data analysis through direct connection without migrating data to DataBuddy. Follow these steps:
Step 1: Configure network connections for the platform cluster
The system automatically creates a platform cluster. Click Edit to configure network connections for the platform cluster so that it can access your data source. After that, make sure to grant workspace members the permission to use the cluster.
Step 2: Create a data source
Direct connection analysis currently supports eight data source types: MySQL, Tencent MySQL, StarRocks, Doris, TCHOUSE-D, PostgreSQL, Tencent PostgreSQL, and GaussDB. After creating a data source, complete the connectivity test on the data source details page and grant usage permissions to the personnel who need to use the data source.
Step 3: Create an external data catalog
You can create an external data catalog through the following two entries:
Entry 1: In the upper-right corner of the data source details page, select Create External Data Catalog.
Entry 2: Go to Workspace > Data Catalog, and click New in the upper-left corner.
Select "External Data Catalog - Direct Connection Analysis" and associate it with the data source created in the previous step.
Step 4 (Optional): Create a semantic model to improve analysis accuracy
A semantic model can help DataBuddy Agent understand your business data and analysis intent more accurately. You can create a semantic model in the following ways:
Method 1: Open the Hi Buddy menu and enter Data Development mode to let Agent assist you in creating semantic models, metrics, and dimensions.
Method 2: Go to Workspace > Ontology Modeling to manually create a business ontology model and a semantic model.
Step 5: Add data to the analysis space
Go to Workspace > Hi Buddy > Data Knowledge Base > Add Data/View, add tables from the direct connection data source to the analysis space, and then you can start query-based analysis.
Permission control note: You need to grant analysts query permissions on tables and metrics. Unauthorized data cannot be accessed. Table permissions can be granted on the details page of the corresponding table in the data catalog, with support for table-level and row-level permissions. Metric permissions can be granted in Metric Management under Ontology Modeling.
5. Data Engineering (AI-Driven)
DataBuddy's data engineering capabilities are driven by AI Agent and support the following core features:
Assist in building a data warehouse for new data sources: AI automatically understands the data structure and analysis goals, and generates a layered data warehouse plan from ODS, DWD, and DWS to ADS.
Intelligent data ingestion: Automatically identify external data sources and table structures, and complete field mapping and data ingestion configuration.
Code generation: Automatically generate high-quality SQL/script code to improve development efficiency.
Automated workflow orchestration: Automatically orchestrate task dependencies and scheduling cycles, and generate a complete data development workflow with one click.
Task diagnosis and repair: Monitor task running status, automatically identify failure causes, and provide repair suggestions.
5.1 Configuring Workspace Rules (Recommended)
DataBuddy supports a workspace-level rule file named workspace_rule.md, which is created by the workspace administrator and placed in the workspace root directory. After the file is created, all Agent sessions of all members in the workspace automatically load the rule as context, eliminating the need for members to manually paste the specification.
Core values:
Output consistency: Layered naming, time parameters, and code style take effect across the entire workspace after a single configuration.
Strong specification constraints: Ensure that the Agent strictly follows internal data specifications.
Evolvability: Discovered issues and conventions can be supplemented at any time, and the Agent takes effect immediately.
Suggestion: If your enterprise already has assets such as data warehouse specifications, metric definition manuals, and SQL coding standards, you can upload the existing specification documents (Word, PDF, Markdown, or Confluence exports are all supported) in the Agent conversation window, and use the following Prompt to let the Agent automatically extract and generate workspace_rule.md:
The customer's existing data specification documents have been uploaded: <document list>.
Please read the above documents and extract the items that can be used as constraints for Agent behavior,
Generate a rule file that complies with the DataBuddy workspace_rule.md format.
Requirements:
1. Organize into sections by topic (data warehouse layering / naming / time parameters / environment isolation / processing constraints / data ingestion / metric definitions / high-risk operations, etc.)
2. Each rule must be executable and verifiable, and avoid vague statements.
3. If key items are missing from the specifications (such as time parameters and idempotency), mark them as "Suggested additions" and provide DataBuddy recommended practices.
4. The output result can be directly written to workspace_rule.md.
5.2 Data Warehouse Solution Design
After switching the DataBuddy Agent to the "Data Development" mode, you can describe business requirements in natural language, and the AI will automatically understand the goals, identify data sources, and generate a data warehouse construction plan that fits the business scenario. Recommended workflow:
1. Have the Agent survey data sources and core business tables: List the data sources and table inventory in the workspace, and filter key tables.
2. Table structure and lineage analysis: The Agent outputs table relationship diagrams, business meanings, and joint analysis scenarios.
3. Plan first: Have the Agent create a plan document in Studio, including requirement understanding, input and output table lists, upstream and downstream dependencies, key metrics and business rules, data risks, and a phased implementation plan.
4. After manual Review and confirmation, have the Agent generate the implementation code according to the plan.
5.3 Workflow Orchestration and Scheduling
The Agent can automatically orchestrate task dependencies and scheduling cycles, and generate a complete data development workflow with one click.
Use the following Prompt to have the Agent configure and publish the workflow:
According to Mobile Game Data Warehouse_Design Proposal.md, help me complete the DAG configuration and release of the workflow.
Note: After a workflow is submitted, it becomes read-only and cannot be modified. Submit it for scheduling only after testing confirms it is correct.
5.4 Task Diagnosis and Repair
DataBuddy supports intelligent task diagnosis and repair capabilities:
You can view the execution information and run logs of each node on the workflow run details page.
When a task is abnormal, AI automatic diagnosis can be triggered to identify the root cause of the failure and provide repair suggestions.
The Agent can automatically fix SQL code and generate a changed version for manual Review.
Supports viewing records of SQL change versions from automatic diagnosis and repair.
Use the following Prompt to have the Agent run the workflow and perform failure diagnosis and repair:
Help me manually run the workflow once to verify its execution. If it fails, automatically diagnose and fix the issue.
6. Data Governance
6.1 Data Quality Monitoring
DataBuddy supports AI-driven data quality rule recommendation and configuration. You can use natural language to have the Agent analyze table data profiles and metadata information, automatically recommend and configure appropriate quality monitoring rules, covering multiple quality dimensions such as completeness, consistency, accuracy, timeliness, and uniqueness.
Core capabilities include:
AI automatically recommends quality rules: It automatically recommends suitable quality rules and advanced rules based on data characteristics.
Quality task trial run: Verify the detection results before officially enabling it.
Alarm notification: Supports configuring notification channels and alarm rules. The workflow automatically detects and alarms during daily scheduling.
Workflow integration: You can add quality tasks to the corresponding workflow nodes so that they are scheduled and executed along with the workflow.
Use the following Prompt to have the Agent automatically generate quality rules:
Based on the data warehouse construction design document, help me analyze the data profiles and metadata of the tables produced for this requirement, explore the data content and patterns of each table, and recommend and configure appropriate quality monitoring rules and advanced rules.
After confirming there are no issues, use the following Prompt to have the Agent add the quality task to the workflow and start scheduling:
Check my current data quality tasks and workflows, and help me add the data quality tasks to the corresponding workflow nodes, configure dependencies, and schedule them along with the workflow.
6.2 Sensitive Field Identification and Masking
DataBuddy supports automatically identifying sensitive fields in tables and setting masking policies to provide dynamic masking protection. Procedure:
1. Use natural language or the UI to have the Agent identify sensitive fields in database tables and automatically apply built-in sensitive tags.
2. Configure user groups in Data Governance.
3. Create or bind recommended masking policies for sensitive tags, such as name masking and partial phone number hiding.
4. If the policy is unreasonable, you can adjust it manually. After confirmation, the masking policy is automatically bound to the field tags and takes effect.
Use the following Prompt to have the Agent identify sensitive fields:
Identify sensitive fields in the existing database tables and set sensitivity tags.
6.3 Data Standards and Field Comments
DataBuddy supports automatically configuring table descriptions and field descriptions for database tables through the Agent, improving the understandability and standardization of data assets. After master data enters the lake, master data field descriptions can be automatically associated through standard ingestion, and data quality configurations can then be performed based on standards.
Use the following Prompt to have the Agent check and configure specifications:
Help me check and configure the workspace rules under the current workspace, including data warehouse and table naming specifications, and configure appropriate table descriptions + field descriptions for existing database tables.
6.4 Semantic Layer Construction
The semantic layer is a critical infrastructure that enables DataBuddy's analysis Agent to retrieve data accurately. By building semantic models, metrics, and dimensions, you can unify business definitions and enable the Agent to accurately understand business semantics.
The semantic layer contains three core components:
Semantic model: defines business objects and their relationships, such as the associations between customers, orders, and products.
Metric: uniformly defines business measures, such as DAU, ARPU, and payment rate, including atomic metrics and derived metrics.
Dimension: defines data slicing perspectives, such as channel, region, and time, and supports multi-level drill-down.
After a semantic model is established, the Agent can understand natural language queries more accurately. For example, without a semantic model, the Agent may not understand the meaning of "play" in "How many people came to play yesterday?" After a semantic model is established, the Agent can automatically interpret it as "calculate DAU."
7. Data Analysis
7.1 Switching Analysis Spaces
After entering the workspace, click "Hi Buddy" to switch to analysis space mode, and then you can start using the intelligent data analysis feature.
7.2 Adding Data Knowledge
In the analysis space, you can add related data tables, views, semantic models, and professional knowledge bases to the analysis space by clicking the "Add Data" button. Only data added to the analysis space can be used for query-based analysis.
7.3 Intelligent Data Query Analysis
DataBuddy supports multiple analysis capabilities, and you can ask questions directly in natural language:
Statistical analysis: Ask questions in natural language, and the Agent automatically generates SQL queries and returns results. For example, "How many people made payments yesterday?" and "Compare revenue across channels last month."
Anomaly detection: The Agent can automatically detect abnormal fluctuations in data, helping you identify business anomalies in a timely manner.
Forecasting and correlation analysis: supports trend forecasting based on historical data and correlation analysis between variables.
7.4 AI Dashboard
You can let the Agent automatically generate an AI dashboard based on current data. Simply describe your requirements in natural language, such as "Generate an AI dashboard for me based on the current data," and the Agent will automatically select appropriate visualization methods to generate an interactive dashboard containing key metrics and charts.
For details, see AI Dashboard.