tencent cloud

Overview

Download
Focus Mode
Font Size
Last updated: 2026-11-17 14:27:51
AI-Translated & Reviewed

Introduction

Cloud Model Routing (CMR) is a unified large model gateway service provided by Tencent Cloud for enterprise-grade AI applications. CMR supports unified client access via standard-compatible APIs and manages various large model services across public clouds, third-party providers, and privatized deployments. Enterprises do not need to adapt to protocols and configure keys separately for each model vendor. By configuring access to CMR, they can access model resources from multiple providers and gain unified authentication, quota management, auditing, and observability capabilities.
CMR is deeply integrated with Virtual Private Cloud (VPC) and supports access to model services deployed within VPCs, in self-built Internet Data Center (IDCs), and on public networks. It can also leverage public network acceleration to optimize cross-region access. Built-in automatic retries, health checks, and multi-level fallback reduce call failure rates and improve service availability while masking API differences across vendors. The product provides monitoring and analysis capabilities across the entire call chain, helping you gain insights into the operational status of your business.


Core Features

Unified Access to Multiple Models

CMR reduces the effort of adapting to each vendor's APIs by centrally managing mainstream models, so that business applications do not need to adapt to each one individually.
Feature Description
Issue Resolved
OpenAI- or Anthropic-compatible API: provides a unified API protocol for the client side. Configuring access to CMR allows for transparent switching of backend vendors.
API formats, authentication methods, and parameters vary across vendors. The application side needs to adapt to multiple Software Development Kits (SDKs) and invocation logic. Switching to or adding a vendor requires repeated modifications.
Multi-vendor management: the model side supports access to mainstream models such as Tencent Hunyuan and DeepSeek, and supports management of endpoints for self-developed, fine-tuned, and privately deployed models.
Self-developed, fine-tuned, and privately deployed models are scattered across different entries with public cloud models, lacking unified management and a unified invocation method.

Intelligent Route Scheduling

Based on routing and forwarding policies of advanced routing engines such as intent routing, requests are automatically distributed to models that meet preset routing policies, eliminating the need to write complex decision logic on the application side.
Feature Description
Issue Resolved
Intent routing: Automatically matches models based on business intents (such as customer service Q&A, code generation, and document summarization), and supports intent classification and dynamic routing based on prompt content.
Model selection relies on manual experience, and application-side if-else rules are complex and difficult to be continuously optimized.
Load-aware scheduling: Dynamically schedules requests to the node with the lowest load based on real-time metrics such as usage, busyness, and latency, avoiding hotspot overload.
Requests concentrate on a few nodes, causing hotspot overload, increased latency, and more frequent rate limiting.
Session affinity: In multi-turn sessions, the same session is always routed to the same instance to maintain context continuity.
Multi-turn sessions are distributed across different instances, causing context breaks, degrading the user experience and cache hit rate.
Canary release and traffic coloring: Supports configuring priorities and weights to route some requests to new models for safe canary release.
New model rollout lacks a controlled low-traffic validation method, and full rollout poses high risks.

Network Acceleration and Hybrid Networking

CMR provides flexible network access solutions for enterprises with different deployment forms, and optimizes the access speed for cross-region model calls1.
Feature Description
Issue Resolved
Hybrid networking: supports unified access from multiple network environments such as the public network, VPC, IDC, Direct Connect, and Cloud Connect Network (CCN).
Models in VPCs or IDCs and public cloud large models are scattered across different network and permission systems, making controlled interconnection between networks difficult.
Public network acceleration: provides a public network acceleration channel to optimize the latency and stability of cross-region calls.
Cross-region model links fluctuate, with high latency and unstable performance, affecting service experience.

High Availability Assurance

CMR provides capabilities such as health checks, automatic retries, and failover, helping improve the availability of the model call chain.
Feature Description
Issue Resolved
Health checks and circuit breaking: Continuously probes the health statuses of backend models, automatically isolating abnormal nodes and reinstating them upon recovery.
When a model is unavailable, requests are still routed to faulty nodes, causing fault spread and requiring manual intervention to remove them.
Automatic retry: Automatically retries when a call fails or times out, and supports configurable retry policies (the number of attempts, interval, and backoff algorithm).
Upstream rate limiting, timeouts, or intermittent errors are directly passed through to the business side, resulting in a high call failure rate.
Multi-level failover: Automatically downgrades to a backup model or backup vendor when the primary model is unavailable, and supports multi-level fallback chain configuration.
A failure of a single model or a single vendor immediately causes service interruption, and the application side lacks automatic disaster recovery capabilities.

Enterprise-Grade Security Management and Control

Security capabilities are deeply integrated into the model call chain, providing configurable features such as network isolation, access control, content risk detection, and audit logs to help customers implement security management and audit retention.
Feature Description
Issue Resolved
Network layer security: Route instances are natively deployed within user VPCs and support access control through security groups and network Access Control Lists (ACLs).
Model access control on the client side needs to meet enterprise network security level requirements.
Content security WAF: The built-in configurable LLM-WAF audit pipeline performs real-time content risk detection on input prompts and returned content based on configured rules, and supports sensitive information masking, interception upon rule hits, and custom rules2.
Input and output content lacks compliance detection, posing risks of sensitive information leakage and non-compliant content.
Fine-grained access control: Combines Cloud Access Management (CAM) and user group capabilities to implement model-level and API Key-level permission management.
Model API Keys are scattered in business code, and permissions cannot be isolated by team or application.
End-to-end audit (CLS): All call records are written to Cloud Log Service (CLS), which supports search, reports, and compliance audit.
The call chain lacks audit logs, and security compliance and post-incident traceability requirements cannot be met.

Cost Governance

Token consumption and cost details are broken down by dimensions such as CMR instance, API Key, user group, Bring Your Own Key (BYOK), and tag. Budgets and quotas are set, with automatic alarms or blocks triggered when limits are exceeded.
Feature Description
Issue Resolved
Multi-dimensional cost breakdown: Precisely calculates token consumption and costs by business dimensions to support internal cost attribution.
Token usage across different teams, applications, and models is difficult to break down, leaving cost attribution and optimization without a basis.
Budget and quota control: Sets call limits for instances, API Keys, and user groups. When limits are exceeded, automatic alarms or blocks are triggered to prevent cost overruns.
Token usage is uncontrollable, and abnormal calls or sudden spikes lead to cost overruns.
Semantic caching: Matches semantically similar repeated requests against cached data to improve the upstream cache hit ratio and reduce unnecessary token consumption.
Repeated or highly similar requests call the model repeatedly, generating a large amount of unnecessary token consumption.
Intelligent cost routing: Schedules requests based on policies such as user configuration, model quality, latency, and price.
A tradeoff mechanism among quality, latency, and price is lacking, and high-priced models are used indiscriminately.

End-to-End Observability

CMR provides monitoring and analysis capabilities across the entire model call chain, enabling real-time visibility into the operational status of AI applications.
Feature Description
Issue Resolved
Unified call monitoring: Supports displaying model call metrics for models that are integrated and have corresponding monitoring collection enabled, such as request volume, success rate, latency distribution, and token consumption of model calls. It also provides distributed tracing capabilities within supported call chains.
The call statuses of multiple models are scattered, and there is no unified view for monitoring request volume, success rate, and consumption metrics.
LLM-specific metrics: Provides metrics such as Time to First Token (TTFT) and token output rate in addition to traditional QPS and latency, and supports P95 and P99 latency percentile statistics.
Traditional Queries Per Second (QPS) and latency metrics cannot reflect large model experience, and the TTFT and token output rate cannot be measured.
End-to-end tracing: Traces the complete chain of a single call based on the OpenTelemetry protocol, from the gateway entry to the completion of model inference.
The duration or failure cause of a single call is difficult to locate, and troubleshooting relies on manual comparison of logs.
Intelligent alarming: Triggers automatic notifications when token consumption surges, the success rate drops sharply, or latency becomes abnormal based on exception detection.
Detection of exceptions and usage surges is delayed, and they are often noticed only after they occur.

Forwarding Path

CMR provides token-level call and cost statistics, and can break down usage and costs by dimensions such as application, team, and model, helping enterprises clearly understand the cost structure of large model calls3. During the forwarding process, it provides end-to-end observability capabilities.


1.A Client Initiating a Request

The client initiates a request to CMR, and the request body contains fields such as the model identifier, Prompt message, and thinking streaming switch.

2.Admission and Authentication

After a client initiates a request, admission and authentication checks are performed based on the policies in the customer-configured security groups or user groups. Only requests that meet the policy requirements are allowed to pass.

3.Rate Limiting

CMR supports setting Tokens Per Minute (TPM) and Requests Per Minute (RPM) rate limits at the CMR instance, user group, and API Key dimensions, or binding credit budgets. It provides effective management capabilities for model usage and resource consumption, helping teams or projects effectively plan, monitor, and optimize token consumption.

4.Inbound Security Review

Tencent Cloud WAF for large model security provides dedicated security protection for large model application scenarios, including core capabilities such as prompt attack defense, input content security review, sensitive data leakage protection, compute consumption protection, and runtime protection2.

5.Intelligent Routing Decisions

CMR provides intent routing, L1 inter-model routing, and L2 intra-model routing. Based on the models and routing policies associated with the requesting customer, CMR performs level-by-level matching and ultimately selects a model.

6.Acceleration

Leveraging the network nodes and lines supported by Tencent Cloud, CMR can schedule based on node proximity to provide network acceleration for cross-region call scenarios, helping reduce cross-region network latency and improve connection stability.

7.Outbound Security Review

Tencent Cloud WAF for large model security provides dedicated security protection for large model application scenarios, including core capabilities such as output content security review, sensitive data leakage protection, compute consumption protection, and runtime protection2.

8.Response Return

Model inference results are returned to the client via streaming (Server-Sent Events, SSE) or in a single response. Call logs and monitoring metrics are synchronously reported to CLS for subsequent auditing and analysis.
Attention:
The data descriptions in the official website documentation, including product overviews, strengths, and scenarios, are all sourced from Tencent's internal experimental test results in 2026. The tests were conducted under specific environments, conditions, and time ranges, and only reflect the situations in the corresponding test scenarios. Actual results may vary depending on business scenarios, configurations, and usage conditions.
Third-party names, trademarks, and software mentioned in the documentation belong to their respective owners and are used only to indicate the compatibility or adaptation scope. They do not constitute endorsement, warranty, or recommendation.
Note:
1. Before using cross-region models or cross-region network links, customers should fulfill necessary data and personal information protection obligations in accordance with the law, based on the actual data types, processing purposes, and applicable regions. The specific supported models, regions, network lines, and data processing scope are subject to the product documentation and console display.
2. When using content detection and processing related data, customers should still fulfill their data and personal information protection obligations in accordance with the law.
3. The actual hit rate and token consumption are subject to specific call scenarios and configurations. Variations in models, scenarios, and other factors may result in different hit rates.



Help and Support

Was this page helpful?

Help us improve! Rate your documentation experience in 5 mins.

Feedback