tencent cloud

Scenarios

Download
Focus Mode
Font Size
Last updated: 2026-11-17 14:27:51
AI-Translated & Reviewed
Cloud Model Routing (CMR) is designed for enterprise-grade unified large model access and intelligent scheduling scenarios. While providing multi-model compatibility, hybrid network connectivity, intelligent routing, and high-availability assurance, it also supports token-level cost governance and end-to-end security auditing. It is widely applicable to diverse business scenarios such as multi-vendor management, cross-domain access acceleration, AI Agent development, and enterprise compliance management and control.

Unified Access and Intelligent Distribution for Multiple Models and Vendors


In the rapid iteration of enterprise AI applications, business teams often need to simultaneously access services from multiple model vendors, including both open-source and self-built models (such as DeepSeek, Llama, and Qwen) and various commercial large models. Different vendors have varying API formats, authentication methods, parameter definitions, and error codes. Even when some vendors provide compatible APIs, the application side still needs to adapt and maintain multiple sets of call logic separately, resulting in high development and maintenance costs.
CMR provides a unified API compatible with OpenAI and Anthropic, bringing mainstream domestic and international model services under the same routing plane. Business applications do not need to write separate adaptation code for each vendor. Simply configuring access to CMR enables transparent call of multiple backend models. Meanwhile, CMR's intent routing engine can automatically distribute requests to appropriate models based on business intents (such as customer service Q&A, code generation, and document summarization), reducing the workload of maintaining complex if-else judgment rules on the application side.
In addition, CMR supports allocating independent API Keys for different business lines, teams, or applications, and configuring call quotas, permission scopes, and cost attribution per key. When multiple teams share the same routing plane, it facilitates usage isolation, cost splitting, and permission management. It also enables rapid location and revocation in the event of key leakage or abnormal calls, reducing the risks of unified management.

Unified Governance for Hybrid Models Across VPC, IDC, and Public Networks


When enterprises deploy large models, they often have multiple deployment forms simultaneously: for data security and self-control considerations, open-source models (such as DeepSeek, Llama, and Qwen) are privatized and deployed in their own Virtual Private Clouds (VPCs) or Internet Data Center (IDCs); at the same time, some scenarios require calling externally procured model services deployed on the public network to supplement specific capabilities. These models scattered across different networks and permission systems are difficult to unify access and management.
CMR natively supports hybrid network deployment. Routing instances can be deployed inside an enterprise's VPC on demand, connecting to the IDC environment through Direct Connect or Cloud Connect Network (CCN), and supporting access to public network models through a secure proxy. Self-deployed open-source models and externally procured public network models can be orchestrated and scheduled together on the same routing plane, with model call traffic managed through a unified CMR instance entry, achieving integrated management in multi-network environments. By combining VPC and CCN, enterprises can incorporate 'self-built open-source models+externally procured public network models' into the same call governance plane without changing their existing network architectures.

Cross-Domain Model Access Acceleration


When enterprises call mainstream cross-domain models domestically, cross-domain links are susceptible to network fluctuations, resulting in increased latency and unstable connections, which affects the business experience. CMR leverages its underlying integrated global acceleration capabilities to optimize network paths for cross-domain calls through compliant acceleration links, combined with intelligent Domain Name System (DNS) resolution for nearby access nodes, helping to reduce transmission latency and improve connection stability. Enterprises do not need to build or additionally configure cross-domain acceleration solutions themselves. They can obtain an out-of-the-box cross-domain access optimization experience through the unified CMR entry.

Multi-Level Fault Tolerance and Business Continuity Assurance


Large model services in production environments inevitably encounter upstream rate limiting, timeouts, or temporary unavailability. Without automatic disaster recovery capabilities, business links may be directly interrupted.
CMR provides a three-level fault tolerance mechanism to ensure requests receive responses through layered fallback:
Inter-model switchover: When the selected model is abnormal, the system automatically switches to another available model.
Intra-model switchover: Multiple API Key (vendor) switchover for the same model. The same model can be configured with multiple API Keys (corresponding to different vendors or channels). When a key is rate-limited or the corresponding vendor is unavailable, the system automatically switches to another available key for the same model.
Fallback: When the above switchovers still cannot satisfy the request, fall back to a pre-configured fallback model to avoid unanswered requests.
Configure targeted health check policies (check interval, timeout threshold, and unhealthy threshold). Abnormal nodes are automatically circuit-broken and isolated, and automatically rejoin the available list after recovery. The entire process is transparent to the caller, helping businesses maintain stable operation in abnormal model scenarios. For core businesses with high availability requirements, it is recommended to configure monitoring and alarming policies in conjunction with Tencent Cloud Observability Platform to detect and intervene at the first moment when model call exceptions occur, further ensuring business continuity.

Fine-Grained Token Consumption Management and Cost Governance


In scenarios where multiple teams and applications share model resources, it is difficult to split token consumption across different business lines and API Keys. Costs therefore lack a basis, making it easy to exceed budgets or waste resources.
CMR provides token consumption breakdowns and cost details by dimensions such as CMR instance, API Key, user group, and BYOK ID, supporting separate statistics for input and output tokens so that every call's cost is attributable and accountable. You can set call quotas and budget limits for different user groups or API Keys, with automatic alarming or blocking when limits are exceeded, keeping costs within a controllable range.
With a clear understanding of cost composition, CMR further provides two proactive cost-reduction capabilities to help move from "cost visibility" to "cost optimization": First, semantic caching, which allows direct reuse of existing results for semantically similar repeated requests, improving cache hit rates and thereby reducing unnecessary token consumption caused by redundant computation; second, intelligent cost routing, which can automatically schedule requests to more cost-effective models based on model quality, latency, and price profiles, reducing model call costs without sacrificing service quality. The two work together with the aforementioned usage splitting and quota management to form a closed loop of "Cost details -> Cost constraints -> Cost optimization."

Enterprise-Grade Security and End-to-End Auditing


When multiple teams and applications within an enterprise simultaneously access large models, model API Keys are often scattered across various business codebases, increasing the risk of leakage and abuse while making call behavior lack unified records, making it difficult to meet internal security management and auditing requirements.
CMR supports unified hosting of BYOKs to the routing instance, so business code no longer directly holds keys. Combined with Cloud Access Management (CAM)'s permission system and CMR's user group capabilities, you can configure fine-grained access policies by model and vendor dimensions: different teams and business lines can only call authorized models, with independent quota and usage accounting, where permission boundaries and resource usage do not affect each other.
The built-in LLM-WAF performs real-time detection on input prompts and model responses, supporting sensitive information masking and non-compliant content blocking. All call records are written in real time to Cloud Log Service (CLS), covering fields such as request source, call parameters, and return results, supporting search and report analysis, providing complete evidence for enterprise internal auditing and compliance.





Help and Support

Was this page helpful?

Help us improve! Rate your documentation experience in 5 mins.

Feedback