
Feature Description | Issue Resolved |
OpenAI- or Anthropic-compatible API: provides a unified API protocol for the client side. Configuring access to CMR allows for transparent switching of backend vendors. | API formats, authentication methods, and parameters vary across vendors. The application side needs to adapt to multiple Software Development Kits (SDKs) and invocation logic. Switching to or adding a vendor requires repeated modifications. |
Multi-vendor management: the model side supports access to mainstream models such as Tencent Hunyuan and DeepSeek, and supports management of endpoints for self-developed, fine-tuned, and privately deployed models. | Self-developed, fine-tuned, and privately deployed models are scattered across different entries with public cloud models, lacking unified management and a unified invocation method. |
Feature Description | Issue Resolved |
Intent routing: Automatically matches models based on business intents (such as customer service Q&A, code generation, and document summarization), and supports intent classification and dynamic routing based on prompt content. | Model selection relies on manual experience, and application-side if-else rules are complex and difficult to be continuously optimized. |
Load-aware scheduling: Dynamically schedules requests to the node with the lowest load based on real-time metrics such as usage, busyness, and latency, avoiding hotspot overload. | Requests concentrate on a few nodes, causing hotspot overload, increased latency, and more frequent rate limiting. |
Session affinity: In multi-turn sessions, the same session is always routed to the same instance to maintain context continuity. | Multi-turn sessions are distributed across different instances, causing context breaks, degrading the user experience and cache hit rate. |
Canary release and traffic coloring: Supports configuring priorities and weights to route some requests to new models for safe canary release. | New model rollout lacks a controlled low-traffic validation method, and full rollout poses high risks. |
Feature Description | Issue Resolved |
Hybrid networking: supports unified access from multiple network environments such as the public network, VPC, IDC, Direct Connect, and Cloud Connect Network (CCN). | Models in VPCs or IDCs and public cloud large models are scattered across different network and permission systems, making controlled interconnection between networks difficult. |
Public network acceleration: provides a public network acceleration channel to optimize the latency and stability of cross-region calls. | Cross-region model links fluctuate, with high latency and unstable performance, affecting service experience. |
Feature Description | Issue Resolved |
Health checks and circuit breaking: Continuously probes the health statuses of backend models, automatically isolating abnormal nodes and reinstating them upon recovery. | When a model is unavailable, requests are still routed to faulty nodes, causing fault spread and requiring manual intervention to remove them. |
Automatic retry: Automatically retries when a call fails or times out, and supports configurable retry policies (the number of attempts, interval, and backoff algorithm). | Upstream rate limiting, timeouts, or intermittent errors are directly passed through to the business side, resulting in a high call failure rate. |
Multi-level failover: Automatically downgrades to a backup model or backup vendor when the primary model is unavailable, and supports multi-level fallback chain configuration. | A failure of a single model or a single vendor immediately causes service interruption, and the application side lacks automatic disaster recovery capabilities. |
Feature Description | Issue Resolved |
Network layer security: Route instances are natively deployed within user VPCs and support access control through security groups and network Access Control Lists (ACLs). | Model access control on the client side needs to meet enterprise network security level requirements. |
Content security WAF: The built-in configurable LLM-WAF audit pipeline performs real-time content risk detection on input prompts and returned content based on configured rules, and supports sensitive information masking, interception upon rule hits, and custom rules2. | Input and output content lacks compliance detection, posing risks of sensitive information leakage and non-compliant content. |
Fine-grained access control: Combines Cloud Access Management (CAM) and user group capabilities to implement model-level and API Key-level permission management. | Model API Keys are scattered in business code, and permissions cannot be isolated by team or application. |
End-to-end audit (CLS): All call records are written to Cloud Log Service (CLS), which supports search, reports, and compliance audit. | The call chain lacks audit logs, and security compliance and post-incident traceability requirements cannot be met. |
Feature Description | Issue Resolved |
Multi-dimensional cost breakdown: Precisely calculates token consumption and costs by business dimensions to support internal cost attribution. | Token usage across different teams, applications, and models is difficult to break down, leaving cost attribution and optimization without a basis. |
Budget and quota control: Sets call limits for instances, API Keys, and user groups. When limits are exceeded, automatic alarms or blocks are triggered to prevent cost overruns. | Token usage is uncontrollable, and abnormal calls or sudden spikes lead to cost overruns. |
Semantic caching: Matches semantically similar repeated requests against cached data to improve the upstream cache hit ratio and reduce unnecessary token consumption. | Repeated or highly similar requests call the model repeatedly, generating a large amount of unnecessary token consumption. |
Intelligent cost routing: Schedules requests based on policies such as user configuration, model quality, latency, and price. | A tradeoff mechanism among quality, latency, and price is lacking, and high-priced models are used indiscriminately. |
Feature Description | Issue Resolved |
Unified call monitoring: Supports displaying model call metrics for models that are integrated and have corresponding monitoring collection enabled, such as request volume, success rate, latency distribution, and token consumption of model calls. It also provides distributed tracing capabilities within supported call chains. | The call statuses of multiple models are scattered, and there is no unified view for monitoring request volume, success rate, and consumption metrics. |
LLM-specific metrics: Provides metrics such as Time to First Token (TTFT) and token output rate in addition to traditional QPS and latency, and supports P95 and P99 latency percentile statistics. | Traditional Queries Per Second (QPS) and latency metrics cannot reflect large model experience, and the TTFT and token output rate cannot be measured. |
End-to-end tracing: Traces the complete chain of a single call based on the OpenTelemetry protocol, from the gateway entry to the completion of model inference. | The duration or failure cause of a single call is difficult to locate, and troubleshooting relies on manual comparison of logs. |
Intelligent alarming: Triggers automatic notifications when token consumption surges, the success rate drops sharply, or latency becomes abnormal based on exception detection. | Detection of exceptions and usage surges is delayed, and they are often noticed only after they occur. |

Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback