Basic Concept Questions
Billing Quota Questions
What Is the Difference Between Rate Limiting and Credit Budget?
When creating an API Key, you need to select a limit type. The two types are compared as follows:
|
Control request frequency and Token throughput rate | Control the total consumption credit cap. |
TPM + RPM | Total credit budget |
Prevent excessive calls to a Key from causing excessive service pressure. | Prevent a Key from consuming excessive credits and exceeding the budget. |
Real-time rate limiting (sliding window) | Cumulative deduction, service stops after exhaustion. |
Assign different rate quotas to different teams/applications. | Allocate fixed credit quotas to different projects/customers. |
Recommendations: If you are concerned about avoiding overly frequent calls, choose rate limiting. If you are concerned about avoiding excessive credit consumption, choose a credit budget. You can also use both together by setting rate limits at the instance level and credit budgets at the Key level.
What Do TPM and RPM Mean, and How Should They Be Configured?
TPM (Tokens Per Minute): The maximum number of tokens that can be processed per minute, measured in thousands/minute. For example, setting it to 100 means up to 100,000 tokens can be processed per minute.
RPM (Requests Per Minute): The maximum number of requests allowed per minute, measured in requests/minute. For example, setting it to 60 means up to 60 requests are accepted per minute.
These two parameters appear in two configuration locations and have different meanings:
|
When Creating an Instance | Maximum Token Throughput per Minute for the Entire Instance | Maximum Request Frequency per Minute for the Entire Instance |
When Creating an API Key (Rate Limit Mode) | Maximum Token Throughput per Minute for a Single Key | Maximum Request Frequency per Minute for a Single Key |
Recommendations:
Set instance-level TPM/RPM appropriately based on your business peaks and budget to avoid request rejections caused by limits that are too low.
Key-level TPM/RPM is used to control quotas for different users/applications at a fine-grained level, and is typically less than or equal to the instance-level quota.
RPM limits are mainly used to prevent sudden traffic spikes. TPM limits are mainly used to control Token costs.
What Is BYOK (Bring Your Own Key)?
BYOK (Bring Your Own Key) is a core access mode of CMR. It allows you to use API keys that you have applied for on various foundation model platforms and manage and schedule them centrally through the CMR gateway.
Which Model Providers Does CMR Support?
The BYOK (Bring Your Own Key) mode allows users to bring their own model keys and flexibly access original vendor models, third-party proxies, or self-built models.
|
BYOK original vendor model provider | Call models from the original AI platform provider through CMR. | Consumes the user's tokens on the original AI platform or computing platform. | Online |
BYOK third-party proxy provider | Call models on third-party AI platforms through CMR. | Consumes the user's tokens on a third-party AI platform. | Online |
BYOK self-built model provider | Call user-privately-deployed models through CMR. | Consumes the user's own tokens on the self-built AI platform. | Online |
What Is Intent Routing in CMR and How to Configure It?
Intent routing is an intelligent distribution capability of CMR. The system automatically identifies the intent category of user messages and routes requests to the optimal model for the corresponding scenario for processing.
How It Works:
1. A client sends a request to the CMR gateway.
2. The gateway performs intent recognition on the request content (for example, general conversation, content generation, coding and technology, and so on).
3. Route requests to the corresponding model based on preconfigured intent-to-model mapping rules.
Configuration Method:
1. Log in to the CMR console. On the Instance Management page, click the target instance ID to go to the instance details page, and then switch to the Model Scheduling Management page. 2. In the routing policy diagram, click Create in the Intent Routing module. Alternatively, click Create Rule in the Intent Routing module on the page.
3. You can select a built-in business template or customize an intent routing business.
4. Configure the mapping between intent categories and their corresponding models.
How Is the CMR Processing Fee Calculated?
For CMR billing items, see Billing Overview. The CMR processing fee is billed by Token volume, which is fundamentally different from the common industry practice of proportional markup billing: |
Total processed tokens (fixed unit price) | Pass through upstream costs proportionally (floating price). |
Fixed unit price, independent of upstream pricing changes. | Charges fluctuate with upstream model pricing. |
Total input and output tokens passing through the gateway | Primarily based on the usage field returned by the upstream |
Calculation formula: CMR processing fee = CMR processing fee unit price (USD per million Tokens) × total Token volume.
What to Do When Quota Is Insufficient and How to Increase It?
For descriptions of CMR quotas, see Usage Constraints. If the default quotas do not meet your requirements, please submit a ticket to contact us and specify the required quota types, target quantities, and business scenarios.