tencent cloud

Tencent Cloud Distributed Cache (Redis OSS-Compatible)

ドキュメントTencent Cloud Distributed Cache (Redis OSS-Compatible)Practical TutorialHot Key and Big Key Troubleshooting and Optimization Practices

Hot Key and Big Key Troubleshooting and Optimization Practices

ダウンロード
フォーカスモード
フォントサイズ
最終更新日: 2026-09-16 16:57:07
AI翻訳
This document describes the definitions, identification methods, solutions, and preventive measures for big keys and hot keys in cloud distributed cache databases (Redis-compatible).

Business Scenarios

Big keys and hot keys are the two most common risks in cache-based businesses. During daily Ops, business users typically encounter the following situations:
Problems are difficult to detect in advance: All instance monitoring metrics are normal, but a large-scale access timeout suddenly occurs during a business peak period.
Root causes are difficult to identify: The overall instance QPS has not reached its limit, but a shard has already exhausted its CPU or connections.
Scaling out cannot solve the problem: After the number of shards is increased, pressure remains concentrated on the original nodes, and access latency does not improve.
The above phenomena are mostly caused by the size or access volume of a single Key deviating from the normal baseline of the instance. This document describes the identification methods, troubleshooting approaches, and optimization solutions for these two types of issues, helping you establish a closed loop across the pre-event prevention, in-event detection, and post-event governance stages.

Definition

Big key

A big Key refers to a Key in Redis whose Value is too large or whose collection contains too many members, consuming a large amount of memory. It is essentially a large Value problem. During Ops, you can determine whether a big Key exists in an instance based on the thresholds in the following table.
Data Type
Criterion
String
A single Value exceeds 10 MB.
Hash
The number of elements exceeds 10,000, or the total size exceeds 100 MB.
List
The number of elements exceeds 10,000, or the total size exceeds 100 MB.
Set
The number of elements exceeds 10,000, or the total size exceeds 100 MB.
ZSet
The number of elements exceeds 10,000, or the total size exceeds 100 MB.
Note that the table above is used to identify existing big keys. During the design phase, stricter recommended control values should be followed (String no more than 10 KB, and collection elements no more than 5,000). For details, see Key and Value Design Principles. Actual determination should also be adjusted based on business scenarios and instance specifications. In cluster architecture, also check whether keys are evenly distributed across shards.

Hot key

A hot key refers to a key whose access volume far exceeds that of other keys over a period of time, causing a large amount of QPS or bandwidth to be concentrated on a specific Redis instance or shard.
Common symptoms:
A Hash Key containing 2,000 fields receives a large number of HGETALL operation requests per second.
A ZSet Key containing 10,000 members receives a large number of ZRANGE operation requests per second.
Identification methods:
There is no absolute QPS threshold for hot keys. A single key with 3,000 QPS accounts for 60% of the total QPS on an instance with 5,000 QPS, making it an obvious hot key. However, the same key accounts for less than 1% on an instance with 500,000 QPS, which is considered normal access. Therefore, the key to identification is not the absolute access volume of a single key, but its deviation from the overall instance. The following table lists three dimensions that can be used for identification.
Determination Dimension
Determination Method
Access concentration
Obtain the access ranking through DBbrain hot key analysis, and observe whether there is an order-of-magnitude difference between the top-ranked key and the keys after it.
Relative proportion
Calculate the ratio of requests for a single Key to the total requests of the instance and the ratio of traffic for a single Key to the instance's outbound bandwidth. A higher ratio indicates a higher risk.
Resource headroom
Evaluate whether the CPU, bandwidth, and connections of the node where the Key resides are approaching their limits based on the instance specifications.
It can be seen that all three dimensions are based on "relative levels" rather than fixed values. Specific alarm thresholds need to be determined by the business side. It is recommended to collect access distribution data for a period of time during stable business periods as a baseline, and then use significant deviations from the baseline as the basis for identification. The same set of thresholds is not universally applicable across different instance specifications and business models.

Symptoms and Impacts

Impacts of Big Keys

Issue
Specific Symptom
Uneven memory usage
In cluster architecture, the memory utilization of a shard is far higher than that of other shards, which may trigger the maxmemory limit and cause important keys to be evicted or even OOM.
Request blocking and timeout
Command execution uses a single-threaded model. Operations on big keys take a long time (such as DEL and HGETALL), during which subsequent requests can only wait in queue.
Sync interruption or master-replica switchover
Deleting big keys or performing RENAME operations may block the master node for a long time, causing master-replica synchronization interruption or failover.
Network congestion
A 1 MB big Key accessed 1,000 times per second generates approximately 1 GB/s of traffic, which may consume the entire instance bandwidth.

Impacts of Hot Keys

Issue
Specific Symptom
Sustained high CPU utilization
Hotspot requests are concentrated on a single node, the CPU of this node becomes a bottleneck, and overall service performance degrades.
Access skew
Shards with hot keys bear much higher access pressure than other shards, which may lead to connection exhaustion or even node exceptions. Since access is concentrated on a single key, horizontally scaling out shards cannot distribute the pressure of that key.
Cache breakdown
When a hot Key expires or its node becomes abnormal, highly concentrated traffic directly penetrates to the backend database. If the database cannot handle the traffic, a cascading failure may occur.

Cause Analysis

Causes of Big Keys

1. Improper Key-Value configuration: Using the String type to store large binary files or large JSON/XML data results in oversized Values.
2. Invalid data not cleaned up in time: Members of List, Set, and other types keep accumulating, but expired or invalid data is not periodically cleaned up.
3. Inaccurate business analysis: The data scale was not fully evaluated before launch, and Keys were not properly split, resulting in an excessive number of members in a single Key.
4. Consumer-side code exception: A failure on the consumer side of the message queue (List) causes data to only increase without decreasing.

Causes of Hot Keys

Unexpected traffic surge:
E-commerce scenario: sudden emergence of popular products and flash sale events.
Content scenario: breaking news with surging traffic.
Live streaming scenario: flooding likes and bullet comments in live chat rooms.
Gaming scenario: a large number of players interact intensively in a specific area.

Troubleshooting

Method 1: Diagnosis in the DBbrain Console (Recommended)

Distributed Cache integrates the diagnostic optimization feature of DBbrain, which helps quickly identify big keys and hot keys in instances:
Big Key inspection: See Memory Analysis.
Hot Key inspection: See Hot Key Analysis.

Method 2: Troubleshooting with Command Line Tools

In scenarios where using the console is inconvenient, you can quickly troubleshoot by using the command-line tool that comes with Redis. To prevent the password from appearing in the process list and Shell history, it is recommended to pass the password through the REDISCLI_AUTH environment variable instead of using the -a parameter:
# Pass the password through an environment variable to avoid plaintext exposure
export REDISCLI_AUTH='<password>'

# Inspect Big Keys: Output the Key with the most elements by data type
redis-cli -h <instance address> -p <port> --bigkeys

# Inspect Hot Keys: Output high-frequency keys based on LFU access frequency statistics
redis-cli -h <instance address> -p <port> --hotkeys
Both commands traverse all keys based on the SCAN cursor and do not block the main thread for a long time. However, they incur additional request overhead when the data volume is large, so it is recommended to run them during off-peak hours. In cluster architecture, you need to connect to each shard node separately, as a single connection can only collect statistics for the current node.
Note the difference in statistical scope between the two commands: --bigkeys counts the number of elements (byte length for strings) rather than actual memory usage, so the result only reflects the element scale. To locate keys by memory usage, use the memory analysis feature in Method 1. --hotkeys depends on the LFU eviction policy, so you must first adjust the eviction policy parameter of the instance before using it.
Note:
Before using --hotkeys, set the instance parameter maxmemory-policy to allkeys-lfu or volatile-lfu. Otherwise, the command will report an error directly. This parameter determines the Key eviction behavior of the instance. Modifying it may change the eviction order of existing data, so evaluate the impact on your business before making the adjustment.

Method 3: Business Layer Instrumentation

Wrap the Redis client in your business code to record the number of accesses and latency of each Key, and then use a scheduled task to periodically output the top N hot keys. The following is a sample component for Java + Jedis.
To use it, replace jedis.get(key) with HotKeyDetector.get(jedis, key) to complete the instrumentation. Then, use ScheduledExecutorService to call printTopN(10) every 60 seconds to output the results and execute reset() to reset. Other clients (Lettuce, Redisson) and commands (set, hget, and so on) can be extended by referring to this component.
import redis.clients.jedis.Jedis;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicLong;

public class HotKeyDetector {

// Key → access count
private static final ConcurrentHashMap<String, AtomicLong> KEY_COUNT = new ConcurrentHashMap<>();
// Key → cumulative latency (nanoseconds)
private static final ConcurrentHashMap<String, AtomicLong> KEY_TIME = new ConcurrentHashMap<>();

/**
* Wraps the Jedis get operation to automatically collect the access count and latency of each Key.
*/
public static String get(Jedis jedis, String key) {
long start = System.nanoTime();
try {
return jedis.get(key);
} finally {
record(key, System.nanoTime() - start);
}
}

private static void record(String key, long nanos) {
KEY_COUNT.computeIfAbsent(key, k -> new AtomicLong()).incrementAndGet();
KEY_TIME.computeIfAbsent(key, k -> new AtomicLong()).addAndGet(nanos);
}

/**
* Prints the top N hot keys. It is recommended to call it through a scheduled task (for example, every 60 seconds).
*/
public static void printTopN(int n) {
System.out.println("===== Top " + n + " Hot Keys =====");
KEY_COUNT.entrySet().stream()
.sorted((a, b) -> Long.compare(b.getValue().get(), a.getValue().get()))
.limit(n)
.forEach(e -> {
String key = e.getKey();
long count = e.getValue().get();
long totalNs = KEY_TIME.getOrDefault(key, new AtomicLong()).get();
double avgMs = count > 0 ? (totalNs / 1_000_000.0) / count : 0;
System.out.printf("Key: %s | Count: %d | Avg: %.2f ms%n",
key, count, avgMs);
});
}

/**
* Resets the statistics. Call it after each print to avoid data accumulation.
*/
public static void reset() {
KEY_COUNT.clear();
KEY_TIME.clear();
}
}

Comparison of Troubleshooting Tools

The three methods above differ in coverage, cost, and accuracy. The following table summarizes their applicable scenarios and limitations.
Methodology
Application scenarios
Strength
Disadvantages
DBbrain Console
Daily Ops and routine troubleshooting
Visualization, historical data analysis support, and no business code modification required.
Requires console access permission
redis-cli --bigkeys
Quickly self-check big keys in the command-line environment
No additional deployment required, ready to use out of the box.
Counts only the number of elements and does not reflect memory usage.
redis-cli --hotkeys
Quickly self-check hot keys in the command-line environment
No additional deployment required, ready to use out of the box.
Depends on the LFU eviction policy and requires adjustment of instance parameters.
Business-Layer Instrumentation
Precise locating and long-term continuous monitoring
Statistical dimensions are customizable and can be associated with business contexts.
Requires code modification and introduces minor performance overhead.
In summary, use DBbrain as the priority for routine inspection because it covers both big keys and hot keys without requiring business code changes. Use command-line self-checks for temporary emergencies or when you do not have console access. Introduce business-layer instrumentation when you need to monitor hot Key distribution over the long term or perform analysis associated with business dimensions.

Solutions

Big Key Optimization

Solution 1: Gradually Cleaning Up Invalid Data

This applies to scenarios where a large number of expired or invalid members have accumulated in collection types such as List, Set, and Hash. Avoid using the DEL command directly, which blocks the main thread. Instead, use progressive cleanup:
# Hash: Use HSCAN + HDEL to progressively delete expired members, looping until cleanup is complete.
HSCAN key cursor COUNT 100
HDEL key field1 field2 ...

# List: Use LTRIM to trim in batches, keeping only the first 1,000 entries.
LTRIM key 0 999

# Redis 4.0+: Use UNLINK to asynchronously delete the entire big Key to avoid blocking.
UNLINK large_key

Solution 2: Compressing the Value of a Big Key

For compressible text data (JSON, XML, or large blocks of text), compress it before writing and decompress it after reading:
// Write: Store the data in binary format after compression.
// You can implement gzipCompress / gzipDecompress based on java.util.zip.GZIPOutputStream.
byte[] compressed = gzipCompress(rawJsonString);
jedis.set(key.getBytes(), compressed);

// Read: Retrieve the binary data and then decompress it to restore the original content.
byte[] data = jedis.get(key.getBytes());
String rawJson = gzipDecompress(data);
Before adopting this solution, evaluate the costs in three aspects: compression consumes additional CPU, and its effectiveness depends on data redundancy. JSON and XML texts with repetitive field structures benefit significantly, while already-compressed binary data such as images and videos yields almost no benefit. After compression, the Value becomes binary data, requiring read/write interfaces in the form of byte arrays. You can no longer view the content directly through Redis commands, which increases troubleshooting difficulty. If the compressed Value still exceeds the threshold, the data scale has exceeded the reasonable capacity of a single Key and must be handled together with a splitting solution.

Solution 3: Splitting Big Keys

Split a single big Key into multiple small keys and store them in shards by business dimension:
# Before Splitting
user:profile:1001 → 2MB JSON

# After Splitting (Grouped by Field)
user:profile:1001:basic → Basic information
user:profile:1001:extend → Extended information
user:profile:1001:settings → Settings information
# Hash Splitting: Apply Hash modulo to the Field and distribute it across N sub-Hashes.
# Shard ID = hash(field) % N. Both writes and reads use the same rule for locating the shard.
user:cart:1001:shard_0 → Stores fields whose modulo result is 0.
user:cart:1001:shard_1 → Stores fields whose modulo result is 1.
...
The splitting granularity must be designed based on business access patterns to prevent a single business operation from requiring multiple network round trips after splitting, which would increase latency.
In cluster architecture, hash slot distribution must also be accounted for: multiple sub-keys generated by splitting may land on different shards due to different hash slots, and multi-key commands such as MGET may not be able to execute across slots. If batch reads are required, use hash tags to pin sub-keys to the same shard, for example, {user:profile:1001}:basic and {user:profile:1001}:extend. As long as the content within the curly braces is the same, the keys are guaranteed to fall into the same slot. However, note that hash tags will concentrate this group of keys on a single shard. If the key is also a hot key, access skew will be exacerbated. Therefore, the need for batch reads and the need for shard distribution should be weighed based on the actual scenario.

Solution 4: Transferring Inapplicable Data

If the String type is used to store large files such as images, videos, and BLOBs, store the data in Cloud Object Storage (COS) instead and keep only the access URL or metadata in Redis.
# Before Offloading
file:report:2025-q4 → 15MB PDF BLOB

# After Offloading
file:report:2025-q4:url → "https://bucket.cos.region.myqcloud.com/report-2025-q4.pdf"
file:report:2025-q4:meta → {"size": 15728640, "md5": "abc123..."}

Hot Key Optimization

Solution 1: Read/Write Separation

If hot Key pressure comes from read requests, enable read/write separation to offload read requests to read-only replicas by adding read-only nodes. This approach is suitable for read-heavy scenarios. Note that data on read-only replicas is obtained through master-replica synchronization and has some latency. Read requests that require extremely high data consistency should be routed to the master node. For detailed operations, see Read/Write Separation.

Solution 2: Local Cache

Use a local cache at the application layer, such as Caffeine or Guava, to cache hot data. Most requests are then returned directly within the application process without accessing Redis.
// Cache hot data locally with Caffeine
private static final LoadingCache<String, String> HOT_KEY_CACHE = Caffeine.newBuilder()
.maximumSize(1000) // Maximum number of cached entries
.expireAfterWrite(50, TimeUnit.MILLISECONDS) // Expiration time, which controls the data inconsistency window
.build(key -> {
// On a cache miss, fetch from Redis and obtain a connection from the connection pool.
try (Jedis jedis = JEDIS_POOL.getResource()) {
return jedis.get(key);
}
});
This approach is suitable for read-heavy hot keys when the business can tolerate brief data inconsistency. The expiration time determines the length of the data inconsistency window and must be set based on the business's tolerance for inconsistency: a shorter time provides better consistency, but it also increases the frequency of fetching from Redis and weakens the pressure reduction effect. The actual value must be balanced between these two factors, and the 50 ms in the preceding code is only an example.

Solution 3: Multi-Replica Distribution

Replicate a hot Key into multiple identical copies, distribute them across different shards, and randomly select one copy for reads:
# Original Hot Key
hot:product:1001

# Split into Multiple Replicas
hot:product:1001:0 → replica 0
hot:product:1001:1 → replica 1
hot:product:1001:2 → replica 2
// During reads, randomly select a replica to distribute requests across different shards.
int index = ThreadLocalRandom.current().nextInt(REPLICA_COUNT);
String key = "hot:product:" + productId + ":" + index;
try (Jedis jedis = JEDIS_POOL.getResource()) {
String value = jedis.get(key);
}
Note:
When data is updated, it must be written to all replicas simultaneously. A write failure on any replica will cause inconsistent read results. Because the read/write logic and consistency guarantees for multiple replicas must be maintained by the application layer, this approach is recommended only as a temporary emergency measure for sudden hot spots, rather than as a long-term architecture.

Solution 4: Multi-Level Cache Architecture

Build a multi-level caching system for ultra-high concurrency scenarios:
User request → Nginx local cache → application local cache (Caffeine) → Redis → database
The vast majority of hot requests are absorbed by the first two cache layers, so Redis only handles a small amount of origin traffic. The expiration time of each cache layer should decrease progressively, with layers closer to the user having shorter expiration times. This reduces backend pressure while limiting the scope of data inconsistency.

Precautions

The best way to manage big keys and hot keys is prevention in advance, which means avoiding risks during the business design phase:

Design Phase

1. Key splitting design: Before launch, evaluate the data size and growth rate, and establish a sharding policy in advance for keys that may grow into large collections.
2. Expiration time setting: All keys must have a reasonable expiration time to prevent unlimited data accumulation.
3. Data structure selection: Select an appropriate data type based on data characteristics:
Description information that requires per-field expiration → Hash + scheduled cleanup
Key values that need to be ordered and evenly distributed → ZSet (use paginated reads instead of ZRANGE 0 -1)
Relational description information that requires a large number of members → Consider splitting it into multiple small Sets.

Runtime Phase

1. Monitoring and alarms: Configure alarms for metrics such as memory utilization, bandwidth utilization, total number of keys, and CPU utilization. To set thresholds, collect metric baselines during stable business periods, and then determine alarm thresholds by reserving sufficient processing time above the baselines. For example, the memory utilization alarm threshold should ensure enough time to complete capacity expansion or cleanup between when an alarm is triggered and when the upper limit is reached. After receiving an alarm, handle it according to the troubleshooting process in this document. For detailed operations, see Monitoring (5-second granularity) and Configuring Alarms.
2. Regular inspection: Regularly check for big keys and hot keys through DBbrain or command-line tools, and intervene before issues affect your business.
3. Capacity planning: Regularly evaluate business growth trends, and perform capacity expansion or splitting in advance.

Coding Standards

1. Do not use full-traversal commands such as KEYS *, SMEMBERS, and HGETALL. Instead, use SCAN, SSCAN, and HSCAN to obtain data in batches.
2. For batch operations, prioritize using Pipeline or MGET/MSET instead of looping single commands.
3. To delete a big Key, use UNLINK for asynchronous deletion (Redis 4.0+).

ヘルプとサポート

この記事はお役に立ちましたか?

フィードバック