Release Notes
Announcements
Subscription Relationship Consistency
This document mainly introduces the core concepts of subscription relationship consistency. It covers the definition and constraint mechanisms, delves into the underlying implementation principles and optimization practices, and combines real-world cases to present the solutions of TDMQ for RocketMQ for subscription relationship inconsistency issues. This helps developers quickly identify the root cause of problems and build stable, reliable messaging systems.
Subscription Relationship Definition
A subscription relationship refers to the rules and status configurations that govern how consumers obtain and process messages in the RocketMQ system. This subscription relationship is dynamically registered by a consumer group with the server. During subsequent message transmission, messages are matched, and consumption progress is maintained according to the filtering rules defined by this subscription relationship.
By configuring subscription relationships, you can control the following consumption behaviors:
Message filtering rules: These rules determine which messages within a topic the consumers select for consumption. Setting consumption filtering rules efficiently filters the set of messages needed by the consumers, allowing flexible adjustment of the message receiving scope based on different business scenarios.
Consumption status: The RocketMQ server provides subscription relationship persistence by default. This means that after a consumer group registers its subscription relationship with the server, if a consumer goes offline and later comes back online, it can retrieve the consumption progress from before it went offline and resume consumption.
In the domain model of RocketMQ, the position and workflow of subscription relationships are as follows:
1. Messages are initialized by producers and sent to the RocketMQ server.
2. Messages are stored in specified queues of a topic in the order they arrive at the RocketMQ server.
3. Consumers obtain and consume messages from the RocketMQ server according to the specified subscription relationships.
Subscription Relationship Consistency Constraints
Subscription relationship consistency requires that all consumer instances within the same consumer group must subscribe to the exact same topics with the exact same filtering rules. This involves three specific constraints:
The consumer group must be consistent.
For most distributed applications, multiple consumer instances are typically attached to a single consumer group. The constraints of subscription relationship consistency apply to all consumers within the same consumer group.
The subscribed topics must be consistent.
All consumers within the same consumer group must subscribe to the same topics. For example, if consumer1 subscribes to TopicA and TopicB, consumer2 must also subscribe to TopicA and TopicB. It cannot subscribe only to TopicA, only to TopicB, or to TopicA and TopicC.
The filtering rules must be consistent.
All consumers within the same consumer group must have the same filtering rules, including both the number and the order of tags. For example, if consumer1 subscribes to TopicB with tags Tag1||Tag2, consumer2 subscribing to TopicB must also use Tag1||Tag2. It cannot use only Tag1, only Tag2, or Tag2||Tag1.
Examples of Consistent Subscription Relationships
The following diagram illustrates two common correct subscription relationships, corresponding to two scenarios:
Example 1: single-topic single-tag subscription
As shown in the diagram, both consumer1 and consumer2 in Group1 subscribe to all messages in TopicA.
Example 2: single-topic multiple-tag subscription
As shown in the diagram, both consumer1 and consumer2 in Group2 subscribe to messages in TopicA with the tag Tag1 or Tag2, and the order is consistently Tag1||Tag2.
Examples of Inconsistent Subscription Relationships
The following diagram illustrates three typical incorrect subscription relationships, corresponding to three scenarios:
Example 1: different subscribed topics
As shown in the diagram, consumer1 and consumer2 in Group1 subscribe to different topics.
Example 2: same topic with different tags
As shown in the diagram, consumer1 in Group2 subscribes to Tag1 of TopicA, while consumer2 subscribes to Tag2 of TopicA.
Example 3: same topic and tags, but different tag order
As shown in the diagram, consumer1 in Group3 subscribes to Tag1||Tag2 of TopicA, while consumer2 subscribes to Tag2||Tag1 of TopicA. Although the subscribed tags are the same, the different order violates the subscription consistency constraint.
Impact of Inconsistent Subscription Relationships
Inconsistent subscription relationships may lead to disordered message consumption logic, resulting in duplicate consumption or message loss.
In the following example, we start two consumers, both of which belong to consumer group Group1 and subscribe to TopicA. However, consumer1 subscribes to messages with Tag1, while consumer2 subscribes to messages with Tag2.
String topic = "TopicA";String consumerGroup = "Group1";FilterExpression filterExpressionTag1 = new FilterExpression("Tag1", FilterExpressionType.TAG);PushConsumer consumer1 = provider.newPushConsumerBuilder().setConsumerGroup(consumerGroup).setSubscriptionExpressions(Collections.singletonMap(topic, filterExpressionTag1)).build();FilterExpression filterExpressionTag2 = new FilterExpression("Tag2", FilterExpressionType.TAG);PushConsumer consumer2 = provider.newPushConsumerBuilder().setConsumerGroup(consumerGroup).setSubscriptionExpressions(Collections.singletonMap(topic, filterExpressionTag2)).build();
What will be the respective behaviors of the two clients in this scenario?
consumer1 will be unable to consume messages with Tag1. This is because when consumer1 pulls messages, the subscription information for its consumer group on the server side has a tag value of Tag2. After server-side filtering, all messages pulled by consumer1 will have Tag2. However, the consumer also performs local filtering upon receiving messages, and these messages will be filtered out as well.
consumer2 will only be able to consume part of the messages with Tag2, because only a portion of the queues is assigned to consumer2.
However, the subscription information of different consumer clients within the same consumer group on the server side will overwrite each other. This results in highly disordered consumption behavior. The consumption patterns of consumer1 and consumer2 in the above example may switch between each other.
Practices for Optimizations by Tencent Cloud
Inconsistent subscription relationships directly lead to abnormal message consumption and require quick identification and fixing. The TDMQ for RocketMQ console provides visual detection capabilities, eliminating the need for manual log or configuration checks. It enables you to discover, locate, and fix issues in 3 steps in the console, reducing Ops complexity.
1. One-click detection
Automatically compares the subscription configurations of all clients within a consumer group and highlights inconsistent subscription relationships.
2. Precise identification
When you click to view the inconsistency details, the associated client instances are displayed, allowing you to quickly identify instances with unexpected subscription relationships.
3. Closed-loop verification
After you make modifications, the status of subscription relationship consistency is synchronized in real time to ensure that the consumer group's subscription relationships meet expectations.
FAQs
What are typical scenarios that cause inconsistent subscription relationships?
The environment isolation is incomplete, and non-production and production environments use the same consumer group to subscribe to different topics.
The subscription relationships are modified in the business code, and old and new versions of consumers coexist during the grayscale release and version deployment process.
Usage Recommendations
Subscription relationship consistency is fundamental to ensuring correct message consumption behavior in RocketMQ messaging systems:
1. Core constraints: All consumers within the same consumer group should strictly adhere to the principles of consistent topics and filtering rules (including tag order). Any deviation in any part may lead to message loss.
2. Tencent Cloud capability: With the one-click detection, precise identification, and closed-loop verification features in the console, developers can quickly identify abnormal instances and resolve issues, reducing the time required for traditional manual troubleshooting from hours to minutes.
3. Best practice: It is recommended to adopt strategies such as consumer group isolation, strict verification of dynamic configurations, and hierarchical governance in multi-topic scenarios to mitigate the risk of inconsistent subscription relationships at the source.
Help and Support
Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback