Cost-Saving Recommendations
If saving costs for Cloud Log Service (CLS), prioritize considering billing items with higher costs. Usually, these are index traffic and index storage, which generally cost several times more than write traffic and log storage. This is mainly because they are billed based on the volume of uncompressed log data. If LogListener is used to upload logs, data compression occurs, and write traffic and log storage are billed based on the volume of compressed log data, with a typical log compression ratio of 1:4 to 1:10. You can adjust the log topic configuration according to your needs in the following ways to achieve cost savings.
Reduce log upload data volume: Most billing items, including index traffic and index storage, are related to log upload data volume. Reducing log upload data volume can save costs from the source. You can appropriately reduce some unnecessary log collection, or use the filter in LogListener Collection Configuration to only collect logs that meet the filter rules, such as only collecting logs containing Error and Warning-level entries. Use STANDARD_IA storage: STANDARD_IA storage addresses the retrieval and storage of mass STANDARD_IA logs at low costs. It is suitable for scenarios where users have no statistical analysis requirements for logs and the log retention time is longer than expected, saving about 80% of the cost. For detailed information, see Storage Class Overview. Shorten the log retention period: The longer the log storage period, the higher the log storage fee and index storage fee. For earlier logs, they can also be transitioned to STANDARD_IA storage. If retrieval is not required at all, the log files can also be shipped to Cloud Object Storage (COS). Simplify index configuration: Simplifying index configuration can reduce index traffic and index storage fees. For details, see Simplifying Index Configuration. Use data processing to distribute log topics: Data processing can ship raw logs to different log topics. For example, classify logs by log level, including ERROR, WARNING, and INFO, and then distribute them to different log topics. You can create a source log topic to receive all log data (this log topic can disable indexing to avoid index traffic and index storage fees, reducing costs). Then, use data processing to ship raw logs to log topics with different retention periods, index configurations, or storage types according to subsequent log retrieval and analysis requirements, better balancing usage needs and costs. For detailed information, see Creating Processing Task. Simplifying Index Configuration
Index configuration is a necessary condition for retrieving and analyzing logs. Only if the indexing is enabled can logs be retrieved and analyzed. Different indexing rules will lead to different outcomes in log retrieval and analysis and incur different fees. 1. If you want to reduce index traffic and index storage fees, you need to disable the full-text index first. If the full-text index is enabled, index traffic will be measured based on the full text of a raw log, and the number of key-value indexes will not affect index traffic and index storage.
Note:
After the full-text index is disabled, you will not be able to directly use keywords to perform full-text retrieval on logs. You can only use key-value retrieval. For example, search for logs with request_id "aaaa-bbbb-cccc-dddd":
[Wrong] Full-text retrieval query statement: "aaaa-bbbb-cccc-dddd". The query failed because no field is specified for the query.
[Correct] Key-value retrieval query statement: request_id:"aaaa-bbbb-cccc-dddd". Query succeeded because the field request_id is specified for the query.
2. After disabling the full-text index, you can reduce the number of key-value index fields. Key-value index traffic is measured based on the length of fields with key-value indexes enabled. You can prioritize deleting fields with no retrieval needs and larger field lengths.
Note:
Disable Automatic Configuration in the key-value index to prevent deleted fields from being automatically added to the key-value index again.
Fields that do not exist in the key-value index do not support key-value retrieval. Since the full-text index has been disabled in the first step, such fields will not support retrieval in any way and can only be displayed.