Basic Concepts
Search and analysis support filtering, searching and statistically analyzing log data, such as querying logs containing error, counting the number of logs by URL grouping, calculating PV change trend, etc. It is the most commonly used function in the Cloud Log Service.
Search and analysis are based on Word Segmentation and Indexing. If you are unfamiliar with the two concepts, it is recommended that you consult this document in advance. Features
Basic Features
Full-text search: Search the full text of logs. The log can be found when any field in the log meets the search conditions, such as using error to search for all logs with errors.
Key-value search: Search for the specified field in the log, and the log will be found when the specified field meets the search criteria. For example, use level:error AND timeCost:>1000 to search for logs whose level is error and that consume time (timeCost) greater than 1,000 ms.
Statistical analysis: You can perform statistical analysis on the logs meeting specified search criteria, and returns the results. For example, you can enter status:404 | select count(*) as logCounts to get the number of logs whose response status code is 404.
The above functions require the use of search and analysis statements. See Syntax Rules for details. Note:
Infrequent storage only supports full-text search, not key-value retrieval or statistical analysis. Advanced Features
Log Search
Multi-topic retrieval: Retrieve multiple log topics simultaneously, but statistical analysis is not supported. It can span log sets but not regions. Contextual search: Retrieve several logs before (previous logs) or after (subsequent logs) the target log data in the original file. By viewing the context of specified logs, you can quickly find fault information during troubleshooting, making it easier to locate issues. Custom redirect: Click the log field value to perform a custom redirect, facilitating further operations, such as querying user information on the internal user management platform based on user_id. Download logs: Download logs that meet the search criteria to the local machine, with a maximum of 50 million logs per download. Log statistical analysis
Rapid analysis: Click the log field name to quickly analyze the distribution of field values, trends over time, and numerical statistics. Sampling analysis: When using the statistical analysis feature, if the number of original logs is very large or the statistical analysis statement (SQL) is complex, the analysis may be slow or even timeout. In this case, you can use the sampling analysis feature to randomly sample the original logs and then perform statistical analysis. Associate external data: When performing statistical analysis on logs, you can associate the logs with external data for analysis. For example, you can query the user gender in the user information database based on the user ID in the logs, and then perform statistical analysis on the logs by gender.
Advantages
Based on pre-built index data, the search and analysis performance is much higher than that of queries with general text scanning methods (such as Linux grep commands and query functions in text editor), making the product suitable for real-time query of massive log data.
Statistical analysis supports SQL and conforms to the SQL-92 specification. In addition, flexible SQL syntax and 400+ SQL functions can be used for log statistics, meeting most statistical analysis requirements.
Fee Description
Search and analysis itself do not incur fees, but since it relies on reporting logs to the CLS platform and enabling indexing, it will generate log and index traffic fees, storage fees, and topic partition fees. Please see Billing Overview. Additionally, using the API to call the search and analysis interface will incur service request fees, and using the log download feature will incur public network read traffic fees. Specifications and Limits