To search for a long log, you usually only need to search for part of the log content. Assume that you need to search for the following complete log that contains sample:
The full text of the log is long, and the log contains many other content in addition to sample. The complete log and the search condition are not identical. Therefore, sample cannot be used as the search condition to search for the log. To meet this search requirement, the full text of the log needs to be divided into multiple fragments, and each fragment is called a word, and this process is called "segmentation".
For example, the example log mentioned above would be split using symbols as delimiters. Whenever any symbol within the range @&()='",;:<>[]{}/ \\n\\t\\r\\\\ appears, the log is split, resulting in the following tokens:
If the search condition is sample, the preceding log after segmentation contains sample and therefore is considered to meet the search condition.
The search conditions themselves are also tokenized, as shown by the following two search conditions:
Search condition A: "/online/sample".
Double quotation marks indicate that the log needs to contain both tokens simultaneously, and the two tokens should be in the exact sequence to meet the search conditions.
The example log mentioned above contains both online and sample, and the tokens appear in the exact order specified by the search condition; therefore, it matches the condition.
Search condition B: "/online/login"
The example log mentioned above does not contain login, and does not meet the search condition.
Segmentation Configuration
Log segmentation is determined based on two types of settings, which can be set in Index Configuration.
Delimiter: You can customize the symbols for log segmentation, including English symbols and \\n\\t\\r\\\\. In the example above, @&()='",;:<>[]{}/ \\n\\t\\r\\\\ are delimiters.
Allow Chinese Characters: Chinese is special and Chinese symbols cannot be used as delimiters. In addition, segmentation according to symbols often fails to achieve the desired effect. For example, if a log in Chinese is 用户登录失败,密码错误 and you want the log to be searched for by 登录失败, segmentation according to symbols cannot be used to meet the search requirement. In this case, you can set the log to Allow Chinese Characters by referring to Configuring Indexes so that CLS will automatically segment every Chinese character or Chinese symbol into a separate word.
Indexes
After segmentation, CLS uses an inverted index to store the position of each word in the log file, allowing users to quickly search for specific raw logs by specifying search conditions. Inverted indexes are the most commonly used data structures in file search systems.
Indexes determine the conditions under which logs can be searched and analyzed. Search and analysis operations can be performed only after indexes are enabled. Before uploading log data, you need to set a reasonable index rule for the corresponding log topic to facilitate subsequent search and analysis. For the configuration method, refer to Index Configuration.
