1. Check whether LogListener is installed on the machine corresponding to the IP address. For installation instructions, see LogListener Installation and Deployment.
3. Check whether the IP address added to the machine group matches the IP address identified by LogListener. You can view the IP address detected by LogListener by executing the following code.
Linux
Windows
/etc/init.d/loglistenerd check
In the returned data, you can confirm the IP address identified by LogListener by viewing group ip, as shown in the following figure:
Using the installation path C:\\Program Files (x86)\\Tencent\\LogListener as an example, run Windows PowerShell as an administrator. Then, in the installation path, run the following command to check the heartbeat and configuration of LogListener:
.\\loglistener_work.exe check
In the returned data, you can confirm the IP address identified by LogListener by viewing group_id, as shown in the following figure:
3. Check whether the machine id of LogListener matches the machine id configured in the machine group. You can refer to the following steps to view the machine id of LogListener.
Linux
Windows
Using the installation path /usr/local/ as an example, you can run the following command in the /usr/local/loglistener/etc directory to view and configure the LogListener configuration file.
vim loglistener.conf
Here, `group_label` is the machine id of LogListener. Ensure that this id matches the one configured in the machine group.
Using the installation path C:\\Program Files (x86)\\Tencent\\LogListener as an example, locate the LogListener configuration file `loglistener.yaml` in the installation directory and open it with any editor, as shown in the following figure:
Here, `agent_label` is defined as the machine id of LogListener. Ensure that this id matches the one configured in the machine group.
Machine Group: Abnormal Machine Status
After completing the machine group configuration, you can view the machine heartbeat associated with that machine group by clicking the machine group name on the Machine Group Management page. A normal status indicates that the machine heartbeat is being reported normally. If the status is not normal (as shown in the following figure), it indicates that LogListener on that machine is abnormal.
Abnormal heartbeat
When the status is abnormal heartbeat, it means that the connection between LogListener and Cloud Log Service backend is interrupted, which will cause LogListener to fail to upload logs normally. You can further troubleshoot the cause of connection interruption by the following methods:
The LogListener quick diagnostic tool can quickly diagnose whether LogListener is started, whether the heartbeat is normal, and whether configuration pulling is normal.
Execute the following command in the command line:
/etc/init.d/loglistenerd check
If LogListener is running normally, the results returned by the diagnostic tool are as follows:
Abnormal LogListener process
If [ERROR] loglistener is not running as shown in the following figure appears, it means that LogListener has not been started. Execute /etc/init.d/loglistenerd start to start. For more operating instructions, see Common Operating Instructions of LogListener.
Abnormal LogListener heartbeat
If [ERROR] check loglistener heareat fail is displayed, the heartbeat of LogListener is abnormal.
There are many causes of abnormal heartbeats in LogListener, and the most common ones include:
Network is abnormal. Run the following command to check whether the network environment is connected. The "cls domain name" in the command is the CLS service domain name. See the Available Regions document to fill it in.
telnet <cls domain name>80
The secret key information is incorrect. Go to the LogListener installation directory and run ./bin/check. If the output at the end is as follows, it indicates that the secret key information is incorrect.
The secret key lacks permissions. Go to the LogListener installation directory and run ./bin/check. If the output at the end is as follows, it indicates that the secret key lacks permissions. Configure the reporting permissions according to the Policy Template.
2. Check for the IP address of the machine group
Check whether the IP address added to the machine group matches the IP address obtained during LogListener installation. Go to the LogListener installation directory and run the following command to check the IP address configured for LogListener.
Note:
Unless otherwise specified, the installation directory of LogListener is usually /usr/local/loglistener.
grep group_ip etc/loglistener.conf
Log in to the CLS console, click Machine Group Management, and check the IP address configured for the machine group. The IP address configured for the machine group must exactly match the IP address obtained by LogListener.
Directory overflow
When the status is directory overrun, it means that the number of collection directories monitored by LogListener has exceeded the specification limit of 5000, which will result in logs from the excessive directory of LogListener cannot be collected. You can repair it in the following ways:
1. Run the following command to view the collection configuration associated with the current LogListener.
/etc/init.d/loglistenerd check
2. In the above return results, the logconfig array contains all collection configurations of the current LogListener. For each collection configuration, you can check whether wildpath may fuzzily hit a large number of directories that do not need to be collected, such as /root/**/error.log, which is a fuzzy path from the root directory.
3. After the target collection configuration that can be optimized is found, view the topicid associated with the collection configuration from the returned data in Step 1, and go to the CLS console to find the target log topic based on this topicid, optimize or delete the collection configuration.
File overflow
When the status is file overrun, it means that the number of collection files monitored by LogListener has exceeded the specification limit of 10000, which will result in LogListener being unable to collect logs from files beyond this limit. You can repair it in the following ways:
1. Run the following command to view the collection configuration associated with the current LogListener.
/etc/init.d/loglistenerd check
2. In the above return results, the logconfig array contains all collection configurations of the current LogListener. For each collection configuration, you can check whether wildpath may fuzzily hit a large number of files that do not need to be collected, such as /root/**/*.log, which fuzzily matches log files with any name from the root directory.
3. After the target collection configuration that can be optimized is found, check the topicid associated with the collection configuration from the returned data in Step 1, and go to the CLS console to find the target log topic based on this topicid, optimize or delete the collection configuration.
CPU overflow
When the status is CPU overrun, it means that the CPU usage of LogListener has exceeded the preset threshold. You can troubleshoot and repair it in the following ways:
Optimizing the Target Collection Directory
1. Run the following command to view the collection configuration associated with the current LogListener.
/etc/init.d/loglistenerd check
2. In the above return results, the logconfig array contains all collection configurations of the current LogListener. For each collection configuration, you can check whether wildpath may fuzzily hit a large number of files that do not need to be collected, such as /root/**/*.log, which fuzzily matches log files with any name from the root directory.
3. After the target collection configuration that can be optimized is found, check the topicid associated with the collection configuration from the returned data in Step 1, and go to the CLS console to find the target log topic based on this topicid, optimize or delete the collection configuration.
Optimizing Regular Expressions
1. Run the following command to view the collection configuration associated with the current LogListener.
/etc/init.d/loglistenerd check
2. In the above return results, the logconfig array contains all collection configurations of the current LogListener. For each collection configuration that has a fullregex or multi_fullregex field, you can check whether the regular expression is overly complex.
3. After the target collection configuration that can be optimized is found, check the topicid associated with the collection configuration from the returned data in Step 1, and go to the CLS console to find the target log topic based on this topicid, optimize or delete the collection configuration.
Increasing the CPU Usage Limit for LogListener
1. Using the LogListener installation directory /usr/local/loglistener as an example, run the following command to open the LogListener configuration file:
vim etc/loglistener.conf
2. In the LogListener configuration file, find cpu_usage_thres and increase the CPU usage limit as needed, as shown below:
3. After completing the configuration, save the file and restart LogListener.
Increasing CPU Resource Quotas
In a K8s environment, ensure that the CPU quota for the Pod: loglistener under the DaemonSet: tke-log-agent is sufficient.
Out of Memory
When the status is memory overrun, it indicates that the memory usage of LogListener has exceeded the preset threshold. You can repair it in the following ways:
1. Using the LogListener installation directory /usr/local/loglistener as an example, run the following command to open the LogListener configuration file.
vim etc/loglistener.conf
2. In the LogListener configuration file, find max_mem and increase the memory usage limit as required, as shown below:
3. After completing the configuration, save the file and restart LogListener.
Authentication Failure
Verifying Key Validity
When the status is authentication failure, it indicates that the secret key configured in LogListener is incorrect or has expired. You can repair it in the following ways:
1. Using the LogListener installation directory /usr/local/loglistener as an example, run the following command to open the LogListener configuration file.
vim etc/loglistener.conf
2. In the LogListener configuration file, locate secret_id and secret_key and verify whether they are correct and valid. If they are invalid, modify them with the correct secret key information, as shown below:
3. After completing the configuration, save the file and restart LogListener.
Server Time Correctness
Ensure that the server time is accurate; otherwise, authentication may fail.
