With the growing popularity of microservices architecture, accurately locating the complete lifecycle of a request in a complex and entropic system has become one of the biggest pain points for R&D teams. Taking the self-testing process of R&D as an example, developers often want to obtain the contextual information of the entire test request in a simple way after initiating a Http/RPC request. These typically include the associated upstream and downstream traces, the method call stacks within each service for a given request, and the logs printed along the trace. The need for unified monitoring of metrics, traces, and logs is therefore becoming increasingly urgent.
Tencent Cloud Application Performance Monitoring (APM), a full-trace monitoring platform that inherently includes metrics-traces, has been integrated with Tencent Cloud Log Service (CLS) to achieve integrated monitoring of metrics-traces-logs. When viewing trace details, you can view the corresponding information to complete the troubleshooting process from traces to logs. Prerequisites
Make sure that the application has been connected to Application Performance Monitoring (APM), and refer to the CLS Getting Started Guide to report business logs to Tencent Cloud Log Service (CLS). Note:
To ensure the reliability of your log data and use the log service more efficiently, it is recommended that you use the CLS - optimized API/Agent to upload structured logs.
Submit via API/SDK. For details, see the contents section in Uploading Logs. Operation Steps
Step 1: Printing TraceID and SpanID to Logs
To enable correlated queries between traces and logs, you need to output the TraceID and SpanID to the logs. For applications accessing APM through the following solutions, the TraceID and SpanID can be automatically injected into the logs without modifying the code:
For applications accessing APM in other ways, please refer to the corresponding access documentation, obtain the TraceID in the code, and output the TraceID in the business log. Step 2: Associating Logs on the System Configuration Page
2. In the left-side menu bar, choose Application Performance Monitoring > System Configuration, and go to the Workspace Configuration page.
3. Select the corresponding business system, and in the Log Association section, click Edit.
4. Configure log correlation information.
Enable Association Logs.
Select the corresponding Log Region, Log Set, and Log Topic, and specify the fields containing TraceID and SpanID.
5. After configuration is complete, click OK.
Step 3: Viewing Log Information on the Trace Query Page
2. In the left-side menu bar, choose Application Performance Monitoring > Distributed Tracing, and go to the Trace Search page.
3. Select the corresponding business system, locate the corresponding TraceID, and click the TraceID to go to the trace details.
4. In the Trace Logs and Span Logs tabs, you can query the log information corresponding to this trace.
Print TraceID and SpanID to logs
Tencent Cloud Enhanced OpenTelemetry Java Agent
For log components that meet the requirements, you can inject TraceID and SpanID by modifying the pattern in the log configuration file. In the log configuration file, TraceID and SpanID are injected using the %X{trace_id} and %X{span_id} fields respectively.
|
Log4j | 1.2+ |
Log4j2 | 2.7+ |
Logback | 1.0+ |
Log4j
log4j.properties file modification example:
log4j.appender.warn.layout=org.apache.log4j.PatternLayout
log4j.appender.warn.layout.ConversionPattern=trace_id: %X{trace_id} span_id: %X{span_id} - %m%n
Log4j2
Log4j2 supports configuration files in multiple formats, including XML, JSON, YAML, and Properties. Taking the XML format as an example, there are multiple ways to write the Pattern. Please refer to one of the existing patterns:
<PatternLayout pattern="trace_id: %X{trace_id} span_id: %X{span_id} - %m%n"/>
<PatternLayout>
<Pattern>trace_id: %X{trace_id} span_id: %X{span_id} - %m%n</Pattern>
</PatternLayout>
<Layout type="PatternLayout" pattern="trace_id: %X{trace_id} span_id: %X{span_id} - %m%n"/>
Logback
logback.xml file modification example:
<encoder>
<pattern>trace_id: %X{trace_id} span_id: %X{span_id} %msg%n</pattern>
</encoder>
SkyWalking Java Agent
For instructions on injecting TraceID and SpanID into logs, refer to the SkyWalking official website based on your log component type.
Tencent Cloud OpenTelemetry-Go Agent
For log components that meet the requirements, TraceID and SpanID can be injected without any additional operations. If the probe detects the presence of an active Span, it will inject TraceID and SpanID into the logs using trace_id and span_id as markers respectively.
|
Zap | go.uber.org/zap/zapcore | 1.20.0+ |
Logrus | github.com/sirupsen/logrus | 1.5.0+ |
Zerolog | github.com/rs/zerolog | 1.10.0+ |
go-kit/log | github.com/go-kit/log | 0.1.0+ |
Go standard library log | log | No limit |
Go standard library slog | log/slog | No limit |
Taking Zap as an example, the generated log content after TraceID and SpanID are injected is as shown below:
{"level":"info","msg":"this is info message","trace_id":"38e6910973e1c0c11351e502329d2d72","span_id":"6f4efff78445ea61"}
OpenTelemetry-Python
For the built-in logging library and log components that make underlying calls to the built-in logging library, you can achieve the injection of TraceID and SpanID by adding the OTEL_PYTHON_LOG_CORRELATION environment variable.
OTEL_PYTHON_LOG_CORRELATION=true
tapm-instrument \\
--traces_exporter otlp \\
--metrics_exporter otlp \\
--logs_exporter none \\
--service_name python-web-test \\
--resource_attributes "token=xxxxx" \\
--exporter_otlp_endpoint xxxxxx \\
python main.py
After log association is enabled, the generated log content is as shown below:
2026-02-25 19:51:25,069 INFO [__main__] [main.py:19] [trace_id=c20f528fa45eebc2cbdae9e341ced39a span_id=fa84ffb16a978ab6 resource.service.name=python-web-test trace_sampled=True]
OpenTelemetry-JS
For log components that meet the requirements, TraceID and SpanID can be injected without any additional operations. If the probe detects the presence of an active Span, it will inject TraceID and SpanID into the logs using trace_id and span_id as markers respectively.
|
Winston | github.com/winstonjs/winston | 1.0.0+ |
Pino | github.com/pinojs/pino | 5.14.0+ |
Bunyan | trentm/node-bunyan | 1.0.0+ |
When we take Bunyan as an example, the generated log content after injecting TraceID and SpanID is as shown below:
INFO: this is info message: GET / | trace_id="e793e3edafb93f98a8a395c13096a26f" span_id="9564de488477c6a8" trace_flags="01"
Please note that if you customize the log formatting in the Winston framework, you need to refer to the following code to manually append TraceID and SpanID to the log content:
const logger = winston.createLogger({
level: "info",
format: winston.format.combine(
winston.format.timestamp(),
winston.format.printf(({ timestamp, level, message, trace_id, span_id }) => {
return `${timestamp} [${level.toUpperCase()}] [trace_id=${trace_id} span_id=${span_id}]: ${message}`;
})
),
transports: [new winston.transports.Console()],
});
OpenTelemetry-dotnet
For log components that meet the requirements, the injection of TraceID and SpanID can be achieved by modifying the pattern in the log configuration file.
Log4Net
In the log configuration file, TraceID and SpanID are injected using the %property{trace_id} and %property{span_id} fields respectively.
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%level] %logger - %message trace_id=%property{trace_id} span_id=%property{span_id} %newline" />
</layout>
NLog
In the log configuration file, TraceID and SpanID are injected using the ${event-properties:TraceId} and span_id=${event-properties:SpanId} fields respectively.
<target xsi:type="File" name="file" fileName="logs/app.log"
layout="${longdate} [${level:uppercase=true}] ${logger} - ${message} trace_id=${event-properties:TraceId} span_id=${event-properties:SpanId}" />
Other solutions
For other solutions, use the OpenTelemetry SDK to obtain TraceID and SpanID from the context of the trace and append them to the log content. For example, for applications integrated via the OpenTelemetry-Go SDK, you can refer to the following code snippet to print TraceID and SpanID to the logs:
func logWithTrace(ctx context.Context, format string, args ...interface{}) {
span := trace.SpanFromContext(ctx)
sc := span.SpanContext()
traceID := sc.TraceID().String()
spanID := sc.SpanID().String()
msg := fmt.Sprintf(format, args...)
log.Printf("[trace_id: %s span_id: %s] %s", traceID, spanID, msg)
}