Observability comprehensively understands the system's health through the combination of Metrics, Logs and Traces. 1. Metrics provides quantitative data snapshots, such as CPU usage, which is suitable for real-time monitoring but cannot locate specific problems; 2. Logs records detailed event information, which is suitable for troubleshooting specific errors but large data volume; 3. Traces tracks request links and locates performance bottlenecks, requiring unique identification and context delivery. They ensure system transparency from three levels: overall state, specific details and process path.

The "observability" of the monitoring system sounds quite technical, but to put it bluntly, we want to figure out what the system is doing. It has three core pillars: Metrics, Logs, and Traces. These three have their own emphasis, and put together can give us a more comprehensive understanding of the system's health.

Metrics is a "snapshot" of data

Metrics are essentially some data points that can be quantified and change over time. For example, CPU usage, memory usage, request success rate, etc. They are usually aggregated data, allowing you to quickly see the overall state of the system.

• Advantages : Lightweight, strong real-time, suitable for alarm and trend-seeking.
• Limitations : Too broad, it only tells you "there is a problem", but it doesn't tell you "whether there is a problem".

For example, you find that the CPU of the server suddenly reaches more than 90%. At this time, you know that something is wrong, but which service or request is caused by it? Metrics can't answer this question by itself.

Common practices:

• Record by time series
• Use tools like Prometheus and Grafana to display
• Set thresholds for automatic alarm

Logs are the "recordor" of details

Logs is what we often call logs, which record everything that happens in the system, such as error messages, user operations, API request details, etc. It's more specific than Metrics and can see what's actually happening.

• Advantages : Rich information and suitable for troubleshooting specific problems.
• Limitations : Large data volume, high retrieval cost, and is not suitable for real-time monitoring.

For example, if you see an interface returning 500 errors, you can check the corresponding log to see if a database connection failed or a parameter parsing problem occurs.

Recommended usage:

• Structured logs are better processed, such as JSON format
• Cooperate with ELK (Elasticsearch, Logstash, Kibana) or Loki for centralized management
• Don't output too much useless information to avoid log explosion

Traces is the "map" of the call chain

Traces focuses on a request flow path throughout the system. Especially under the microservice architecture, a request may involve calls between multiple services. Traces can help you clearly see this process and find out the source of performance bottlenecks or failures.

• Advantages : Helps understand the request process, locate slow service or failed links
• Limitations : It is slightly more complex to implement and requires a context propagation mechanism

For example, if you call an API, the response will be very slow. Through Trace tools (such as Jaeger or OpenTelemetry), you can see the entire request link and find that it turns out that a downstream service has spent a lot of time processing data.

Key points:

• Each request must have a unique identity (trace ID)
• Transfer context information between services
• Support sampling control to avoid the pressure caused by full collection

These three things combine to be like installing the system with "eyes", "ears" and "brain". Metrics tells you if there is any problem in the system, Logs helps you find out where the problem is, and Traces tells you how the problem occurs.

Basically that's it.

The above is the detailed content of What are the three pillars of observability (metrics, logs, traces)?. For more information, please follow other related articles on the PHP Chinese website!