[Feature] Tracing: Fine-Grained Tracing for Request Latency Part1

.gitignore

@@ -127,7 +127,7 @@ checkpoints/
 checkpoints_origin/
 result/
 result_lora/
-
+prfo/
 # npu kernel cache
 kernel_meta*
 

docs/observability/README.md

@@ -0,0 +1,173 @@
+## Observability Example Configuration (`examples/observability`)
+
+This directory provides a complete, Docker Compose–based observability example environment, including:
+
+* **Prometheus**: Metrics collection
+* **Grafana**: Metrics visualization
+* **OpenTelemetry Collector**: Distributed tracing data ingestion and processing
+
+Developers can use this example to **launch a local monitoring and tracing system with a single command**.
+
+---
+
+### Prerequisites
+
+Please make sure the following components are installed in advance:
+
+* Docker
+* Docker Compose (or a newer Docker CLI version that supports `docker compose`)
+
+---
+
+### Usage
+
+#### Start All Services
+
+Enter the directory:
+
+```bash
+cd examples/observability
+```
+
+Run the following command to start the complete monitoring and tracing stack:
+
+```bash
+docker compose -f docker-compose.yaml up -d
+```
+
+After startup, you can access:
+
+* **Prometheus**: [http://localhost:9090](http://localhost:9090)
+* **Grafana**: [http://localhost:3000](http://localhost:3000)
+* **OTLP receiver**: Applications should send traces to the default ports of the OTel Collector (usually `4317` or `4318`)
+
+  * gRPC: `4317`
+  * HTTP: `4318`
+* **Jaeger UI**: [http://localhost:16886](http://localhost:16886)
+
+**Notes:**
+
+* Update the Prometheus scrape targets to match your actual application endpoints.
+* Map Grafana’s service port to a port that is accessible on your machine.
+* Map the Jaeger UI port to a port that is accessible on your machine.
+* When starting the full stack, there is no need to start individual sub-services separately.
+
+---
+
+#### Start Metrics Services Only
+
+Enter the directory:
+
+```bash
+cd examples/observability/metrics
+```
+
+Run the following command:
+
+```bash
+docker compose -f prometheus_compose.yaml up -d
+```
+
+After startup, you can access:
+
+* **Grafana**: [http://localhost:3000](http://localhost:3000)
+
+---
+
+#### Start Tracing Services Only
+
+Enter the directory:
+
+```bash
+cd examples/observability/tracing
+```
+
+Run the following command:
+
+```bash
+docker compose -f tracing_compose.yaml up -d
+```
+
+After startup, you can access:
+
+* **OTLP receiver**: Applications should send traces to the default ports of the OTel Collector (usually `4317` or `4318`)
+
+  * gRPC: `4317`
+  * HTTP: `4318`
+* **Jaeger UI**: [http://localhost:16886](http://localhost:16886)
+
+---
+
+### Directory Structure and File Descriptions
+
+#### Core Startup File
+
+| File Name             | Purpose    | Description                                                                                                                                                         |
+| --------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `docker-compose.yaml` | Main entry | Defines and starts the full observability stack (Prometheus, Grafana, OTel Collector, and Jaeger). This is the single entry point to launch the entire environment. |
+
+---
+
+#### Metrics and Monitoring Configuration
+
+| File / Directory                                    | Purpose                  | Description                                                                                                               |
+| --------------------------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------- |
+| `metrics`                                           | Metrics root directory   | Contains all Prometheus- and metrics-related configurations.                                                              |
+| `prometheus.yaml`                                   | Prometheus main config   | Defines scrape targets, global scrape parameters, and optional recording rules. All monitored endpoints are defined here. |
+| `prometheus_compose.yaml`                           | Prometheus Docker config | Defines the Prometheus container, volume mounts, and network settings.                                                    |
+| `grafana/datasources/datasource.yaml`               | Datasource configuration | Configures how Grafana connects to Prometheus.                                                                            |
+| `grafana/dashboards/config/dashboard.yaml`          | Dashboard provisioning   | Specifies the locations of dashboard JSON files to be loaded.                                                             |
+| `grafana/dashboards/json/fastdeploy-dashboard.json` | Dashboard definition     | Contains visualization layouts and queries for `fastdeploy` monitoring metrics.                                           |
+
+---
+
+#### Distributed Tracing Configuration
+
+| File / Directory                                                                | Purpose                | Description                                                            |
+| ------------------------------------------------------------------------------- | ---------------------- | ---------------------------------------------------------------------- |
+| `tracing`                                                                       | Tracing root directory | Contains all configurations related to distributed tracing.            |
+| `opentelemetry.yaml`                                                            | OTel Collector config  | Defines the Collector data pipelines:                                  |
+| • **receivers**: receive OTLP data (traces, metrics, logs)                      |                        |                                                                        |
+| • **processors**: data processing and batching                                  |                        |                                                                        |
+| • **exporters**: export data to tracing backends (such as Jaeger) or files      |                        |                                                                        |
+| • **extensions**: health check, pprof, and zpages                               |                        |                                                                        |
+| • **pipelines**: define complete processing flows for traces, metrics, and logs |                        |                                                                        |
+| `tracing_compose.yaml`                                                          | Tracing Docker config  | Defines the container configuration for the OTel Collector and Jaeger. |
+
+---
+
+### Customization
+
+#### 4.1 Modify Metrics Scrape Targets
+
+If your application’s metrics endpoint, port, or path changes, edit:
+
+```plain
+metrics/prometheus.yaml
+```
+
+---
+
+#### 4.2 Adjust Tracing Sampling Rate or Processing Logic
+
+Edit:
+
+```plain
+tracing/opentelemetry.yaml
+```
+
+---
+
+#### 4.3 Add Custom Grafana Dashboards
+
+1. Add the new dashboard JSON file to:
+
+```plain
+grafana/dashboards/json/
+```
+
+2. Register the dashboard so Grafana can load it automatically by editing:
+
+```plain
+grafana/dashboards/config/dashboard.yaml
+```