Better Tracing

[sjfhsjfh]

TODO

• Update Jaeger implementation to OTLP (deprecated)
• Better tracing transmission from Node to Daemon (Related: Terminal logging freezes after ~30 seconds #1001, refactor(tracing): use builder style #1009)
• Better feature management for tracing feature gate (currently a bit mixed)

[sjfhsjfh]

@phil-opp @haixuanTao I'd like to discuss the approach for passing tracing information from Node to Daemon. Currently, we're using stdout/stderr to send unstructured string data line by line. I have concerns about this approach:

• stdout may conflict with actual node output in certain dataflows
• The lack of structure makes it difficult to properly parse and process trace data

I've temporarily disabled this problematic approach in #1009 as it was causing issues.

I have two questions regarding the implementation:

1. OpenTelemetry Integration: Are we planning to adopt OpenTelemetry for structured trace/span data? I understand it's still in beta, but it would provide a standardized format for telemetry data. src

2. Structured Communication: Should I extend the DaemonRequest enum in node_to_daemon.rs to support structured trace data transfer via Arrow+shmem?

   My concern here is version compatibility - if Node and Daemon are running different versions of Dora, we might encounter serialization/deserialization issues. How should we handle this problem?

[phil-opp]

As a first step, I would like to see support for Rust's and Python standard log packages. For Rust, we could set up a log-compatible logger, which forwards log events to the daemon. Python has the logging module, which appears to doing something similar (or is there some other logging framework that is more common?)

By supporting these standard log frameworks we also see log output from dependencies, which are not written for dora.