add logfire opentelemetry instrumentation for observability #4

motivation#

add distributed tracing and observability to understand search performance, debug issues, and monitor production behavior.

solution#

use logfire's official rust SDK with actix-web integration. logfire provides:

distributed tracing across all service calls
request/response metrics
custom spans for critical operations (embedding generation, vector search, RRF fusion)
structured logging with rich context
production-ready OTLP backend

implementation#

1. add dependencies#

update Cargo.toml:

[dependencies]
# existing dependencies...

# logfire and opentelemetry
logfire = "0.8"
opentelemetry = { version = "0.27", features = ["trace", "metrics"] }
opentelemetry-instrumentation-actix-web = { version = "0.27", features = ["metrics"] }
opentelemetry-otlp = { version = "0.27", features = ["trace", "http-proto", "reqwest-client", "reqwest-rustls"] }
tracing = "0.1"

note: reqwest-rustls feature is required to avoid export failures

2. environment setup#

update .env.example:

# existing vars...

# logfire configuration
LOGFIRE_TOKEN=your_logfire_write_token_here
RUST_LOG=info  # control verbosity

get token from: https://logfire.pydantic.dev → settings → write tokens

3. update main.rs#

use opentelemetry_instrumentation_actix_web::{RequestMetrics, RequestTracing};

#[actix_web::main]
async fn main() -> Result<()> {
    dotenv::dotenv().ok();
    
    // replace env_logger::init() with logfire
    let logfire = logfire::configure()
        .finish()
        .map_err(|e| anyhow::anyhow!("failed to initialize logfire: {}", e))?;
    
    let _guard = logfire.shutdown_guard();
    
    let config = Config::from_env()?;
    let host = config.host.clone();
    let port = config.port;

    logfire::info!("starting bufo search server", 
        host = %host, 
        port = port
    );

    HttpServer::new(move || {
        App::new()
            // add opentelemetry middleware FIRST
            .wrap(RequestTracing::new())
            .wrap(RequestMetrics::default())
            
            // existing middleware
            .wrap(middleware::Logger::default())
            .wrap(cors)
            // ... rest of config
    })
    .bind((host.as_str(), port))?
    .run()
    .await?;

    Ok(())
}

4. instrument search operations#

add spans to src/search.rs:

use tracing::instrument;

#[instrument(skip(config), fields(query = %query.query, top_k = query.top_k))]
pub async fn search(
    query: web::Json<SearchQuery>,
    config: web::Data<Config>,
) -> ActixResult<HttpResponse> {
    // embedding generation span
    let query_embedding = {
        let _span = logfire::span!("generate_embedding", query = %query.query);
        embedding_client.embed_text(&query.query).await.map_err(|e| {
            logfire::error!("failed to generate embedding", error = %e);
            // ...
        })?
    };

    // vector search span
    let vector_results = {
        let _span = logfire::span!("vector_search", top_k = query.top_k * 2);
        tpuf_client.query(vector_request).await.map_err(|e| {
            logfire::error!("vector search failed", error = %e);
            // ...
        })?
    };

    // bm25 search span
    let bm25_results = {
        let _span = logfire::span!("bm25_search", query = %query.query);
        tpuf_client.bm25_query(&query.query, query.top_k * 2).await.map_err(|e| {
            logfire::error!("bm25 search failed", error = %e);
            // ...
        })?
    };

    // rrf fusion span
    let _span = logfire::span!("reciprocal_rank_fusion", 
        vector_results = vector_results.len(),
        bm25_results = bm25_results.len()
    );
    
    // ... existing rrf logic ...
    
    logfire::info!("search completed", 
        query = %query.query,
        results_count = results.len(),
        top_score = results.first().map(|r| r.score).unwrap_or(0.0)
    );

    Ok(HttpResponse::Ok().json(SearchResponse { results }))
}

5. replace log macros#

throughout codebase, replace:

log::info!() → logfire::info!()
log::error!() → logfire::error!()
log::warn!() → logfire::warn!()

benefits#

automatic http tracing: all requests/responses traced with timing
distributed tracing: follow request flow through embedding → vector search → bm25 → rrf fusion
custom metrics: track search latency, result counts, error rates
rich context: structured logs with query details, result quality
zero vendor lock-in: uses standard opentelemetry, can export anywhere
minimal changes: mostly adding middleware + changing logger init

what you'll see in logfire dashboard#

request waterfall showing embedding time, search time, fusion time
error rates and p50/p95/p99 latencies
custom spans showing which search method (vector vs bm25) is slower
structured logs with full query context
distributed traces if you add more services later

references#

logfire rust SDK: https://github.com/pydantic/logfire-rust
docs: https://docs.rs/logfire/latest/logfire/
actix-web integration: https://docs.rs/opentelemetry-instrumentation-actix-web/latest/
example: https://github.com/pydantic/logfire-rust/blob/main/examples/actix-web.rs

rollout plan#

add dependencies and update local .env with logfire token
test locally with logfire middleware
verify traces appear in logfire dashboard
add custom spans to key operations
set LOGFIRE_TOKEN in fly secrets
deploy to production