traffic-replayer

Production-grade Rust CLI that replays HTTP traffic from NDJSON logs — with PII sanitization, concurrent dispatch, S3 streaming, and rich latency statistics.

Rust stable Tokio async reqwest aws-sdk-s3
View on GitHub
Project Overview
What traffic-replayer does and why it matters

Streaming Input

Reads NDJSON line-by-line from a local file or S3 object. Never buffers the full dataset — constant memory regardless of file size.

PII Sanitization

Strips sensitive headers (Authorization, Cookie), redacts query params, and recursively redacts JSON body fields before any network call.

Concurrent Replay

Semaphore-bounded concurrency (default 50). Tokio tasks are spawned only up to the configured limit — no unbounded task spawning.

Live Stats Dashboard
Drop a stats.json file produced by --output-stats to visualise your run, or click to load a demo.

Drop stats.json here, or click to browse

— or — load demo data

Architecture Diagram
Module layout with data-flow arrows and design-pattern annotations
ENTRY INPUT LAYER PROCESSING LAYER OUTPUT LAYER main.rs Entry · thin cli.rs clap args config.rs validated Config SourceFactory Factory pattern LocalSource tokio::fs S3Source aws-sdk-s3 LineStream Pin<Box<dyn Stream>> FilterChain Chain of Responsibility Method Filter PathPrefix Filter Status Filter RecordSanitizer Facade + Strategy Header Sanitizer Query Sanitizer Body Sanitizer SenderFactory Strategy pattern HttpSender reqwest DryRunSender no-op pipeline.rs Semaphore · RateLimiter · SIGINT StatsAccumulator hdrhistogram · mpsc AggregatedStats p50 / p95 / p99 stdout summary human-readable stats.json --output-stats file error.rs TrafficReplayerError models.rs ReplayRecord + Outcome LEGEND Factory Chain of Responsibility Strategy / Facade Pipeline orchestration Data flow OCP = every border box is a trait-based extension point. Add implementations without modifying existing code.
Request Pipeline Flow
Every record passes through this sequence — from raw bytes to recorded outcome
1
Read Line
LineStream
→
2
Parse JSON
ReplayRecord
→
3
Filter
FilterChain
→
4
Sanitize
RecordSanitizer
→
5
Rate Limit
governor
→
6
Acquire Permit
Semaphore
→
7
Spawn Task
tokio::spawn
→
8
Send Request
RequestSender
→
9
Record Stats
mpsc → Acc

Error handling at each stage

Parse error
warn! + parse_errors++ + continue
Filter reject
filtered_count++ + continue
Sanitize error
warn! + parse_errors++ + continue
Transport error
failed++ + error_categories++ + continue
Non-2xx response
failed++ — exit code stays 0
SIGINT
CancellationToken → drain → exit 2
Design Patterns
Extension Points — you can add these without modifying existing code
Want to add Implement trait Register in Existing code changes
New input backend
(stdin, HTTP, Kafka)		 RecordSource SourceFactory::create None
New filter type
(header, body-size, regex)		 Filter FilterFactory::from_config None
New sanitization strategy Sanitizer + sanitize() RecordSanitizer::new None
New dispatch strategy
(gRPC, WebSocket, throttled)		 RequestSender SenderFactory::from_config None
CLI Reference
All flags with types, defaults, and descriptions
Flag Type Default Description
--base-url String required Target base URL
--input-file Path — Local NDJSON file (mutually exclusive with S3)
--s3-bucket + --s3-key String — S3 source (both required together)
--concurrency u32 50 Max in-flight requests (Semaphore bound)
--timeout-ms u64 30000 Total request timeout in milliseconds
--connect-timeout-ms u64 5000 TCP connect timeout in milliseconds
--method String… — Filter: HTTP method(s), repeatable, OR logic
--path-prefix String… — Filter: path prefix(es), repeatable, OR logic
--status u16… — Filter: original status codes, repeatable, OR logic
--rate u32 — Max req/s (governor RateLimiter)
--dry-run flag false Parse/filter/sanitize — do NOT send requests
--verbose flag false Enable debug-level tracing to stderr
--output-stats Path — Write full JSON stats to this file (never stdout)
--max-records u64 — Stop after N records
--insecure-skip-tls-verify flag false Disable TLS certificate verification
--follow-redirects flag false Follow HTTP redirects (default: OFF)
--aws-region String — AWS region (falls back to AWS_DEFAULT_REGION)
--sanitize-passthrough-keys String — Comma-separated keys exempt from sanitization
--drain-timeout-ms u64 5000 Max wait for in-flight requests on SIGINT
Example Commands 
# Local replay with filters and rate limit
traffic-replayer \
  --base-url http://localhost:8080 \
  --input-file ./traffic.ndjson \
  --method GET --method POST \
  --path-prefix /api/v1 --path-prefix /health \
  --status 200 \
  --concurrency 200 \
  --rate 1000 \
  --output-stats stats.json

# S3 source with region
traffic-replayer \
  --base-url http://staging.internal \
  --s3-bucket my-bucket \
  --s3-key logs/replay.ndjson \
  --aws-region us-east-1

# Dry-run validation without network traffic
traffic-replayer \
  --base-url http://localhost:8080 \
  --input-file ./traffic.ndjson \
  --dry-run --verbose

# Exempt 'session_id' from PII sanitization
traffic-replayer \
  --base-url http://localhost:8080 \
  --input-file ./traffic.ndjson \
  --sanitize-passthrough-keys session_id,key
Rust traffic replayer By Biniyam Gebreyohannes