Hey all, sharing my latest alternative to elasticsearch, cloudwatch etc that either require heavy cloud tax or cost more for a managed solution. This is a known pattern but maybe not well known enough to write logs to s3 as durable storage with Parquet and query rapidly with DuckDB. This has become the primary way I ingest logs for all my side projects and don't ever have to worry about losing logs and possibly free storage.

Features

    Format Agnostic - Works with any JSON log format via configurable field extraction
    Fast - 28K+ entries/sec ingestion
    Efficient - Parquet + Snappy (3.7x compression)
    Quick Queries - DuckDB queries in <50ms on 56K logs
    S3-Compatible - AWS S3, MinIO, DigitalOcean Spaces, R2, etc.
    Partitioned - Hive-style partitioning by date/level (no redundant part suffixes)
    Auto-Flush - Configurable automatic flushing (default: 90s)
    Dedupe - Optional deduplication

Happy to answer any questions about internals