weastel

weastel

Drumil Patel

I'm an accidental query engine engineer.

Building query engines at petabyte scale which power data warehouses for India’s largest enterprises.

Resume Work

Technologies I love

01 Interests

Query engines & planners

DataFusion, Tantivy, Ballista — parallel scans, sane plans, and turning messy telemetry into SQL that actually finishes.

Columnar formats & cheap reads

Parquet row groups, bloom filters, byte-range GETs — micro-optimizations that become macro wins at PB scale.

Observability as a warehouse

Metrics, logs, traces, and sessions in one store — ingestion that doesn't flinch and retrieval that feels interactive.

Platform primitives

BYOB/BYOK, materialized views, UDFs — the unglamorous features that let enterprises actually adopt the thing.

02 Work
Resume PDF

Experience

  1. Senior Member of Technical Staff

    DevRev

    Jul 2024 – Present Bangalore, India

    Owned the architecture for DevRev's native observability warehouse (MELTS — Metrics, Events, Logs, Traces & Sessions), powering observability for enterprise customers including Indian banks (HDFC, ICICI, Axis Bank, Bajaj, and others).

    Query Engine

    • Designed and built a distributed SQL query engine using Apache DataFusion and Ballista, integrating Tantivy for full-text search, Bloom filter indexes for trace search, and Parquet columnar storage on S3 for highly parallel, low-latency queries at PB scale.
    • Architected custom materialized view integration in DataFusion's logical plan conversion, so the same SQL interface queries raw and precomputed data.
    • Built an extensible UDF framework so FDEs can add domain-specific logic, with anomaly detection and log pattern mining (Drain3) as example use cases.

    Ingestion

    • Designed a horizontally scalable pipeline sustaining 35 TB/hour of compressed telemetry and 5–6 PB monthly, replacing a legacy MongoDB/EBS architecture with a self-healing elastic S3 storage layer that decouples compute from storage.
    • Reached 5× higher ingestion capacity while cutting infrastructure cost from $100K to $17K/month (~$1M annually) through compute/storage decoupling and usage-based S3, and extended it with compliance-grade BYOB and BYOK for regulated customers.

    Consumer Ecosystem

    • Architected MELTS' downstream extensibility layer for commerce, incident management, LLM/Harness observability, and agent-improvement pipelines.

    Tech stack RustGolangDataFusionBallistaFluentdTantivyParquetS3

  2. Platform Engineer

    Newton School · Incanus Technologies

    May 2022 – Jun 2024 Bangalore, India

    Learning environments

    • Kubernetes and AWS control plane that spawns interactive student pods, including Jupyter notebooks and Node.js + Express, in the style of Replit and GitHub Codespaces. GoKubernetes SDKCrossplane

    ML platform

    • Event-driven Kubernetes platform for production ML, with Dapr and Kafka as brokers. Foundation for 15+ models in production. KafkaDaprFlaskGunicornKubernetes

    Cluster platform

    • Moved the backend from AMIs to Docker on EKS with Flux GitOps. Grew from 3 to 20+ services, including self-managed Airbyte, Airflow, GitLab, and Karpenter. KubernetesAWSFluxHelm
    • Karpenter bin packing and node provisioning saved about $160,000 a year and beat a 60% cost-reduction target across clusters. KarpenterKubecostKubernetesEC2AWS Fargate
    • KEDA and a custom gRPC server scaled pods with demand, at 99.999% availability and 10× traffic capacity, scaling down when idle. KEDAgRPCGo

  3. Software Engineering Intern

    Amazon

    Jun 2021 – Jul 2021 Hyderabad, India
    • Centralized configuration store on Amazon's internal tree-structured database with a Java/Spring library for geo-aware config resolution.

  4. Google Summer of Code

    CNCF · Prometheus

    May 2020 – Aug 2020 Remote
    • EKS support for Prombench — Kubernetes benchmarking control plane orchestrating ~1,000 pods to stress-test Prometheus at scale.

  5. Google Summer of Code

    RPI · Submitty

    May 2019 – Aug 2019 Remote
    • Randomized I/O for authentic assessment; RabbitMQ workers processing 10K students/hour and 60K submissions during exams (PostgreSQL, Symfony).

Education

Indian Institute of Technology, Roorkee

B.Tech, Electrical Engineering

Jun 2018 – Jun 2022 · Roorkee, India

Publication

Random Input and Automated Output Generation in Submitty

ACM SIGCSE Technical Symposium on Computer Science Education · 2020