How Zerodha Reduced PDF Processing Time from 8 Hours to 25 Minutes While Handling 1.5M+ Files

The Scale of Financial PDF Processing

Every day, millions of traders execute transactions on financial platforms, and by regulation, each trader must receive a detailed PDF contract note within hours of market close. For Zerodha, India's largest stock broker, this means generating, digitally signing, and emailing over 1.5 million PDF reports daily, a number that can easily double during volatile market conditions.

According to the Zerodha engineering team, their legacy system had grown increasingly sluggish over time, taking up to 8 hours to complete the entire process. With strict regulatory deadlines requiring delivery before the next trading day begins, and exchange data often arriving with delays, the team faced a critical bottleneck that threatened their ability to serve customers effectively.

The technical debt had reached a breaking point, demanding a complete architectural overhaul that would transform their approach to high-volume document processing.

The Business Challenge Behind the Technical Problem

When you're processing 1.5+ million financial documents daily, every minute of delay compounds into significant business risk. The Zerodha team discovered that their monolithic Python-based system, running on a single vertically scaled server, simply couldn't keep pace with their growing user base.

The challenge extended beyond just processing speed. Each contract note varies dramatically in complexity, some users might have two-page reports while high-frequency traders could generate documents spanning thousands of pages. This unpredictable workload variation made capacity planning nearly impossible with traditional scaling approaches.

The regulatory pressure added another layer of complexity. Stock market contract notes aren't just business documents, they're legally mandated reports that must be digitally signed, encrypted, and delivered before the start of the next trading session. Miss that deadline, and you're facing regulatory penalties and customer dissatisfaction.

The team's existing infrastructure was buckling under these combined pressures, with processing times extending even longer during high-volume trading days when customers needed their reports most urgently.

The Strategic Decision to Rebuild from Scratch

Rather than attempting incremental improvements to their existing system, the Zerodha team made a bold decision: completely reimagine their architecture with horizontal scalability at its core. They recognized that conventional serverless functions wouldn't meet their needs due to cold-start problems and cost overhead for such CPU-intensive batch processing.

Their breakthrough insight was elegantly simple: spin up large ephemeral server instances, saturate every CPU core with concurrent work, then destroy the infrastructure immediately upon completion. This approach would maximize resource utilization while minimizing costs, exactly what you need for predictable, time-bound batch workloads.

The team evaluated various distributed processing frameworks but ultimately chose to build their own lightweight job management system called Tasqueue, giving them precise control over workload distribution and resource allocation.

This decision reflected a key strategic principle: sometimes the most efficient solution requires stepping away from existing tools and building exactly what your specific use case demands.

Architectural Innovation: From Monolith to Distributed Powerhouse

The new architecture treats PDF generation as a series of independent, chainable jobs: data processing → PDF generation → digital signing → email delivery. Each stage runs on dedicated worker pools, processing jobs concurrently and passing results through shared storage.

Here's how the team structured their workflow:

Worker Specialization: Different instance types handle different job categories. PDF generation requires significantly more CPU and memory than digital signing, so these processes run on separate auto-scaling groups optimized for their specific resource needs.

Storage Strategy: After testing AWS EFS and finding it unsuitable for high-throughput small file operations, they moved to S3 with a clever partitioning scheme. By distributing files across 10 fixed partition keys (0-9 with different file types), they achieved 10x higher request rates and overcame S3's per-prefix limitations.

Orchestration with Nomad: The team leverages Nomad's system jobs feature, similar to Kubernetes DaemonSets, to ensure each node runs exactly one instance of each worker type. This guarantees optimal resource utilization without complex scheduling overhead.

Dynamic Infrastructure: Every night, Terraform provisions a fresh Nomad cluster, deploys the job workers, processes the workload, then tears down everything automatically. This ephemeral approach eliminates infrastructure drift and ensures predictable performance.

Technical Breakthroughs That Transformed Performance

Three key technical decisions drove the dramatic performance improvements:

Switching from Chrome/Puppeteer to Typst

Initially, the team generated PDFs from HTML using headless Chrome instances, a resource-intensive approach that didn't scale. After experimenting with LaTeX (which provided 10x speed improvements but had memory limitations), they discovered Typst, a modern typesetting system written in Rust.

Typst delivered remarkable performance gains: 2-3x faster than LaTeX for small documents, and dramatically better for large ones. A 2,000-page contract note that took 18 minutes with LaTeX now compiles in just 1 minute with Typst. The single-binary deployment also significantly reduced Docker image sizes and startup times.

Solving the S3 Rate Limiting Challenge

When storing millions of temporary files during processing, the team initially hit S3's request rate limits. Their solution involved moving from random UUID prefixes to a fixed partitioning scheme that distributes requests evenly across pre-warmed S3 partitions, effectively multiplying their throughput capacity by 10x.

Email Infrastructure Overhaul

The team replaced their Postal SMTP server with Haraka, enabling horizontal scaling of email delivery. Combined with their custom smtppool library for managing concurrent connection pools, this change eliminated email delivery as a bottleneck entirely.

Impressive Results: 25 Minutes vs 8 Hours

The architectural transformation delivered remarkable business impact:

• Processing Time: Reduced from 8 hours to just 25 minutes, a 92% improvement in processing speed
• Scalability: Can now handle 1.5+ million PDFs daily with room to scale indefinitely
• Cost Efficiency: Negligible infrastructure costs due to ephemeral resource usage
• Reliability: Consistent performance regardless of workload variation
• Resource Utilization: Nearly 100% CPU utilization during processing with zero waste

The system now processes approximately 7 million ephemeral files throughout the complete workflow, with each step optimized for maximum throughput. On volatile trading days when document volume doubles, the architecture simply spins up additional instances automatically.

Perhaps most importantly, the team eliminated the regulatory compliance risk that came with extended processing times, ensuring customers always receive their contract notes before the next trading session begins.

Key Lessons for High-Volume Document Processing

The Zerodha team's experience offers several valuable insights for organizations facing similar scaling challenges:

Embrace Ephemeral Infrastructure: For predictable batch workloads, spinning up dedicated resources, using them intensively, then destroying them immediately can be more cost-effective than maintaining persistent infrastructure.

Match Architecture to Workload Patterns: Understanding that workload varies dramatically (2-page vs 2,000-page documents) led to a design that handles this variation gracefully through concurrent processing rather than complex resource planning.

Don't Overlook Storage Layer Performance: High-throughput applications can be bottlenecked by storage systems not designed for small file operations at scale. Testing storage solutions under realistic workloads prevents expensive surprises in production.

Tool Selection Matters More at Scale: Technologies that work fine for moderate workloads (like HTML-to-PDF conversion) may become prohibitive bottlenecks at scale. Sometimes newer, specialized tools (like Typst) offer dramatic improvements over established solutions.

Build vs Buy for Core Differentiators: While the team used existing tools where possible, they built custom solutions (Tasqueue, smtppool) for workflow orchestration where existing options didn't meet their specific performance requirements.

Looking Forward: A Framework for Future Growth

The Zerodha team views this project as more than just solving their PDF processing challenge, it's a reusable framework for any high-volume batch processing workload in the financial services industry.

They're now planning to migrate other time-consuming bulk jobs to follow the same architectural patterns: ephemeral infrastructure, job chaining, horizontal scaling, and automatic resource management. The modular design means individual components can be swapped out as better tools emerge, while the core orchestration patterns remain stable.

The success demonstrates that sometimes the best solution to scaling challenges isn't found in existing platforms, but in carefully combining simple, well-understood technologies in innovative ways. By focusing on first principles, maximize resource utilization, minimize waste, eliminate bottlenecks, they achieved remarkable results while keeping their architecture comprehensible and maintainable.

For financial technology companies facing similar document processing challenges, the Zerodha approach offers a proven blueprint for transforming batch processing workloads from operational liability to competitive advantage.