Node.js, GraphQL, MongoDB, Next.js
Journal of Medical Insights
Performance optimization and bug resolution for JOMI — a live medical journal platform where expert surgeons share on-demand virtual shadowing experiences for medical students, residents, and attending physicians.
What needed to be solved
Two specific performance problems existed on a live production platform. First, CSV exports for large datasets were timing out — loading all records into memory before sending caused long waits and server timeouts that blocked medical professionals from accessing their data. Second, cron jobs updating millions of records were causing database contention, slowing down the live system while the jobs ran.
Key Decisions & Challenges
Response Streaming + Cursor Pagination for CSV Exports
Situation
CSV exports for large datasets were timing out. The existing implementation loaded all matching records into memory before sending the response — as dataset size grew, this caused server timeouts and minutes-long waits for users who needed their data.
Options Considered
- Continue loading all records into memory before sending — simple, but causes timeouts on large exports
- Stream the response: read records in batches using cursor pagination and pipe each batch to the HTTP response as it arrives
Decision
Implemented Node.js response streaming with cursor-based pagination. Data is read from MongoDB in ordered cursor batches and written directly to the HTTP response stream as each batch is fetched. This eliminated the memory bottleneck entirely — download times dropped from minutes to seconds, a 70% improvement — without any infrastructure changes or new services.
Cursor-Based Pagination for Cron Jobs Updating Millions of Records
Situation
Cron jobs needed to process and update millions of records on a schedule. Using offset/limit pagination gets progressively slower as offsets grow. Fetching all records at once would lock the database and block live queries during the entire job.
Options Considered
- Offset/limit pagination — standard approach, but slows down at high offsets and increases database pressure
- Fetch all records at once — simple but locks the database during the full job run
- Cursor-based pagination — track the last processed document ID and continue from there in each batch
Decision
Cursor-based pagination. Each batch reads records starting from the last processed document ID, updates them, saves the cursor position, and moves to the next batch. Batch sizes stay consistent regardless of how far into the dataset the job is, lock contention is avoided, and the database continues serving live queries between batches.
Rewriting the MongoDB Aggregation Pipeline to Eliminate N+1
Situation
The diagnosis of the CSV timeout revealed a second problem: an N+1 query pattern buried inside the aggregation pipeline. For each document in the result set, additional database queries were being triggered — meaning total database calls scaled linearly with dataset size.
Options Considered
- Keep the existing pipeline structure — simple, but database calls multiply with every record in the result
- Restructure the aggregation using $lookup and proper stage ordering to fetch all related data in a single pipeline pass
Decision
Rewrote the MongoDB aggregation pipeline using Typegoose to combine the related data fetches into a single pipeline pass, eliminating the N+1 pattern. This reduced database call counts from O(n) to a fixed number of pipeline stages regardless of result set size.
Features
Streaming CSV Export
Node.js response streaming with cursor-based pagination for large dataset exports. Records are read and piped to the HTTP response in batches — eliminating memory buildup and reducing export times from minutes to seconds.
Cursor-Based Cron Jobs for Millions of Records
Cron jobs that update millions of records using cursor pagination, processing in ordered batches without locking the database or degrading live query performance.
Aggregation Pipeline Optimization
Rewrote complex MongoDB aggregation pipelines using Typegoose to eliminate N+1 query patterns, reducing database call count from proportional to result set size down to a fixed number of pipeline stages.
Bug Fixes Across Frontend and Backend
Resolved critical bugs across the Next.js frontend and Express.js/GraphQL backend, covering functionality issues across multiple areas of the live platform.
In Action
Article details page — navigate the video by selecting sections from the left
My Role
I was brought in as a contributor to work on specific performance improvements and bug fixes on a live production platform — not the full system. My scope covered the CSV export streaming, the cron job cursor pagination, the aggregation pipeline optimization, and resolving bugs across frontend and backend. I also managed client communications and task tracking via ClickUp.
Tech Stack
Outcome
CSV export speed improved by 70%. Cron jobs now update millions of records without locking the database. Both improvements were achieved through code changes alone — no infrastructure changes, no new services, no additional cost.