Get the source code for this video for FREE → https://the-dotnet-weekly.kit.com/outbox-scaling
Want to master Clean Architecture? Go here: https://bit.ly/3PupkOJ
Want to unlock Modular Monoliths? Go here: https://bit.ly/3SXlzSt
Join a community of 1000+ .NET developers: https://www.patreon.com/milanjovanovic
Learn how I scaled a transactional outbox pattern to handle over 2 billion messages per day using C# and parallel processing. I'll show you the exact code implementation using Parallel.ForEachAsync for efficient message processing, and break down the architecture that makes it possible. Perfect for developers dealing with high-throughput distributed systems.
Scaling the Outbox Pattern (2B+ messages per day)
https://www.milanjovanovic.tech/blog/scaling-the-outbox-pattern
Check out my courses:
https://www.milanjovanovic.tech/courses
Read my Blog here:
https://www.milanjovanovic.tech/blog
Join my weekly .NET newsletter:
https://www.milanjovanovic.tech
Chapters
0:00 Baseline Outbox performance
4:15 Increasing the batch size (~1,200 MPS)
4:54 Measuring each processing step
9:06 Optimizing SELECT queries
12:31 Batch publishing messages
18:37 Optimizing UPDATE queries
22:13 Results after optimization (~2,500 MPS)
22:57 Scaling out the Outbox processing
27:20 Enabling RabbitMQ batch publish
28:09 FOR UPDATE SKIP LOCKED (~27,000 MPS)
30:19 Dangerous thing you can do for more speed
31:19 Recap - How I scaled Outbox processing