Fan-out happens when a microservice handling one request calls several other services, which may call still more services and cascade work down the stack.
Fanout across a service graph
A browser sends a low-rate stream to the front end. The front end fans requests out to two mid-tier services, and each mid-tier service calls two back-end services. The third back-end service receives an intentionally high rate of 2.5 requests per second, five times the 0.5 requests per second on another normalized path. Bright orbs represent requests only; responses are not shown.
Sometimes a service may accrue tech debt in such a way that the read path ends up calling the same downstream multiple times. The downstream may make multiple queries against its database per request. These translate into database reads that add up (and multiply) very quickly!
One request, six database reads
A continuous stream enters Service A at 5 requests per second. Each request produces three downstream requests, averaging 15 requests per second at Service B. Each downstream request produces two database reads, averaging 30 reads per second. The three rails stay singular; moving dots preserve the causal 1-to-3-to-6 bursts.
War story: redundant fanout
Now for the motivation of this post: at work I identified a codepath that ran at ~8,000 queries per second and found that each request sent 2-3 requests to the same downstream service. 99% of these requests weren't needed, and their results were discarded. We were DDOSing a partner team! Traffic settled near 10 QPS after fixing the issue.
Traffic over time
A two-day traffic timeline shows Service A oscillating between 1,000 and 4,000 queries per second. Before the fix, two downstream calls per query pushed 2,000 to 8,000 queries per second downstream. After the fix was merged and deployed at hour 36, only about 10 downstream queries per second remained because 99 percent of the fanout was redundant.
Fix deployed. Downstream traffic settled near 10 queries per second, avoiding 99 percent of the redundant fanout.