Scaling and load balancing
Handle more traffic by scaling up and out, spreading load, and protecting your system with rate limits.
13 min read
You will learn
- Compare vertical and horizontal scaling
- Explain how load balancers distribute traffic and handle failures
- Find the bottleneck in a system
- Implement a token-bucket rate limiter
More users, same app
Your app works fine with 50 users. Then a marketing campaign brings 50,000. Scalability is the ability to handle more load by adding resources — ideally without rewriting anything.
Scaling up vs scaling out
| Vertical (scale up) | Horizontal (scale out) | |
|---|---|---|
| How | Bigger machine: more CPU, RAM | More machines running the same app |
| Pros | Simple, no code changes | No hard ceiling, survives a machine failing, scale down when quiet |
| Cons | Hard limit, expensive at the top, single point of failure | Needs stateless servers and a load balancer |
Web servers are usually scaled out. Databases are often scaled up first, then out with read replicas, caching or sharding.
Load balancers
A load balancer sits in front of your instances and spreads requests across them:
- Round robin — each instance in turn.
- Least connections — the instance with the fewest active requests.
- Hash-based — the same client (IP or key) goes to the same instance.
It also performs health checks (GET /health every few seconds) and stops routing to instances that fail — so one crash doesn't take the site down. Cloud platforms provide load balancers as a service; in .NET you can also run YARP as a reverse proxy.
Finding the bottleneck
A system is only as fast as its slowest tier. Add servers when the database is the problem and nothing improves — the load just moves. Play with traffic and capacity, and watch where things break first:
Scaling a web app
Each request needs one database query unless the cache answers it
Users
200 req/s
Load balancer
1 target
Server 1
200 rps
DB primary
200 q/s
Latency (typical)
342 ms
Failed requests
25%
Busiest tier
> 100%
Monthly cost
$240
Notice how latency explodes before a tier reaches 100%: as utilization approaches capacity, requests queue up. Keep headroom — running at 60–70% leaves room for spikes.
Typical scaling moves, in order of effort:
- Measure — find the actual bottleneck with metrics (Operations lesson).
- Cache repeated reads (previous lesson).
- Scale out stateless web servers behind a load balancer.
- Optimize queries and add indexes.
- Read replicas for read-heavy databases; queues to smooth write bursts.
- Partition/shard data — powerful but complex; last resort.
Autoscaling
Cloud platforms can add and remove instances automatically based on CPU, memory, request count or queue length — scaling out for the Monday morning peak and back in at night to save money. It only works if instances are stateless and start quickly.
Rate limiting
Sometimes the right answer to excessive traffic is no. Rate limiting protects your system (and your bill) from abusive clients, buggy retry loops and brute-force login attempts. Excess requests get 429 Too Many Requests with a Retry-After header.
Common algorithms:
- Fixed window — at most 100 requests per minute, counter resets each minute.
- Sliding window — smoother version of the same.
- Token bucket — a bucket holds up to N tokens and refills at a steady rate; each request takes one. Allows short bursts while enforcing an average rate. (You'll implement it below.)
ASP.NET Core has all of these built in:
builder.Services.AddRateLimiter(o => o.AddTokenBucketLimiter("api", opt =>
{
opt.TokenLimit = 20;
opt.TokensPerPeriod = 10;
opt.ReplenishmentPeriod = TimeSpan.FromSeconds(1);
}));
app.UseRateLimiter();
app.MapGet("/search", Search).RequireRateLimiting("api");Key takeaways
- Scale up with bigger machines, out with more instances behind a load balancer.
- Load balancers distribute requests and use health checks to route around failures.
- Always find the bottleneck before scaling — and keep headroom, because latency grows sharply near capacity.
- Rate limiting (e.g. token bucket) protects the system and returns
429withRetry-After.
Exercise
Implement a token-bucket rate limiter. The bucket starts full with `capacity` tokens and gains `refillPerSecond` tokens per second (never exceeding capacity). TryTake(now) refills based on the time elapsed since the last call, then takes one token if available and returns true — otherwise false. Time is passed in as seconds so it's easy to test.
C# runtime not loaded
Hint 1
Keep two fields: double tokens (start at capacity) and double lastRefill (start at 0).
Hint 2
On each call: tokens = Math.Min(capacity, tokens + (now - lastRefill) * refillPerSecond); lastRefill = now;
Hint 3
Then: if (tokens >= 1) { tokens -= 1; return true; } return false;
Show solution
var bucket = new TokenBucket(capacity: 3, refillPerSecond: 1);
for (int i = 0; i < 5; i++)
Console.WriteLine($"request {i + 1}: {(bucket.TryTake(now: 0) ? "allowed" : "rejected (429)")}");
class TokenBucket
{
private readonly int capacity;
private readonly double refillPerSecond;
private double tokens;
private double lastRefill;
public TokenBucket(int capacity, double refillPerSecond)
{
this.capacity = capacity;
this.refillPerSecond = refillPerSecond;
tokens = capacity;
}
public bool TryTake(double now)
{
tokens = Math.Min(capacity, tokens + (now - lastRefill) * refillPerSecond);
lastRefill = now;
if (tokens >= 1)
{
tokens -= 1;
return true;
}
return false;
}
}Quiz
4 questions · 3 correct to pass1.What is horizontal scaling?
2.How does a load balancer avoid sending traffic to a crashed server?
3.You add more web servers but response times don't improve. What's the likely cause?
4.What status code should a rate-limited request receive?
To complete this lesson
- ○ Solve the exercise
- ○ Pass the quiz (3/4 correct)