.NET Academy

Microservices, CAP and resilience

Monoliths vs microservices, the trade-offs of distributed data, and patterns that keep failures contained.

14 min read

You will learn

  • Compare a monolith with microservices honestly
  • Explain the CAP theorem and eventual consistency
  • Apply timeouts, retries with backoff and circuit breakers
  • Implement retry with exponential backoff

Monolith or microservices?

A monolith is one deployable application containing all features. Microservices split a system into small services — orders, payments, catalog, shipping — each deployed independently and owning its own database.

MonolithMicroservices
DevelopmentSimple: one codebase, one deploy, in-process callsTeams work and deploy independently
ScalingScale the whole appScale only the busy parts
FailuresA bug can take everything downFailures can be isolated (if designed for it)
DataOne database, easy transactions and joinsEach service owns its data; no cross-service transactions
OperationsOne thing to run and monitorMany services, networks, versions, distributed debugging

The honest advice: start with a modular monolith — one deployable app with clear internal module boundaries (Track 12). Extract services when you have a concrete reason: separate teams, very different scaling needs, or isolation requirements.

Everything fails, eventually

Once components talk over a network, you must assume that calls can be slow, fail, or succeed without you knowing (the response got lost). The famous fallacies of distributed computing begin with "the network is reliable" — it isn't.

The CAP theorem

When data is copied across several machines and the network between them partitions (they can't communicate), a system must choose:

  • Consistency — refuse to answer (or error) rather than return possibly stale data, or
  • Availability — keep answering, accepting that some answers may be stale.

You can't have both during a partition. Banking ledgers lean towards consistency; shopping carts, social feeds and product catalogs usually lean towards availability with eventual consistency — all copies converge once the partition heals.

Resilience patterns

Timeouts

Never wait forever. Every outgoing call (HTTP, database, cache) needs a timeout; a hung dependency should cost you seconds, not all your threads.

Retries with exponential backoff (and jitter)

Many failures are transient — a blip, a restart, a lock timeout. Retrying usually works, if done carefully:

  • Wait longer after each failure: 100 ms, 200 ms, 400 ms… (exponential backoff).
  • Add random jitter so thousands of clients don't retry in lockstep.
  • Cap the number of attempts, and only retry idempotent operations (or use idempotency keys).

Circuit breaker

If a dependency keeps failing, retries just pile on more load. A circuit breaker watches failures; after a threshold it opens and fails immediately for a cool-down period, then lets a trial request through (half-open) to test whether the dependency has recovered.

 CLOSED ──(too many failures)──▶ OPEN ──(after 30 s)──▶ HALF-OPEN
   ▲                                                    │
   └────────────(trial request succeeds)────────────────┘

Bulkheads and fallbacks

  • Bulkhead — limit concurrency per dependency so one slow service can't consume every thread or connection.
  • Fallback — degrade gracefully: show cached prices, hide recommendations, queue the request for later.

In .NET: Polly and resilience pipelines

You rarely hand-write these. Polly and Microsoft.Extensions.Http.Resilience add them to HttpClient in one line:

builder.Services.AddHttpClient<PaymentClient>(c => c.BaseAddress = new("https://payments.example"))
    .AddStandardResilienceHandler();   // timeouts, retries with backoff + jitter, circuit breaker

Writing a retry loop yourself once is still the best way to understand what these libraries do — that's the exercise below.

Key takeaways

  • Prefer a modular monolith; adopt microservices for concrete reasons, accepting their complexity.
  • CAP: during a network partition, choose consistency or availability; many systems accept eventual consistency.
  • Use timeouts everywhere, retries with exponential backoff and jitter for transient faults, and circuit breakers to stop hammering failing dependencies.
  • In .NET, Polly / AddStandardResilienceHandler provide these patterns.

Exercise

Implement RetryAsync. It calls the operation; if it throws, it waits and tries again — up to maxAttempts attempts in total. The delay doubles each time: baseDelayMs, then 2×, 4×… If the last attempt also fails, rethrow that exception.

Program.cs
Loading editor…

C# runtime not loaded

Hint 1

Loop with for (int attempt = 1; ; attempt++) and wrap the call in try { return await operation(); }.

Hint 2

Use an exception filter so the last failure isn't caught: catch (Exception) when (attempt < maxAttempts) { ... }

Hint 3

Inside the catch, await Task.Delay(baseDelayMs * (1 << (attempt - 1))); — that's base, 2×base, 4×base…

Show solution
int calls = 0;
string result = await RetryAsync(async () =>
{
    calls++;
    await Task.Delay(10);
    if (calls < 3) throw new HttpRequestException($"attempt {calls} failed");
    return "success";
}, maxAttempts: 5, baseDelayMs: 20);
Console.WriteLine($"{result} after {calls} calls");

async Task<string> RetryAsync(Func<Task<string>> operation, int maxAttempts, int baseDelayMs)
{
    for (int attempt = 1; ; attempt++)
    {
        try
        {
            return await operation();
        }
        catch (Exception) when (attempt < maxAttempts)
        {
            await Task.Delay(baseDelayMs * (1 << (attempt - 1)));
        }
    }
}

Quiz

4 questions · 3 correct to pass

  1. 1.What's a fair description of microservices?

  2. 2.According to the CAP theorem, what must a distributed data store choose between during a network partition?

  3. 3.What does a circuit breaker do?

  4. 4.Why add jitter (randomness) to retry delays?

To complete this lesson

  • ○ Solve the exercise
  • ○ Pass the quiz (3/4 correct)