
<Sponsorships>

EU compliance shouldn't mean slower inference.
[**Infercom**](https://fandf.co/4kwsmzT) runs OpenAI-compatible APIs on dedicated SambaNova RDU infrastructure in Europe.
Deploy models like DeepSeek and Llama, or build agents with frameworks like OpenClaw.
European jurisdiction, CLOUD Act protection, zero prompt retention.
[**Start Building For Free**](https://fandf.co/4kwsmzT).

Treat your credit card like a programmable dev environment.
[**Klutch**](https://fandf.co/4tFOLyL) gives you full API access to script your own spending rules, view real-time transaction behavior,
and deploy unlimited virtual cards directly from your stack.
[**Start coding your card with Klutch**](https://fandf.co/4tFOLyL).

</Sponsorships>

Most teams get the macro architecture right.
They build a [**modular monolith**](what-is-a-modular-monolith) with clear module boundaries, [**public APIs**](internal-vs-public-apis-in-modular-monoliths), and proper [**data isolation**](modular-monolith-data-isolation).

But then they stop thinking about architecture.
Every module gets the same internal structure, usually some form of layered architecture.

The thing is, [**Clean Architecture**](/pragmatic-clean-architecture) and [**Vertical Slice Architecture**](vertical-slice-architecture) aren't as far apart as people think.
Both focus on use cases and maximizing cohesion.
Clean Architecture just adds a rule for the direction of dependencies, which often leads to more abstractions and ceremony.
My [**Pragmatic Clean Architecture**](/pragmatic-clean-architecture) approach takes a middle ground, and it's quite similar to VSA in nature.

The real question isn't which one is "better".
It's where each one shines inside your modular monolith.
And the beautiful part is: you can mix and match.

## Two Levels of Architecture

There are two architectural decisions you need to make when building a modular monolith:

1. **Macro architecture** - How do you decompose the system into modules?
   This covers module boundaries, [**communication patterns**](modular-monolith-communication-patterns),
   [**data isolation**](modular-monolith-data-isolation), public API design, and how modules are deployed.
2. **Micro architecture** - How do you organize code _inside_ each module?
   This covers folder structure, the direction of dependencies, how you implement use cases,
   where validation lives, and how you access the database.

Most articles about modular monoliths focus entirely on the macro level.
And for good reason.
Getting **module boundaries** wrong is **expensive to fix**.

<Image
  src="blogs/mnw_182/modular_monolith.png"
  width={2934}
  height={1554}
  alt="Modular monolith."
/>

But the micro level is equally important.
It determines how easy it is to add features, navigate code, and onboard new developers within a module.
It's the architecture your team interacts with every day.

The key insight is this:

> The macro architecture constrains how modules interact.
> The micro architecture is a local decision each module can make independently.

Your `Ticketing` module doesn't have to follow the same internal structure as your `Notifications` module.
The modular boundary gives you this freedom.

### Vertical Slices Are Not Modules

I've seen people conflate vertical slices with modules.
On the macro level, a module can look like a "vertical slice" of the business domain.
But that analogy breaks down at the application level.

A module is a bounded context.
It owns its data, exposes a public API, and encapsulates a business capability.
A vertical slice is a feature implementation pattern.
It groups the request, handler, validation, and data access for a single use case.

Modules and vertical slices operate at different levels.
Modules define the boundaries of the system.
Vertical slices organize the code within those boundaries.

<Image
  src="blogs/mnw_182/vertical_slices.png"
  width={2159}
  height={1276}
  alt="Vertical slices."
/>

## Vertical Slices Inside a Module

[**Vertical Slice Architecture**](vertical-slice-architecture-structuring-vertical-slices)
organizes code by feature instead of by technical layer.
Each feature is a self-contained unit: request, handler, validation, data access, all in one place.

Inside a modular monolith module, this is a natural fit.
The module boundary already enforces separation from the rest of the system.
You don't need layers to protect you.
The module's [**public API**](internal-vs-public-apis-in-modular-monoliths) does that.

Here's what a `Ticketing` module looks like with vertical slices using one file per feature:

```
📁 Modules/
|__ 📁 Ticketing
    |__ 📁 Features
        |__ 📁 AddItemToCart
            |__ #️⃣ AddItemToCart.cs
        |__ 📁 SubmitOrder
            |__ #️⃣ SubmitOrder.cs
        |__ 📁 GetOrder
            |__ #️⃣ GetOrder.cs
        |__ 📁 CancelOrder
            |__ #️⃣ CancelOrder.cs
        |__ 📁 RefundPayment
            |__ #️⃣ RefundPayment.cs
    |__ 📁 Data
        |__ #️⃣ TicketingDbContext.cs
    |__ 📁 Entities
        |__ #️⃣ Order.cs
        |__ #️⃣ Ticket.cs
    |__ #️⃣ ITicketingModule.cs
    |__ #️⃣ TicketingModule.cs
```

You can also use separate files per component if you prefer more granularity:

```
📁 Modules/
|__ 📁 Ticketing
    |__ 📁 Features
        |__ 📁 SubmitOrder
            |__ #️⃣ SubmitOrderRequest.cs
            |__ #️⃣ SubmitOrderResponse.cs
            |__ #️⃣ SubmitOrderHandler.cs
            |__ #️⃣ SubmitOrderValidator.cs
            |__ #️⃣ SubmitOrderEndpoint.cs
        |__ 📁 GetOrder
            |__ #️⃣ GetOrderRequest.cs
            |__ #️⃣ GetOrderResponse.cs
            |__ #️⃣ GetOrderHandler.cs
            |__ #️⃣ GetOrderEndpoint.cs
    |__ 📁 Data
    |__ 📁 Entities
    |__ #️⃣ ITicketingModule.cs
    |__ #️⃣ TicketingModule.cs
```

Both approaches keep everything for a feature in one folder.
Compare this to having `Application`, `Domain`, and `Infrastructure`
folders with dozens of files scattered across layers.
The vertical slice version is flat, scannable, and easy to navigate.

Here's a concrete example using a static class to group the feature components:

```csharp
public static class SubmitOrder
{
    public record Request(string CartId);
    public record Response(string OrderId, decimal Total);

    public class Validator : AbstractValidator<Request>
    {
        public Validator()
        {
            RuleFor(x => x.CartId).NotEmpty();
        }
    }

    public class Endpoint : IEndpoint
    {
        public void MapEndpoint(IEndpointRouteBuilder app)
        {
            app.MapPost("orders", Handler).WithTags("Ticketing");
        }

        public static async Task<IResult> Handler(
            Request request,
            IValidator<Request> validator,
            TicketingDbContext context)
        {
            var result = validator.Validate(request);
            if (!result.IsValid)
            {
                return Results.BadRequest(result.Errors);
            }

            var cart = await context.Carts
                .Include(c => c.Items)
                .FirstOrDefaultAsync(c => c.Id == request.CartId);

            if (cart is null)
            {
                return Results.NotFound();
            }

            var order = Order.Create(cart);

            context.Orders.Add(order);
            await context.SaveChangesAsync();

            return Results.Ok(
                new Response(order.Id, order.Total));
        }
    }
}
```

Adding a new feature means adding a new folder.
You're not touching shared code across layers.
You're not worrying about side effects.

## Choosing the Internal Architecture

A common misconception is that VSA is only for simple modules and Clean Architecture is for complex ones.
That's not how it works.

Vertical slices work great with rich domain models.
You can have domain entities, value objects, and domain events inside a vertical slice.
The slice organizes the entry point and orchestration. The domain model handles the business rules.
As the slice grows in complexity, you [**push logic into the domain**](refactoring-from-an-anemic-domain-model-to-a-rich-domain-model)
just like you would in Clean Architecture.

Similarly, [**Pragmatic Clean Architecture**](/pragmatic-clean-architecture) works well for simpler modules too.
The structure is lightweight when the domain is simple.

So the decision isn't about complexity.
It's about what your team is comfortable with and what gives you the most clarity.

Here are a few things I consider:

- **Team familiarity.** If your team already thinks in layers and dependency direction,
  Clean Architecture will feel natural. If they prefer organizing around features, VSA reduces friction.
- **Shared domain logic.** When many use cases share the same domain entities,
  having a dedicated `Domain` layer can make that sharing explicit.
  With VSA, you'd extract shared logic into a separate folder, which also works.
- **Independence of features.** When features are mostly independent and rarely share behavior,
  vertical slices keep things simple. Each feature is self-contained, and the mental model is straightforward.

The modular boundary protects the rest of the system regardless of what you choose inside the module.
So pick what makes your team most productive and be willing to change as the module evolves.

## Takeaway

[**Modular Monolith**](what-is-a-modular-monolith) answers the macro question:
how to decompose the system into modules with clear boundaries.
[**Vertical Slice Architecture**](vertical-slice-architecture) answers the micro question:
how to organize code by feature inside those modules.

They operate at different levels.
Modular Monolith gives you high cohesion within each module and helps you manage coupling between modules.
Vertical Slice Architecture gives you high cohesion within each feature inside a module.

You don't have to pick one internal architecture for the entire system.
Each module can choose what works best for its context.
Some modules will use [**Pragmatic Clean Architecture**](/pragmatic-clean-architecture).
Others will use vertical slices.
A well-defined module boundary makes this safe.

If you want to see how I build modular monoliths with this approach,
check out [**Modular Monolith Architecture**](/modular-monolith-architecture).

See you next week.

---



<Sponsorships>

As AI moves into production, legacy identity becomes a bottleneck. Anonymous execution and static credentials don't scale.
Join Teleport's [**free Feb 19 webinar**](https://fandf.co/3MnTC71) to learn how to move faster with AI,
without compromising security or reliability.

NEW: Use [**Augment Code's Context Engine**](https://fandf.co/3MI2WTh) with your favorite models!
[**Connect via MCP**](https://fandf.co/4rfBsTX) to see how you can get faster, higher-quality results using fewer tokens.

</Sponsorships>

I wanted to see how well a **vision model** (LLM) could parse grocery receipts into structured data.

And I don't just mean describe what it sees.
I want to be able to actually extract line items, quantities, and prices into clean JSON.
Running entirely on my machine with [**Ollama**](https://ollama.com/) and a **llama3.2-vision** model.

It started as a quick experiment. I ended up spending a whole evening on it.

In this week's issue, I'll walk you through:

- Setting up Ollama with a vision model in .NET
- Sending images to the model and getting structured output
- Iterating on the system prompt to improve accuracy
- Deserializing LLM responses into strongly typed C# objects
- Testing whether the results are actually consistent

Let's dive in.

## Setting Up Ollama With Microsoft.Extensions.AI

[Ollama](https://ollama.com/) lets you run large language models locally.
You pull a model the same way you'd pull a Docker image, and it runs on your hardware.

```bash
ollama pull llama3.2-vision:latest

# Then run the model locally
ollama run llama3.2-vision:latest
```

On the .NET side, [**Microsoft.Extensions.AI**](https://learn.microsoft.com/en-us/dotnet/ai/ai-extensions)
provides a unified `IChatClient` interface for talking to any LLM provider.
Combined with [OllamaSharp](https://github.com/awaescher/OllamaSharp), the setup is minimal:

```csharp {3-6}
var builder = Host.CreateApplicationBuilder();

builder.Services.AddChatClient(
    new OllamaApiClient(
        new Uri("http://localhost:11434"),
        "llama3.2-vision:latest"));

var app = builder.Build();

var chatClient = app.Services.GetRequiredService<IChatClient>();
```

That gives us an `IChatClient` backed by a local vision model.
You don't have to manage any API keys or cloud dependencies.

The nice thing about `IChatClient` is that it's **provider-agnostic**.
If you want to swap Ollama for OpenAI or Azure later, your application code doesn't change.

## Sending an Image to the Model

The simplest thing to try: send a receipt image and ask what's in it.

```csharp
var message = new ChatMessage(
    ChatRole.User, "What's in this image?");

message.Contents.Add(
    new DataContent(
        File.ReadAllBytes("receipts/receipt_1.png"),
        "image/png"));

var response = await chatClient.GetResponseAsync([message]);

Console.WriteLine(response.Text);
```

You read the image bytes, wrap them in a `DataContent` with the appropriate MIME type, and attach it to a `ChatMessage`.

Here's the receipt I used for testing.

<div className="centered">
  <Image
    src="blogs/mnw_181/receipt_1.png"
    width={430}
    height={363}
    alt="A grocery receipt with line items, quantities, and prices."
  />
</div>

And here's the raw text response from the model:

```text
This image appears to be a receipt or invoice in a foreign language, likely Russian or another
Slavic language. The document is in black and white and features a large QR code at the bottom.
The text is in a blocky, old-style font and includes several columns of numbers and words.
The top of the page has a header with some information in a foreign language, followed by a
series of columns with various details such as date, time, and product information. The document
also includes some calculations and a total at the bottom. The overall design and layout suggest
that this is a receipt or invoice from a store or restaurant, but the specific details and
language make it difficult to understand without further context or translation.
```

The model correctly identified it as a receipt and listed the items on it.
That's impressive for a first try with zero fine-tuning.
But it's not very useful if we want to extract structured data.

The good thing is we can refine our prompt and ask for a more specific output format.

## Asking for JSON Output

A text description isn't very useful programmatically. So I asked for structured JSON instead:

```csharp
var message = new ChatMessage(ChatRole.User,
    @"""
    Extract all line items from this receipt.
    Respond in JSON format with this structure:
    {
        "items": [
            {
                "name": "item name",
                "quantity": 1.500,
                "unitPrice": 0.00,
                "totalPrice": 0.00
            }
        ],
        "subtotal": 0.00
    }
    """);
message.Contents.Add(
    new DataContent(
        File.ReadAllBytes("receipts/receipt_1.png"),
        "image/png"));

var response = await chatClient.GetResponseAsync([message]);

Console.WriteLine(response.Text);
```

Here's the new response from the model:

```text
Here is the receipt in JSON format with all line items extracted:

{
    "items": [
        {
            "name": "limun /kg (A)",
            "quantity": 280.00,
            "unitPrice": 1.105,
            "totalPrice": 309.40
        },
        {
            "name": "salata /kom (A)",
            "quantity": 70.00,
            "unitPrice": 3.00,
            "totalPrice": 210.00
        },
        {
            "name": "susam 100g trpeza /kom (A)",
            "quantity": 90.00,
            "unitPrice": 1.00,
            "totalPrice": 90.00
        }
    ],
    "subtotal": 609.40
}
```

This worked surprisingly well on the first try.
The model returned a JSON blob with item names, quantities, and prices.
Some quantities were slightly off but the structure was correct.

For zero fine-tuning, I was impressed.

## Iterating on the System Prompt

Here's what I found: when the model misreads a digit or rounds a number, you don't fix a bug in C#.
You refine the system prompt.

What is a system prompt?
It's the initial instruction you give the model to set the context for the conversation.
Notice the `ChatRole.System` in the code below.
Conversely, we use `ChatRole.User` for the message where we ask the question.

After a few rounds of this, my system prompt ended up reading like a specification document:

```csharp
var systemMessage = new ChatMessage(ChatRole.System,
    @"""
    You are a receipt parsing assistant. Extract all line items from the receipt image.
    For each line item, extract the name, quantity, unit price, and total price.
    Quantity can be a decimal number (e.g. weight in kg like 0.550 or 1.105).
    Extract the subtotal which is the final total amount shown on the receipt.
    IMPORTANT: Read every digit exactly as printed on the receipt.
    Pay very close attention to each decimal digit - do NOT round or approximate.
    For example, if the receipt shows 1.105, report exactly 1.105, not 1.1 or 1.2.
    Verify that quantity * unitPrice = totalPrice for each line item.
    Don't invent items that aren't on the receipt.

    DECIMAL FORMAT: Receipts may use different number formats depending on locale.
    - Some use period as decimal separator: 7,499.00
    - Some use comma as decimal separator: 7.499,00
    First, detect which format the receipt uses by examining the numbers on it.
    Then, always output numbers in the JSON using a period as the decimal separator.
    For example: 7499.00, not 7.499,00 or 7,499.00.
    """);
```

Every instruction in that prompt exists because the model got something wrong.

"Read every digit exactly as printed": it was rounding `1.105` to `1.1`.

"Don't invent items": it hallucinated a line item that wasn't on the receipt.

The entire decimal format section: my receipts use commas as decimal separators (European locale),
and the model kept confusing thousands separators with decimal points.

Each prompt iteration is like a debugging session with words instead of code.
It's not the most fun part, but it's necessary to get accurate results.

And to think we used to write code to tell computers what to do.
Now we write prompts to tell models how to think.
I digress...

## Strongly Typed Responses

This is where [`Microsoft.Extensions.AI`](working-with-llms-in-dotnet-using-microsoft-extensions-ai) gets interesting.

Instead of parsing raw JSON strings yourself, you can call `GetResponseAsync<T>` and get back a **strongly typed** object:

```csharp {1, 5}
var response = await chatClient.GetResponseAsync<Receipt>(
    [systemMessage, message],
    new ChatOptions { Temperature = 0 });

if (response.Result is { } receipt)
{
    Console.WriteLine(
        $"\nExtracted {receipt.Items.Count} line items:");

    foreach (var item in receipt.Items)
    {
        Console.WriteLine(
            $"  {item.Name} - " +
            $"Qty: {item.Quantity} x {item.UnitPrice:C}" +
            $" = {item.TotalPrice:C}");
    }

    Console.WriteLine($"  Subtotal: {receipt.Subtotal:C}");
}
```

The `Receipt` and `LineItem` classes are plain C# objects:

```csharp
public class Receipt
{
    public List<LineItem> Items { get; set; } = [];
    public decimal Subtotal { get; set; }
}

public class LineItem
{
    public string Name { get; set; } = string.Empty;
    public decimal Quantity { get; set; }
    public decimal UnitPrice { get; set; }
    public decimal TotalPrice { get; set; }
}
```

The library generates the JSON schema, sends it to the model, and deserializes the response.
You get back a `Receipt` object directly.

I also set `Temperature = 0` to make the output as deterministic as possible.
For data extraction, you want accuracy.
This isn't foolproof, but it helps.

Here's the object we get back from the model in Visual Studio:

<Image
  src="blogs/mnw_181/chat_client_typed_response_in_vs.png"
  width={986}
  height={528}
  alt="Visual Studio screenshot showing the Receipt object with line items and subtotal extracted from the model response."
/>

## Testing Consistency

One thing I wanted to verify: if I send the same receipt with the same prompt five times, do I get the same result?

```csharp
const int runs = 5;
Console.WriteLine($"\n--- Consistency test ({runs} runs) ---");

var results = new List<Receipt>();
for (int i = 0; i < runs; i++)
{
    Console.WriteLine($"\nRun {i + 1}...");
    var testResponse = await chatClient.GetResponseAsync<Receipt>(
        [systemMessage, message],
        new ChatOptions { Temperature = 0 });

    if (testResponse.Result is { } r)
    {
        results.Add(r);
        Console.WriteLine(
            $"  Items: {r.Items.Count}, " +
            $"Subtotal: {r.Subtotal:C}");

        foreach (var item in r.Items)
        {
            Console.WriteLine(
                $"    {item.Name} - " +
                $"Qty: {item.Quantity} x {item.UnitPrice:C}" +
                $" = {item.TotalPrice:C}");
        }
    }
}
```

Then I compare every run against the baseline:

```csharp
var baseline = results[0];
for (int i = 1; i < results.Count; i++)
{
    bool match = baseline.Subtotal == results[i].Subtotal
        && baseline.Items.Count == results[i].Items.Count
        && baseline.Items.Zip(results[i].Items).All(pair =>
            pair.First.Name == pair.Second.Name
            && pair.First.Quantity == pair.Second.Quantity
            && pair.First.UnitPrice == pair.Second.UnitPrice
            && pair.First.TotalPrice == pair.Second.TotalPrice);

    Console.WriteLine(
        $"  Run 1 vs Run {i + 1}: " +
        $"{(match ? "MATCH" : "DIFFERENT")}");
}
```

Temperature 0 helps, but vision models aren't perfectly deterministic.
Most runs matched. Some didn't.
The differences were usually small - a misread digit, a slightly different item name.

This is worth keeping in mind when working with LLMs.
They're probabilistic systems.
Even with temperature 0, the same input can produce slightly different output.
If you need guaranteed accuracy, you'll want validation layers on top of this.

## Where I Want to Take This

The receipt scanner is a starting point.
Once you have structured data from receipt images, you can build on top of it.

I've been thinking about extending this into a **personal finance tracker**.
Scan receipts, store the data, and use the same LLM to categorize purchases.
Groceries, household, electronics - let the model figure it out.

From there, you could generate **weekly and monthly spending summaries**.
How much did I spend on groceries this month?
How does that compare to last month?

You could also do **multi-receipt aggregation** for business trips.
Scan a stack of receipts and generate an expense report.

Or **price tracking** over time - detect when items at your usual store go up in price.

We could defintiely build [**semantic search**](building-semantic-search-with-amazon-s3-vectors-and-semantic-kernel)
on top of this too.
Search through your past receipts for specific items or price ranges.
This works by embedding the structured data and using [**vector search**](what-is-vector-search-a-concise-guide)
to find relevant entries.

The vision model handles the hard part: turning unstructured images into structured data.
Everything after that is regular application development.

I might build some of this out. We'll see.

## Summary

Running a vision model locally with Ollama is straightforward to set up.
`Microsoft.Extensions.AI` and `OllamaSharp` make the .NET integration clean.
You get a provider-agnostic `IChatClient` with support for strongly typed responses.
You could also run this with [**Aspire and GitHub models**](building-generative-ai-applications-with-github-models-and-dotnet-aspire)
if you want to keep everything in the Microsoft ecosystem.

The system prompt is where most of the work happens.
Every line in mine was the result of the model getting something wrong and me adding a corrective instruction.

If you want to try this yourself:

1. Install [Ollama](https://ollama.com/)
2. Pull the vision model: `ollama pull llama3.2-vision:latest`
3. Create a .NET console app and add the `OllamaSharp` and `Microsoft.Extensions.AI` NuGet packages
4. Point it at a receipt and see what comes back

Hope this was useful. See you next week.

---



<Sponsorships>

[**NServiceBus**](https://particular.net/pricing/small-business-program?utm_source=newsletter1&utm_medium=email&utm_campaign=spb_promotion) -
The gold standard for async .NET microservices on Azure, AWS, and On-Prem.
**Free** for orgs under $1M, and with extended discounts, under $25/month for 10 microservices.
Sign up for the [**Small Business Program**](https://particular.net/pricing/small-business-program?utm_source=newsletter1&utm_medium=email&utm_campaign=spb_promotion)
today!

[**OutSystems**](https://fandf.co/45RwwfE) is hosting a [**$1,000 Groundhog Day AI Challenge**](https://fandf.co/45RwwfE)
that tests real development skills. Your challenge?
Progress from UI-level puzzle solving to using AI inside the IDE,
and race to save Phil the Groundhog faster than anyone else.

</Sponsorships>

**Authentication** is one of those things that's easy to get wrong and **expensive to fix later**.
Rolling your own auth system means dealing with password hashing, token management, session handling,
and a never-ending stream of security patches.

I was never a fan of this...

What if you could outsource all of that to a battle-tested **identity provider**?

[Keycloak](https://www.keycloak.org/) is an open-source identity and access management solution.
It handles user authentication, authorization, and identity brokering (social logins, enterprise SSO) out of the box.
You get a polished admin console, built-in support for **OAuth 2.0** and **OpenID Connect**, and it runs anywhere Docker does.

We'll spin up Keycloak as a container, create a realm with a public client,
and wire up Swagger UI to authenticate using the [OAuth 2.0](https://oauth.net/2/) **Authorization Code flow**.
Then we'll add [JWT](https://www.rfc-editor.org/rfc/rfc7519.html) validation to our .NET backend and
trace the entire authentication flow using the [**Aspire Dashboard**](standalone-aspire-dashboard-setup-for-distributed-dotnet-applications).

## Running Keycloak as a Container

The fastest way to spin up **Keycloak** is with [Docker](<https://en.wikipedia.org/wiki/Docker_(software)>).
We'll run it in development mode, which disables HTTPS and uses an embedded H2 database.
This is perfect for local development but **not suitable for production** (more on that later).

Here's a minimal `docker-compose.yml`:

```yaml
services:
  keycloak:
    image: quay.io/keycloak/keycloak:26.5.2
    container_name: keycloak
    environment:
      - KC_BOOTSTRAP_ADMIN_USERNAME=admin
      - KC_BOOTSTRAP_ADMIN_PASSWORD=admin
    ports:
      - '8080:8080'
    command: start-dev
```

Start it with:

```bash
docker compose up -d
```

Once Keycloak is running, navigate to `http://localhost:8080` and log in with `admin` / `admin`.

<Image
  src="blogs/mnw_180/keycloak_admin_login.png"
  width={1919}
  height={1079}
  alt="The Keycloak admin login screen with username and password fields."
/>

You should see the Keycloak admin console.

<Image
  src="blogs/mnw_180/keycloak_admin_console.png"
  width={1919}
  height={1079}
  alt="The Keycloak admin console showing the master realm dashboard."
/>

## Setting Up a Realm and Client

Keycloak organizes everything into **realms**.
A realm is a space where you manage users, roles, and applications.
The `master` realm is reserved for Keycloak administration, so we'll create a new one for our application.

### Creating a Realm

1. Click the **Manage Realms** button in the top-left corner
2. Click **Create realm**
3. Enter a name (e.g., `keycloak-demo`) and click **Create**

<Image
  src="blogs/mnw_180/keycloak_create_realm_dialog.png"
  width={1117}
  height={809}
  alt="The Keycloak create realm dialog with 'demo' entered as the realm name."
/>

### Creating a Public Client

Now we need to register our application.
In OAuth 2.0 terms, this is a **client**.
Since Swagger UI runs in the browser, we'll create a **public client** (no client secret).

1. Go to **Clients** → **Create client**
2. Set **Client ID** to `demo-api`
3. Leave **Client type** as `OpenID Connect`
4. Click **Next**

<Image
  src="blogs/mnw_180/keycloak_create_client_step1.png"
  width={1919}
  height={1079}
  alt="The first step of creating a client in Keycloak, showing the Client ID field."
/>

5. Enable **Client authentication**: Off (public client)
6. Check **Standard flow** (Authorization Code)
7. Choose **PKCE Method**: S256 (SHA-256)
8. Click **Next**

<Image
  src="blogs/mnw_180/keycloak_create_client_step2.png"
  width={1919}
  height={1079}
  alt="The second step of creating a client showing authentication settings."
/>

9. Configure the redirect URIs:
   - **Valid redirect URIs**: `https://localhost:5001/*` (your API's Swagger URL)
   - **Web origins**: `https://localhost:5001`
10. Click **Save**

<Image
  src="blogs/mnw_180/keycloak_create_client_step3.png"
  width={1919}
  height={1079}
  alt="The third step showing redirect URI configuration."
/>

### Creating a Test User

We need a user to authenticate with.

1. Go to **Users** → **Add user**
2. Fill in the details (username, email, etc.)
3. Leave **Email Verified** checked to avoid email confirmation
4. Click **Create**
5. Go to the **Credentials** tab
6. Click **Set password** and create a password (disable "Temporary")

<Image
  src="blogs/mnw_180/keycloak_create_user.png"
  width={1919}
  height={1079}
  alt="The Keycloak user creation form."
/>

You're now ready to authenticate users against Keycloak!

## The Authorization Code Flow

Before we dive into code, let's understand what happens when a user authenticates.
The [**Authorization Code flow**](https://www.rfc-editor.org/rfc/rfc6749#section-4.1) is the recommended OAuth 2.0 flow for browser-based applications.

There's an important security enhancement called **PKCE** ([Proof Key for Code Exchange](https://www.rfc-editor.org/rfc/rfc7636))
that prevents authorization code interception attacks.
It works by having the client generate a random secret (the code verifier) and deriving a hash (the code challenge) sent in the initial authorization request.
When exchanging the authorization code for tokens, the client must present the original code verifier.

Here's the sequence:

<Image
  src="blogs/mnw_180/authorization_code_flow.png"
  width={2725}
  height={2208}
  alt="A sequence diagram showing the OAuth 2.0 Authorization Code flow between Browser, API, and Keycloak."
/>

1. **User clicks "Authorize"** in Swagger UI
2. **Browser redirects** to Keycloak's authorization endpoint
3. **User logs in** at Keycloak
4. **Keycloak redirects back** with an authorization code
5. **Swagger UI exchanges** the code for tokens (access token, refresh token, ID token)
6. **Swagger UI attaches** the access token to API requests
7. **API validates** the token signature and claims

The beauty of this flow is that credentials never touch your application.
The user authenticates directly with Keycloak, and your API only sees signed tokens.

## Configuring Swagger UI with OAuth 2.0

Now let's set up our .NET API to use Swagger UI as our OAuth 2.0 test client.

First, install the required packages:

```bash
dotnet add package Swashbuckle.AspNetCore
```

Configure Swagger in your `Program.cs`:

```csharp
var keycloakAuthority = builder.Configuration["Keycloak:Authority"]!;
var keycloakClientId = builder.Configuration["Keycloak:ClientId"]!;

builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen(options =>
{
    options.SwaggerDoc("v1", new OpenApiInfo
    {
        Title = "Demo API",
        Version = "v1"
    });

    // Define the OAuth 2.0 security scheme
    options.AddSecurityDefinition(nameof(SecuritySchemeType.OAuth2), new OpenApiSecurityScheme
    {
        Type = SecuritySchemeType.OAuth2,
        Flows = new OpenApiOAuthFlows
        {
            AuthorizationCode = new OpenApiOAuthFlow
            {
                AuthorizationUrl = new Uri($"{keycloakAuthority}/protocol/openid-connect/auth"),
                TokenUrl = new Uri($"{keycloakAuthority}/protocol/openid-connect/token"),
                Scopes = new Dictionary<string, string>
                {
                    { "openid", "OpenID Connect scope" },
                    { "profile", "User profile" }
                }
            }
        }
    });

    // Apply security to all operations
    options.AddSecurityRequirement(doc => new OpenApiSecurityRequirement
    {
        {
            new OpenApiSecuritySchemeReference(nameof(SecuritySchemeType.OAuth2), doc),
            []
        }
    });
});
```

And configure the Swagger UI middleware:

```csharp
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI(options =>
    {
        options.OAuthClientId(keycloakClientId); // Default Client ID
        options.OAuthUsePkce(); // Proof Key for Code Exchange (security enhancement)
    });
}
```

Your `appsettings.Development.json`:

```json
{
  "Keycloak": {
    "Authority": "http://localhost:8080/realms/keycloak-demo",
    "ClientId": "demo-api",
    "Audience": "account",
    "Issuer": "http://localhost:8080/realms/keycloak-demo",
    // Here we use the Docker service name for Keycloak
    "MetadataAddress": "http://keycloak:8080/realms/keycloak-demo/.well-known/openid-configuration"
  }
}
```

Now when you open Swagger UI, you'll see an **Authorize** button.
Clicking it opens the OAuth flow, redirecting you to Keycloak to log in.

<Image
  src="blogs/mnw_180/swagger_authorize_form.png"
  width={1919}
  height={1079}
  alt="Swagger UI showing the Authorize form for OAuth 2.0."
/>

## Adding JWT Validation

At this point, Swagger UI can obtain tokens, but our API isn't validating them yet.
Let's add JWT Bearer authentication.

Install the authentication package:

```bash
dotnet add package Microsoft.AspNetCore.Authentication.JwtBearer
```

Configure authentication in `Program.cs`:

```csharp
builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(options =>
    {
        options.MetadataAddress = builder.Configuration["Keycloak:MetadataAddress"]!;
        options.Audience = builder.Configuration["Keycloak:Audience"];

        options.TokenValidationParameters = new TokenValidationParameters
        {
            ValidIssuer = builder.Configuration["Keycloak:Issuer"]
        };

        // Required for HTTP in development (Keycloak uses HTTP by default in dev mode)
        options.RequireHttpsMetadata = !builder.Environment.IsDevelopment();
    });

builder.Services.AddAuthorization();
```

The default `TokenValidationParameters` will validate the token signature, expiration, issuer, and audience.

Add the middleware:

```csharp
app.UseAuthentication();
app.UseAuthorization();
```

Create a protected endpoint:

```csharp
app.MapGet("users/me", (ClaimsPrincipal user) =>
{
    return Results.Ok(new
    {
        UserId = user.FindFirstValue(ClaimTypes.NameIdentifier),
        Email = user.FindFirstValue(ClaimTypes.Email),
        Name = user.FindFirstValue("preferred_username"),
        Claims = user.Claims.Select(c => new { c.Type, c.Value })
    });
})
.RequireAuthorization();
```

## How JWT Validation Works

When a request hits your protected endpoint, here's what happens under the hood:

<Image
  src="blogs/mnw_180/jwt_validation_flow.png"
  width={2030}
  height={2108}
  alt="A sequence diagram showing how JWT validation works in ASP.NET Core."
/>

1. **Middleware extracts** the `Authorization: Bearer <token>` header
2. **JWT Handler fetches** Keycloak's public keys from the JWKS endpoint (cached)
3. **Signature validation** proves the token wasn't tampered with
4. **Claims are extracted** and the `ClaimsPrincipal` is populated
5. **Authorization middleware** checks if the user meets the endpoint requirements
6. **Endpoint executes** with access to `HttpContext.User`

The key insight here is that your API **never contacts Keycloak** to validate individual tokens.
It fetches the signing keys once and validates tokens locally.
This is what makes JWT-based authentication so fast.

## Observing the Flow with Aspire Dashboard

If you're using [**Aspire**](https://aspire.dev), you can observe the entire authentication flow in the distributed traces.

Here's what a successful authentication looks like:

<Image
  src="blogs/mnw_180/aspire_auth_trace.png"
  width={1919}
  height={1079}
  alt="Aspire Dashboard showing a distributed trace of the authentication flow."
/>

You can see:

1. The initial request to `users/me` (with the Bearer token)
2. The outbound call to Keycloak's `.well-known/openid-configuration` endpoint
3. The outbound call to Keycloak's JWKS endpoint (fetching signing keys)
4. The response back to the client

On subsequent requests, you won't see the JWKS call because the keys are cached.

This is why JWT validation adds virtually no latency after the initial key fetch.

## Production Considerations

What we've built is great for development.
For production, you'll want to address a few things:

**1. HTTPS Everywhere**

Keycloak should run behind HTTPS.
Set `KC_HOSTNAME` and configure TLS certificates.

**2. Persistent Storage**

Replace the embedded H2 database with PostgreSQL or MySQL:

```yaml
environment:
  - KC_DB=postgres
  - KC_DB_URL=jdbc:postgresql://postgres:5432/keycloak
  - KC_DB_USERNAME=keycloak
  - KC_DB_PASSWORD=secret
```

**3. Require HTTPS Metadata**

Remove `options.RequireHttpsMetadata = false` in production.

## Summary

In about 10 minutes, we've set up:

- A [**containerized**](containerize-your-dotnet-applications-without-a-dockerfile) Keycloak instance
- A realm with a public OAuth 2.0 client
- Swagger UI acting as an OAuth client with Authorization Code + PKCE
- JWT validation in ASP.NET Core
- Observability with [**OpenTelemetry**](introduction-to-distributed-tracing-with-opentelemetry-in-dotnet) into the authentication flow

What I really like about Keycloak is how easy it is to extend.
Want Google login? Configure it in Keycloak.
Need enterprise SSO? Add a SAML provider.
Your API code stays exactly the same because it just validates tokens.

If you want to see how I integrate Keycloak in a real-world system with role-based access control,
check out [**Pragmatic Clean Architecture**](/pragmatic-clean-architecture) and
[**Modular Monolith Architecture**](/modular-monolith-architecture).

See you next week.

---


<Sponsorships>

Developers report that [**over 40% of committed code is now AI-assisted**](https://fandf.co/4qexwBR), yet they still spend about 25% of their week on toil.
Sonar's new developer survey shows AI didn't eliminate work; it shifted it into reviewing, fixing, and verifying code that looks correct but isn't.
[**Download the report**](https://fandf.co/4qexwBR) to understand why your workload feels the same, just harder to reason about.
(No form fill required.)

[**Webinar: How to Build Faster with AI Agents**](https://www.telerik.com/webinars/devcraft/ai-for-developers-how-to-achieve-a-50-percent-productivity-boost?utm_medium=cpm&utm_source=milanjovanovich&utm_campaign=dt_ai_productivity_webinar_bitovi)
Learn how full-stack developers boost productivity by 50% with AI agents that automate layout,
styling, and component generation through RAG and LLM pipelines.
See how orchestration and spec-driven workflows keep you in control of quality and consistency.
[**Check it out**](https://www.telerik.com/webinars/devcraft/ai-for-developers-how-to-achieve-a-50-percent-productivity-boost?utm_medium=cpm&utm_source=milanjovanovich&utm_campaign=dt_ai_productivity_webinar_bitovi)

</Sponsorships>

Containers have become the standard for deploying modern applications.
But if you've ever written a [Dockerfile](https://docs.docker.com/reference/dockerfile/), you know it can be tedious.
You need to understand multi-stage builds, pick the right base images, configure the right ports, and remember to copy files in the correct order.

What if I told you that **you don't need a Dockerfile at all**?

Since .NET 7, the SDK has built-in support for publishing your application directly to a container image.
You can do this with a single `dotnet publish` command.

In this week's newsletter, we'll explore:

- Why Dockerfile-less publishing matters
- How to enable container publishing in your project
- Customizing the container image
- Publishing to container registries
- **How I'm using this to deploy to a VPS**

## The Traditional Approach: Writing a Dockerfile

Before we look at the SDK approach, let's see what we're replacing.

A typical multi-stage Dockerfile for a .NET application looks like this:

```bash
FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS base
WORKDIR /app
EXPOSE 8080
EXPOSE 8081

FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY ["src/MyApi/MyApi.csproj", "src/MyApi/"]
RUN dotnet restore "src/MyApi/MyApi.csproj"

COPY . .
WORKDIR "/src/src/MyApi"
RUN dotnet build "MyApi.csproj" -c $BUILD_CONFIGURATION  -o /app/build

FROM build AS publish
ARG BUILD_CONFIGURATION=Release
RUN dotnet publish "MyApi.csproj" -c $BUILD_CONFIGURATION -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "MyApi.dll"]
```

This works, but there's a learning curve and maintenance overhead:

- **Maintenance burden**: You need to update base image tags manually
- **Layer caching**: Getting the COPY order wrong kills your build cache
- **Duplication**: Every project needs a similar Dockerfile
- **Context switching**: You're writing Docker DSL, not .NET code

The .NET SDK approach eliminates all of this.

## Enabling Container Publishing

If you're running on .NET 10, you don't need to do anything special to enable container publishing.
This will work for ASP.NET Core apps, worker services, and console apps.

You can publish directly to a container image:

```bash
dotnet publish --os linux --arch x64 /t:PublishContainer
```

That's it. The .NET SDK will:

1. Build your application
2. Select the appropriate base image
3. Create a container image with your published output
4. Load it into your local OCI-compliant daemon

The most popular option is Docker, but it also works with Podman.

<Image
  src="blogs/mnw_179/dotnet_publish_container.png"
  width={1891}
  height={457}
  alt="An image showing the output of the dotnet publish command creating a container image."
/>

## Customizing the Container Image

The SDK provides sensible defaults, but you'll often want to customize the image.
For a more comprehensive list of options, see the [official docs](https://learn.microsoft.com/en-us/dotnet/core/containers/publish-configuration).

I'll cover the most common customizations here.

### Setting the Image Name and Tag

The `ContainerRepository` property sets the image name (repository).
The `ContainerImageTags` property sets one or more tags (separated by semicolons).
If you want a single tag, you can use `ContainerImageTag` instead.

```xml
<PropertyGroup>
  <ContainerRepository>ghcr.io/USERNAME/REPOSITORY</ContainerRepository>
  <ContainerImageTags>1.0.0;latest</ContainerImageTags>
</PropertyGroup>
```

From .NET 8 and onwards, when a tag isn't provided the default is `latest`.

### Choosing a Different Base Image

By default, the SDK uses the following base images:

- `mcr.microsoft.com/dotnet/runtime-deps` for self-contained apps
- `mcr.microsoft.com/dotnet/aspnet` image for ASP.NET Core apps
- `mcr.microsoft.com/dotnet/runtime` for other cases

You can switch to a smaller or different image:

```xml
<PropertyGroup>
  
  <ContainerBaseImage>mcr.microsoft.com/dotnet/aspnet:10.0-alpine</ContainerBaseImage>
</PropertyGroup>
```

You could also do this by setting `ContainerFamily` to `alpine`, and letting the rest be inferred.

Here's the size difference between the default and Alpine images for an ASP.NET Core app:

<Image
  src="blogs/mnw_179/container_image_size.png"
  width={1623}
  height={409}
  alt="An image showing the size difference between the default and Alpine base images for ASP.NET Core applications."
/>

```text
| Base Image                                  | Size (MB) |
| ------------------------------------------- | --------- |
| mcr.microsoft.com/dotnet/aspnet:10.0        | 231.73    |
| mcr.microsoft.com/dotnet/aspnet:10.0-alpine | 122.65    |
```

You can see a significant size reduction by switching to `alpine`.

### Configuring Ports

For web applications, the default exposed ports are `8080` and `8081` for HTTP and HTTPS.
These are inferred from ASP.NET Core environment variables (`ASPNETCORE_URLS`, `ASPNETCORE_HTTP_PORT`, `ASPNETCORE_HTTPS_PORT`).
The `Type` attribute can be `tcp` or `udp`.

```xml
<PropertyGroup>
  <ContainerPort Include="8080" Type="tcp" />
  <ContainerPort Include="8081" Type="tcp" />
</PropertyGroup>
```

## Publishing to a Container Registry

Publishing locally is useful for development, but you'll want to push to a registry for deployment.
You can specify the target registry during publishing.

Here's an example publishing to GitHub Container Registry:

```bash
dotnet publish --os linux --arch x64  /t:PublishContainer /p:ContainerRegistry=ghcr.io
```

**Authentication**: The SDK uses your local Docker credentials.
Make sure you've logged in with `docker login` before publishing to a remote registry.

However, **I don't use the above approach**.
I prefer using docker CLI for the publishing step, as it gives me more control over authentication and tagging.

## CI/CD Integration

Here's what I'm doing in my [**GitHub Actions workflow**](how-to-build-ci-cd-pipeline-with-github-actions-and-dotnet) to build and push my .NET app container.
I left out the boring bits of seting up the .NET environment and checking out code.

This will build the container image, tag it, and push it to [GitHub Container Registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry):

```yaml
- name: Publish
  run: dotnet publish "${{ env.WORKING_DIRECTORY }}" --configuration ${{ env.CONFIGURATION }} --os linux -t:PublishContainer
# Tag the build for later steps
- name: Log in to ghcr.io
  run: echo "${{ env.DOCKER_PASSWORD }}" | docker login ghcr.io -u "${{ env.DOCKER_USERNAME }}" --password-stdin
- name: Tag Docker image
  run:
    docker tag ${{ env.IMAGE_NAME }}:${{ github.sha }} ghcr.io/${{ env.DOCKER_USERNAME }}/${{ env.IMAGE_NAME }}:${{ github.sha }} |
    docker tag ${{ env.IMAGE_NAME }}:latest ghcr.io/${{ env.DOCKER_USERNAME }}/${{ env.IMAGE_NAME }}:latest
- name: Push Docker image
  run:
    docker push ghcr.io/${{ env.DOCKER_USERNAME }}/${{ env.IMAGE_NAME }}:${{ github.sha }} |
    docker push ghcr.io/${{ env.DOCKER_USERNAME }}/${{ env.IMAGE_NAME }}:latest
```

Once my images are up in the registry, I can deploy them to my VPS.

I'm using [Dokploy](https://dokploy.com/) (a simple but powerful deployment tool for Docker apps) to pull the latest image and restart my service.

```yaml
deploy:
  runs-on: ubuntu-latest
  needs: build-and-publish
  steps:
    - name: Trigger deployment
      run: |
        curl -X POST ${{ env.DEPLOYMENT_TRIGGER_URL }} \
          -H 'accept: application/json' \
          -H 'Content-Type: application/json' \
          -H 'x-api-key: ${{ env.DEPLOYMENT_TRIGGER_API_KEY }}' \
          -d '{
            "applicationId": "${{ env.DEPLOYMENT_TRIGGER_APP_ID }}"
          }'
```

This kicks off a deployment on my VPS, pulling the latest image and restarting the container.

<Image
  src="blogs/mnw_179/dokploy_deployment.png"
  width={1206}
  height={833}
  alt="An image showing the output of the dokploy deployment command restarting the container."
/>

By the way, I'm running my VPS on [Hetzner Cloud](https://www.hetzner.com/cloud) - highly recommended if you're looking for affordable and reliable VPS hosting.

## When You Still Need a Dockerfile

The SDK container support is powerful, but it doesn't cover every scenario.

You'll still need a Dockerfile when:

- **Installing system dependencies**: If your app needs native libraries (like `libgdiplus` for image processing)
- **Complex multi-stage builds**: When you need to run custom build steps
- **Non-.NET components**: If your container needs additional services or tools

For most web APIs and background services, the SDK approach is sufficient.

## Summary

The .NET SDK's built-in container support removes the friction of containerization.

You get:

- **No Dockerfile to maintain** - one less file to worry about
- **Automatic base image selection** - always uses the right image for your framework version
- **MSBuild integration** - configure everything in your `.csproj`
- **CI/CD friendly** - works anywhere `dotnet` runs

The days of copy-pasting Dockerfiles between projects are over.

Just enable the feature, customize what you need, and publish.

Thanks for reading.

And stay awesome!

---


<Sponsorships>

[**Push updates live faster with RunPod**](https://fandf.co/4aVMokD)\
Spin up compute and storage in seconds with RunPod.
Deploy serverless inference that scales automatically, with no manual tuning or capacity planning.
Push updates live, iterate fast, and monitor everything in one place with logs, and metrics built in.
RunPod is infrastructure for AI startups and technical teams who want to stay in flow, move fast,
and spend less time on cloud plumbing and more time shipping real products.
[**Explore Runpod**](https://fandf.co/4aVMokD)

[**Data Engineering Design Patterns - Book Giveaway from Buf**](https://fandf.co/49GvtA9)\
Buf Schema Registry brings type safety to data pipelines with Protobuf.
Manage schemas centrally, validate changes, and generate Python code automatically.
Plus: Buf is giving away an O'Reilly Media book, Data Engineering Design Patterns with proven solutions to
idempotency, error handling, observability challenges and more.
[**Get your copy now**](https://fandf.co/49GvtA9).

</Sponsorships>

Security requirements evolve.
What was considered "secure enough" five years ago might not pass a security audit today.

You need to upgrade to a modern algorithm like [Argon2](https://en.wikipedia.org/wiki/Argon2) or [Bcrypt](https://en.wikipedia.org/wiki/Bcrypt).
But here is the problem: hashing is a **one-way operation**.
You cannot reverse-engineer the existing hashes to "upgrade" them.

If you simply swap your `IPasswordHasher` implementation, you break the application.
Every single existing user who tries to log in will fail authentication because your new hasher doesn't understand the old format.

In this article, I want to demo a **zero-downtime migration** concept in practice.

Real systems have more constraints (and you should not build auth from scratch).
But this is a clean example of a **pattern** you can reuse for [**database migrations**](efcore-migrations-a-detailed-guide):

- Move from old format to new format
- Keep existing behavior working
- Gradually migrate data
- Delete legacy only when you are done

Let's dive in.

## The Naive Approach and Why It Fails

Let's imagine you have a simple authentication system.
You want to replace your legacy `PBKDF2` hasher with a standard `Argon2` implementation.

You might think, "I'll just register the new implementation in the dependency injection container."

```csharp
// Switching from LegacyHasher to ModernHasher
builder.Services.AddSingleton<IPasswordHasher, ModernHasher>();
```

Here is the failure scenario:

1. **New Users:** They register and log in perfectly.
   Their passwords are hashed with Argon2 from day one.
2. **Existing Users:** A user enters their correct password.
   The system fetches the _old_ [PBKDF2](https://en.wikipedia.org/wiki/PBKDF2) hash from the database.
3. **The Crash:** The `ModernHasher` tries to verify the PBKDF2 hash.
   It fails immediately, returning `401 Unauthorized`.

You have inadvertently locked out your entire user base.
We need a way to support _both_ algorithms simultaneously without making the login code a mess.

## The Solution: Migration on Login

The strategy is simple: we don't migrate the database in a batch job.
We migrate users lazily when they prove their identity.

The flow looks like this:

1. **Attempt 1:** Try to verify the password using the **New** algorithm.
2. **Attempt 2 (Fallback):** If that fails, check if the **Legacy** algorithm can verify it.
3. **The Migration:** If the _Legacy_ verification succeeds:
   - Log the user in (Success).
   - **Immediately re-hash** their password using the **New** algorithm.
   - Update the database record.

Future logins for this user will now succeed via the standard flow.

<Image
  src="blogs/mnw_178/migration_on_login_flow.png"
  width={959}
  height={1189}
  alt="A sequence diagram showing the migration on login flow, with attempts to verify using new and legacy hashers."
/>

## Implementation with .NET Keyed Services

In .NET 8, Microsoft introduced **Keyed Services**, which are perfect for this scenario.
They allow us to register multiple implementations of the same interface and retrieve them by name.

### 1. Registering the Services

We register both hashers in our `Program.cs`, assigning them unique keys:

```csharp
// Register the implementations with specific keys
builder.Services.AddKeyedSingleton<IPasswordHasher, Pbdkf2PasswordHasher>("legacy");
builder.Services.AddKeyedSingleton<IPasswordHasher, Argon2PasswordHasher>("modern");

// (Optional) Register the modern one as the default for other services
builder.Services.AddSingleton<IPasswordHasher, Argon2PasswordHasher>();
```

### 2. The Login Command Handler

Now we implement the migration logic.
We inject both hashers using the `[FromKeyedServices]` attribute.

```csharp
public class LoginCommandHandler(
    IUserRepository userRepository,
    [FromKeyedServices("modern")] IPasswordHasher newHasher,
    [FromKeyedServices("legacy")] IPasswordHasher legacyHasher)
{
    public async Task<AuthenticationResult> Handle(LoginCommand command)
    {
        var user = await userRepository.GetByEmailAsync(command.Email);
        if (user is null)
        {
            return AuthenticationResult.Fail();
        }

        // 1. Try the new algorithm first (Happy Path)
        if (newHasher.Verify(user.PasswordHash, command.Password))
        {
            return AuthenticationResult.Success(user);
        }

        // 2. Fallback: Check if it's a legacy hash
        if (legacyHasher.Verify(user.PasswordHash, command.Password))
        {
            // 3. MIGRATION STEP: Re-hash and save
            var newHash = newHasher.Hash(command.Password);

            user.UpdatePasswordHash(newHash);
            await userRepository.SaveChangesAsync();

            return AuthenticationResult.Success(user);
        }

        return AuthenticationResult.Fail();
    }
}

```

This code ensures that active users are automatically upgraded.
After a few months, the vast majority of your user base will be on the new algorithm.

## Real-World Improvements

While the implementation above works, here are two improvements to make it production-ready.

### 1. Algorithm Prefixes

Relying on "trial and error" verification works, but it's cleaner to know exactly which algorithm was used to create a hash.

Standard algorithms often include a prefix (e.g., Bcrypt starts with `$2a$` or `$2b$`).
You can use this to route the request efficiently:

```csharp
public bool IsLegacyHash(string hash)
{
    // This assumes we're storing a prefix for PBKDF2 hashes. Something to consider.
    return hash.StartsWith("pbkdf2$");
}
```

Another benefit this unlocks is being able to query the database for users still on the legacy format.

### 2. Feature Flags

Performing a database write during a login request adds latency.
If you have high traffic, you might want to control this roll-out.

By wrapping the migration logic behind a [**Feature Flag**](feature-flags-in-dotnet-and-how-i-use-them-for-ab-testing), you can disable the "write" step if your database comes under load,
while still allowing users to log in via the read-only fallback.

```csharp
if (await featureManager.IsEnabledAsync(FeatureFlags.MigratePasswords) &&
    legacyHasher.Verify(user.PasswordHash, command.Password))
{
    // Perform migration...
}
```

## Finishing the Migration

After you run this for a while (usually a few months), most active accounts will be upgraded.
You can then run a cleanup script to identify any remaining legacy hashes and force those users to reset their passwords on the next login attempt.

At that point, you can remove:

- The legacy hasher registration
- The legacy verification code path
- The feature flag

And the migration is complete.

## Summary

A "simple" hashing upgrade is really a **data migration**.
This article is about the migration pattern.
Not about reinventing auth.

The **zero-downtime pattern** looks like this:

1. New format for new writes
2. Support both formats for reads
3. Migrate old data gradually (migrate-on-login is a great trick)
4. Put it behind a feature flag
5. Delete legacy when you are done

By allowing the old and new formats to coexist for a period of time, you achieve a seamless transition.
Once your monitoring shows that 99% of active users have migrated,
you can identify the users on the legacy format and force a password reset on their next attempt.

If you want to see a practical demo of this, [**check out this video I made**](https://youtu.be/7YUV4O9eMjQ).

Hope this was helpful!

---



<Sponsorships>

[**MongoDB's new Voyage 4 series**](https://fandf.co/3NbvbcW), a unified family of text embedding models featuring shared embedding spaces,
lets you mix and match models for documents and queries in the same pipeline for a flexible,
cost-effective embedding solution for local development (voyage-4-nano) and production (voyage-4-large).
Mix, match, and optimize: [**start building with Voyage 4 (First 200M tokens free)**](https://fandf.co/3NbvbcW)

Building the next big thing?
[**MongoDB for Startups**](https://fandf.co/49JEDwU) gives you the tools and support to accelerate your growth.
Jumpstart your development with MongoDB Atlas credits; build on a unified data platform that integrates document storage, vector search, and database scalability;
and leverage MongoDB's technical expertise to [**grow from idea to IPO**](https://fandf.co/49JEDwU).

</Sponsorships>

Distributed systems are great for scalability, but they introduce a whole new class of problems.
One of the hardest problems to solve is **cache invalidation**.

In .NET 9, [**Microsoft introduced `HybridCache`**](hybrid-cache-in-aspnetcore-new-caching-library) to simplify caching.
It's a fantastic library that combines the speed of in-memory caching (L1) with the durability of distributed caching (L2) like Redis.
It also handles "cache stampede" protection out of the box.

However, there is a catch.

When you run multiple instances of your application, `HybridCache` doesn't automatically synchronize the local L1 cache across all nodes.
If you update data on **Node A**, **Node B** will continue serving stale data from its in-memory cache until the entry expires.

While HybridCache is a massive step forward, the lack of a built-in backplane for invalidation is a known limitation.
In fact, there is an active discussion on the [dotnet/extensions GitHub repository](https://github.com/dotnet/extensions/issues/5517) tracking this exact feature request.
Until that ships, we have to roll our own solution.

In this week's newsletter, we'll explore:

- The distributed caching dilemma
- Why `HybridCache` doesn't solve this alone
- Using Redis Pub/Sub as a backplane
- Implementing real-time cache invalidation

Let's dive in.

## The Distributed Caching Dilemma

Let's imagine a typical production scenario.
You have an API running on multiple servers (or pods) behind a load balancer.

To improve performance, you introduce [**caching**](caching-in-aspnetcore-improving-application-performance).
You want the speed of local memory, so you use `HybridCache`.

Here is the failure scenario:

1. **User A** updates their profile on **Server 1**.
2. **Server 1** updates the database and clears its local cache.
3. **User A** (or User B) hits **Server 2**.
4. **Server 2** still holds the _old_ profile data in its local `HybridCache`.
5. The user sees outdated information, since the local cache hasn't been invalidated.

<Image
  src="blogs/mnw_177/hybridcache_out_of_sync_scenario.png"
  width={2007}
  height={1378}
  alt="A sequence diagram showing two servers with HybridCache out of sync after a user update."
/>

**Why Not Just Shorten the Cache Duration?**

A common "hack" to solve this is to simply reduce the L1 cache duration (TTL).
For example, setting the local cache to expire every 10 seconds.

While this reduces the window of inconsistency, it doesn't solve the problem.
It just masks it.

This approach introduces two new issues:

- **Increased Latency**: You are now forcing your application to reach out to the distributed L2 cache (Redis) or the database much more frequently.
- **Lost Efficiency**: The main benefit of L1 caching is avoiding network requests entirely.
  If you expire data too fast, you lose the performance gain for the majority of your traffic.

For things like user permissions, feature flags, or pricing, "mostly correct" is often not good enough. You need immediate consistency.

## The Solution: Redis Pub/Sub Backplane

To solve this, we need a **backplane**.
It's a communication channel that connects all our application nodes.

When a cache entry is removed or updated on one node, we publish a message to the backplane.
All other nodes subscribe to this channel and, upon receiving the message, remove the corresponding key from their local cache.

Redis is already a popular choice for the L2 cache, so it makes perfect sense to use its [**Pub/Sub feature**](simple-messaging-in-dotnet-with-redis-pubsub)
for this signaling mechanism.

It works like this:

1. **Publisher:** The node that modifies data publishes a `cache-invalidation` message with the cache key.
2. **Subscriber:** All nodes listen to this channel.
3. **Action:** When a message arrives, they call `HybridCache.RemoveAsync(key)`.

<Image
  src="blogs/mnw_177/cache_invalidation_message_fanout.png"
  width={1873}
  height={1354}
  alt="A sequence diagram showing cache invalidation messages being published and received by multiple servers."
/>

## Implementing the Solution

We will need the `StackExchange.Redis` library to handle the messaging.

Let's start by defining a simple service to handle the publishing.
This service will be responsible for notifying the rest of the system that a key has changed.

```csharp
public interface ICacheInvalidator
{
    Task InvalidateAsync(string key, CancellationToken cancellationToken = default);
}

public class RedisCacheInvalidator(
    IConnectionMultiplexer connectionMultiplexer,
    ILogger<RedisCacheInvalidator> logger)
    : ICacheInvalidator
{
    private const RedisChannel Channel = RedisChannel.Literal("cache-invalidation");

    public async Task InvalidateAsync(string key, CancellationToken cancellationToken = default)
    {
        var subscriber = connectionMultiplexer.GetSubscriber();

        await subscriber.PublishAsync(Channel, new RedisValue(key));

        logger.LogInformation("Published invalidation for key: {Key}", key);
    }
}
```

Now, whenever you update an entity in your Command Handler or Service, you just call `ICacheInvalidator.InvalidateAsync`.

```csharp {3,20}
public class UpdateUserProfileHandler(
    AppDbContext dbContext,
    ICacheInvalidator cacheInvalidator,
    ILogger<UpdateUserProfileHandler> logger)
{
    public async Task Handle(int userId, string newName, CancellationToken ct)
    {
        // 1. Update the database
        var user = await dbContext.Users.FindAsync([userId], ct);
        if (user is null)
        {
             return;
        }

        user.Name = newName;
        await dbContext.SaveChangesAsync(ct);

        // 2. Invalidate the cache (Distributed)
        var cacheKey = $"user:{userId}";
        await cacheInvalidator.InvalidateAsync(cacheKey, ct);

        logger.LogInformation("Updated user and invalidated cache for {UserId}", userId);
    }
}
```

### The Background Listener

Next, we need a [**background service**](running-background-tasks-in-asp-net-core) that runs on every node.
It will subscribe to the Redis channel and evict keys from the local `HybridCache`.

**A quick note on self-publishing**:
Because Redis Pub/Sub broadcasts to everyone subscribed, the node that published the invalidation will also receive the message.
In this implementation, we simply remove the key again.
It's redundant but harmless, and it keeps the code simple.

Note that we are injecting `HybridCache` directly into our background service.
An alternative is using `IMemoryCache`, since that is the L1 cache inside `HybridCache`.

```csharp
public class CacheInvalidationService(
    IConnectionMultiplexer connectionMultiplexer,
    HybridCache hybridCache,
    ILogger<CacheInvalidationService> logger)
    : BackgroundService
{
    private const RedisChannel Channel = RedisChannel.Literal("cache-invalidation");

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var subscriber = connectionMultiplexer.GetSubscriber();

        await subscriber.SubscribeAsync(Channel, (channel, value) =>
        {
            string key = value.ToString();

            logger.LogInformation("Invalidating local cache for: {Key}", key);

            // This removes the item from the local L1 cache
            var task = hybridCache.RemoveAsync(key, stoppingToken);

            if (!task.IsCompleted)
            {
                task.GetAwaiter().GetResult();
            }
        });
    }
}

```

### Wiring It All Together

Finally, we need to register these services in our DI container.

```csharp
builder.Services.AddSingleton<IConnectionMultiplexer>(sp =>
    ConnectionMultiplexer.Connect("<REDIS_CONNECTION_STRING>"));

// Register HybridCache (defaults generally work fine for L1)
builder.Services.AddHybridCache();

// Register our invalidation services
builder.Services.AddSingleton<ICacheInvalidator, RedisCacheInvalidator>();
builder.Services.AddHostedService<CacheInvalidationService>();

```

Now, when **Node A** calls `InvalidateAsync("user:123")`, Redis pushes that message to **Node B**, **Node C**, and so on.
They all trigger `hybridCache.RemoveAsync("user:123")`, ensuring the next request fetches fresh data from the source (or the shared L2).

## A Better Way: FusionCache

If building your own backplane feels like reinventing the wheel, you should look at [**FusionCache**](https://github.com/ZiggyCreatures/FusionCache).

FusionCache is a mature, battle-tested library that has solved this exact problem for years.
It has a built-in backplane feature that automatically handles the Pub/Sub messaging for you.

Even better, FusionCache recently added an implementation of the HybridCache abstract class.
This means you can swap it in without changing much of your existing code.

```csharp {3}
// Using FusionCache's implementation of HybridCache
builder.Services.AddFusionCache()
    .WithBackplane(
        new RedisBackplane(new RedisBackplaneOptions { Configuration = "<REDIS_CONNECTION_STRING>" }))
    .AsHybridCache();
```

## Summary

`HybridCache` is a powerful addition to the .NET ecosystem, effectively merging the benefits of `IMemoryCache` and `IDistributedCache`.
However, for multi-node setups requiring high consistency, you still need a mechanism to synchronize the local caches.

Redis Pub/Sub offers a lightweight, effective solution to this problem.

By implementing a simple "bus" for invalidation messages, you get the best of both worlds:
the extreme performance of local caching and the data consistency of a distributed system.

Good luck out there, and see you next week.

---



<Sponsorships>

[**Meet your AI App and Agent Factory**](https://fandf.co/49R20Ur)\
It's never been easier to build, optimize, and govern AI apps and agents that understand your business context and deliver business impact.
Microsoft Foundry helps teams accelerate innovation with open-source frameworks, the widest model selection on any cloud, RAG on data stored anywhere,
secure MCP tools, and VS Code integration.
Govern the entire AI lifecycle with end-to-end observability, evaluations, and security controls.
Stop wrestling with disconnected AI tools.
Find out why Microsoft Foundry is used by developers at more than 80,000 organizations, including 80% of Fortune 500 companies.
👉 [**Get started in minutes**](https://fandf.co/49R20Ur)

Code your personal credit card to meet your needs.
With [**Klutch**](https://fandf.co/3NxWqhS), your card becomes an API-powered dev environment, offering direct access to financial data, unlimited virtual cards,
and full programmatic control over spending rules, automations, and real-time transaction behavior.
[**Check out what you can do with a Klutchcard**](https://fandf.co/3NxWqhS).

</Sponsorships>

Most systems don't need _global_ message ordering.

They need something simpler and more useful: **events must be handled in order per aggregate**.

Per `OrderId`, per `InvoiceId`, per `CustomerId`, or whatever your aggregate boundary is.
You can make this as broad or as narrow as you need.

This starts as an eventing problem, but if you follow the requirements to their logical conclusion, you'll end up with a workflow.
And that workflow has a name: **a saga**.

## Domain Events Feel Like the Clean Solution

[**Domain events**](how-to-use-domain-events-to-build-loosely-coupled-systems) are attractive because they come from first principles:

- An aggregate changes state
- It emits events describing what happened
- Handlers react and do useful work

You also get a nice mental model:

> State change → Event → Reaction

A typical example:

- `OrderPlaced`
- `PaymentCaptured`
- `OrderShipped`

But there's a catch.

Domain events are **brittle** when you try to use them for integration.

<Image
  src="blogs/mnw_176/domain_events_flow.png"
  width={2158}
  height={1056}
  alt="A sequence diagram showing an Aggregate committing a change, emitting Domain Events, which are then processed by multiple Event Handlers."
/>

If you publish directly from your transaction, you're coupling business correctness to an unreliable side effect:

- The transaction succeeds but publishing fails
- Publishing succeeds but the transaction rolls back
- Consumers process duplicates
- Retries cause reordering

So we keep the model, but harden the delivery.

## The Outbox Makes Publishing Reliable (but not ordered)

With an [**Outbox**](implementing-the-outbox-pattern), we store outgoing events in the same transaction as the aggregate update.

Then a background publisher reads the Outbox and pushes events to a queue.

This fixes the reliability problem:

- If the transaction commits, the event is persisted
- If the publisher crashes, it can resume later
- We can retry safely

<Image
  src="blogs/mnw_176/outbox_flow.png"
  width={3473}
  height={549}
  alt="A diagram showing an Aggregate committing a change along with an Outbox entry, which is then published to a Queue by a background process."
/>

Now we've made event publishing reliable.

But we haven't made event handling ordered.

## Competing consumers Are Great, Until Order Matters

The moment events hit a queue, we usually scale with the simplest lever: [**competing consumers**](event-driven-architecture-in-dotnet-with-rabbitmq).

Multiple instances consume from the same queue to increase throughput.

That works… until ordering matters.

Two events for the same `OrderId` can be processed at the same time:

- Consumer A receives `PaymentCaptured`
- Consumer B receives `OrderPlaced`
- Side effects run out of order

Even if the events were published in order, retries and redelivery can scramble processing order.

And now you have a subtle bug that only appears under load.

<Image
  src="blogs/mnw_176/competing_consumers.png"
  width={2093}
  height={1050}
  alt="A sequence diagram showing multiple consumers processing messages from a Queue concurrently, leading to out-of-order handling of events for the same aggregate ID."
/>

That's the key realization: queues scale work.
They don't preserve your invariants.

## What We Really Want is Per-Aggregate Ordering

You don't need one ordered line for _everything_.

You need many independent ordered lines, one per aggregate.

That usually holds true because:

- Aggregates already define consistency boundaries
- Events are naturally produced in order (v1, v2, v3…)
- The "correct" order is the aggregate's own timeline

If we could guarantee that **only one handler processes events for a given aggregate at a time**, most of the problem disappears.

The most direct solution is also the simplest: **use a single consumer for the whole stream.**

That enforces ordering, assuming events are published in order.

<Image
  src="blogs/mnw_176/single_consumer.png"
  width={1730}
  height={952}
  alt="A sequence diagram showing a single consumer processing messages from a Queue, leading to ordered handling of events for the same aggregate ID."
/>

But it has an obvious drawback.

## A Single Consumer Solves Ordering But Limits Scale

One consumer means:

- Throughput ceiling (one worker)
- Latency spikes under load
- Scaling becomes vertical, not horizontal

Even if your events are lightweight, you're artificially bottlenecking the system.

So we want:

- **Per-aggregate ordering**
- **Horizontal scaling**
- **Reliability (Outbox still stays)**

This is where teams often "invent" the next step.

## Publish the Next Message From the Handler

If competing consumers break ordering, one natural idea is:

> Don't let the queue decide what's next, we decide.

Instead of dumping all events into the queue and letting consumers race, we move to a chained approach:

1. Handle one message for an aggregate
2. When done, publish the **next** message to be handled

Now, the system processes a single message at a time per aggregate.

<Image
  src="blogs/mnw_176/choreographed_saga.png"
  width={2842}
  height={920}
  alt="A diagram showing a sequence of events being processed one at a time per aggregate ID, with each event handler publishing the next event upon completion."
/>

This is the key moment:

You've stopped building "event handlers".

You've started building a **workflow**.

And that workflow is… a saga.

## Congratulations, You Built a Choreographed Saga

A [**choreographed saga**](orchestration-vs-choreography) is a workflow where:

- Each step reacts to an event
- Performs work
- Emits the next event to trigger the next step

There isn't a single central coordinator.

Instead, we have a chain of events: "when X happens, do Y, then publish Z".

This pattern naturally fits your new requirement:

- Per-aggregate ordering is preserved (the chain is sequential)
- You can scale across aggregates (many chains in flight)
- Each step is isolated and retryable

It also forces a useful discipline:

- "What's the next step?" becomes explicit
- Boundaries between steps become clearer
- You can observe the workflow as a sequence

But choreography has a limitation: **control is distributed**, so tracking progress and handling exceptions can get messy.

So we take the final step.

## If You Want Control, Introduce a State Machine Saga

When the workflow becomes important, you often want:

- A single place that knows the current state
- Visibility into progress ("where are we stuck?")
- Explicit timeouts and retries
- Compensating actions when something fails

That's when you move from choreography to **orchestration** via a [**state machine saga**](implementing-the-saga-pattern-with-masstransit):

- The saga holds the workflow state
- Events drive transitions
- The saga decides what message to publish next
- You gain control and observability

<Image
  src="blogs/mnw_176/state_machine.png"
  width={2901}
  height={2171}
  alt="A state machine diagram showing states and transitions for a saga managing ordered message processing per aggregate ID."
/>

This doesn't replace the Outbox, by the way.

You still want reliable publishing.

You've just made the workflow explicit.

## Broker Support Helps with Ordering, not Correctness

It's worth calling out that you don't always have to build this yourself.

Many popular message brokers provide technical primitives for **ordered processing per key** (your aggregate ID):

- [Amazon SQS FIFO message groups](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html) (per key)
- [Azure Service Bus sessions](https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-sessions) (per key)
- [Kafka Partitions](https://developer.confluent.io/courses/apache-kafka/partitions/) in a log (key → partition → ordered stream)
- [RabbitMQ "single active consumer"](https://www.rabbitmq.com/docs/consumers?#single-active-consumer) style semantics (per queue)

These features can eliminate the most common failure mode of competing consumers: **concurrent handling of messages for the same aggregate**.

But even with perfect per-aggregate ordering, you still need patterns around it to keep the system correct:

- **Outbox** to publish reliably (ordering is useless if events are lost)
- [**Idempotent consumers**](idempotent-consumer-handling-duplicate-messages) / **Inbox** because retries and duplicates still happen
- **Consistency boundaries** (what's safe to do inside the transaction vs outside)
- **Timeouts** + **compensation** when the "ordered sequence" is actually a business workflow that can partially fail

So broker-level ordering is a great foundation.
It reduces accidental complexity.
It just doesn't remove the need to model long-running work explicitly when the business demands it.

## Takeaway

If you follow the problem from first principles:

- Aggregates define the boundary where ordering matters
- The Outbox makes event publishing reliable
- Competing consumers break per-aggregate order
- A single consumer restores order but caps throughput
- Publishing "the next message" creates sequential progress per aggregate
- That sequential progress is a saga (choreographed first, state machine when you need control)

So you didn't reinvent something by accident.

You discovered that "ordered handling per aggregate at scale" is not a queue feature.

It's a workflow. And sagas are how we model workflows in distributed systems.

Once you see it that way, you stop fighting queues for ordering guarantees.

You design the workflow the business actually needs.

Hope this was helpful!

---



<Sponsorships>

[**Fly.io**](https://fandf.co/4pPGJRw) runs .NET apps where they belong - at the edge, close to users - with low latency and global reach.
Package your .NET app, deploy with flyctl, and scale worldwide with no ops overhead.
[**Learn how to deploy .NET with Fly.io**](https://fandf.co/4pPGJRw)

Oracle's new [**Database 26ai**](https://fandf.co/491wd3X) lets you run vector search, JSON relational duality,
and AI workloads directly in the database, without moving data between systems.
If you care about simpler architectures and fewer integration points, it's worth exploring.
[**Free here**](https://fandf.co/491wd3X)

</Sponsorships>

There is a specific kind of restlessness that only developers understand.
It's a quiet, persistent itch.
A mental "background process" that runs while you're eating, showering, or trying to sleep.
I call it **the urge to build**.

Looking back at [my GitHub contribution graphs](https://github.com/m-jovanovic) from the last few years, I see more than just green squares.
I see a timeline of my own evolution.

I was **chasing that dream** many developers have: build a small product, ship it, get users, maybe turn it into a business.
Most of those ideas didn't become a successful SaaS.

But that wasn't the point.

The point was that I was building constantly.
Learning constantly.
Shipping constantly.

And I loved it.

## The Indie Hacker Dream

Back in 2020, my contribution graph shows the classic "burst" pattern.
I was chasing the indiehacker dream.
You know the one: build a SaaS, get to ramen profitability, escape the 9-to-5, tweet about your MRR milestones.
I consumed every bootstrapper podcast, every "I made $10K/month" blog post, every ProductHunt launch breakdown.

I was working on a project called [Expensely](https://github.com/m-jovanovic/expensely-server).
I spent late nights pouring myself into this "genius" idea: a budgeting app.

Nothing revolutionary.
The world didn't need another expense tracker.
But I needed to build it.

Looking back now, it was far from genius.
But at the time, I was convinced that this was _the one_.

<Image
  src="blogs/mnw_175/github_contributions_2020.png"
  width={1350}
  height={600}
  alt="A GitHub contribution graph from 2020 showing a burst of activity in the end of the year."
/>

In 2021, that momentum stayed steady for five months before life (or perhaps reality) intervened.
Those green squares represent the **fun periods**.
They represent the thrill of architecting an application, the satisfaction of a passing test suite,
and the hope that you're building something that might change your life.

<Image
  src="blogs/mnw_175/github_contributions_2021.png"
  width={1350}
  height={600}
  alt="A GitHub contribution graph from 2021 showing steady activity for the first five months and then a tapering off."
/>

Expensely never became a business.
Most side projects don't.
But here's what I've come to understand: the outcome wasn't the point.

Even though those projects didn't become the next big SaaS, they were the forge where my skills were sharpened.
You don't "waste" time building something that fails.
You only waste time when you don't build at all.
And I mean this both metaphorically and literally.
Humans are meant to create.
When we stop creating, we stagnate.

## What Building Actually Gives You

When you build something from scratch - when you own every decision from the database schema to the button colors - you learn differently.
There's no senior developer to ask.
No established patterns to follow.
Every problem is yours to solve.

You can use [Railway-Oriented Programming](https://fsharpforfunandprofit.com/rop/) if you want to, because who is going to stop you?

Jokes aside, **functional programming** is **very useful**. You should learn it.

<Image
  src="blogs/mnw_175/rop_sample_code.png"
  width={938}
  height={890}
  alt="A screenshot of sample code implementing Railway-Oriented Programming in C#."
/>

Through my side projects, I wrestled with authentication, background jobs, payment integrations, deployment pipelines.
I made mistakes and spent late nights fixing them.
I built features nobody asked for and skipped features everyone needed.

There is **one thing side projects can't teach you** (unless you get users): dealing with the **consequences of your decisions**.
When you build for yourself, the only person affected by your mistakes is you.
When you build for users, every bug, every outage, every poor design choice has real consequences.

By mid 2021, the project had wound down.
I was settling into a comfortable routine at my day job, working as a senior engineer at a big corporation.
There was no time left for side projects.

## The Unexpected Pivot

By 2022, something changed.
My contribution graph exploded to over 1,600 commits.
But the focus shifted.
I wasn't building a product anymore.
I was building something more meaningful.
The repositories shifted to my tech blog and [YouTube projects](https://www.youtube.com/@MilanJovanovicTech).

I had **discovered** content creation.

The urge to build hadn't disappeared.
I found a new outlet.
Instead of building products for users, I was building educational content for developers.
Instead of SaaS metrics, I was tracking video views and newsletter subscribers.
To me it was something new, something exciting.

<Image
  src="blogs/mnw_175/github_contributions_2022.png"
  width={1350}
  height={600}
  alt="A GitHub contribution graph from 2022 showing steady activity throughout the year."
/>

I discovered that while I loved building software, I loved **explaining** it even more.
I transitioned from an aspiring founder to a software engineering educator.
At first it was just me writing and recording **things I wish someone had explained to me earlier**.

Then something interesting happened.

People started reading.
Watching.
Replying.
Asking questions.
More people showed up.
And then even more.

<Image
  src="blogs/mnw_175/youtube_milanjovanovictech.png"
  width={837}
  height={946}
  alt="A screenshot of the Milan Jovanovic Tech YouTube channel with over 140,000 subscribers."
/>

Today, I get to help thousands of developers around the world improve their careers.
It is, without a doubt, the **most rewarding work I've ever done**.

<div className="centered">
  <Image
    src="blogs/mnw_175/pca_testimonials.png"
    width={570}
    height={759}
    alt="A screenshot of testimonials from students of the Pragmatic Clean Architecture course."
  />
</div>

<figure className="figure-center">
  <Image
    src="blogs/mnw_175/pca_testimonials.png"
    width={570}
    height={759}
    alt="A screenshot of testimonials from students of the Pragmatic Clean Architecture course."
  />
  <figcaption>
    Student testimonials from [**Pragmatic Clean
    Architecture**](/pragmatic-clean-architecture).
  </figcaption>
</figure>

This is genuinely fulfilling work.
When someone messages me saying my content helped them land a job
or finally understand a concept they'd struggled with for years, that feeling is hard to describe.
I'm doing something meaningful.

And yet.

## The Lingering Itch

The urge to build something still remains.

It's not dissatisfaction.
It's not that content creation isn't "real" building, it absolutely is.
It's a different kind of building, certainly.
But this feeling I have is something more fundamental.
A part of my brain that wants to work on a fresh project.
That wants to solve a problem nobody has asked me to solve.
That wants to take an idea from nothing to something.

That's why I have a [Hetzner](https://www.hetzner.com/) server sitting idle right now.
Never know when I might need it.
Right?
Also, I'm getting a great deal for €3.29/month.
Can't argue with that.

<Image
  src="blogs/mnw_175/hetzner_server.png"
  width={1500}
  height={520}
  alt="A screenshot of a Hetzner server management dashboard showing a server."
/>

I think this urge never fully goes away for people like us.
We can channel it, redirect it, find new outlets for it.
But it's always there, lingering.
And maybe that's okay.
Maybe that restlessness is what makes us who we are.

## Why You Should Build Too

If you're reading this with your own idea rattling around in your head - some app concept, some tool you wish existed,
some problem you think you could solve - I want to tell you something.

**Build it.**

Not because it will make you rich.
Statistically, it won't.
Not because it will become the next big thing.
It probably won't be that either.

**Build it because the person who finishes that project will not be the same person who started it.**
You will learn things no tutorial can teach.
You will develop judgment that only comes from making real decisions with real consequences.
You will have stories, opinions, and experiences that set you apart.

The outcome is almost beside the point.
The transformation is the product.

Maybe you'll end up with a successful SaaS.
Maybe you'll end up with a failed project and a mass of hard-won knowledge.
Maybe, like me, you'll end up somewhere you never expected - doing work you couldn't have imagined when you wrote that first line of code.

You won't know until you build.

So open your terminal.
Type `git init`.
And start.

Let this be a year of building.

---



<Sponsorships>

Oracle's new [**Database 26ai**](https://fandf.co/491wd3X) lets you run vector search, JSON relational duality,
and AI workloads directly in the database, without moving data between systems.
If you care about simpler architectures and fewer integration points, it's worth exploring.
[**Free here**](https://fandf.co/491wd3X)

[**AI runs on data. Data runs on dbt.**](https://fandf.co/3MI0Tys)\
Give your data the structured context it needs for consistent, governed insights.
The new O'Reilly report, “Structured for Intelligence: Why AI Needs Governed, Discoverable and Provisioned Data,” is the essential roadmap.
Learn how to use dbt Labs and MCP to build safe, scalable AI driven analytics and confidently shift from dashboards to dialogue.\
[**Read the Full Report**](https://fandf.co/3MI0Tys)

</Sponsorships>

A couple of days ago, I was looking at a piece of code that's doing too much work.
I'm sure you'll be able to draw a parallel to something in your own applications.
Maybe it's a database call that should be faster, or an external API that's starting to bill you by the thousands.

My first instinct is: **"I'll just cache it"**.

In .NET, that usually means reaching for `IMemoryCache` or plugging in a distributed cache like Redis.

But have you ever stopped to wonder what's actually happening inside those libraries?\
Why do we need all that complexity just to store a value in memory?

So I spent the afternoon trying to build a [**high-performance cache**](caching-in-aspnetcore-improving-application-performance) from scratch.

I don't recommend DIY-ing your own caching library for production use.
But I learn best by doing something myself.
Understanding these patterns (concurrency, race conditions, and [**locking**](introduction-to-locking-and-concurrency-control-in-dotnet-6))
is what separates a "coder" from an engineer.

## The Starting Point

I was working on a simple currency conversion handler.
We're calling a third-party API to get exchange rates.
The API returns the current exchange rate for a given currency code (like EUR, GBP, JPY) against USD.

This is the initial implementation:

```csharp {23}
public static class CurrencyConversion
{
    public static async Task<IResult> Handle(
		string currencyCode,
		decimal amount,
		CurrencyApiClient currencyClient)
	{
		// Validate currency code format (3 uppercase letters)
		if (string.IsNullOrWhiteSpace(currencyCode) ||
			currencyCode.Length != 3 ||
			!currencyCode.All(char.IsLetter))
		{
			return Results.BadRequest(
				new { error = "Currency code must be a 3-letter uppercase code (e.g., EUR, GBP)" });
		}

		// Validate amount (must be positive)
		if (amount < 0)
		{
			return Results.BadRequest(new { error = "Amount must be a positive number" });
		}

		var rate = await currencyClient.GetExchangeRateAsync(currencyCode);

		if (rate == null)
		{
			return Results.NotFound(
				new { error = $"Exchange rate for {currencyCode} not found or API error occurred" });
		}

		var convertedAmount = amount * rate.Value;

		return Results.Ok(new ExchangeRateResponse(
			Currency: currencyCode,
			BaseCurrency: "USD",
			Rate: rate.Value,
			Amount: amount,
			ConvertedAmount: convertedAmount
        ));
    }
}
```

This works fine in your local dev environment.
But in production, if 100 people hit this at the same time, you're making 100 identical network calls.
Your API provider will hate you (and you may even get rate limited), and your latency will spike.

Now let's build a cache to fix this without using any external libraries.
Remember, we're doing this for learning purposes only.

## Level 1: Adding a `ConcurrentDictionary`

Your first thought is probably to store the rates in a `ConcurrentDictionary`. It’s thread-safe, so it feels like the right tool.

```csharp {4, 11}
private static readonly ConcurrentDictionary<string, decimal> Cache = new();

// In the Handler:
if (Cache.TryGetValue(currencyCode, out var cachedRate))
{
	return cachedRate;
}

var rate = await currencyClient.GetExchangeRateAsync(currencyCode);

Cache.TryAdd(currencyCode, rate.Value);
```

This definitely helps with performance under load.
Multiple threads can read and write to the dictionary without crashing.
But `ConcurrentDictionary` protects the _dictionary structure_, not your _logic_.

If 100 users request "EUR" at the exact same time, `TryGetValue` will return false for all of them.
They will all proceed to call the API.
This is a classic [**race condition**](solving-race-conditions-with-ef-core-optimistic-locking).
You've protected your memory, but you haven't protected the external API.

There's also another problem with this approach: the rates never expire.

## Level 2: Adding Cache Expiration

Currency rates don't stay the same forever.
We need a way to expire them.
Since `ConcurrentDictionary` doesn't have a "Time to Live" (TTL), we have to wrap our data.

```csharp {9,10}
// Store both the rate and the time it was created
private record CacheEntry(decimal Rate, DateTime CreatedAt);

// Our cache now stores CacheEntry objects
private static readonly ConcurrentDictionary<string, CacheEntry> Cache = new();
private static readonly TimeSpan CacheDuration = TimeSpan.FromMinutes(5);

// Check: Is it there? And is it still "fresh"?
if (Cache.TryGetValue(currencyCode, out var entry) &&
    (DateTime.UtcNow - entry.CreatedAt) < CacheDuration)
{
	return entry.Rate;
}
```

Now we have expiration.
But we've actually created a new problem: [The Thundering Herd](https://en.wikipedia.org/wiki/Thundering_herd_problem) (a.k.a [Cache Stampede](https://en.wikipedia.org/wiki/Cache_stampede)).

Every 5 minutes, when the cache expires, all incoming traffic will see "stale" data and try to refresh it at once.

So we need to fix that next.

## Level 3: Solving the "Cache Stampede"

To fix the stampede, we need to ensure that only _one_ person can fetch the update while everyone else waits.

How do we do that in C#?

We use a `SemaphoreSlim` and a pattern called [Double-Checked Locking](https://en.wikipedia.org/wiki/Double-checked_locking).
We check the cache once (the "fast path"), then we lock, and then we check _again_ to see if someone else filled it while we were waiting for the lock.

```csharp {12,30,36}
// Basically a mutex but async-friendly
private static readonly SemaphoreSlim Lock = new(1, 1);

public static async Task<decimal> GetRateAsync(string code, CurrencyApiClient client)
{
	// Fast path: No locking needed
	if (Cache.TryGetValue(code, out var entry) && IsFresh(entry))
	{
		return entry.Rate;
	}

    var acquired = await Lock.WaitAsync(TimeSpan.FromSeconds(10)); // Avoid deadlocks
    if (!acquired)
    {
        throw new Exception("Could not acquire lock to fetch exchange rate.");
    }
	try
	{
		// Double-check: Did someone else finish the API call while we waited?
		if (Cache.TryGetValue(code, out entry) && IsFresh(entry))
		{
			return entry.Rate;
		}

		var rate = await client.GetExchangeRateAsync(code);
		var newEntry = new CacheEntry(rate.Value, DateTime.UtcNow);

		// Atomically update the cache
		// This is safe because we're inside the lock
		Cache.AddOrUpdate(code, newEntry, (_, _) => newEntry);
		return rate.Value;
	}
	finally
	{
		// Always release the lock
		Lock.Release();
	}
}
```

This is an improvement.
But something still feels off.

Can you spot the _problem_ with this code?

Our lock behaves like a global lock.
This means that if one thread is fetching "EUR", all other threads (even those requesting "JPY") are blocked until the "EUR" fetch completes.
This problem is called **lock contention**.

Let's fix that next.

## Level 4: Scaling with Keyed Locking

The "pro" move here is **Keyed Locking**.
We create a lock for every specific currency.
Since the number of currencies is finite, this isn't too memory-intensive.

We need an additional `ConcurrentDictionary` to hold our semaphores, per currency code.

```csharp {4}
private static readonly ConcurrentDictionary<string, SemaphoreSlim> Locks = new();

// In the Handler:
var semaphore = Locks.GetOrAdd(currencyCode, _ => new SemaphoreSlim(1, 1));
if (!Cache.TryGetValue(currencyCode, out var cachedRate) &&
	DateTime.UtcNow - cachedRate?.CreatedAt < CacheDuration)
{
    var acquired = await semaphore.WaitAsync(TimeSpan.FromSeconds(10));
    if (!acquired)
    {
        throw new Exception("Could not acquire lock to fetch exchange rate.");
    }
	try
	{
		// Fetch and update logic...
	}
	finally { semaphore.Release(); }
}
```

The only thing that changes is how we acquire the lock.
Now, if one thread is fetching "EUR", other threads requesting "JPY" can proceed without waiting.
This is the most scalable version of our cache.

**But...** It only works in memory.
So it's not suitable for distributed systems or multiple server instances.
There are also a few more edge cases to consider, but you can explore those as an exercise.

## The Final Code

Here's the final version of our caching logic:

```csharp
public static class CurrencyConversion
{
    private record CacheEntry(decimal Rate, DateTime CreatedAt);
    private static readonly ConcurrentDictionary<string, CacheEntry> Cache = new();
    private static readonly TimeSpan CacheDuration = TimeSpan.FromMinutes(5);

    private static readonly ConcurrentDictionary<string, SemaphoreSlim> Locks = new();

    public static async Task<IResult> Handle(
        string currencyCode,
        decimal amount,
        CurrencyApiClient currencyClient)
    {
        // Validate currency code format (3 uppercase letters)
        if (string.IsNullOrWhiteSpace(currencyCode) ||
            currencyCode.Length != 3 ||
            !currencyCode.All(char.IsLetter))
        {
            return Results.BadRequest(
                new { error = "Currency code must be a 3-letter uppercase code (e.g., EUR, GBP)" });
        }

        // Validate amount (must be positive)
        if (amount < 0)
        {
            return Results.BadRequest(new { error = "Amount must be a positive number" });
        }

        decimal? rate;
        var semaphore = Locks.GetOrAdd(currencyCode, _ => new SemaphoreSlim(1, 1));
        if (!Cache.TryGetValue(currencyCode, out var cachedRate) &&
            DateTime.UtcNow - cachedRate?.CreatedAt < CacheDuration)
        {
            var acquired = await semaphore.WaitAsync(TimeSpan.FromSeconds(10));
            if (!acquired)
            {
                throw new Exception("Could not acquire lock to fetch exchange rate.");
            }

            try
            {
                // Double-check locking pattern: check again inside the lock
                if (!Cache.TryGetValue(currencyCode, out cachedRate) &&
                    DateTime.UtcNow - cachedRate?.CreatedAt < CacheDuration)
                {
                    rate = await currencyClient.GetExchangeRateAsync(currencyCode);

                    if (rate == null)
                    {
                        return Results.NotFound(
                            new { error = $"Exchange rate for {currencyCode} not found or API error occurred" });
                    }

                    Cache.AddOrUpdate(currencyCode,
                        _ => new CacheEntry(rate.Value, DateTime.UtcNow),
                        (_, _) => new CacheEntry(rate.Value, DateTime.UtcNow));
                }
                else
                {
                    rate = cachedRate!.Rate;
                }
            }
            finally
            {
                semaphore.Release();
            }
        }
        else
        {
            rate = cachedRate!.Rate;
        }

        var convertedAmount = amount * rate.Value;

        return Results.Ok(new ExchangeRateResponse(
            Currency: currencyCode,
            BaseCurrency: "USD",
            Rate: rate.Value,
            Amount: amount,
            ConvertedAmount: convertedAmount
        ));
    }
}
```

The next step would be to extract the core caching logic into its own reusable class.
That way, you can use it in other parts of your application.

## Takeaway

Why go through all this trouble?

It's easy to look at a simple `ConcurrentDictionary` and think you're done.
But as we've seen, the gap between "it works" and "it scales" is filled with edge cases that can bring a production system to its knees.

When you use a library, it handles these edge cases for you.
But building it yourself teaches you about the "three pillars" of high-performance code:

1. **Thread safety**
2. **Lock contention**
3. **Resource protection**

Sometimes the most "boring" parts of our infrastructure, like a cache, are actually the most architecturally interesting.

There's also that 1% of the time when you need a custom solution that no library can provide.
So it's worth knowing the fundamentals of how these things work.

Modern libraries like [**HybridCache**](hybrid-cache-in-aspnetcore-new-caching-library) or
[FusionCache](https://github.com/ZiggyCreatures/FusionCache) handle this for you,
but understanding these patterns ensures you know exactly why your application behaves the way it does under load.

And since this is the last issue of the year, I wish you a fantastic New Year! 🎉

---


Previous issues of The .NET Weekly