Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions docs/.vuepress/configs/sidebar.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,29 @@ export const sidebarEn: EsSidebarOptions = {
]
}
]
},
{
text: "Outbox Out-of-the-Box",
collapsible: true,
expanded: false,
group: "Outbox Out-of-the-Box",
children: [
"/getting-started/use-cases/outbox/introduction.md",
{
text: "Tutorial",
collapsible: true,
expanded: false,
group: "Outbox Tutorial",
children: [
"/getting-started/use-cases/outbox/tutorial-intro.md",
"/getting-started/use-cases/outbox/tutorial-1.md",
"/getting-started/use-cases/outbox/tutorial-2.md",
"/getting-started/use-cases/outbox/tutorial-3.md",
"/getting-started/use-cases/outbox/tutorial-4.md",
"/getting-started/use-cases/outbox/tutorial-summary.md"
]
}
]
}
]
},
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
75 changes: 75 additions & 0 deletions docs/getting-started/use-cases/outbox/introduction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
---
title: Introduction
next: ./tutorial-intro.md
---

## Outbox Out-of-the-Box

### Dual Write Problem
Without distributed transactions, operations that write to multiple resources are not atomic, potentially leading to inconsistencies in the system. This issue is commonly known as the dual write problem.

Although named "Dual Write," this pattern can involve writing to more than two resources. A common use case is updating a relational database and simultaneously sending a notification message via a message queue to another system.

![Dual Write Problem](./images/dual-write-problem.png#light)

![Dual Write Problem](./images/dual-write-problem-dark.png#dark)

Failure to write to one resource introduces inconsistency with the other. For instance, if the database update succeeds but message delivery fails, downstream systems remain uninformed.

![Out of sync when database updated but messaging failed](./images/dual-write-problem-failed-messaging.png#light)

![Out of sync when database updated but messaging failed](./images/dual-write-problem-failed-messaging-dark.png#dark)

Conversely, a failed database update paired with a successful message dispatch leads to downstream systems acting on non-existent data.

### Transactional Outbox Pattern
The transactional outbox pattern ensures consistency by writing business data and outgoing messages atomically within the same database transaction. A separate process later dispatches these messages to downstream systems, mitigating the dual write problem.

The outbox pattern promotes reducing the number of resources we write to, preferably to just one, and technically, to one transaction. This
implies that the effect on the other resources gets deferred.

Practically, the outbox acts as a persistent queue holding messages until they're dispatched asynchronously. This trades immediate consistency for eventual consistency, reducing complexity and the risk associated with simultaneous writes.
There are several common implementation approaches to this pattern.

#### Using a Relational Outbox Table

![Outbox with relational table](./images/outbox-with-database-table.png#light)

![Outbox with relational table](./images/outbox-with-database-table-dark.png#dark)

In a relational database, the outbox could manifest itself as one or more tables that keep track of how the other resources need to be affected. In the canonical example, the outbox would be a table containing messages to be sent to the other system.

The salient point is to atomically commit regular database changes and outbox messages as part of a single transaction. A separate process (outbox relay) can now pick up the messages from the outbox and send them out. Once a message has been sent, it can be marked as sent or removed from the outbox, whichever option is preferred.

While straightforward and intuitive, this approach introduces latency because messages are delivered via periodic polling rather than immediate notification. Additionally, the outbox pattern inherently depends on the database’s scalability, meaning throughput and performance are constrained by database capacity and resource contention.

#### Using Change Data Capture (CDC)

![Outbox with change data capture](./images/outbox-with-cdc.png#light)

![Outbox with change data capture](./images/outbox-with-cdc-dark.png#dark)

Instead of directly managing an outbox table, another implementation of the outbox pattern involves generating a CDC feed from the affected data table and transforming these changes into messages for external systems. Streaming platforms commonly favor this method.

This approach avoids polling overhead and propagates updates with significantly lower latency. However, it relies on additional CDC tooling, increasing complexity if such tools aren't already part of the existing infrastructure. It also risks exposing the internal data model of the source tables, potentially creating undesirable coupling with other systems.

### Outbox Out-of-the-Box with KurrentDB

![Outbox with KurrentDB](./images/outbox-with-kurrentdb.png#light)

![Outbox with KurrentDB](./images/outbox-with-kurrentdb-dark.png#dark)

When working with streams and events, an important shift tends to happen. That is, the events written to a stream turn out to be triggers for the messages we want to send to the other system with minimal translation.

This occurs because when KurrentDB is used as an event store for event sourcing, an event stream behaves like a hybrid between a database table and a message queue.

Like a database table, the stream provides atomic, durable, and immediately consistent operations. And like a message queue, it offers subscription mechanisms to propagate updates to multiple systems in an eventually consistent way.
In effect, the stream is the outbox, out of the box.

The native subscription capabilities, such as persistent and catch-up subscriptions and connectors, act as cheap mechanisms for writing the glue code that sends messages to the other system.

### How to Approach the Dual Write Problem with KurrentDB
1. Record each business change as an event and append it to a stream in KurrentDB, using the stream as the definitive source of truth.
2. Do not update other systems or read models directly as part of the same append operation.
3. Set up subscriptions to listen for new events in the stream.
4. Process these events asynchronously by triggering actions on external systems.
33 changes: 33 additions & 0 deletions docs/getting-started/use-cases/outbox/tutorial-1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
title: Part 1 - Set up Codespaces
---

# Part 1: Set up Codespaces

In this part, you will start a GitHub Codespaces session in your browser.

::: info
GitHub Codespaces provides an instant and preconfigured development environment all within your browser. This environment contains all the tools and code to complete this tutorial. To learn more about Github Codespaces, [click here](https://github.com/features/codespaces).
:::

## Step 1: Set up Your Codespaces

1. Click the button below to initiate Codespaces and ensure following values are selected:

[![](https://github.com/codespaces/badge.svg)](https://github.com/codespaces/new?hide_repo_select=true&ref=main&repo=951198039&skip_quickstart=true&devcontainer_path=.devcontainer%2Foutbox%2Fdevcontainer.json)


| Configuration Option | Selection |
|--------------------------------|----------------------|
| Branch | `main` |
| Dev container configuration | `Outbox` |
| Region | Any value |
| Machine type | Any value |

Log in to GitHub if required.

2. Wait for your Codespace to build. This can take up to a few minutes.

::: tip
For this quickstart, you can safely ignore and close any Codespaces notifications that appear on the bottom right of the page.
:::
126 changes: 126 additions & 0 deletions docs/getting-started/use-cases/outbox/tutorial-2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
---
title: Part 2 - Trigger Writes to External Data Stores
---

# Part 2: Trigger Writes to External Data Stores

With KurrentDB, eventually consistent update to multiple resources often begin with an event that triggers the entire process. Subsequently, each downstream system subscribes to this event and updates its data store on its own without the need for a distributed transaction.

In this tutorial, an `OrderPlaced` event will trigger the start of an order fulfillment process in the fulfillment system.

For the purpose of this tutorial, how this event is created is not important. For simplicity, it will be created by a data generator based on checkout-related events.

::: note How Triggering Events are Created
In general, just like any event, a triggering event can be created in many ways. For example, a `CouponUsed` event may need transactional mechanisms and patterns such as aggregate and deciders to ensure race condition from multiple actors don't violate business rules (e.g. a coupon can only be used 100 times).

On the other hand, an event like `OrderPlaced` in this tutorial may not require these practices since it is only a summary event that collects information from the relevant shopping cart and checkout events.
:::

## Step 2: Start Databases and Append OrderPlaced Event to KurrentDB

1. Once your Codespace is loaded, run this command in the terminal to start KurrentDB and PostgreSQL, and append sample events to KurrentDB:

```sh
./scripts/1-start-dbs-and-generate-data.sh
```

2. You will see the following message printed in the terminal:

```
Appended data to KurrentDB

🚀 KurrentDB Server has started!! 🚀

URL to the KurrentDB Admin UI 👉: https://XXXXXXXXX.XXX
```

3. Copy the URL printed in the terminal from the last step.

4. Open a new browser tab.

5. In the address bar of the new tab, paste the URL and navigate to it.

6. This will display the KurrentDB Admin UI.

![KurrentDB Admin UI Dashboard](images/admin-ui.png =300x)

## Step 3: Browse OrderPlaced Events in KurrentDB's Admin UI

1. Click the `Stream Browser` link from the top navigation bar.

2. Under `Recently Changed Streams`, click the `$ce-order` link.

::: info Understanding Category System Projection
The `$ce-order` stream contains events from all the carts in KurrentDB. This uses KurrentDB's "by category" system projection stream feature. For more information, see [System Projections](https://docs.kurrent.io/server/v25.0/features/projections/system.html#by-category).
:::

3. You should see a sequenced list of the appended events associated with the two distinct orders.

4. Click on one of them to see the details of the order.

::: details Sample detail of an `OrderPlaced` event

```json
{
"orderId": "order-b0d1a15a21d24ffa97785ce7b345a87e",
"customerId": "customer-185176238",
"checkoutOfCart": "cart-631dd4d51e6b4f4d9f9e26e55f1cd587@9",
"lineItems": [
{
"productId": "3906362089844",
"productName": "Glamorise Women's Plus Size MagicLift Natural Shape Bra Wirefree #1210",
"quantity": 5,
"pricePerUnit": "USD601.05",
"taxRate": 0.21
},
{
"productId": "4579864912959",
"productName": "Simple Designs LT3039-PRP 14.17” Contemporary Mosaic Tiled Glass Genie Standard Table Lamp with Matching Fabric Shade for Home Décor, Bedroom, Living Room, Foyer, Office, Purple",
"quantity": 3,
"pricePerUnit": "USD392.81",
"taxRate": 0.06
}
],
"shipping": {
"recipient": {
"title": "Ms.",
"fullName": "Beulah Schmidt",
"emailAddress": "Beulah.Schmidt@yahoo.com",
"phoneNumber": "1-819-847-8206 x80714"
},
"address": {
"country": "IR",
"lines": [
"Clementine Mountain 445",
"75096-8505 North Chadport",
"Bedfordshire"
]
},
"instructions": "",
"method": "express"
},
"billing": {
"recipient": {
"title": "Ms.",
"fullName": "Beulah Schmidt",
"emailAddress": "Beulah.Schmidt@yahoo.com",
"phoneNumber": "1-819-847-8206 x80714"
},
"address": {
"country": "IR",
"lines": [
"Clementine Mountain 445",
"75096-8505 North Chadport",
"Bedfordshire"
]
},
"paymentMethod": "wireTransfer"
},
"at": "2025-01-01T01:11:30.976938+00:00"
}
```
:::

::: tip
You may have noticed other streams and events in KurrentDB. You can safetly ignore them for the purpose of this tutorial.
:::
Loading
Loading