-
Notifications
You must be signed in to change notification settings - Fork 20
Added time traveling use case tutorial #886
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
da3b0d0
Added time traveling use case tutorial
stktung 4bbeb45
A few minor cosmetic update to the time travel tutorial
stktung e42113f
Added images and improved some representations in the auditing capabi…
stktung a7f7be8
Changed the terminologies around about synchronous vs asynchronous re…
stktung 8d21d5c
Added more pre-compute vs on-demand read model narrative
stktung 5179815
Added section about granularity of snapshot
stktung cbd753d
Updated the curl request and response for event api for clarity
stktung 88b3057
Added missing message from script
stktung 40b9694
Added time travel introduction
stktung 0f84151
Added introduction content
stktung 23742ce
Added hero image and improved clarity of article
stktung 9b1e0a6
Updated title and header for better SEO for time travel
stktung 3c96f58
Updated title and sidebar desc of older tutorials to improve seo
stktung fc15f74
Added og:image to time travel
stktung 1e2ed13
Apply suggestions from code review
stktung e0f48e3
Applied fix as suggested by mark
stktung d2eb91c
Fixed path to og:image
stktung fce9655
Added outbox hero image
stktung f6a0fb1
Update docs/getting-started/use-cases/time-travel/tutorial-2.md
stktung c01928d
Apply suggestions from code review
stktung 7f0b583
Refined points about denormalized data models
stktung 095143c
Minor wording change
stktung 9767608
Realigned denormalize info section
stktung 7e882f8
Changed sidebar use case label from 'introduction' to 'overview'
stktung d58d850
Removed bracket at the end of comment
stktung 2beb939
Update docs/getting-started/use-cases/time-travel/tutorial-5.md
stktung 44bab38
Update docs/getting-started/use-cases/time-travel/tutorial-4.md
stktung 399227f
Switched tutorial branch to main
stktung 88aac04
Merge branch 'use-case-time-travel' of https://github.com/kurrent-io/…
stktung File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 changes: 1 addition & 3 deletions
4
docs/getting-started/use-cases/mix-and-match-database/introduction.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
1
docs/getting-started/use-cases/mix-and-match-database/tutorial-1.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 changes: 2 additions & 1 deletion
3
docs/getting-started/use-cases/mix-and-match-database/tutorial-intro.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| --- | ||
| title: Part 1 - Set up Codespaces | ||
| prev: ./tutorial-intro.md | ||
| --- | ||
|
|
||
| # Part 1: Set up Codespaces | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| --- | ||
| title: Introduction | ||
| title: Outbox Tutorial Introduction | ||
| prev: ./introduction.md | ||
| --- | ||
|
|
||
| # Introduction | ||
|
|
||
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+29.7 KB
docs/getting-started/use-cases/time-travel/images/events-for-daily-sales.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+55 KB
...getting-started/use-cases/time-travel/images/events-for-total-monthly-sales.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+168 KB
docs/getting-started/use-cases/time-travel/images/month-end-sales-report.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+191 KB
docs/getting-started/use-cases/time-travel/images/time-travel-hero.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions
4
docs/getting-started/use-cases/time-travel/images/time-travel-hero.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+12.4 MB
docs/getting-started/use-cases/time-travel/images/time-travel-report.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
83 changes: 83 additions & 0 deletions
83
docs/getting-started/use-cases/time-travel/introduction.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| --- | ||
| title: Time Traveling with KurrentDB | ||
| image: /getting-started/use-cases/time-travel/images/time-travel-hero.png | ||
| next: ./tutorial-intro.md | ||
| --- | ||
|
|
||
|  | ||
|
|
||
| ## What is Time Traveling? | ||
| Time traveling in data systems means being able to query or reconstruct your data as it existed at any point in the past. This is useful for many scenarios, such as auditing, debugging, compliance, and analytics. | ||
|
|
||
| | Use Case | Example | | ||
| |--------------------------------------|---------| | ||
| | Business process reconstruction | See the full lifecycle of a loan application, including every approval, rejection, and amendment, to resolve disputes or meet regulations | | ||
| | Point-in-time state diffing | Show exactly what changed in a customer’s profile or account settings between two points in time for support or compliance reviews | | ||
| | Simulation and what-if analysis | Simulate the impact of a pricing change or business rule adjustment by replaying all related product and sales events to see how outcomes would differ | | ||
| | Business event tracing | Audit the sequence of business events for a transaction (order placed, payment received, shipment sent, delivery confirmed) to verify SLAs or investigate complaints | | ||
| | Timeline-Based State Inspection | Let users view a timeline of how their account or order changed, with each step tied to a business event (like address updated, item added, return requested) | | ||
| | Temporal aggregate calculation | Track how total sales for a product changed over time by calculating monthly sales at the end of each month | | ||
|
|
||
| ## Time Traveling with Traditional Approaches | ||
| Traditional databases struggle with historical queries because they are designed to store only the current state. Retrieving or reconstructing previous versions often requires extra mechanisms like audit logs, snapshots, or CDC with time-partitioned data lakes. These approaches add complexity, slow down queries, and may not always provide a complete or reliable view of historical data. | ||
|
|
||
| #### Audit Logs | ||
| Audit logs are a common approach in traditional databases for tracking changes. They record each modification as a new entry in a separate log table, typically including details such as who made the change, when it occurred, and what operation was performed. | ||
|
|
||
| **Advantages:** | ||
| - Provide a detailed record of changes. | ||
| - Can answer questions about who changed what and when. | ||
| - Useful for meeting regulatory requirements for tracking changes. | ||
|
|
||
| **Limitations:** | ||
| - Audit logs are separate from main data, risking inconsistencies if not commited together. | ||
| - Often capture only high-level operations. | ||
| - Gaps or missed events make it difficult to reconstruct past states. | ||
|
|
||
| #### Snapshots | ||
| Snapshots are another traditional method for preserving historical data. A snapshot captures the entire state of a table or dataset at a specific point in time, often on a scheduled basis (e.g., nightly or weekly), and stores it as a separate copy. | ||
|
|
||
| **Advantages:** | ||
| - Provide a complete view of data. | ||
| - Useful for restoring data after accident. | ||
| - Easy to implement without major changes to existing applications. | ||
|
|
||
| **Limitations:** | ||
| - Snapshots are taken infrequently. | ||
| - Storage requirements can be high. | ||
| - Snapshots lack a detailed audit trail of individual changes. | ||
|
|
||
| #### Change Data Capture (CDC) and Data Lake | ||
| Change Data Capture (CDC) is a technique that tracks and records changes (inserts, updates, deletes) in source databases and delivers them to downstream systems, often storing them in time-partitioned data lakes for analytics and historical queries. | ||
|
|
||
| **Advantages:** | ||
| - Captures detailed change events, enabling near real-time replication and analytics. | ||
| - Supports integration with data warehouses and lakes for large-scale historical analysis. | ||
| - Can be used to build audit trails and reconstruct state changes over time. | ||
|
|
||
| **Limitations:** | ||
| - CDC pipelines add operational complexity and require many moving parts | ||
| - Do not persist the actual change from the source, so lost events mean state can't be reliably reconstructed. | ||
| - Captured changes are usually low-level and technical, making business intent unclear. | ||
|
|
||
| ## Time Traveling with KurrentDB | ||
|
|
||
| KurrentDB makes time travel simple by recording every change as an immutable, ordered event. This lets you accurately reconstruct any previous state, making historical queries, audits, and analysis straightforward without complex workarounds. | ||
|
|
||
| KurrentDB makes this possible because: | ||
|
|
||
| - It stores every change as an immutable event, preserving a complete and accurate history. | ||
| - It allows event streams to be replayed at any time for simulation, testing, and debugging—even far into the future. | ||
| - Its historical event log is naturally suited for regulatory and compliance needs, making audits and reporting straightforward. | ||
| - It supports outbox pattern out of the box, ensuring reliable delivery of historical events to downstream systems. | ||
| - It promotes events are modeled around business intent, providing clear context for historical queries. | ||
| - Events are small and focused, making them efficient to store and process. | ||
| - It handles storage, indexing, and pushing events natively, reducing the need for extra components in your time travel solution. | ||
|
|
||
| ## How to Time Travel with KurrentDB | ||
|
|
||
| - **Store events in ordered streams:** Record every relevant change as an event in KurrentDB, ensuring events are stored in the order they occurred. | ||
| - **Replay events on-demand:** Replay all the relevant events from stream up to the desired timestamp or version. | ||
| - **Pre-compute state for efficiency:** For objects with long histories or frequent queries, you can periodically store snapshots of state. This reduces the number of events that need to be replayed for common queries. | ||
|
|
||
| KurrentDB’s approach makes time travel queries reliable, flexible, and efficient, supporting both detailed investigations and high-performance analytics. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| --- | ||
| title: Part 1 - Set up Codespaces | ||
| prev: ./tutorial-intro.md | ||
| --- | ||
|
|
||
| # Part 1: Set up Codespaces | ||
|
|
||
| In this part, you will start a GitHub Codespaces session in your browser. | ||
|
|
||
| ::: info | ||
| GitHub Codespaces provides an instant and preconfigured development environment all within your browser. This environment contains all the tools and code to complete this tutorial. To learn more about Github Codespaces, [click here](https://github.com/features/codespaces). | ||
| ::: | ||
|
|
||
| ## Step 1: Set up Your Codespaces | ||
|
|
||
| 1. Click the button below to initiate Codespaces and ensure following values are selected: | ||
|
|
||
| [](https://github.com/codespaces/new?hide_repo_select=true&ref=time-travel&repo=951198039&skip_quickstart=true&devcontainer_path=.devcontainer%2Ftime-travel%2Fdevcontainer.json) | ||
|
|
||
|
|
||
| | Configuration Option | Selection | | ||
| |--------------------------------|----------------------| | ||
| | Branch | `main` | | ||
| | Dev container configuration | `Time Travel` | | ||
| | Region | Any value | | ||
| | Machine type | Any value | | ||
|
|
||
| Log in to GitHub if required. | ||
|
|
||
| 2. Wait for your Codespace to build. This can take up to a few minutes. | ||
|
|
||
| ::: tip | ||
| For this quickstart, you can safely ignore and close any Codespaces notifications that appear on the bottom right of the page. | ||
| ::: |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how do we debug in the future?