Skip to content

V1/feature/47 #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eneshoxha merged 2 commits into from
Dec 3, 2024
Merged

V1/feature/47 #48

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0ae8c20
v1/feature/47 Re-implementation of Windows:
eneshoxha Dec 3, 2024
53aabd0
Update versions; ready for release v1.0.1
eneshoxha Dec 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 18 additions & 17 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@

**Cortex Data Framework** is a robust, extensible platform designed to facilitate real-time data streaming, processing, and state management. It provides developers with a comprehensive suite of tools and libraries to build scalable, high-performance data pipelines tailored to diverse use cases. By abstracting underlying streaming technologies and state management solutions, Cortex Data Framework enables seamless integration, simplified development workflows, and enhanced maintainability for complex data-driven applications.

![GitHub forks](https://img.shields.io/github/forks/buildersoftio/cortex)
![GitHub License](https://img.shields.io/github/license/buildersoftio/cortex)
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams?label=Cortex.Streams)
![GitHub contributors](https://img.shields.io/github/contributors/buildersoftio/cortex) [![Discord Shield](https://discord.com/api/guilds/1310034212371566612/widget.png?style=shield)](https://discord.com/invite/4Gfe6mhJ)
[![GitHub forks](https://img.shields.io/github/forks/buildersoftio/cortex)](https://github.com/buildersoftio/cortex/fork)
[![GitHub License](https://img.shields.io/github/license/buildersoftio/cortex)](https://github.com/buildersoftio/cortex/blob/master/LICENSE)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams?label=Cortex.Streams)](https://www.nuget.org/packages/Cortex.Streams)
[![GitHub contributors](https://img.shields.io/github/contributors/buildersoftio/cortex)](https://github.com/buildersoftio/cortex)
[![Discord Shield](https://discord.com/api/guilds/1310034212371566612/widget.png?style=shield)](https://discord.com/invite/4Gfe6mhJ)

## Key Features
- **Modular Architecture**: Comprises distinct, interchangeable modules for streaming, state management, and connectors, allowing developers to choose components that best fit their requirements.
Expand Down Expand Up @@ -41,44 +42,44 @@
## Project Structure

- **Cortex.Streams:** Core streaming capabilities for building data pipelines.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams?label=Cortex.Streams)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams?label=Cortex.Streams)](https://www.nuget.org/packages/Cortex.Streams)

- **Cortex.Streams.Kafka:** Integration with Apache Kafka for robust data streaming.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.Kafka?label=Cortex.Streams.Kafka)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.Kafka?label=Cortex.Streams.Kafka)](https://www.nuget.org/packages/Cortex.Streams.Kafka)

- **Cortex.Streams.Pulsar:** Integration with Apache Pulsar for versatile messaging needs.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.Pulsar?label=Cortex.Streams.Pulsar)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.Pulsar?label=Cortex.Streams.Pulsar)](https://www.nuget.org/packages/Cortex.Streams.Pulsar)

- **Cortex.Streams.RabbitMQ:** Integration with RabbitMQ for versatile messaging needs.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.RabbitMQ?label=Cortex.Streams.RabbitMQ)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.RabbitMQ?label=Cortex.Streams.RabbitMQ)](https://www.nuget.org/packages/Cortex.Streams.RabbitMQ)

- **Cortex.Streams.AWSSQS:** Integration with Amazon SQS for messaging needs in the cloud.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.AWSSQS?label=Cortex.Streams.AWSSQS)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.AWSSQS?label=Cortex.Streams.AWSSQS)](https://www.nuget.org/packages/Cortex.Streams.AWSSQS)


- **Cortex.Streams.AzureServiceBus:** Integration with Azure Messaging Service Bus for messaging needs in the cloud.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.AzureServiceBus?label=Cortex.Streams.AzureServiceBus)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.AzureServiceBus?label=Cortex.Streams.AzureServiceBus)](https://www.nuget.org/packages/Cortex.Streams.AzureServiceBus)

- **Cortex.Streams.AzureBlobStorage:** Integration with Azure Blob Storage for sinking messages.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.AzureBlobStorage?label=Cortex.Streams.AzureBlobStorage)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.AzureBlobStorage?label=Cortex.Streams.AzureBlobStorage)](https://www.nuget.org/packages/Cortex.Streams.AzureBlobStorage)

- **Cortex.Streams.S3:** Integration with AWS S3 for sinking messages.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.S3?label=Cortex.Streams.S3)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.S3?label=Cortex.Streams.S3)](https://www.nuget.org/packages/Cortex.Streams.S3)

- **Cortex.Streams.Files:** Implementation of File Source and Sink operators.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.Files?label=Cortex.Streams.Files)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Streams.Files?label=Cortex.Streams.Files)](https://www.nuget.org/packages/Cortex.Streams.Files)

- **Cortex.States:** Core state management functionalities.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.States?label=Cortex.States)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.States?label=Cortex.States)](https://www.nuget.org/packages/Cortex.States)

- **Cortex.States.RocksDb:** Persistent state storage using RocksDB.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.States.RocksDb?label=Cortex.States.RocksDb)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.States.RocksDb?label=Cortex.States.RocksDb)](https://www.nuget.org/packages/Cortex.States.RocksDb)

- **Cortex.Telemetry:** Core library to add support for Tracing and Matrics.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Telemetry?label=Cortex.Telemetry)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Telemetry?label=Cortex.Telemetry)](https://www.nuget.org/packages/Cortex.Telemetry)

- **Cortex.Telemetry.OpenTelemetry:** Adds support for Open Telemetry.
![NuGet Version](https://img.shields.io/nuget/v/Cortex.Telemetry.OpenTelemetry?label=Cortex.Telemetry.OpenTelemetry)
[![NuGet Version](https://img.shields.io/nuget/v/Cortex.Telemetry.OpenTelemetry?label=Cortex.Telemetry.OpenTelemetry)](https://www.nuget.org/packages/Cortex.Telemetry.OpenTelemetry)


## Getting Started
Expand Down
6 changes: 3 additions & 3 deletions src/Cortex.States.RocksDb/Cortex.States.RocksDb.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
<PropertyGroup>
<TargetFrameworks>net8.0;net7.0</TargetFrameworks>

<AssemblyVersion>1.0.0</AssemblyVersion>
<FileVersion>1.0.0</FileVersion>
<AssemblyVersion>1.0.1</AssemblyVersion>
<FileVersion>1.0.1</FileVersion>
<Product>Buildersoft Cortex Framework</Product>
<Company>Buildersoft</Company>
<Authors>Buildersoft,EnesHoxha</Authors>
Expand All @@ -16,7 +16,7 @@
<RepositoryUrl>https://github.com/buildersoftio/cortex</RepositoryUrl>
<PackageTags>cortex vortex mediator eda streaming distributed streams states kafka pulsar rocksdb</PackageTags>

<Version>1.0.0</Version>
<Version>1.0.1</Version>
<PackageLicenseFile>license.md</PackageLicenseFile>
<PackageIcon>cortex.png</PackageIcon>
<PackageId>Cortex.States.RocksDb</PackageId>
Expand Down
21 changes: 21 additions & 0 deletions src/Cortex.States.RocksDb/RocksDbStateStore.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
public class RocksDbStateStore<TKey, TValue> : IStateStore<TKey, TValue>
{
private readonly RocksDbSharp.RocksDb _db;
private readonly ColumnFamilyHandle _handle;

Check warning on line 12 in src/Cortex.States.RocksDb/RocksDbStateStore.cs

View workflow job for this annotation

GitHub Actions / build-and-publish 

The field 'RocksDbStateStore<TKey, TValue>._handle' is never used
private readonly Func<TKey, byte[]> _keySerializer;
private readonly Func<TValue, byte[]> _valueSerializer;
private readonly Func<byte[], TKey> _keyDeserializer;
Expand Down Expand Up @@ -121,5 +121,26 @@
_db?.Dispose();
_lock?.Dispose();
}

public IEnumerable<TKey> GetKeys()
{
_lock.EnterReadLock();
try
{
using (var iterator = _db.NewIterator())
{
for (iterator.SeekToFirst(); iterator.Valid(); iterator.Next())
{
var keyBytes = iterator.Key();
var key = _keyDeserializer(keyBytes);
yield return key;
}
}
}
finally
{
_lock.ExitReadLock();
}
}
}
}
1 change: 1 addition & 0 deletions src/Cortex.States/Abstractions/IStateStore.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,6 @@ public interface IStateStore<TKey, TValue> : IStateStore
bool ContainsKey(TKey key);
void Remove(TKey key);
IEnumerable<KeyValuePair<TKey, TValue>> GetAll();
IEnumerable<TKey> GetKeys();
}
}
6 changes: 3 additions & 3 deletions src/Cortex.States/Cortex.States.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
<PropertyGroup>
<TargetFrameworks>net9.0;net8.0;net7.0;netstandard2.1;netstandard2.0</TargetFrameworks>

<AssemblyVersion>1.0.0</AssemblyVersion>
<FileVersion>1.0.0</FileVersion>
<AssemblyVersion>1.0.1</AssemblyVersion>
<FileVersion>1.0.1</FileVersion>
<Product>Buildersoft Cortex Framework</Product>
<Company>Buildersoft</Company>
<Authors>Buildersoft,EnesHoxha</Authors>
Expand All @@ -16,7 +16,7 @@
<RepositoryUrl>https://github.com/buildersoftio/cortex</RepositoryUrl>
<PackageTags>cortex vortex mediator eda streaming distributed streams states kafka pulsar rocksdb</PackageTags>

<Version>1.0.0</Version>
<Version>1.0.1</Version>
<PackageLicenseFile>license.md</PackageLicenseFile>
<PackageIcon>cortex.png</PackageIcon>
<PackageId>Cortex.States</PackageId>
Expand Down
5 changes: 5 additions & 0 deletions src/Cortex.States/InMemoryStateStore.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,10 @@ public IEnumerable<KeyValuePair<TKey, TValue>> GetAll()
{
return _store;
}

public IEnumerable<TKey> GetKeys()
{
return _store.Keys;
}
}
}
6 changes: 3 additions & 3 deletions src/Cortex.Streams.AWSSQS/Cortex.Streams.AWSSQS.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
<TargetFrameworks>net8.0</TargetFrameworks>
<Nullable>enable</Nullable>

<AssemblyVersion>1.0.0</AssemblyVersion>
<FileVersion>1.0.0</FileVersion>
<AssemblyVersion>1.0.1</AssemblyVersion>
<FileVersion>1.0.1</FileVersion>
<Product>Buildersoft Cortex Framework</Product>
<Company>Buildersoft</Company>
<Authors>Buildersoft,EnesHoxha</Authors>
Expand All @@ -16,7 +16,7 @@
<RepositoryUrl>https://github.com/buildersoftio/cortex</RepositoryUrl>
<PackageTags>cortex vortex mediator eda streaming distributed streams states kafka pulsar rocksdb</PackageTags>

<Version>1.0.0</Version>
<Version>1.0.1</Version>
<PackageLicenseFile>license.md</PackageLicenseFile>
<PackageIcon>cortex.png</PackageIcon>
<PackageId>Cortex.Streams.AWSSQS</PackageId>
Expand Down
6 changes: 3 additions & 3 deletions src/Cortex.Streams.AzureBlobStorage/Cortex.Streams.AzureBlobStorage.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
<TargetFrameworks>net8.0</TargetFrameworks>
<Nullable>enable</Nullable>

<AssemblyVersion>1.0.0</AssemblyVersion>
<FileVersion>1.0.0</FileVersion>
<AssemblyVersion>1.0.1</AssemblyVersion>
<FileVersion>1.0.1</FileVersion>
<Product>Buildersoft Cortex Framework</Product>
<Company>Buildersoft</Company>
<Authors>Buildersoft,EnesHoxha</Authors>
Expand All @@ -16,7 +16,7 @@
<RepositoryUrl>https://github.com/buildersoftio/cortex</RepositoryUrl>
<PackageTags>cortex vortex eda streaming distributed streams states kafka pulsar rocksdb rabbitMq AWS Azure</PackageTags>

<Version>1.0.0</Version>
<Version>1.0.1</Version>
<PackageLicenseFile>license.md</PackageLicenseFile>
<PackageIcon>cortex.png</PackageIcon>
<PackageId>Cortex.Streams.AzureBlobStorage</PackageId>
Expand Down
6 changes: 3 additions & 3 deletions src/Cortex.Streams.AzureServiceBus/Cortex.Streams.AzureServiceBus.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
<TargetFrameworks>net8.0</TargetFrameworks>
<Nullable>enable</Nullable>

<AssemblyVersion>1.0.0</AssemblyVersion>
<FileVersion>1.0.0</FileVersion>
<AssemblyVersion>1.0.1</AssemblyVersion>
<FileVersion>1.0.1</FileVersion>
<Product>Buildersoft Cortex Framework</Product>
<Company>Buildersoft</Company>
<Authors>Buildersoft,EnesHoxha</Authors>
Expand All @@ -16,7 +16,7 @@
<RepositoryUrl>https://github.com/buildersoftio/cortex</RepositoryUrl>
<PackageTags>cortex vortex eda streaming distributed streams states kafka pulsar rocksdb rabbitMq AWS Azure</PackageTags>

<Version>1.0.0</Version>
<Version>1.0.1</Version>
<PackageLicenseFile>license.md</PackageLicenseFile>
<PackageIcon>cortex.png</PackageIcon>
<PackageId>Cortex.Streams.AzureServiceBus</PackageId>
Expand Down
6 changes: 3 additions & 3 deletions src/Cortex.Streams.Files/Cortex.Streams.Files.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
<TargetFrameworks>net9.0;net8.0</TargetFrameworks>
<Nullable>enable</Nullable>

<AssemblyVersion>1.0.0</AssemblyVersion>
<FileVersion>1.0.0</FileVersion>
<AssemblyVersion>1.0.1</AssemblyVersion>
<FileVersion>1.0.1</FileVersion>
<Product>Buildersoft Cortex Framework</Product>
<Company>Buildersoft</Company>
<Authors>Buildersoft,EnesHoxha</Authors>
Expand All @@ -16,7 +16,7 @@
<RepositoryUrl>https://github.com/buildersoftio/cortex</RepositoryUrl>
<PackageTags>cortex vortex eda streaming distributed streams states kafka pulsar rocksdb rabbitMq files azure</PackageTags>

<Version>1.0.0</Version>
<Version>1.0.1</Version>
<PackageLicenseFile>license.md</PackageLicenseFile>
<PackageIcon>cortex.png</PackageIcon>
<PackageId>Cortex.Streams.Files</PackageId>
Expand Down
6 changes: 3 additions & 3 deletions src/Cortex.Streams.Kafka/Cortex.Streams.Kafka.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
<TargetFrameworks>net8.0</TargetFrameworks>
<Nullable>enable</Nullable>

<AssemblyVersion>1.0.0</AssemblyVersion>
<FileVersion>1.0.0</FileVersion>
<AssemblyVersion>1.0.1</AssemblyVersion>
<FileVersion>1.0.1</FileVersion>
<Product>Buildersoft Cortex Framework</Product>
<Company>Buildersoft</Company>
<Authors>Buildersoft,EnesHoxha</Authors>
Expand All @@ -16,7 +16,7 @@
<RepositoryUrl>https://github.com/buildersoftio/cortex</RepositoryUrl>
<PackageTags>cortex vortex eda streaming distributed streams states kafka pulsar rocksdb</PackageTags>

<Version>1.0.0</Version>
<Version>1.0.1</Version>
<PackageLicenseFile>license.md</PackageLicenseFile>
<PackageIcon>cortex.png</PackageIcon>
<PackageId>Cortex.Streams.Kafka</PackageId>
Expand Down
6 changes: 3 additions & 3 deletions src/Cortex.Streams.Pulsar/Cortex.Streams.Pulsar.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
<TargetFrameworks>net8.0;net7.0</TargetFrameworks>
<Nullable>enable</Nullable>

<AssemblyVersion>1.0.0</AssemblyVersion>
<FileVersion>1.0.0</FileVersion>
<AssemblyVersion>1.0.1</AssemblyVersion>
<FileVersion>1.0.1</FileVersion>
<Product>Buildersoft Cortex Framework</Product>
<Company>Buildersoft</Company>
<Authors>Buildersoft,EnesHoxha</Authors>
Expand All @@ -17,7 +17,7 @@
<RepositoryUrl>https://github.com/buildersoftio/cortex</RepositoryUrl>
<PackageTags>cortex vortex mediator eda streaming distributed streams states kafka pulsar rocksdb</PackageTags>

<Version>1.0.0</Version>
<Version>1.0.1</Version>
<PackageLicenseFile>license.md</PackageLicenseFile>
<PackageIcon>cortex.png</PackageIcon>
<PackageId>Cortex.Streams.Pulsar</PackageId>
Expand Down
6 changes: 3 additions & 3 deletions src/Cortex.Streams.RabbitMQ/Cortex.Streams.RabbitMQ.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
<TargetFrameworks>net8.0</TargetFrameworks>
<Nullable>enable</Nullable>

<AssemblyVersion>1.0.0</AssemblyVersion>
<FileVersion>1.0.0</FileVersion>
<AssemblyVersion>1.0.1</AssemblyVersion>
<FileVersion>1.0.1</FileVersion>
<Product>Buildersoft Cortex Framework</Product>
<Company>Buildersoft</Company>
<Authors>Buildersoft,EnesHoxha</Authors>
Expand All @@ -16,7 +16,7 @@
<RepositoryUrl>https://github.com/buildersoftio/cortex</RepositoryUrl>
<PackageTags>cortex vortex mediator eda streaming distributed streams states kafka pulsar rocksdb rabbitMq AWS Azure</PackageTags>

<Version>1.0.0</Version>
<Version>1.0.1</Version>
<PackageLicenseFile>license.md</PackageLicenseFile>
<PackageIcon>cortex.png</PackageIcon>
<PackageId>Cortex.Streams.RabbitMQ</PackageId>
Expand Down
6 changes: 3 additions & 3 deletions src/Cortex.Streams.S3/Cortex.Streams.S3.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
<TargetFrameworks>net9.0;net8.0</TargetFrameworks>
<Nullable>enable</Nullable>

<AssemblyVersion>1.0.0</AssemblyVersion>
<FileVersion>1.0.0</FileVersion>
<AssemblyVersion>1.0.1</AssemblyVersion>
<FileVersion>1.0.1</FileVersion>
<Product>Buildersoft Cortex Framework</Product>
<Company>Buildersoft</Company>
<Authors>Buildersoft,EnesHoxha</Authors>
Expand All @@ -16,7 +16,7 @@
<RepositoryUrl>https://github.com/buildersoftio/cortex</RepositoryUrl>
<PackageTags>cortex vortex eda streaming distributed streams states kafka pulsar rocksdb rabbitMq S3 Azure</PackageTags>

<Version>1.0.0</Version>
<Version>1.0.1</Version>
<PackageLicenseFile>license.md</PackageLicenseFile>
<PackageIcon>cortex.png</PackageIcon>
<PackageId>Cortex.Streams.S3</PackageId>
Expand Down
Loading
Loading