feat: encrypt datasets - add routes and write on db by Manuthor · Pull Request #17 · Cosmian/findex-server

Manuthor · 2024-11-15T14:18:15Z

Doc:

Add README.md
Add doc for public_documentation repo

CI:

add build for Windows, macos and Docker

Server:

add list Permissions route
add routes to save/get/delete encrypted dataset
split Database trait in 3 (Findex, Permissions and Datasets)

CLI:

add tests with a 20K-lines datasets
reuse cosmian_http_client and cosmian_config_utils from https://www.github.com/Cosmian/http_client_server
make Clap action reusable by https://www.github.com/Cosmian/cli
add list Persmissions action
save conf in TOML (uniformize with server)

tbrezot

Great work, I will need more time to fully understand it, though. Maybe a deeper review can be performed upon first release.

Cargo.toml

README.md

tbrezot · 2024-11-26T13:39:33Z

crate/structs/src/uuids.rs

+impl Display for Uuids {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        for uuid in &self.uuids {
+            writeln!(f, "UUID: {uuid}")?;
+        }
+        Ok(())
+    }
+}


We already talked about this, but encapsulating data in ad-hoc wrappers for the same of automatic Display logging really bugs me. Is there really no way to use an automatic Debug mode instead? I agree it wouldn't be as pretty, but the overhead of these wrappers is really big (79LoC here, 134LoC for the encrypted entries... and it is contaminating).

Yes we already talked about this. I didn't find a better way to have neat logs where data are not raw u8 arrays. IMHO, Debug default trait is insufficient if we plan to debug code later. Debugging could be done by someone else. (The Display trait impl' themselves do not take 79LoC and 134LoC)

the Display implementation isn't, but if you don't need it, then the wrappers are not used and the whole uuids.rs and encrypted_entries.rs files and surely others become useless.

documentation/README.md

documentation/build_pdf.sh

documentation/docs/index.md

rust-toolchain

* fix: windows tests * fix: clean permissions redis writing in server side * fix: make dataset redis writing sync * fix(redis): delete datasets * test: make consistent the Redis writing. Use pipe everywhere except transaction on permissions creation * fix: increase server waiting for redis to start * fix: move cli conf in client

HatemMn

Amazing work !

A generic comment : there are many TODO / todo(manu) all around the code, maybe should be cleaned ? I didn't know if some of them are still up to date or no

crate/cli/src/actions/login.rs

crate/server/src/findex_server.rs

crate/server/Cargo.toml

crate/server/src/database/redis/instance.rs

HatemMn · 2024-12-04T13:45:43Z

crate/server/src/database/redis/instance.rs

+    ";
+
+pub(crate) struct Redis {
+    pub(crate) client: Client,


I don't think we should store this, is it used anywhere ? We don't need it in cloudproof v6 and v7 too

it can be declared inline ( line 34 to be exact ) as it won't be used again anyway

well, in that case, I would need an upsert script to prevent concurrent permission writes

not sure I understand, but LUA scripts are run in an atomic manner. Do we need another LUA script ? I might work on that, if needed

Another alternative is marking it as issue where you show me where the script should be and where you mention my PR

Then close this, and I will try to do it on my own branch

crate/server/src/database/redis/mod.rs

crate/server/src/routes/utils.rs

…er::context

tbrezot

Will read all that in #15 once merged.

* feat: add findex impl - client and server sides * feat: add CSV datasets support in CLI * fix: rename IndexAction to AddAction and flatten Findex actions * Update .pre-commit-config.yaml Co-authored-by: Hatem M'naouer <19950216+HatemMn@users.noreply.github.com> * fix(auth0): use proper findex-server token * test: add JWT authentication tests * feat: add Delete Findex operation * fix: remove Secret for now * fix: prepare sqlite impl * feat: add authorization access mechanism * feat: revoke role * docs: cli access * chore: rename FindexClient to RestClient * fix: permissions mechanism (#16) * ci: fix * test: authentication failing * test: ignore auth tests * test: re-enable test server auth tests * test: re-enable client auth tests * test: disable client auth tests * test: do not clear db * chore: rename access to permission * refactor: check_permission * test: try mt tests * chore: remove all lints * test: remove dependency_on_unit_never_type_fallback * test: make default server port to 6666 * fix: lint * feat: encrypt datasets - add routes and write on db (#17) * feat: encrypt datasets - add routes and write on db * build: docker image * fix: permissions on datasets (add/del/get) * chore: split database traits * fix: reuse config_utils * feat: add list_permissions * fix: strip error conversions * ci: merge workflows between debug and release * docs: add main README * docs: add mkdocs * docs: small changes in mkdocs * chore: reduce verbosity * ci: include windows build by default * docs: fix menu * fix(config): use only toml format (client and server) * test: index a 3MB dataset * fix(redis): use pipe everywhere Redis needs to read/write * feat: support docker on arm cpu * ci: build ARM docker only on nightly build * fix: remove pandoc files * docs: add CLI usage * fix: use develop branch from http_client_server repo * ci(docker build): remove useless arg * fix: PR review * fix(only docs): PR review * test: add/delete/get datasets * fix: Redis writing concurrency (on permissions) and Windows tests (#19) * fix: windows tests * fix: clean permissions redis writing in server side * fix: make dataset redis writing sync * fix(redis): delete datasets * test: make consistent the Redis writing. Use pipe everywhere except transaction on permissions creation * fix: increase server waiting for redis to start * fix: move cli conf in client * feat: use Serializable trait * docs: add rust doc on database traits * fix: avoid a user to lookup who are the more powerful users * fix: database parameters not optional but with default values * fix: reenable cli auth tests * fix: remove useless r2d2 redis-feature and FindexRestClientResultHelper::context * fix: removing useless clippy::blocks_in_conditions * fix: remove uneeded RUSTSECs from deny.toml * ci: fix Github actions warnings * ci: fix Github actions warnings - download-artifact to v4 * fix: remove some todos * fix: remove ahash feature from redis crate * fix: remove useless .rustfmt.toml * fix: Cargo.toml cleanup * fix: bad links * fix: clean useless comments in rust doc * fix: in datasets actions, use &[Uuid] instead of Vec<Uuid> * fix: uniformize in clap action all process() to run() * fix: remove console * fix: PR review --------- Co-authored-by: Hatem M'naouer <19950216+HatemMn@users.noreply.github.com>

feat: encrypt datasets - add routes and write on db

14110b3

Manuthor force-pushed the reuse_cli branch from ef4d31a to 14110b3 Compare November 15, 2024 15:04

Manuthor added 6 commits November 15, 2024 17:10

build: docker image

f8fea53

fix: permissions on datasets (add/del/get)

8f976c3

chore: split database traits

c1f0fa7

fix: reuse config_utils

e0d484b

feat: add list_permissions

32d25b7

fix: strip error conversions

e3d3e35

Manuthor force-pushed the reuse_cli branch 5 times, most recently from 6fdb46e to d4466b0 Compare November 18, 2024 10:17

ci: merge workflows between debug and release

06a3183

Manuthor force-pushed the reuse_cli branch from d4466b0 to 06a3183 Compare November 18, 2024 10:40

docs: add main README

338770e

Manuthor force-pushed the reuse_cli branch from 8475f68 to 338770e Compare November 18, 2024 14:33

Manuthor and others added 12 commits November 19, 2024 11:43

docs: add mkdocs

8b0483a

docs: small changes in mkdocs

d3c8392

chore: reduce verbosity

7a2e965

ci: include windows build by default

ddb91b6

docs: fix menu

e8f71f3

fix(config): use only toml format (client and server)

88d6772

test: index a 3MB dataset

027c5d8

test: index a 3MB dataset

8ceaad4

fix(redis): use pipe everywhere Redis needs to read/write

9fe49ba

feat: support docker on arm cpu

0cc0d16

ci: build ARM docker only on nightly build

a0ce487

fix: remove pandoc files

d09ab9a

tbrezot reviewed Nov 26, 2024

View reviewed changes

Manuthor added 6 commits November 26, 2024 15:30

docs: add CLI usage

88e2652

fix: use develop branch from http_client_server repo

5f87dab

ci(docker build): remove useless arg

458cd6d

fix: PR review

7f9876b

fix(only docs): PR review

326c3e1

test: add/delete/get datasets

2f093bc

Manuthor force-pushed the reuse_cli branch from d0e58eb to 2f093bc Compare November 28, 2024 10:03

Manuthor mentioned this pull request Dec 3, 2024

Make KMS encryption optional #21

Closed

Manuthor added 3 commits December 3, 2024 22:01

feat: use Serializable trait

db51c4e

docs: add rust doc on database traits

80fc868

fix: avoid a user to lookup who are the more powerful users

c0b06e9

Manuthor marked this pull request as ready for review December 4, 2024 07:53

Manuthor requested a review from HatemMn December 4, 2024 07:53

HatemMn approved these changes Dec 4, 2024

View reviewed changes

Manuthor added 8 commits December 5, 2024 08:51

fix: database parameters not optional but with default values

137299e

fix: reenable cli auth tests

270b279

fix: remove useless r2d2 redis-feature and FindexRestClientResultHelp…

9fd10d8

…er::context

fix: removing useless clippy::blocks_in_conditions

042c764

fix: remove uneeded RUSTSECs from deny.toml

0ca41f4

ci: fix Github actions warnings

b678279

ci: fix Github actions warnings - download-artifact to v4

9878dc0

fix: remove some todos

ceb6111

tbrezot approved these changes Dec 5, 2024

View reviewed changes

fix: remove ahash feature from redis crate

e8c68ba

Manuthor merged commit 61ee1be into findex_v6 Dec 5, 2024

Manuthor deleted the reuse_cli branch December 5, 2024 12:09

HatemMn mentioned this pull request Dec 5, 2024

Atomicity of permisssion database operations #22

Open

Comments

Conversation

Manuthor commented Nov 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tbrezot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tbrezot Nov 26, 2024

Choose a reason for hiding this comment

Uh oh!

Manuthor Nov 27, 2024

Choose a reason for hiding this comment

Uh oh!

tbrezot Nov 27, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HatemMn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HatemMn Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

Manuthor Dec 5, 2024

Choose a reason for hiding this comment

Uh oh!

HatemMn Dec 5, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tbrezot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Manuthor commented Nov 15, 2024 •

edited

Loading