Add shared_location() API for accessing shared directory files#329
Open
Edwardvaneechoud wants to merge 7 commits intofeauture/kernel-implementationfrom
Open
Add shared_location() API for accessing shared directory files#329Edwardvaneechoud wants to merge 7 commits intofeauture/kernel-implementationfrom
Edwardvaneechoud wants to merge 7 commits intofeauture/kernel-implementationfrom
Conversation
Adds a new `shared_location(filename)` function to the kernel's Python API that returns an absolute path on the shared volume where files can be written and read from any FlowFile service. This enables users to easily write files (CSV, Parquet, etc.) to a shared directory that persists across kernel executions and is accessible from core, worker, and kernel containers. Changes: - kernel/manager.py: Pass FLOWFILE_KERNEL_SHARED_DIR env var to kernel containers - flowfile_client.py: Add shared_location() function with auto-mkdir - test_flowfile_client.py: Add 6 tests for shared_location() - FlowfileApiHelp.vue: Add File Utilities section and usage pattern example https://claude.ai/code/session_018xyft77B2g2s3Whm746Qmh
Three new integration-style tests that actually write Polars DataFrames to shared_location paths and read them back, verifying the full write-read cycle with no mocking. https://claude.ai/code/session_018xyft77B2g2s3Whm746Qmh
Rename the function to make it clearer that it's a getter, not a property or constant. https://claude.ai/code/session_018xyft77B2g2s3Whm746Qmh
…m:Edwardvaneechoud/Flowfile into claude/add-flowfile-shared-location-fmwpx
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Patch os.makedirs in the fallback test to avoid PermissionError when /shared is not writable on CI runners. Also asserts makedirs was called with the correct path. https://claude.ai/code/session_018xyft77B2g2s3Whm746Qmh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a new
flowfile.shared_location()API that provides a convenient way for kernel scripts to write files to the shared directory that persists across executions and is accessible from all FlowFile services.Changes
FLOWFILE_KERNEL_SHARED_DIRenvironment variable that tells the kernel the absolute path of the shared directory as seen from inside the kernel containershared_location(filename)function that:/shared/user_files/if environment variable is not setshared_location()Implementation Details
The
shared_location()function constructs paths under{FLOWFILE_KERNEL_SHARED_DIR}/user_files/to organize user-written files separately from system files. Parent directories are created withos.makedirs(exist_ok=True)to handle nested paths transparently.