Add code to reconstitute deliver() from a cache area#713
Add code to reconstitute deliver() from a cache area#713
deliver() from a cache area#713Conversation
ponyisi
requested review from
ArturU043,
BenGalewsky,
MattShirley,
gordonwatts and
kyungeonchoi
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #713 +/- ##
==========================================
+ Coverage 98.35% 98.37% +0.02%
==========================================
Files 30 31 +1
Lines 2190 2218 +28
==========================================
+ Hits 2154 2182 +28
Misses 36 36
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Hi Is there a reason for wanting to set Maybe I am missing something about this point. |
|
Hi @ArturU043 - I tend to prefer not to lock choices in too early if we don't have to - there could certainly be debugging situations where we would want to use the URLs, or even a situation where you would want to test locally with downloads but then parallelize on nodes which need to use URLs instead. (Note that we don't actually verify whether the files exist locally before returning the list...) |
|
After some discussions and thought I might prefer this function to be called |
|
|
||
| if local_preferred: | ||
| return { | ||
| _[0]: GuardList(_[1].file_list if _[1].file_list else _[1].signed_url_list) |
There was a problem hiding this comment.
Can this be refactored into functions stored in servicex_client.py (that is already loaded into this file and is where the duplicated dictionary comprehensions are stored)?
| f"{path} does not contain a valid ServiceX download area" | ||
| ) | ||
| config = Configuration(api_endpoints=[], cache_path=str(path)) | ||
| cache = QueryCache(config) |
There was a problem hiding this comment.
Should the cache object be closed as in the tests (sidebar: should QueryCache be a context manager?)?
3 participants
Addresses #710 . New function
servicex.read_dir(name is up for discussion) which gives a mockdeliver()return value corresponding to data in a cache area. Assumes cache is in fact valid (i.e. doesn't validate that files or URLs actually work).