open list

[MikeBadescu]

Open list / wishes / references. Once fully defined, move them to separate issues
Mostly brainstorming, half baked ideas.

[MikeBadescu]

R6Eddy

• add S3 print

R6Flow

• add S3 print
• is_rflow, to avoid rflow() an R6Flow or an R6FlowElement
• forget()

Other:

• reactivity
• consider future
• consider storr
• option to use only one envir and only one dir instead of one env/dir per fn_key
• AWS cache

[MikeBadescu]

Reference: best practices (for now) to document R6

• https://stackoverflow.com/questions/45431845/documenting-r6-classes-and-methods-within-r-package-in-rstudio
• https://github.com/r-lib/processx/blob/bc7483237b0fbe723390cbb74951221968fdb963/R/process.R#L2

[MikeBadescu]

Notes:

• writing an wrapping (_rf) function for each function is annoying
• the most important option in the wrapping function is the eddy ==> option to set a default eddy
• consider an special operator, e.g. %rflow% (similar to %>% and %dopar%) that provides the caching

f1(a, b) %rflow%
f2(c) %rflow%
rf(f3(d), rflow_options = ...)

• details:
  • the first %rflow% does f1(a,b) %>% f2(c) while recognizing f1 and f2 as functions, adding them to the eddy and inspecting their arguments; it also can identify whether each input is already an rflow
  • the second %rflow% takes the first input (an rflow now) and performs %>% f3(d); it recognizes rf() function as a special function that provides additional options to the f3's rflow.
• we cannot have f1 %rflow% f2 %rflow% rf(options...) since rf() will be parsed afterf1 %rflow% f2 and it is confusing w.r.t. the order of operations
  • maybe f1 %rflow% rf(options...) %rflow% f2 could work under the convention that rf always applies to the next argument, e.g. (f2),
  • thus, in this case, rf does not apply to f1 and it is "preserved" within the rflow of f1 to be passed to f2
  • but it is still confusing and goes against the standard %>% logic
  • maybe setting options to the "downstream / outward" rflow using with_rf(...) or set_rf(...) ==> implies it applies downstream (to all downstream or just to the next?)
• by default (without %rflow%), rf does nothing, it is the identity function
  • changing the pipe op to %>% gives the standard pipe
  • maybe an option for the %rflow% op to behave like %>% for debug purposes?
• not keen to store these settings in options
  • maybe store them in a package env ?
  • or set them at the beginning of each pipeline? e.g., with_rf(options) %rflow% ...
• better op name:(?) %>rf>%, %rf%, %rf>%; %rf>T% for the T op?

[MikeBadescu]

• the most efficient way (so far) to calc rowwise hash: df %>% pmap_chr( ~ .EDDY$digest(list(...)) )

[MikeBadescu]

others:

• input_hash: which (named) arguments represent vector / list of file paths (and multiple files)
• input_hash: which (named) arguments not to consider when calc input hash
• option to cache split if %in% c(list, data.frame)
• both sources and sinks must be anchored to the files / variables, not to functions
  • if at function level, the "state" changes when reading from a second file ==> triggers update, even if the first file did not change (same reasoning for sinks)
• there is also a need for a simple operator / function to force the re-calc (delete output and recalc)
  • this implies fn to be pure (do not exclude args) and to save args to disk (not only in temp env)
  • if so, it can be implemented by the user as needed [very low priority]

[MikeBadescu]

eager/lazy: the calculations are eager but the cache retrieval is lazy

• solution: make calculations lazy as well, to truly reflect the concept of "collect"
  • lazy calc concept is closer to shiny and helps with reactivity
• another advantage to lazy calc is that we can inquiry the state (e.g., before a long calculation takes place, to provide an warning to the user)
  • now we can inquire, but it is a hack

[AnaNiculescu36]

• For add_group(group) : what happens if group already exists? (TBD)
  Answer (edit): all cache functions are "group" friendly, they create / delete groups as needed when the context is clear; this also helps with multi-layer cache engines where different groups might be present in different layers.

[MikeBadescu]

• enhancement: similar to flow_dfr, consider two-table functions, e.g. left_join between two df's

[MikeBadescu]

• can we use later to flush the cache when R is done / idle?

[MikeBadescu]

• the custom input hash create problems (we need to be able to get an hash for each arg, with or without the data) ==>
• instead of excluded args and custom hash function, consider a named list of functions (one for each named argument), each such function returning the data to be hashed.
• to exclude args from hash, in the list of named functions just provide a value instead of a function, e.g. NA or NULL
• in2_hash column that takes into account out_hash of the arg to avoid repeating calcs

[MikeBadescu]

• file sink

[MikeBadescu]

• function to split and extract elements (since the code repeats in 3 places) ; it will be called from $compute() + tests