Quarto
Replies: 9 comments 20 replies
-
|
You are talking about the R package but the R package is a wrapper around the real tool which is not an R package contrary to knitr or RMarkdown (so it is not loaded in R session, does not use your R session or anything from it)
Quarto's document reproducibility is ensured by the fact that by the document itself or the Quarto project. For short, if you wanted to load R packages, render a Quarto document using the loaded functions, this can't be achieved. |
Beta Was this translation helpful? Give feedback.
-
|
ok thanks looks to me like a loss of feature compared to rmarkdown or knitr is there any plan for a knit2pdf that would compile from typst instead of Latex ? this would be great, and keep the facility to pick current R session as in regular knitr/rmarkdown |
Beta Was this translation helpful? Give feedback.
-
I don't believe this was ever a feature in the first but more of a side effect of knitr/rmarkdown running in R.
This repository is about Quarto not knitr specific or whatever package this function is from.
This idea to use unknown things from a session goes against the idea of reproducibility and the idea of having self-contained projects, so this is unlikely to ever be possible in Quarto. Could you explain what your workflow is exactly and why you need things from an R session that can't live in your document? |
Beta Was this translation helpful? Give feedback.
-
|
the workflow is the following quarto is oversized for that simple usage my workflow is more programmatic I would say, don't need the visual cells and don't use jupyter like approaches as they tend in my opinion to get in the way I don't aim at reproducibility with sharing to others with uploading in a cloud, I just need reproducibility for my own scripts of course currently you get typst with R sweave easely only in quarto, there is not knit2dpf using typst instead of latex... I know it's theoretically possible and relatively easy to tweak to do it with knitr, but I did'nt get to it yet when you put everything in a quarto cell you lose reusability of the scripts for other purpose, the scripts is trapped in the quarto document, ok for reproducibility, but not for reusablity of building block R scripts in my opinion there should be only minimal R instruction in a Rnw or quarto template, just the necessary to print the tables and the graphs namely, but not anything heavy to compute, and the rest is just writing thanks |
Beta Was this translation helpful? Give feedback.
-
|
Based on what you said, I don't think Quarto is the tool you need/want as it would require from you to change your entire workflow to use external script that you source/load in a code cell (knitr), to use tools such as
Quarto does not use R Sweave (it's for LaTeX/R). As far as I know there is nothing in R that currently supports Typst. For reference, as you have opened a feature request for a native support at the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for your feedback @blset ! Really appreciated to know more about users' workflow and how they try to use Quarto
As a developer of knitr and maintainer of R Markdown, I'll take the opportunity to comment on the workflow and compare it with Quarto. The workflow you have is very different from what Quarto offers. Quarto is closer to how the R Markdown workflow works. A document (respectively .qmd or .Rmd) is expected to contain all computations in cells. They are assumed to be reproducible source files. When preparations are in a bunch of R scripts, they can be loaded for the document in a cell at the top. There are features to easily load them:
So, regarding your comment
They are not trapped. You can do all your code content in an R script and then source it as a script to make computation available for knitr. To share experience, some even structure their work as an R package for their preparation function and data and load the package inside the document to make everything available easily, and have the package available for easy reusability outside the document. The document can then be rendered in a background session without problem. The assumption of having a Rmd file be reproducible by containing everything is the default, and it is even bigger with Quarto as rendering a .qmd will always run in a new clean R session. So, what to do when pre-calculation is needed, when data needs to be shared, or pre-computed ? So in a nutshell, yes, using Quarto will require a change in your workflow. It would have been the same if you were using R Markdown, probably. By using
I understand you are only considering Quarto for Typst support. So thank you for opening This is probably to be considered—PR is welcome there, too. We'll discuss it in the issue in the knitr repo.
For this part, I did not completely follow. To have a BTW, Quarto has support for So |
Beta Was this translation helpful? Give feedback.
-
|
Hello thank you for your response
yes but there is a hitch, you have to set very carefully the path of the files, when you run the scripts in advance you don't have to worry about the file paths (they are set independently), but once they are in a cell it's more complicated you can use here library but is so much simpler to inherit current session
yes ok but then you need to use cache for long prepared scripts, and cache always reserve surprises it's not always in sync, you don't see clearly what's inside and it is a layer of complexity, so much easier to just inherit the session
indeed I tried quarto but knitr / session / latex is more natural for my use case. I retried quarto because it can compile to typst I would like to switch to typst instead of latex, and so I am looking for knit / session /typst. also I would like to be able to compose the template file not in markdown but directly in typst, what I mean is that in my workflow, all the heavy work is done in advance by independent programs and I pass the session to knit2pdf and it is available in Rnw templates, and in these templates I just write text and if something goes wrong I can browse the object in the R session and everything can be fixed quickly, with debug points if necessary, even re running part of the scripts only if needed. very agile workflow. you don't put debug points in caches you don't slice the cache... or then your heavy code is disseminated in cells, a total nightmare. thank you very much |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your additional details. What you are looking at then is not Quarto. This is indeed knitr support for
Sorry if I have misunderstood your workflow. Sourcing before calling knit or running inside a cell should be the same as long as the working directory is the same. Now I understand this may not be the case.
Caching objects does not mean not recomputing. I use this term as a way to name the fact of flowchart LR
A[Running some code] --> B[Saving object]
B --> C[Reusing object by loading them]
C --> D[Rendering document]
Doing everything in the current R session is only "caching in memory" which works if the R session is shared - which is not always the case. This is a layer of complexity, which can be required in some situations, and also it can be simplified with some tools. targets is one of them.
The cached object is only there to pass from an R script run in one session to rendering the document in another. If you need to debug, this would happen in your script. But this is also where using the R package structure to organize code has benefits. The document could use only functions from this package, and any function can be easily debugged as a package. Anyhow, just wanted to add precision on what I meant. All of this is a matter of workflow, and there is no best one; good ones vary by person. This is a personal matter. Using Quarto is like calling another external tool when you are inside R. So, if your workflow is not compatible with this, using R package only will be the best. Just the knit to typst feature does not exist yet. You could use R markdown with custom formats, but depending on which R package you use, they may not know what to do in the Typst context. For example, gt for producing an HTML table would not output anything in Typst. Let's continue in knitr repo ! Thanks a lot ! |
Beta Was this translation helpful? Give feedback.
-
|
I second the sentiment about this feeling like a major feature loss when coming from |
Beta Was this translation helpful? Give feedback.
-
|
This topic is all about workflow.
So the broader question here is "How do I work with a non-self-contained document? How do I make available pre-computed data to my document ?" This is a workflow question to me. First answer would be to use workflow tools. R has an excellent option with targets R 📦 . It has support for Quarto: https://books.ropensci.org/targets/literate-programming.html Another option is to manually save some disk space. This could be done using caching (available in knitr or another R package) or the more classic The idea is the same each time: the R object required by the Quarto document needs to be accessed from somewhere to be loaded in memory. So regarding Hope it helps. |
Beta Was this translation helpful? Give feedback.
-
|
Targets is a great package but I don't think it needs to be used for every "big" workflow. I have used it before and it's excellent, especially when the benefits are worth the added effort to fit within it's framework. However, for other (non trivial) workflows (I can think of several), the simple and flexible option for an input for passing data objects directly into the quarto render call would serve as a great compromise and stop gap. There are several discussions online from my quick search discussing this topic, so I won't go into specific use cases beyond stating that a 'resources' input (data, objects, etc) directly in the render call would be a user friendly addition for some legitimate workflow patterns. |
Beta Was this translation helpful? Give feedback.
-
|
I agree and I would add that if one has to prepare everything upfront before using quarto then, for what concerns a typst output, it's simpler and more efficient to just use typst within vs code for instance. you just include the graphs already made and the tables already made in typst format and then |
Beta Was this translation helpful? Give feedback.
-
|
yes that is definitely an option if you need only PDF through typst and you know how to write Typst syntax to get all the feature you need for your document. |
Beta Was this translation helpful? Give feedback.
-
|
I agree this feels like a loss in functionality. My common pattern is to query/clean data for the whole country, and then use a loop to create HTML outputs for each region from the same Rmd. It's more efficient to do my data preparation in one go, and I'll realise sooner if there are any errors. It seems that a pattern like this isn't possible with Quarto, without extra complications like saving data to disk etc. If Quarto just isn't designed for that, so be it I guess. I can keep using R Markdown. But it seems like a shame, if Quarto is the format that will be more actively developed. |
Beta Was this translation helpful? Give feedback.
-
It is not as easy as it sounds. And with current architecture, this would be probably done by saving on a disk the R session (or part of it) and reloading it after. A bit like the So for now, we decided that instead of doing that for the user internally a bit magically, it would be best and safer to let users be explicit about what needs to be save, and when it should be reloaded. Probably more optimized memorywise too. Some users do it manually in cells, using tools like that saves on disk, and then reload when needed in another document. Others leverage tooling like targets R package that allows to track all computation of assets and cleverly reuse them and recompute them when needed. Obviously this can be discuss, but inherently one thing will not change: for multiple documents project, for each of them Quarto open a R session, run knitr, close the R session, do others stuff, then on next doc, open a new R session, etc... So problem is passing object between independent sessions. In R Markdown ecosystem it is the same in bookdown if you are using the knit-and-merge mode (https://bookdown.org/yihui/bookdown/new-session.html). Same if you are calling So I understand this could feels like a loss, but this can also be seen as an opportunity to improve one workflow toward better reproducibility and safer computation. At least, that is how I see it. I'll keep thinking about a solution - it would probably not be internal saving, but maybe having a way to have some content loaded at R session startup (which .Rprofile can offers already, or some clever tool like startup R package. Hope it helps. And thank you a lot for sharing your thoughts! That is really helpful. |
Beta Was this translation helpful? Give feedback.
-
|
It would appear this is a common request
|
Beta Was this translation helpful? Give feedback.
-
|
yes it is. |
Beta Was this translation helpful? Give feedback.
-
I do get this point, but it feels like a slightly narrow perspective. Reproducibility is extremely important for scientific analysis etc, but it's not the top priority for automation of routine outputs. That's a lot of what my organisation uses R Markdown for, we've got tons of weekly reports etc that are based on R Markdown. Sometimes we run them from a parent script for efficiency, to load the data once where it's the same report for multiple areas. Other times we might use the parent script to authenticate with MS SharePoint, which we can't reliably do in a non-interactive session (I know it should be possible, but it would take cooperation from IT, would never get off the ground). Being able to do things like that is more important for us than perfect reproducibility, which we'd never have anyway because our data changes constantly.
Perhaps I'm naive, but I'd have more faith in a solution you implemented centrally for Quarto than I'd have in my own (or my colleagues') individual manual saving/reading from disk solutions. I am surprised actually that you'd also be constrained to a saving/reading pattern, I would have imagined you could pass things directly between the R sessions. But that's beyond my expertise. Something like targets might be a reliable option, but for my workplace it adds too much complexity. Overall, unless there's a simple way to get behaviour like |
Beta Was this translation helpful? Give feedback.
-
No we can't. Currently, with knitr engine, all R sessions are independent. So each process is independant, and opened / closed one after the other. No shared session, no shared process. Only way to "share" content would be to save them and read them (on disk, database, online service, ...) - any solution that can help preserver state.
Your current usage of Other are doing quite similar but they usually have a So to me this is a workflow perspective, and I am sure a specific solution narrowed to specific usecase is better. Some do use targets and other use other method. Anyhow, in current design we can't make session communicate on memory only. Maybe we'll be able in the future to create a new R engine for Quarto documents that would open a session once, and keep it open for all render so that a single R environment is shared. Overall this is about how you organize your studies and projects, how data are created and managed, how documents and projects needs to be self contained or not, and all of this is very specific usage with no better way. This is why it is hard to come with a "good for all solution".
rmarkdown behavior to use object in current session for evaluation of what is inside the document is a side effect of it being coded in R. If you need that feature primarily then I understand you stay in R Markdown, and this is ok. We're (I am with Yihui) still maintaining it, and you can still use it. You are not left behind. Hope this clarifies a little, and it helps understand. If we can find a solution, we'll offer something (or community could) - but I think this needs to pass through a new Quarto computation engine that works slightly different (and will have its own limitation). But just my current thoughts that need research (when I'll have time after all Quarto issues) |
Beta Was this translation helpful? Give feedback.
-
|
Quarto can use other than Python, but in real work you are not going to use python in one cell and R in another or something like that I ended up using neither quarto nor Rmardown but generate all images in svg and typst modules as text files containing several related tables all from regular R scripts and then I just compose the typst document with vscode using typst directly. There is a instant preview it's blazingly fast, faster than RMardown, zero complications |
Beta Was this translation helpful? Give feedback.
-
|
Have you only used Typst editing in VSCode, curious about the experience in Positron? |
Beta Was this translation helpful? Give feedback.
-
|
The experience is the same as it relies on an extension: Tinymist (https://github.com/Myriad-Dreamin/tinymist). There are no perfect tools. You always need to find the right one for the right task. |
Beta Was this translation helpful? Give feedback.
-
|
For raw typst it is better in vscode, once all resources are generated |
Beta Was this translation helpful? Give feedback.
-
|
@blset Avoid replying in different threads. It scatters the discussion making it harder to follow. Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
There has been a lot of discussion on this topic and some are starting to get a bit circular. I will share two save/load options I have used to possibly help fellow R users that have to use Quarto for one reason or another; maybe they are forced to, maybe they simply don't want to use the Rmarkdown styling (default is still BS3?). Both options could be improved if the quarto Method 1: Temporary save and load by nameIn a # Function to save objects to a tempdir as qs2 object and return all the paths
auto_save_by_name <- function(...) {
tmp_dir <- tempdir()
en_obj <- rlang::enquos(...)
en_names <- purrr::map_chr(en_obj, rlang::as_label)
loc_paths <- purrr::map(en_names, \(x) file.path(tmp_dir, paste0(x, '.qs2')))
names(loc_paths) <- en_names
purrr::walk2(list(...), loc_paths, \(x, y) qs2::qs_save(x, y))
loc_paths
}
# List your objects needed for Quarto and pass thru function
list_data_loc <- auto_save_by_name(myfavdata1, myfavdata2, myfavvector1)
# Execute the Quarto param with the data locs
quarto::quarto_render(
input = "./path/to/my.qmd",
execute_params = list(data_loc = list_data_loc)
)
Now, in the QMD file, you can restore all these files at once... # Recalls obj from R tempdir, so dont need to reformat on load
loadlist <- purrr::map(params$data_loc, \(x) qs2::qs_read(x))
setNames(loadlist, names(params$data_loc))
list2env(loadlist, envir = knitr::knit_global());
rm(loadlist);Method 2: Stow/RetrieveTaking this one step further, we can formalize the save and loading procedures with the addition of classes, making them easier to discover within a workflow and pass to another session. This is what I call {targets}-lite, to myself usually. One thing I'm not sure about in the Quarto case, is if the object can be passed as an execution parameter, if not, the path itself could still be sent along (making this a similar method to (1)). Again, this is something I hope Quarto as a package will be able to support, the parameter inputs are not flexible in my experience. There are two paired functions, Similar to (1), you # Example data
temp_data <- data.frame(col1 = rpois(100, 1), col2 = runif(100))
tmp_dir <- tempdir()
temp_data_stowed <- stow(object = temp_data, path = tmp_dir, method = 'fst')
temp_data_stowed2 <- stow(object = temp_data, path = tmp_dir, method = 'fst')You can list all the objects stowed and, as before, pass this to the render call all_stowaways <- lapply(ls()[sapply(ls(), \(x) inherits(get(x), 'stow_temp'))], get)
quarto::quarto_render(
input = "./path/to/my.qmd",
execute_params = list(data_loc = all_stowaways)
)Then in the QMD, you can use purrr::walk(params$data_loc, \(x) retrieve(x, keep_name = TRUE)) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Description
hello
using knit2pdf at the end of an R script the Rnw templates with latex inherit current session with all libraries loaded
I cannot translate that workflow with quarto
using quarto_render at the end of an R script the qmd files have no access to the current session
I would have to pass individually all objects and load all libraries and source scripts from inside the qmd files
this is very inconvenient
how would you pass the whole session to quarto simply ?
I'm aware the render button in rstudio executes everything in a brand new environment for reproducibility
but I thought using quarto_render would allow the use case of inheriting the current session which is very convenient when sharing is not the goal or to experiment easely
Beta Was this translation helpful? Give feedback.
All reactions