Quick dirty html build

[ascholerChemeketa]

I rebased this code off of #2328 as the logic re: runestone-services overlaps and that other PR seems more fundamental.

This adds html.quick-dirty string param. It cuts out build steps when used on its own. When used with -r it prunes the source tree early in the assembly phase and really shines.

The results are not all that dirty... quick and dirty produces clean diffs.

Sample book                                           ~35 seconds
Sample book -r inactive-code                          ~20 seconds
Sample book -x html.quick-dirty yes                   ~23 seconds
Sample book -r inactive-code -x html.quick-dirty yes  ~1.6 seconds

[ascholerChemeketa]

@rbeezer @oscarlevin Our discussion in drop in mostly assumed quick and dirty would in fact be dirty. Since it is cleaner than expected, it might make sense to always enable it when -r is used. I'm not sure I see a use case for just -r without html.quick-dirty yes. Especially since -r is already marked experimental and possibly not reliable. But in the pretext script as swiss army knife philosophy it might make sense to keep them discrete.

For the CLI, the case seems stronger to make building a subset always use quick and dirty. Possibly with an opt-out switch and a message that subset building is not guaranteed to be artifact free.

[bnmnetp]

Kind of jumping in to the middle here, but I'm seeing a potential use case on Runestone.

An instructor types in a question in PreTeXt, we take the PreTeXt and add it to a mimimal book, where we then build the subset of that book containing this one exercise. Looks like it could be quite fast!

Then we extract the generated html to preview and save to the DB. This is essentially what we do today, but I think we could do it with the cli and not have to spin up another process.

[ascholerChemeketa]

Rebased and updated against master post @rbeezer update to Runestone logic.

Retested.

[ascholerChemeketa]

Force push after rebase on current master

[ascholerChemeketa]

Force push after rebase to recent version of master

[rbeezer]

This has merge conflicts I can't work out on my own. First one is with the addition of hooks to allow _static to be cached on the CLI side. I think. With clean-up I should be able to attack this anew. Sorry for all the bitrot lately.

[ascholerChemeketa]

Rebased and cleaned up. Merged fixups into existing commits and force pushed.

Pretty sure about logic for integrating with @oscarlevin's portable code. It makes the first choice - include files or not. If including files, then quick and dirty does its check.

I'm on the fence re: skipping copy_managed_directories. Skipping it does appear to make a measurable difference. But means you don't see any new images. Part of the speed difference is because I usually am running linux via WSL and all file operations have a significant overhead. @rbeezer - do you want to try with and without the conditional on line 3994 of pretext.py?

My sample-book build:
Stock ~15s
-r activecode ~7.5s
-r activecode -x html.quick-dirty yes ~2.1s
When I disable 3994 and always copy_managed_directories
-r activecode -x html.quick-dirty yes ~3s

[rbeezer]

Stock ~2.5s
-r activecode ~1.5s
-r activecode -x html.quick-dirty yes ~0.76s
When I disable 3994 and always copy_managed_directories
-r activecode -x html.quick-dirty yes ~0.75s

So little cost in copying the managed directories. Which side of the fence are you on now?

Do we want build_or_copy_theme()? Maybe in the quick-dirty case we just try a copy from the pre-built un-minified versions?

Let's do a more careful job of defining $b-quick-dirty. Only inputs of yes or no get the job done. See the definition of b-version-only in -assembly.

Just a first pass - there's a bit more I want to look at.

[ascholerChemeketa]

Great, will modify to always copy managed dirs and clean up flag.

Strongly think theme should not be built/copied. If you are using a custom theme it would be actively confusing/bad to had a pre-built swapped in. IMO quick and dirty is for rapidly previewing content changes, not for seeing changes to theme or major structural options.

[rbeezer]

Well, with pretext/pretext I get no CSS, so this is not really of much value. I don't think a feature should be contingent on the choice of a front end.

I'm not 100% clear on the use case for this anyway. We can chat Friday morning?

[ascholerChemeketa]

Yes, we can discuss tomorrow. For the record here, the feature is now more "incremental build" than "quick and dirty build". It assumes you have already built the book and are just looking to see how some small set of changes to the source change the output.

My standard authoring setup at this point is to have nodemon watch my source directory for changes and do a quick and dirty build to the chapter I am working on every time I change the source. It isn't quite live preview, but it is darn close. I think setting up something like that as an option in the CLI would be quite popular.

[rbeezer]

the feature is now more "incremental build"

Good, that makes more sense than whatever I was thinking. ;-) For the record, let's think of changing the name of the string parameter.

[rbeezer]

Maybe this should be on -dev, but here goes. Suppose a stylesheet made a mapping from every structural element to the ID of the structural element of the chunk that contains it. And say this is in a form that Python can read it and make a dictionary easily.

Now, a source file is edited. The mapping takes the ID of the root element of that file and figures out which output chunk reflects the edit. That ID becomes the -r argument. Or something like that. Translate a source file edit into an HTML-output-chunk file and rebuild that only.

Then maybe Python can manage the monitoring and rebuilding on some port/server...?

[ascholerChemeketa]

I like it. But I think that is a separate PR (that is matched with a CLI PR to consume that data).

[ascholerChemeketa]

Just pushed another commit. Not intended as final code - lots of extra timing messages. Converted to draft while code isn't final.

Extended notes here as we probably will have a lull before next discussing this.

Note:
Have figured out that Windows Defender makes file copies REALLY slow. As in each call to copy2 to copy a simple html file ends up taking at least 0.3 seconds. Not something pretext can fix, but worth knowing about while trying to make "real timeish" previews of changes.

Changes:
Realized the merge of the pruning code in pretext-assembly wasn't pruning correctly. Fixed that.

html() in pretext.py was getting unwieldy with the various switches for "incremental preview" and "portable". And the incremental code was mostly just turning everything off. So I have made a new html-incremental entry point in both pretext and pretext.py. The pretext script does a little setup stringparams/pubvars, then calls html-incremental.

The logic for html-incremental is that a front end tool can:

• one time startup: run the pubvar report and augment stringparams
• watch for changes in source dir and call html-incremental() when they happen
• watch for changes in external/generated and call a function to copy those (or trigger a non-incremental rebuild)

The code in the pretext script is essentially a super simple version of a frontend tool that doesn't watch for changes.

Current stats on my machine, native Windows:
sample-article -f html: ~9s
sample-article -f html -r lists: ~4.2s
sample-article -f html-incremental -r lists: ~0.5s

A front end does need to know what to rebuild. To avoid the user having to specify an xml:id as part of the "start watching" command, we will need something like Rob's clever mapping. That could be built once as part of the "one time startup" code.

[rbeezer]

Visual scan while away from a real machine. (Not because you just happened to push.)

I'll need to look closer, and a chat might help. WeBWorK in assembly already has me concerned, so I need to look closely at which tree is being used.

[ascholerChemeketa]

I'm still working. Will add a comment and mark "Ready for review" when I think it is worth a deeper look.

[ascholerChemeketa]

Done with cleanups. I think this is ready for serious review

Broke the functionality into two features:

• build-incremental is an attempt to build pages with clean diffs, but only build specified pages (no search, theme, etc...)
• build-preview is an even more aggressive level of partial build that prunes the subtree. Here we can guarantee some things will be broken (cross page links/knowls in particular).

Assembly tree change should only affect later levels when use html.build-preview. That mode already promises that what you get will not be correct enough for a final build.

ol-markers generating directory when needed seems unrelated, but lack of dir caused an issue at one point for me.

[ascholerChemeketa]

Recent timing tests.

TourCS book - Using windows on windows file system:
-f assembly-dynamic 1.1s
-f assembly-dynamic-x html.build-preview yes subtree vectors_creating-vectors 0.35s
-f html 7.2s
-f html-incremental 4.8s
-f html -r vectors_creating-vectors 4.2s
-f html-incremental -r vectors_creating-vectors 2s
-f html-incremental -r vectors_creating-vectors -x html.build-preview yes 0.56s

TourCS book - Using WSL build on windows file system:
-f assembly-dynamic 3.1s
-f assembly-dynamic-x html.build-preview yes subtree vectors_creating-vectors 2.5s
-f html 26.4s
-f html-incremental 8s
-f html -r vectors_creating-vectors 20.5s
-f html-incremental -r vectors_creating-vectors 4.1s
-f html-incremental -r vectors_creating-vectors -x html.build-preview yes 2.8s

[ascholerChemeketa]

Rebase against more recent master. All looks good.

[ascholerChemeketa]

Rebase.

[ascholerChemeketa]

Rebase to a0d5516