From fa73d7c8fa94eb4cec3402756c995944ec6d2ef8 Mon Sep 17 00:00:00 2001 From: Indrayudd Roy Chowdhury Date: Sat, 24 May 2025 20:47:36 -0400 Subject: [PATCH 01/26] HelpersTask741: Add commonly used commands in a how-to guide --- .../all.notes_toolchain.how_to_guide.md | 299 ++++++++++++++++++ 1 file changed, 299 insertions(+) create mode 100644 docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md new file mode 100644 index 000000000..cd20eaba2 --- /dev/null +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -0,0 +1,299 @@ + + +- [Notes Documentation Toolchain](#notes-documentation-toolchain) + * [1 · Generate slides & PDFs — `notes_to_pdf.py`](#1-%C2%B7-generate-slides--pdfs--notes_to_pdfpy) + + [What it does](#what-it-does) + + [Most‑used flags](#most%E2%80%91used-flags) + + [Quick‑start recipes](#quick%E2%80%91start-recipes) + + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet) + + [Worked examples](#worked-examples) + - [Slides with navigation breadcrumbs](#slides-with-navigation-breadcrumbs) + - [Focus on a subsection](#focus-on-a-subsection) + - [Plain PDF article (no slides)](#plain-pdf-article-no-slides) + * [2 · Auto‑render figures — `render_images.py`](#2-%C2%B7-auto%E2%80%91render-figures--render_imagespy) + + [Supported File types and Code blocks](#supported-file-types-and-code-blocks) + + [Quick Start Recipes](#quick-start-recipes) + - [Render to a new file](#render-to-a-new-file) + - [Render in‑place (Markdown or LaTeX)](#render-in%E2%80%91place-markdown-or-latex) + - [HTML preview of already‑rendered images](#html-preview-of-already%E2%80%91rendered-images) + - [Dry‑run (test parsing / comments only)](#dry%E2%80%91run-test-parsing--comments-only) + + [Flags](#flags) + * [3 · Lint & prettify — `lint_notes.py`](#3-%C2%B7-lint--prettify--lint_notespy) + + [Quickstart recipes](#quickstart-recipes) + - [In‑place prettify with Dockerised Prettier + TOC rebuild](#in%E2%80%91place-prettify-with-dockerised-prettier--toc-rebuild) + - [Custom print width & selective actions](#custom-print-width--selective-actions) + + [Flags](#flags-1) + * [4 · Notebook image scraping — `extract_notebook_images.py`](#4-%C2%B7-notebook-image-scraping--extract_notebook_imagespy) + + [Flag Options](#flag-options) + * [5 · LLM‑powered transforms — `llm_transform.py`](#5-%C2%B7-llm%E2%80%91powered-transforms--llm_transformpy) + + [Minimum viable command](#minimum-viable-command) + + [Finding available prompts](#finding-available-prompts) + + [Flags](#flags-2) + + [Example recipes](#example-recipes) + + + +# Notes Documentation Toolchain + +This is a high‑level guide to the helper scripts that turn raw `.txt` lecture +notes into polished PDFs, slide decks, and more. + +--- + +## 1 · Generate slides & PDFs — `notes_to_pdf.py` + +### What it does + +Convert plain‑text notes into polished **PDF, HTML, or Beamer slides** with a +single command. + +> ```bash +> notes_to_pdf.py --input --output --type [pdf|html|slides] +> ``` + +### Most‑used flags + +* `--type {pdf|html|slides}` +* `--toc_type {none|pandoc_native|navigation}` +* `--debug_on_error`, `--skip_action ...`, `--filter_by_lines A:B` + +### Quick‑start recipes + +| Goal | Command | +| ----------------------------------- | ------------------------------------------------------------------ | +| Compile to **Beamer slides** | `notes_to_pdf.py -i lesson.txt -o lesson.pdf --type slides` | +| Produce a **stand‑alone HTML** page | `notes_to_pdf.py -i cheatsheet.txt -o cheatsheet.html --type html` | +| Build a **PDF article** (LaTeX) | `notes_to_pdf.py -i paper.txt -o paper.pdf --type pdf` | +| Skip the final viewer **open** step | `... --skip_action open` | + +_TIP  Run with `--preview_actions` to print the exact steps without executing +them._ + +### CLI flags cheat‑sheet + +| Flag | Purpose | Notes | +| -------------------------------------------- | ------------------------------------------------------------------------- | ----------------------------------------------------- | +| `--type {pdf,html,slides}` | Output format | "slides" uses Beamer | +| `--toc_type {none,pandoc_native,navigation}` | TOC style | `navigation` inserts slide‑friendly breadcrumb frames | +| `--filter_by_header "# Intro"` | Build artefact from a _section subset_ | Useful for testing | +| `--filter_by_lines 120:250` | Compile only a range of lines | Accepts `None` sentinel | +| `--debug_on_error` | On Pandoc failure, generate _.tex_ and drop you a helpful log | | +| `--script myrun.sh` | Save every shell command executed | Repro build pipelines | +| Docker knobs | `--dockerized_force_rebuild`, `--dockerized_use_sudo`, `--use_host_tools` | Control container vs host pandoc/latex | + +_Run `notes_to_pdf.py -h` for the exhaustive list._ + +### Worked examples + +#### Slides with navigation breadcrumbs + +TODO(indro): `--toc_type navigation` fails because of the preprocess step. + +```bash +notes_to_pdf.py \ + --input MSML610/Lesson5-Theory_Statistical_learning.txt \ + --output Lesson5.pdf \ + --type slides \ + --toc_type navigation \ + --debug_on_error \ + --skip_action cleanup_after +``` + +_Highlights_: adds breadcrumb navigation, keeps intermediates for inspection. + +#### Focus on a subsection + +```bash +notes_to_pdf.py \ + --input Lesson8-Reasoning_over_time.txt \ + --output Focus.pdf \ + --type slides \ + --filter_by_lines 362:None \ + --skip_action cleanup_after +``` + +Compiles only from line 362 to EOF—fast iteration when debugging slides. + +#### Plain PDF article (no slides) + +```bash +notes_to_pdf.py -i book_notes.txt -o book_notes.pdf --type pdf +``` + +--- + +## 2 · Auto‑render figures — `render_images.py` + +Detects fenced code blocks (PlantUML, Mermaid, TikZ, Graphviz, ...), renders +them into images and swaps the block with `![](img)` markup. + +Example: + +```bash +render_images.py -i notes/MSML610/Lesson9-Causal_inference.txt \ + -o lesson9.images.txt --run_dockerized +``` + +### Supported File types and Code blocks + +| File extension | Rendering syntax allowed | Output embeds as | +| -------------- | ---------------------------------------------- | ----------------------- | +| `.md`, `.txt` | `plantuml / mermaid / graphviz / tikz / latex` | `` | +| `.tex` | same tags (TikZ & LaTeX especially) | `\includegraphics{...}` | + +### Quick Start Recipes + +#### Render to a new file + +```bash +render_images.py -i lesson.md -o lesson.rendered.md --action render --run_dockerized +``` + +#### Render in‑place (Markdown or LaTeX) + +```bash +render_images.py -i lesson.md --action render --run_dockerized +``` + +#### HTML preview of already‑rendered images + +```bash +render_images.py -i lesson.md --action open --run_dockerized +``` + +#### Dry‑run (test parsing / comments only) + +```bash +render_images.py -i lesson.md -o /tmp/out.md --dry_run +``` + +### Flags + +| Flag | Default | Purpose | +| ----------------------------------- | ------------ | ---------------------------------------------------------- | +| `-i/--in_file_name` | **required** | Input `.md`, `.tex`, or `.txt` | +| `-o/--out_file_name` | `` | Output path (must share extension) | +| `--action` | `render` | `render` ↔ `open` | +| `--dry_run` | _False_ | Skip actual rendering, still rewrites markup | +| `--run_dockerized / --dockerized_*` | _False_ | Use pre‑built container images for PlantUML, Mermaid, etc. | +| `--verbosity/-v` | `INFO` | Logging verbosity | + +--- + +## 3 · Lint & prettify — `lint_notes.py` + +A tidying‑up tool for Markdown/LaTeX notes: normalise G‑Doc artifacts, run +Prettier, fix bullet/heading quirks, refresh the Table of Contents – all from a +single command or straight from vim. + +### Quickstart recipes + +#### In‑place prettify with Dockerised Prettier + TOC rebuild + +```bash +lint_notes.py -i Lesson10.md --in_place \ + --use_dockerized_prettier \ + --use_dockerized_markdown_toc +``` + +#### Custom print width & selective actions + +```bash +lint_notes.py -i draft.txt -o tidy.txt -w 100 \ + --action preprocess,prettier,postprocess +``` + +### Flags + +| Flag | Default | Purpose | +| ------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------- | +| `-i/--infile` | **stdin** | Input `.txt` or `.md` (also via pipe) | +| `-o/--outfile` | **stdout** | Destination file (omit for pipe) | +| `--in_place` | _False_ | Overwrite the input file | +| `-w/--print-width` | _None_ → Prettier default | Line wrap width | +| `--use_dockerized_prettier` | _False_ | Run Prettier inside helper container | +| `--use_dockerized_markdown_toc` | _False_ | Refresh TOC via containerised `markdown-toc` | +| `--action` | all five stages | Comma‑separated subset of: `preprocess`, `prettier`, `postprocess`, `frame_chapters`, `refresh_toc` | +| `-v/--verbosity` | `INFO` | Logging level | + +--- + +## 4 · Notebook image scraping — `extract_notebook_images.py` + +Spins up a docker container and dumps every `png/svg` output cell into a folder. +You can then publish or reuse the static plots/diagrams already rendered in a +Jupyter notebook. + +Minimal call: + +```bash +extract_notebook_images.py \ + --in_notebook_filename notebooks/Lesson8.ipynb \ + --out_image_dir notebooks/screenshots +``` + +### Flag Options + +| Flag | Purpose | Default | +| ---------------------------------- | ------------------------------------------------------------- | ------------ | +| `-i / --in_notebook_filename PATH` | Notebook to scan | **required** | +| `-o / --out_image_dir DIR` | Folder where images land | **required** | +| `--dockerized_force_rebuild` | Re‑build the Docker image (use if you changed extractor code) | _false_ | +| `--dockerized_use_sudo` | Prepend `sudo docker ...` | auto‑detects | +| `-v INFO/DEBUG` | Log verbosity | `INFO` | + +--- + +## 5 · LLM‑powered transforms — `llm_transform.py` + +Apply a GPT‑style transformation (rewrite, summarise, critique code, convert to +slides, etc.) to any text file _without_ leaving the terminal / editor. + +> _Note: You need to have an `OPENAI_API_KEY` and an internet connection._ + +### Minimum viable command + +```bash +llm_transform.py -i draft.txt -o polished.txt -p rewrite_clearer +``` + +### Finding available prompts + +```bash +llm_transform.py -p list -i - -o - +``` + +### Flags + +| Flag | Role | Notes | +| -------------------------------------------------------------------- | ------------------------------------------------------------- | ---------------------- | +| `-i / --input` | Source text (`-` = stdin) | — | +| `-o / --output` | Destination (`-` = stdout) | — | +| `-p / --prompt` | **Prompt tag** (`list`, `code_review`, `slide_colorize`, ...) | required | +| `-c / --compare` | Print _both_ original & transformed blocks to stdout | helpful for quick diff | +| `-b / --bold_first_level_bullets` | Post‑format tweak for slide prompts | | +| `-s / --skip-post-transforms` | Return raw LLM output, skip prettier/cleanup | | +| Docker flags (`--dockerized_force_rebuild`, `--dockerized_use_sudo`) | Control container lifecycle | + +### Example recipes + +* **Turn a code file into a review checklist** + + ```bash + llm_transform.py -i foo.py -o cfile -p code_review + vim cfile + ``` + +* **Color‑accent the bold bullets for slides** + + ```bash + llm_transform.py -i deck.md -o - -p slide_colorize | tee deck.color.md + ``` + +* **Inline use in Vim** – visual‑select a block, then: + + ```vim + :'<,'>!llm_transform.py -p summarize -i - -o - + ``` + +--- \ No newline at end of file From 40f2cc732d41ac38dba80f9666503219f23f8a32 Mon Sep 17 00:00:00 2001 From: Indrayudd Roy Chowdhury Date: Sat, 24 May 2025 22:56:26 -0400 Subject: [PATCH 02/26] HelpersTask741: Remove --in_place flag bit (presumably obsolete) and add explanation for notes_to_pdf.py --- .../all.notes_to_pdf.explanation.md | 87 +++++++++++++++++++ .../all.notes_toolchain.how_to_guide.md | 12 +-- 2 files changed, 93 insertions(+), 6 deletions(-) create mode 100644 docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md diff --git a/docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md b/docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md new file mode 100644 index 000000000..b174dd1ff --- /dev/null +++ b/docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md @@ -0,0 +1,87 @@ + + +- [`notes_to_pdf.py` - Flow Explanation](#notes_to_pdfpy---flow-explanation) + * [Goal](#goal) + * [Architecture diagram](#architecture-diagram) + * [Steps](#steps) + * [Dependencies](#dependencies) + + [`preprocess_notes.py`](#preprocess_notespy) + + [`render_images.py`](#render_imagespy) + + [`latex_abbrevs.sty`](#latex_abbrevssty) + + + +# `notes_to_pdf.py` - Flow Explanation + +This document walks through the **architecture** of the `notes_to_pdf.py` + +## Goal + +Convert a lightweight, annotation‑heavy **plain‑text notes file** (usually +`*.txt`) into a share‑ready document – **PDF**, **HTML**, or **Beamer slide +deck** – while: + +* honouring rich Markdown features and custom shorthand, +* inlining auto‑generated diagrams (PlantUML, Mermaid, TikZ, Graphviz, LaTeX + tables). + +--- + +## Architecture diagram + +```mermaid +%%{init: {'theme':'default'}}%% +C4Context + +System_Ext(user, "User", "Author preparing course notes") +System(doc, "Documentation Toolchain", "Python & LaTeX") + +System_Boundary(doc_boundary, "") { + Container(notes2pdf, "notes_to_pdf.py", "Python CLI", "Orchestrates conversion: clean → images → Pandoc → LaTeX") + Container(render, "render_images.py", "Python module", "Renders diagram blocks to images. Caches results.") + Container(preproc, "preprocess_notes.py", "Python module", "Cleans notes & expands macros (pre-Pandoc)") + Container(style, "latex_abbrevs.sty", "LaTeX style", "Provides LaTeX math & formatting shortcuts") +} + +Rel(user, notes2pdf, "Invokes via CLI") +Rel(notes2pdf, preproc, "Uses for Note Pre-processing") +Rel(notes2pdf, render, "Calls to Render Images") +Rel(notes2pdf, style, "Injects LaTeX Style (.sty)") +``` + +--- + +## Steps + +| Stage | Performed by | Key idea | +| ------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **1 – Clean‑up & augmentation** | `preprocess_notes.py` | ‑ Normalises headers, expands arrow shorthand (`->` → `\rightarrow`), deals with comments, inserts Pandoc YAML front‑matter & optional navigation slides. | +| **2 – Diagram extraction** | `render_images.py` | ‑ Scans code blocks (` ```plantuml``, etc.), renders via Docker containers, replaces the code with `![](figs/...)\` include and comments out original block.
‑ Uses a SHA‑256 cache so unchanged diagrams are skipped. | +| **3 – Orchestration** | `notes_to_pdf.py` | ‑ Calls **1** & **2**, then Pandoc, then (for PDF) LaTeX. Flags control each sub‑action so you can skip, debug or re‑run steps individually. | +| **4 – Document synthesis** | **Pandoc + LaTeX** | ‑ Pandoc converts Markdown → LaTeX (or HTML / Beamer).
‑ `latex_abbrevs.sty` is copied next to the generated `.tex` so vector/matrix macros (`\vv{}`, `\mat{}`), deep lists and colour helpers compile correctly. | + +--- + +## Dependencies + +###   `preprocess_notes.py` + +* **Input →** raw notes. +* **Output →** Pandoc‑ready Markdown. +* Handles banner frames, question formatting, colour commands (`\red{}` → + `\textcolor{red}{...}`), TOC injection. + +###   `render_images.py` + +* Docker‑wrapper around _PlantUML_, _Mermaid CLI_, _TikZ → Ghostscript_, + _Graphviz_. +* Produces deterministic filenames: `figs/..png` → + diff‑friendly. + +###   `latex_abbrevs.sty` + +* House style for math: bold‑underlined vectors (`\vv{x}`), matrices, colour + presets, 9‑level `enumitem` lists, symbol shorthands. +* Copied automatically; you rarely touch this unless you need new macros. + +--- \ No newline at end of file diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index cd20eaba2..539698c53 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -136,10 +136,10 @@ render_images.py -i notes/MSML610/Lesson9-Causal_inference.txt \ ### Supported File types and Code blocks -| File extension | Rendering syntax allowed | Output embeds as | -| -------------- | ---------------------------------------------- | ----------------------- | -| `.md`, `.txt` | `plantuml / mermaid / graphviz / tikz / latex` | `` | -| `.tex` | same tags (TikZ & LaTeX especially) | `\includegraphics{...}` | +| File extension | Rendering syntax allowed | Output embeds as | +| -------------- | ---------------------------------------------- | -------------------------- | +| `.md`, `.txt` | `plantuml / mermaid / graphviz / tikz / latex` | `` | +| `.tex` | same tags (TikZ & LaTeX especially) | `\includegraphics{...}` | ### Quick Start Recipes @@ -188,10 +188,10 @@ single command or straight from vim. ### Quickstart recipes -#### In‑place prettify with Dockerised Prettier + TOC rebuild +#### Prettify with Dockerised Prettier + TOC rebuild ```bash -lint_notes.py -i Lesson10.md --in_place \ +lint_notes.py -i Lesson10.md \ --use_dockerized_prettier \ --use_dockerized_markdown_toc ``` From 05d0ca804d8279dcd79489acf2d80c0b4096e3ac Mon Sep 17 00:00:00 2001 From: Indrayudd Roy Chowdhury Date: Sat, 24 May 2025 23:12:25 -0400 Subject: [PATCH 03/26] HelpersTask741: add run_pandoc how-to guide --- .../all.run_pandoc.how_to_guide.md | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 docs/tools/documentation_toolchain/all.run_pandoc.how_to_guide.md diff --git a/docs/tools/documentation_toolchain/all.run_pandoc.how_to_guide.md b/docs/tools/documentation_toolchain/all.run_pandoc.how_to_guide.md new file mode 100644 index 000000000..4460987c9 --- /dev/null +++ b/docs/tools/documentation_toolchain/all.run_pandoc.how_to_guide.md @@ -0,0 +1,52 @@ + + +- [run_pandoc.py – How‑to Guide](#run_pandocpy-%E2%80%93-how%E2%80%91to-guide) + * [What the script does](#what-the-script-does) + * [Quick‑use commands](#quick%E2%80%91use-commands) + * [Flags](#flags) + + + +# run_pandoc.py – How‑to Guide + +A **minimal wrapper** around Pandoc that lets you transform files between types. + +--- + +## What the script does + +```text +stdin / file ──► run_pandoc.py ──► stdout / file + │ + └─► helpers.hlatex.convert_pandoc_md_to_latex() +``` + +* Reads **Markdown** from _stdin_ or `--input` file. +* Dispatches to a named **action** (currently only `convert_md_to_latex`). +* Pushes the Pandoc output to _stdout_ or the `--output` file. + +--- + +## Quick‑use commands + +| Goal | Command | +| ------------------------------------- | -------------------------------------------- | +| Convert a Markdown file to LaTeX | `run_pandoc.py -i note.md -o note.tex` | +| Same, but stream from STDIN to STDOUT | `cat note.md \| run_pandoc.py -i - -o -` | +| Inside **Vim** (visual range) | `:'<,'>!run_pandoc.py -i - -o - -v CRITICAL` | + +> **Tip :** pass `-v CRITICAL` to silence helper logging when piping into +> editors. + +--- + +## Flags + +| Flag | Default | Meaning | +| ------------------ | --------------------- | --------------------------------------------------------- | +| `-i / --input` | `-` | Source file or `-` for STDIN. | +| `-o / --output` | `-` | Destination file or `-` for STDOUT. | +| `--action` | `convert_md_to_latex` | Transformation to apply. Future‑proofed for more actions. | +| `-v / --log_level` | `INFO` | Standard helper‑library verbosity. | + +--- \ No newline at end of file From 9818c78976a796a5c20cec8697ee36dca6e70257 Mon Sep 17 00:00:00 2001 From: Indrayudd Roy Chowdhury Date: Sat, 24 May 2025 23:39:00 -0400 Subject: [PATCH 04/26] HelpersTask741: Put all the documentation into one md --- .../all.notes_toolchain.how_to_guide.md | 160 +++++++++++++++++- 1 file changed, 158 insertions(+), 2 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 539698c53..95aae4c08 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -20,7 +20,7 @@ + [Flags](#flags) * [3 · Lint & prettify — `lint_notes.py`](#3-%C2%B7-lint--prettify--lint_notespy) + [Quickstart recipes](#quickstart-recipes) - - [In‑place prettify with Dockerised Prettier + TOC rebuild](#in%E2%80%91place-prettify-with-dockerised-prettier--toc-rebuild) + - [Prettify with Dockerised Prettier + TOC rebuild](#prettify-with-dockerised-prettier--toc-rebuild) - [Custom print width & selective actions](#custom-print-width--selective-actions) + [Flags](#flags-1) * [4 · Notebook image scraping — `extract_notebook_images.py`](#4-%C2%B7-notebook-image-scraping--extract_notebook_imagespy) @@ -30,6 +30,24 @@ + [Finding available prompts](#finding-available-prompts) + [Flags](#flags-2) + [Example recipes](#example-recipes) + * [6 · Pandoc Wrapper — `run_pandoc.py`](#6-%C2%B7-pandoc-wrapper--run_pandocpy) + + [What the script does](#what-the-script-does) + + [Quick‑use commands](#quick%E2%80%91use-commands) + + [Flags](#flags-3) + * [7 · Automate notes transformations — `transform_notes.py`](#7-%C2%B7-automate-notes-transformations--transform_notespy) + + [What it does](#what-it-does-1) + + [Supported actions](#supported-actions) + + [Examples](#examples) + + [Flags](#flags-4) + * [8 · Scrape headers from a markdown — `extract_headers_from_markdown.py`](#8-%C2%B7-scrape-headers-from-a-markdown--extract_headers_from_markdownpy) + + [Goal](#goal) + + [Examples](#examples-1) + + [Flags](#flags-5) + * [9 · TikZ to Bitmap — `dockerized_tikz_to_bitmap.py`](#9-%C2%B7-tikz-to-bitmap--dockerized_tikz_to_bitmappy) + + [Examples](#examples-2) + * [10 · MacOS screenshot helper — `save_screenshot.py`](#10-%C2%B7-macos-screenshot-helper--save_screenshotpy) + + [What it does](#what-it-does-2) + + [Flags](#flags-6) @@ -209,7 +227,6 @@ lint_notes.py -i draft.txt -o tidy.txt -w 100 \ | ------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------- | | `-i/--infile` | **stdin** | Input `.txt` or `.md` (also via pipe) | | `-o/--outfile` | **stdout** | Destination file (omit for pipe) | -| `--in_place` | _False_ | Overwrite the input file | | `-w/--print-width` | _None_ → Prettier default | Line wrap width | | `--use_dockerized_prettier` | _False_ | Run Prettier inside helper container | | `--use_dockerized_markdown_toc` | _False_ | Refresh TOC via containerised `markdown-toc` | @@ -296,4 +313,143 @@ llm_transform.py -p list -i - -o - :'<,'>!llm_transform.py -p summarize -i - -o - ``` +--- + +## 6 · Pandoc Wrapper — `run_pandoc.py` + +### What the script does + +* Reads **Markdown** from _stdin_ or `--input` file. +* Dispatches to a named **action** (currently only `convert_md_to_latex`). +* Pushes the Pandoc output to _stdout_ or the `--output` file. + +### Quick‑use commands + +| Goal | Command | +| ------------------------------------- | -------------------------------------------- | +| Convert a Markdown file to LaTeX | `run_pandoc.py -i note.md -o note.tex` | +| Same, but stream from STDIN to STDOUT | `cat note.md \| run_pandoc.py -i - -o -` | +| Inside **Vim** (visual range) | `:'<,'>!run_pandoc.py -i - -o - -v CRITICAL` | + +> **Tip :** pass `-v CRITICAL` to silence helper logging when piping into +> editors. + +### Flags + +| Flag | Default | Meaning | +| ------------------ | --------------------- | --------------------------------------------------------- | +| `-i / --input` | `-` | Source file or `-` for STDIN. | +| `-o / --output` | `-` | Destination file or `-` for STDOUT. | +| `--action` | `convert_md_to_latex` | Transformation to apply. Future‑proofed for more actions. | +| `-v / --log_level` | `INFO` | Standard helper‑library verbosity. | + +--- + +## 7 · Automate notes transformations — `transform_notes.py` + +### What it does + +* Accepts a **text/Markdown** stream (file or `-`). +* Applies a named **action** (`-a/--action`). +* Writes the result to the given output (in‑place, file, or `-`). + +### Supported actions + +| Run `-a list` to print. | Tag | Effect | Typical Vim one‑liner | | +| -------------------------------------------------------------- | -------------------------------------------------- | ------------------------ | --------------------- | --- | +| -------------------------------------------------------------- | +| -------------------------------------------------- | | `toc` | Generate a bullet | +| TOC (top‑level by default) | `:!transform_notes.py -a toc -i % -l 1` | | +| `format_headers` | Re‑flow / indent headers (≤ `--max_lev`) | +| `:%!transform_notes.py -a format -i - --max_lev 3` | | `increase_headers_level` | +| Bump all headers down one level | `:%!transform_notes.py -a increase -i -` | | +| `md_list_to_latex` | Convert a Markdown list to LaTeX `\begin{itemize}` | +| `:%!transform_notes.py -a md_list_to_latex -i -` | | `md_*` family | Formatting | +| clean‑ups (bold bullets, colourise bold text, etc.) | see `-a list` | + +### Examples + +```bash +# Re‑flow & clean a file in place +transform_notes.py -a md_format -i notes/lecture.txt --in_place + +# Generate a 2‑level TOC to STDOUT +transform_notes.py -a toc -i notes/lecture.md -o - -l 2 +``` + +### Flags + +| Flag | Default | Purpose | +| ---------------- | ------------ | -------------------------------------------------- | +| `-a / --action` | _(required)_ | Choose the transformation. | +| `-l / --max_lev` | `5` | Header depth for `format_headers`. | +| `-i / --input` | `-` | File path or `-` (STDIN). | +| `-o / --output` | `-` | File path or `-` (STDOUT). | +| `--in_place` | _False_ | Overwrite input file instead of writing elsewhere. | + +--- + +## 8 · Scrape headers from a markdown — `extract_headers_from_markdown.py` + +### Goal + +Turn a Markdown document into either: + +* a **plain list** of headers, +* a **nested header map**, or +* a \*_Vim_ quick‑fix\*\* (`cfile`) that lets you jump between sections with + `:cnext`. + +### Examples + +```bash +# Human‑readable map (levels 1‑3) to STDOUT +extract_headers_from_markdown.py -i README.md -o - --mode list --max-level 3 + +# Build a quick‑fix file and open Vim on it +extract_headers_from_markdown.py -i README.md -o headers.cfile --mode cfile +vim -c "cfile headers.cfile" +``` + +### Flags + +| Flag | Default | Meaning | +| ------------- | ------- | ------------------------------ | +| `--mode` | `list` | `list`, `headers`, or `cfile`. | +| `--max-level` | `3` | Maximum `#` depth to parse. | + +--- + +## 9 · TikZ to Bitmap — `dockerized_tikz_to_bitmap.py` + +### Examples + +```bash +# Plain 300 DPI conversion +./dockerized_tikz_to_bitmap.py -i figure.tikz -o figure.png + +# Custom ImageMagick options (e.g. 600 DPI) +./dockerized_tikz_to_bitmap.py -i fig.tikz -o fig.png -- -density 600 -quality 90 +``` + +_Any extra tokens after `--` are passed verbatim to `convert`._ + +--- + +## 10 · MacOS screenshot helper — `save_screenshot.py` + +### What it does + +1. Prompts you to select a screen region (`⌘ + Ctrl + 4`). +2. Saves it as `screenshot.YYYY‑MM‑DD_HH‑MM‑SS.png` (or your chosen name). +3. Prints and copies the Markdown embed `![](path/to/file.png)`. + +### Flags + +| Flag | Purpose | +| --------------------- | ---------------------------------------- | +| `--dst_dir DIR` | Target directory (e.g. `notes/figures`). | +| `--filename NAME.png` | Override default timestamped name. | +| `--override` | Allow clobbering an existing file. | + --- \ No newline at end of file From f542f53e93f7658a5406e4f46272e1c73d191cb3 Mon Sep 17 00:00:00 2001 From: Indrayudd Roy Chowdhury Date: Sat, 24 May 2025 23:40:05 -0400 Subject: [PATCH 05/26] HelpersTask741: Remove all.run_pandoc.how_to_guide.md --- .../all.run_pandoc.how_to_guide.md | 52 ------------------- 1 file changed, 52 deletions(-) delete mode 100644 docs/tools/documentation_toolchain/all.run_pandoc.how_to_guide.md diff --git a/docs/tools/documentation_toolchain/all.run_pandoc.how_to_guide.md b/docs/tools/documentation_toolchain/all.run_pandoc.how_to_guide.md deleted file mode 100644 index 4460987c9..000000000 --- a/docs/tools/documentation_toolchain/all.run_pandoc.how_to_guide.md +++ /dev/null @@ -1,52 +0,0 @@ - - -- [run_pandoc.py – How‑to Guide](#run_pandocpy-%E2%80%93-how%E2%80%91to-guide) - * [What the script does](#what-the-script-does) - * [Quick‑use commands](#quick%E2%80%91use-commands) - * [Flags](#flags) - - - -# run_pandoc.py – How‑to Guide - -A **minimal wrapper** around Pandoc that lets you transform files between types. - ---- - -## What the script does - -```text -stdin / file ──► run_pandoc.py ──► stdout / file - │ - └─► helpers.hlatex.convert_pandoc_md_to_latex() -``` - -* Reads **Markdown** from _stdin_ or `--input` file. -* Dispatches to a named **action** (currently only `convert_md_to_latex`). -* Pushes the Pandoc output to _stdout_ or the `--output` file. - ---- - -## Quick‑use commands - -| Goal | Command | -| ------------------------------------- | -------------------------------------------- | -| Convert a Markdown file to LaTeX | `run_pandoc.py -i note.md -o note.tex` | -| Same, but stream from STDIN to STDOUT | `cat note.md \| run_pandoc.py -i - -o -` | -| Inside **Vim** (visual range) | `:'<,'>!run_pandoc.py -i - -o - -v CRITICAL` | - -> **Tip :** pass `-v CRITICAL` to silence helper logging when piping into -> editors. - ---- - -## Flags - -| Flag | Default | Meaning | -| ------------------ | --------------------- | --------------------------------------------------------- | -| `-i / --input` | `-` | Source file or `-` for STDIN. | -| `-o / --output` | `-` | Destination file or `-` for STDOUT. | -| `--action` | `convert_md_to_latex` | Transformation to apply. Future‑proofed for more actions. | -| `-v / --log_level` | `INFO` | Standard helper‑library verbosity. | - ---- \ No newline at end of file From 8c1808d47366cf7e503b2f05e1e5ea91fd8a71c9 Mon Sep 17 00:00:00 2001 From: Indrayudd Roy Chowdhury Date: Sat, 24 May 2025 23:47:48 -0400 Subject: [PATCH 06/26] HelpersTask741: Add examples --- .../all.notes_toolchain.how_to_guide.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 95aae4c08..7af1c0f9e 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -375,6 +375,9 @@ transform_notes.py -a md_format -i notes/lecture.txt --in_place # Generate a 2‑level TOC to STDOUT transform_notes.py -a toc -i notes/lecture.md -o - -l 2 + +# Tidy ChatGPT‑generated Markdown (visual mode in Vim) +:'<,'>!transform_notes.py -i - -o - -a md_fix_chatgpt_output ``` ### Flags From 2e78a8b04dd5f1255bb5f3c0fcf5c570a1174f52 Mon Sep 17 00:00:00 2001 From: Indrayudd Roy Chowdhury Date: Sun, 25 May 2025 22:40:24 -0400 Subject: [PATCH 07/26] HelpersTask741: Add Aayush's part to the how_to_guide as well --- .../all.notes_toolchain.how_to_guide.md | 249 +++++++++++++++++- 1 file changed, 246 insertions(+), 3 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 7af1c0f9e..08dbbf91c 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -45,8 +45,33 @@ + [Flags](#flags-5) * [9 · TikZ to Bitmap — `dockerized_tikz_to_bitmap.py`](#9-%C2%B7-tikz-to-bitmap--dockerized_tikz_to_bitmappy) + [Examples](#examples-2) - * [10 · MacOS screenshot helper — `save_screenshot.py`](#10-%C2%B7-macos-screenshot-helper--save_screenshotpy) + * [10 · Graphviz Renderer — `dockerized_graphviz.py`](#10-%C2%B7-graphviz-renderer--dockerized_graphvizpy) + [What it does](#what-it-does-2) + + [Most‑used flags](#most%E2%80%91used-flags-1) + + [Quick‑start recipes](#quick%E2%80%91start-recipes-1) + + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet-1) + * [11 · LaTeX Renderer — `dockerized_latex.py`](#11-%C2%B7-latex-renderer--dockerized_latexpy) + + [What it does](#what-it-does-3) + + [Most‑used flags](#most%E2%80%91used-flags-2) + + [Quick‑start recipes](#quick%E2%80%91start-recipes-2) + + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet-2) + * [12 · Mermaid Renderer — `dockerized_mermaid.py`](#12-%C2%B7-mermaid-renderer--dockerized_mermaidpy) + + [What it does](#what-it-does-4) + + [Most‑used flags](#most%E2%80%91used-flags-3) + + [Quick‑start recipes](#quick%E2%80%91start-recipes-3) + + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet-3) + * [13 · Pandoc Renderer — `dockerized_pandoc.py`](#13-%C2%B7-pandoc-renderer--dockerized_pandocpy) + + [What it does](#what-it-does-5) + + [Most‑used flags](#most%E2%80%91used-flags-4) + + [Quick‑start recipes](#quick%E2%80%91start-recipes-4) + + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet-4) + * [14 · Prettier Formatter — `dockerized_prettier.py`](#14-%C2%B7-prettier-formatter--dockerized_prettierpy) + + [What it does](#what-it-does-6) + + [Most‑used flags](#most%E2%80%91used-flags-5) + + [Quick‑start recipes](#quick%E2%80%91start-recipes-5) + + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet-5) + * [15 · MacOS screenshot helper — `save_screenshot.py`](#15-%C2%B7-macos-screenshot-helper--save_screenshotpy) + + [What it does](#what-it-does-7) + [Flags](#flags-6) @@ -376,7 +401,7 @@ transform_notes.py -a md_format -i notes/lecture.txt --in_place # Generate a 2‑level TOC to STDOUT transform_notes.py -a toc -i notes/lecture.md -o - -l 2 -# Tidy ChatGPT‑generated Markdown (visual mode in Vim) +# Tidy ChatGPT‑generated Markdown (visual mode in Vim) :'<,'>!transform_notes.py -i - -o - -a md_fix_chatgpt_output ``` @@ -439,7 +464,225 @@ _Any extra tokens after `--` are passed verbatim to `convert`._ --- -## 10 · MacOS screenshot helper — `save_screenshot.py` +## 10 · Graphviz Renderer — `dockerized_graphviz.py` + +### What it does + +Converts a Graphviz `.dot` file into a `.png` image using a Dockerized +container. + +> ```bash +> graphviz_wrapper.py --input input.dot --output output.png +> ``` + +This script serves as a thin wrapper around Dockerized Graphviz for consistent +rendering across systems. + +### Most‑used flags + +- `--input`: path to the `.dot` file +- `--output`: destination `.png` image file +- `--dockerized_force_rebuild`: rebuild the container from scratch +- `--dockerized_use_sudo`: use `sudo` for Docker commands + +### Quick‑start recipes + +| Goal | Command | +| --------------------- | ------------------------------------------------------------------------------ | +| Convert DOT to PNG | `graphviz_wrapper.py -i diagram.dot -o diagram.png` | +| Rebuild Docker image | `graphviz_wrapper.py -i diagram.dot -o diagram.png --dockerized_force_rebuild` | +| Use `sudo` for Docker | `graphviz_wrapper.py -i diagram.dot -o diagram.png --dockerized_use_sudo` | + +### CLI flags cheat‑sheet + +| Flag | Purpose | Notes | +| ---------------------------- | ---------------------------- | ------------- | +| `-i / --input` | Path to input `.dot` file | **required** | +| `-o / --output` | Output path for `.png` image | **required** | +| `--dockerized_force_rebuild` | Force Docker image rebuild | Optional | +| `--dockerized_use_sudo` | Run Docker with `sudo` | Optional | +| `-v / --verbosity` | Logging verbosity | Default: INFO | + +--- + +## 11 · LaTeX Renderer — `dockerized_latex.py` + +### What it does + +Compiles a LaTeX `.tex` file into a PDF using `pdflatex` inside a Docker +container. +Automatically rebuilds the Docker image if needed. + +> ```bash +> latex_wrapper.py --input doc.tex --output doc.pdf +> ``` + +Supports optional rerun of LaTeX for proper references or table of contents +generation. + +### Most‑used flags + +- `--input`: LaTeX source file to compile +- `--output`: Output PDF path +- `--run_latex_again`: Compile the LaTeX file twice +- `--dockerized_force_rebuild`: Force container rebuild +- `--dockerized_use_sudo`: Run Docker with `sudo` + +### Quick‑start recipes + +| Goal | Command | +| ------------------------ | ------------------------------------------------------------------------- | +| Compile `.tex` to `.pdf` | `latex_wrapper.py -i report.tex -o report.pdf` | +| Rebuild Docker image | `latex_wrapper.py -i report.tex -o report.pdf --dockerized_force_rebuild` | +| Use `sudo` for Docker | `latex_wrapper.py -i report.tex -o report.pdf --dockerized_use_sudo` | +| Run LaTeX twice | `latex_wrapper.py -i paper.tex -o paper.pdf --run_latex_again` | + +### CLI flags cheat‑sheet + +| Flag | Purpose | Notes | +| ---------------------------- | -------------------------- | ----------------------------- | +| `-i / --input` | Path to input `.tex` file | **required** | +| `-o / --output` | Output PDF file path | **required** | +| `--run_latex_again` | Run LaTeX a second time | Optional, useful for TOC/refs | +| `--dockerized_force_rebuild` | Force Docker image rebuild | Optional | +| `--dockerized_use_sudo` | Run Docker with `sudo` | Optional | +| `-v / --verbosity` | Logging verbosity | Default: INFO | + +--- + +## 12 · Mermaid Renderer — `dockerized_mermaid.py` + +### What it does + +Renders Mermaid `.mmd` or `.md` diagrams into image files using a Dockerized +container. + +> ```bash +> mermaid_wrapper.py --input flowchart.mmd --output flowchart.png +> ``` + +Automatically sets output to match input name if `--output` is omitted. + +### Most‑used flags + +- `--input`: Source Mermaid file +- `--output`: Destination image file (optional) +- `--dockerized_force_rebuild`: Rebuild Docker image +- `--dockerized_use_sudo`: Use `sudo` for Docker + +### Quick‑start recipes + +| Goal | Command | +| ----------------------------- | ----------------------------------------------------------------------------- | +| Render Mermaid diagram | `mermaid_wrapper.py -i diagram.mmd -o diagram.png` | +| Use input as output (default) | `mermaid_wrapper.py -i diagram.mmd` | +| Rebuild container | `mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_force_rebuild` | +| Use `sudo` for Docker | `mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_use_sudo` | + +### CLI flags cheat‑sheet + +| Flag | Purpose | Notes | +| ---------------------------- | ---------------------------------- | -------------------------- | +| `-i / --input` | Path to input `.mmd` or `.md` file | **required** | +| `-o / --output` | Output image file | Defaults to input filename | +| `--dockerized_force_rebuild` | Force Docker image rebuild | Optional | +| `--dockerized_use_sudo` | Run Docker with `sudo` | Optional | +| `-v / --verbosity` | Logging verbosity | Default: INFO | + +--- + +## 13 · Pandoc Renderer — `dockerized_pandoc.py` + +### What it does + +Converts documents using `pandoc` inside a Docker container. +Supports output to Beamer slides, PDFs, and more with custom CLI flags. + +> ```bash +> pandoc_wrapper.py --input notes.md --output slides.pdf -- docker_args... +> ``` + +Internally builds a Docker container and passes the full `pandoc` command +string. + +### Most‑used flags + +- `--input`: source file (e.g., `.md`, `.txt`) +- `--output`: output file (e.g., `.pdf`, `.html`) +- `--container_type`: use `pandoc_only`, `pandoc_latex`, or `pandoc_texlive` +- `--dockerized_force_rebuild`: rebuild image from scratch +- `--dockerized_use_sudo`: run Docker with `sudo` + +### Quick‑start recipes + +| Goal | Command | +| ------------------------ | ---------------------------------------------------------------------------------------------------- | +| Convert Markdown to PDF | `pandoc_wrapper.py --input notes.md --output notes.pdf --container_type pandoc_latex` | +| Convert to Beamer slides | `pandoc_wrapper.py --input slides.md --output slides.pdf --container_type pandoc_latex -- -t beamer` | +| Rebuild Docker image | `pandoc_wrapper.py --input notes.md --output notes.pdf --dockerized_force_rebuild` | +| Run with sudo | `pandoc_wrapper.py --input notes.md --output notes.pdf --dockerized_use_sudo` | + +### CLI flags cheat‑sheet + +| Flag | Purpose | Notes | +| ---------------------------- | ------------------------------------------------------ | ---------------------- | +| `--input` | Input source file for Pandoc | **required** | +| `--output` | Output file path | Defaults to input name | +| `--data_dir` | Additional resource/data path | Optional | +| `--container_type` | Docker image type: `pandoc_only`, `pandoc_latex`, etc. | Default: `pandoc_only` | +| `--dockerized_force_rebuild` | Force rebuild of Docker image | Optional | +| `--dockerized_use_sudo` | Use `sudo` for Docker execution | Optional | +| `-v / --verbosity` | Logging level | Default: INFO | + +--- + +## 14 · Prettier Formatter — `dockerized_prettier.py` + +### What it does + +Formats text files (`.md`, `.txt`, `.tex`, etc.) using Prettier within a Docker +container. +Avoids environment-specific issues and ensures consistent formatting. + +> ```bash +> dockerized_prettier.py --parser markdown --write test.md +> ``` + +Supports full Prettier CLI flexibility via passthrough of additional options. + +### Most‑used flags + +- `--parser`: Prettier parser (e.g. `markdown`) +- `--write`: Apply formatting in-place +- `--tab-width`: Number of spaces per indentation level +- `--dockerized_force_rebuild`: Force rebuild of Docker container +- `--dockerized_use_sudo`: Use `sudo` for Docker commands + +### Quick‑start recipes + +| Goal | Command | +| --------------------------------- | -------------------------------------------------------------------------------------------- | +| Format a Markdown file | `dockerized_prettier.py --parser markdown --write test.md` | +| Use `sudo` for Docker execution | `dockerized_prettier.py --use_sudo --parser markdown --write test.md` | +| Rebuild the Docker image | `dockerized_prettier.py --dockerized_force_rebuild --parser markdown --write test.md` | +| Change indentation and wrap style | `dockerized_prettier.py --parser markdown --tab-width 4 --prose-wrap always --write test.md` | + +### CLI flags cheat‑sheet + +| Flag | Purpose | Notes | +| ---------------------------- | ----------------------------------------------------- | ------------------------------------- | +| `-i / --input` | Input file path | Required | +| `-o / --output` | Output file path | Optional (defaults to input) | +| `--parser` | Prettier parser type (e.g. `markdown`, `babel`, etc.) | Required via passthrough | +| `--write` | Format and overwrite input file | Common usage flag | +| `--tab-width` | Number of spaces per tab | Optional, defaults to Prettier config | +| `--dockerized_force_rebuild` | Force Docker image rebuild | Optional | +| `--dockerized_use_sudo` | Use `sudo` for Docker commands | Optional | +| `-v / --verbosity` | Logging level | Default: INFO | + +--- + +## 15 · MacOS screenshot helper — `save_screenshot.py` ### What it does From 43060551af9f32f36938293afdb8098ee743ebfd Mon Sep 17 00:00:00 2001 From: Indrayudd Roy Chowdhury Date: Sun, 25 May 2025 22:55:14 -0400 Subject: [PATCH 08/26] HelpersTask741: Change headers to be compatible with Linter toc --- .../all.notes_toolchain.how_to_guide.md | 148 +++++++++--------- 1 file changed, 74 insertions(+), 74 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 08dbbf91c..6a72e8ac0 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -1,16 +1,16 @@ - [Notes Documentation Toolchain](#notes-documentation-toolchain) - * [1 · Generate slides & PDFs — `notes_to_pdf.py`](#1-%C2%B7-generate-slides--pdfs--notes_to_pdfpy) + * [1. Generate slides & PDFs — `notes_to_pdf.py`](#1-generate-slides--pdfs--notes_to_pdfpy) + [What it does](#what-it-does) - + [Most‑used flags](#most%E2%80%91used-flags) - + [Quick‑start recipes](#quick%E2%80%91start-recipes) - + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet) + + [Most used flags](#most-used-flags) + + [Quickstart recipes](#quickstart-recipes) + + [CLI flags cheatsheet](#cli-flags-cheatsheet) + [Worked examples](#worked-examples) - [Slides with navigation breadcrumbs](#slides-with-navigation-breadcrumbs) - [Focus on a subsection](#focus-on-a-subsection) - - [Plain PDF article (no slides)](#plain-pdf-article-no-slides) - * [2 · Auto‑render figures — `render_images.py`](#2-%C2%B7-auto%E2%80%91render-figures--render_imagespy) + - [Plain PDF article](#plain-pdf-article) + * [2. Auto render figures — `render_images.py`](#2-auto-render-figures--render_imagespy) + [Supported File types and Code blocks](#supported-file-types-and-code-blocks) + [Quick Start Recipes](#quick-start-recipes) - [Render to a new file](#render-to-a-new-file) @@ -18,59 +18,59 @@ - [HTML preview of already‑rendered images](#html-preview-of-already%E2%80%91rendered-images) - [Dry‑run (test parsing / comments only)](#dry%E2%80%91run-test-parsing--comments-only) + [Flags](#flags) - * [3 · Lint & prettify — `lint_notes.py`](#3-%C2%B7-lint--prettify--lint_notespy) - + [Quickstart recipes](#quickstart-recipes) - - [Prettify with Dockerised Prettier + TOC rebuild](#prettify-with-dockerised-prettier--toc-rebuild) - - [Custom print width & selective actions](#custom-print-width--selective-actions) + * [3. Lint & prettify — `lint_notes.py`](#3-lint--prettify--lint_notespy) + + [Quickstart recipes](#quickstart-recipes-1) + - [Prettify with Dockerised Prettier and TOC rebuild](#prettify-with-dockerised-prettier-and-toc-rebuild) + - [Custom print width and selective actions](#custom-print-width-and-selective-actions) + [Flags](#flags-1) - * [4 · Notebook image scraping — `extract_notebook_images.py`](#4-%C2%B7-notebook-image-scraping--extract_notebook_imagespy) + * [4. Notebook image scraping — `extract_notebook_images.py`](#4-notebook-image-scraping--extract_notebook_imagespy) + [Flag Options](#flag-options) - * [5 · LLM‑powered transforms — `llm_transform.py`](#5-%C2%B7-llm%E2%80%91powered-transforms--llm_transformpy) + * [5. LLM powered transforms — `llm_transform.py`](#5-llm-powered-transforms--llm_transformpy) + [Minimum viable command](#minimum-viable-command) + [Finding available prompts](#finding-available-prompts) + [Flags](#flags-2) + [Example recipes](#example-recipes) - * [6 · Pandoc Wrapper — `run_pandoc.py`](#6-%C2%B7-pandoc-wrapper--run_pandocpy) + * [6. Pandoc Wrapper — `run_pandoc.py`](#6-pandoc-wrapper--run_pandocpy) + [What the script does](#what-the-script-does) - + [Quick‑use commands](#quick%E2%80%91use-commands) + + [Quickstart commands](#quickstart-commands) + [Flags](#flags-3) - * [7 · Automate notes transformations — `transform_notes.py`](#7-%C2%B7-automate-notes-transformations--transform_notespy) + * [7. Automate notes transformations — `transform_notes.py`](#7-automate-notes-transformations--transform_notespy) + [What it does](#what-it-does-1) + [Supported actions](#supported-actions) + [Examples](#examples) + [Flags](#flags-4) - * [8 · Scrape headers from a markdown — `extract_headers_from_markdown.py`](#8-%C2%B7-scrape-headers-from-a-markdown--extract_headers_from_markdownpy) + * [8. Scrape headers from a markdown — `extract_headers_from_markdown.py`](#8-scrape-headers-from-a-markdown--extract_headers_from_markdownpy) + [Goal](#goal) + [Examples](#examples-1) + [Flags](#flags-5) - * [9 · TikZ to Bitmap — `dockerized_tikz_to_bitmap.py`](#9-%C2%B7-tikz-to-bitmap--dockerized_tikz_to_bitmappy) + * [9. TikZ to Bitmap — `dockerized_tikz_to_bitmap.py`](#9-tikz-to-bitmap--dockerized_tikz_to_bitmappy) + [Examples](#examples-2) - * [10 · Graphviz Renderer — `dockerized_graphviz.py`](#10-%C2%B7-graphviz-renderer--dockerized_graphvizpy) + * [10. Graphviz Renderer — `dockerized_graphviz.py`](#10-graphviz-renderer--dockerized_graphvizpy) + [What it does](#what-it-does-2) - + [Most‑used flags](#most%E2%80%91used-flags-1) - + [Quick‑start recipes](#quick%E2%80%91start-recipes-1) - + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet-1) - * [11 · LaTeX Renderer — `dockerized_latex.py`](#11-%C2%B7-latex-renderer--dockerized_latexpy) + + [Most used flags](#most-used-flags-1) + + [Quickstart recipes](#quickstart-recipes-2) + + [CLI flags cheatsheet](#cli-flags-cheatsheet-1) + * [11. LaTeX Renderer — `dockerized_latex.py`](#11-latex-renderer--dockerized_latexpy) + [What it does](#what-it-does-3) - + [Most‑used flags](#most%E2%80%91used-flags-2) - + [Quick‑start recipes](#quick%E2%80%91start-recipes-2) - + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet-2) - * [12 · Mermaid Renderer — `dockerized_mermaid.py`](#12-%C2%B7-mermaid-renderer--dockerized_mermaidpy) + + [Most used flags](#most-used-flags-2) + + [Quickstart recipes](#quickstart-recipes-3) + + [CLI flags cheatsheet](#cli-flags-cheatsheet-2) + * [12. Mermaid Renderer — `dockerized_mermaid.py`](#12-mermaid-renderer--dockerized_mermaidpy) + [What it does](#what-it-does-4) - + [Most‑used flags](#most%E2%80%91used-flags-3) - + [Quick‑start recipes](#quick%E2%80%91start-recipes-3) - + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet-3) - * [13 · Pandoc Renderer — `dockerized_pandoc.py`](#13-%C2%B7-pandoc-renderer--dockerized_pandocpy) + + [Most used flags](#most-used-flags-3) + + [Quickstart recipes](#quickstart-recipes-4) + + [CLI flags cheatsheet](#cli-flags-cheatsheet-3) + * [13. Pandoc Renderer — `dockerized_pandoc.py`](#13-pandoc-renderer--dockerized_pandocpy) + [What it does](#what-it-does-5) - + [Most‑used flags](#most%E2%80%91used-flags-4) - + [Quick‑start recipes](#quick%E2%80%91start-recipes-4) - + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet-4) - * [14 · Prettier Formatter — `dockerized_prettier.py`](#14-%C2%B7-prettier-formatter--dockerized_prettierpy) + + [Most used flags](#most-used-flags-4) + + [Quickstart recipes](#quickstart-recipes-5) + + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet) + * [14. Prettier Formatter — `dockerized_prettier.py`](#14-prettier-formatter--dockerized_prettierpy) + [What it does](#what-it-does-6) - + [Most‑used flags](#most%E2%80%91used-flags-5) - + [Quick‑start recipes](#quick%E2%80%91start-recipes-5) - + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet-5) - * [15 · MacOS screenshot helper — `save_screenshot.py`](#15-%C2%B7-macos-screenshot-helper--save_screenshotpy) + + [Most used flags](#most-used-flags-5) + + [Quickstart recipes](#quickstart-recipes-6) + + [CLI flags cheatsheet](#cli-flags-cheatsheet-4) + * [15. MacOS screenshot helper — `save_screenshot.py`](#15-macos-screenshot-helper--save_screenshotpy) + [What it does](#what-it-does-7) + [Flags](#flags-6) @@ -83,7 +83,7 @@ notes into polished PDFs, slide decks, and more. --- -## 1 · Generate slides & PDFs — `notes_to_pdf.py` +## 1. Generate slides & PDFs — `notes_to_pdf.py` ### What it does @@ -94,13 +94,13 @@ single command. > notes_to_pdf.py --input --output --type [pdf|html|slides] > ``` -### Most‑used flags +### Most used flags * `--type {pdf|html|slides}` * `--toc_type {none|pandoc_native|navigation}` * `--debug_on_error`, `--skip_action ...`, `--filter_by_lines A:B` -### Quick‑start recipes +### Quickstart recipes | Goal | Command | | ----------------------------------- | ------------------------------------------------------------------ | @@ -112,7 +112,7 @@ single command. _TIP  Run with `--preview_actions` to print the exact steps without executing them._ -### CLI flags cheat‑sheet +### CLI flags cheatsheet | Flag | Purpose | Notes | | -------------------------------------------- | ------------------------------------------------------------------------- | ----------------------------------------------------- | @@ -157,7 +157,7 @@ notes_to_pdf.py \ Compiles only from line 362 to EOF—fast iteration when debugging slides. -#### Plain PDF article (no slides) +#### Plain PDF article ```bash notes_to_pdf.py -i book_notes.txt -o book_notes.pdf --type pdf @@ -165,7 +165,7 @@ notes_to_pdf.py -i book_notes.txt -o book_notes.pdf --type pdf --- -## 2 · Auto‑render figures — `render_images.py` +## 2. Auto render figures — `render_images.py` Detects fenced code blocks (PlantUML, Mermaid, TikZ, Graphviz, ...), renders them into images and swaps the block with `![](img)` markup. @@ -223,7 +223,7 @@ render_images.py -i lesson.md -o /tmp/out.md --dry_run --- -## 3 · Lint & prettify — `lint_notes.py` +## 3. Lint & prettify — `lint_notes.py` A tidying‑up tool for Markdown/LaTeX notes: normalise G‑Doc artifacts, run Prettier, fix bullet/heading quirks, refresh the Table of Contents – all from a @@ -231,7 +231,7 @@ single command or straight from vim. ### Quickstart recipes -#### Prettify with Dockerised Prettier + TOC rebuild +#### Prettify with Dockerised Prettier and TOC rebuild ```bash lint_notes.py -i Lesson10.md \ @@ -239,7 +239,7 @@ lint_notes.py -i Lesson10.md \ --use_dockerized_markdown_toc ``` -#### Custom print width & selective actions +#### Custom print width and selective actions ```bash lint_notes.py -i draft.txt -o tidy.txt -w 100 \ @@ -260,7 +260,7 @@ lint_notes.py -i draft.txt -o tidy.txt -w 100 \ --- -## 4 · Notebook image scraping — `extract_notebook_images.py` +## 4. Notebook image scraping — `extract_notebook_images.py` Spins up a docker container and dumps every `png/svg` output cell into a folder. You can then publish or reuse the static plots/diagrams already rendered in a @@ -286,7 +286,7 @@ extract_notebook_images.py \ --- -## 5 · LLM‑powered transforms — `llm_transform.py` +## 5. LLM powered transforms — `llm_transform.py` Apply a GPT‑style transformation (rewrite, summarise, critique code, convert to slides, etc.) to any text file _without_ leaving the terminal / editor. @@ -340,7 +340,7 @@ llm_transform.py -p list -i - -o - --- -## 6 · Pandoc Wrapper — `run_pandoc.py` +## 6. Pandoc Wrapper — `run_pandoc.py` ### What the script does @@ -348,7 +348,7 @@ llm_transform.py -p list -i - -o - * Dispatches to a named **action** (currently only `convert_md_to_latex`). * Pushes the Pandoc output to _stdout_ or the `--output` file. -### Quick‑use commands +### Quickstart commands | Goal | Command | | ------------------------------------- | -------------------------------------------- | @@ -370,7 +370,7 @@ llm_transform.py -p list -i - -o - --- -## 7 · Automate notes transformations — `transform_notes.py` +## 7. Automate notes transformations — `transform_notes.py` ### What it does @@ -417,7 +417,7 @@ transform_notes.py -a toc -i notes/lecture.md -o - -l 2 --- -## 8 · Scrape headers from a markdown — `extract_headers_from_markdown.py` +## 8. Scrape headers from a markdown — `extract_headers_from_markdown.py` ### Goal @@ -448,7 +448,7 @@ vim -c "cfile headers.cfile" --- -## 9 · TikZ to Bitmap — `dockerized_tikz_to_bitmap.py` +## 9. TikZ to Bitmap — `dockerized_tikz_to_bitmap.py` ### Examples @@ -464,7 +464,7 @@ _Any extra tokens after `--` are passed verbatim to `convert`._ --- -## 10 · Graphviz Renderer — `dockerized_graphviz.py` +## 10. Graphviz Renderer — `dockerized_graphviz.py` ### What it does @@ -478,14 +478,14 @@ container. This script serves as a thin wrapper around Dockerized Graphviz for consistent rendering across systems. -### Most‑used flags +### Most used flags - `--input`: path to the `.dot` file - `--output`: destination `.png` image file - `--dockerized_force_rebuild`: rebuild the container from scratch - `--dockerized_use_sudo`: use `sudo` for Docker commands -### Quick‑start recipes +### Quickstart recipes | Goal | Command | | --------------------- | ------------------------------------------------------------------------------ | @@ -493,7 +493,7 @@ rendering across systems. | Rebuild Docker image | `graphviz_wrapper.py -i diagram.dot -o diagram.png --dockerized_force_rebuild` | | Use `sudo` for Docker | `graphviz_wrapper.py -i diagram.dot -o diagram.png --dockerized_use_sudo` | -### CLI flags cheat‑sheet +### CLI flags cheatsheet | Flag | Purpose | Notes | | ---------------------------- | ---------------------------- | ------------- | @@ -505,7 +505,7 @@ rendering across systems. --- -## 11 · LaTeX Renderer — `dockerized_latex.py` +## 11. LaTeX Renderer — `dockerized_latex.py` ### What it does @@ -520,7 +520,7 @@ Automatically rebuilds the Docker image if needed. Supports optional rerun of LaTeX for proper references or table of contents generation. -### Most‑used flags +### Most used flags - `--input`: LaTeX source file to compile - `--output`: Output PDF path @@ -528,7 +528,7 @@ generation. - `--dockerized_force_rebuild`: Force container rebuild - `--dockerized_use_sudo`: Run Docker with `sudo` -### Quick‑start recipes +### Quickstart recipes | Goal | Command | | ------------------------ | ------------------------------------------------------------------------- | @@ -537,7 +537,7 @@ generation. | Use `sudo` for Docker | `latex_wrapper.py -i report.tex -o report.pdf --dockerized_use_sudo` | | Run LaTeX twice | `latex_wrapper.py -i paper.tex -o paper.pdf --run_latex_again` | -### CLI flags cheat‑sheet +### CLI flags cheatsheet | Flag | Purpose | Notes | | ---------------------------- | -------------------------- | ----------------------------- | @@ -550,7 +550,7 @@ generation. --- -## 12 · Mermaid Renderer — `dockerized_mermaid.py` +## 12. Mermaid Renderer — `dockerized_mermaid.py` ### What it does @@ -563,14 +563,14 @@ container. Automatically sets output to match input name if `--output` is omitted. -### Most‑used flags +### Most used flags - `--input`: Source Mermaid file - `--output`: Destination image file (optional) - `--dockerized_force_rebuild`: Rebuild Docker image - `--dockerized_use_sudo`: Use `sudo` for Docker -### Quick‑start recipes +### Quickstart recipes | Goal | Command | | ----------------------------- | ----------------------------------------------------------------------------- | @@ -579,7 +579,7 @@ Automatically sets output to match input name if `--output` is omitted. | Rebuild container | `mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_force_rebuild` | | Use `sudo` for Docker | `mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_use_sudo` | -### CLI flags cheat‑sheet +### CLI flags cheatsheet | Flag | Purpose | Notes | | ---------------------------- | ---------------------------------- | -------------------------- | @@ -591,7 +591,7 @@ Automatically sets output to match input name if `--output` is omitted. --- -## 13 · Pandoc Renderer — `dockerized_pandoc.py` +## 13. Pandoc Renderer — `dockerized_pandoc.py` ### What it does @@ -605,7 +605,7 @@ Supports output to Beamer slides, PDFs, and more with custom CLI flags. Internally builds a Docker container and passes the full `pandoc` command string. -### Most‑used flags +### Most used flags - `--input`: source file (e.g., `.md`, `.txt`) - `--output`: output file (e.g., `.pdf`, `.html`) @@ -613,7 +613,7 @@ string. - `--dockerized_force_rebuild`: rebuild image from scratch - `--dockerized_use_sudo`: run Docker with `sudo` -### Quick‑start recipes +### Quickstart recipes | Goal | Command | | ------------------------ | ---------------------------------------------------------------------------------------------------- | @@ -636,7 +636,7 @@ string. --- -## 14 · Prettier Formatter — `dockerized_prettier.py` +## 14. Prettier Formatter — `dockerized_prettier.py` ### What it does @@ -650,7 +650,7 @@ Avoids environment-specific issues and ensures consistent formatting. Supports full Prettier CLI flexibility via passthrough of additional options. -### Most‑used flags +### Most used flags - `--parser`: Prettier parser (e.g. `markdown`) - `--write`: Apply formatting in-place @@ -658,7 +658,7 @@ Supports full Prettier CLI flexibility via passthrough of additional options. - `--dockerized_force_rebuild`: Force rebuild of Docker container - `--dockerized_use_sudo`: Use `sudo` for Docker commands -### Quick‑start recipes +### Quickstart recipes | Goal | Command | | --------------------------------- | -------------------------------------------------------------------------------------------- | @@ -667,7 +667,7 @@ Supports full Prettier CLI flexibility via passthrough of additional options. | Rebuild the Docker image | `dockerized_prettier.py --dockerized_force_rebuild --parser markdown --write test.md` | | Change indentation and wrap style | `dockerized_prettier.py --parser markdown --tab-width 4 --prose-wrap always --write test.md` | -### CLI flags cheat‑sheet +### CLI flags cheatsheet | Flag | Purpose | Notes | | ---------------------------- | ----------------------------------------------------- | ------------------------------------- | @@ -682,7 +682,7 @@ Supports full Prettier CLI flexibility via passthrough of additional options. --- -## 15 · MacOS screenshot helper — `save_screenshot.py` +## 15. MacOS screenshot helper — `save_screenshot.py` ### What it does From 9dba518ea9d078b3ab6a7dbef856159d30d04bba Mon Sep 17 00:00:00 2001 From: Aayush Verma <44445921+aver81@users.noreply.github.com> Date: Mon, 26 May 2025 14:01:54 -0500 Subject: [PATCH 09/26] Fixing TOC navigation in file --- .../all.notes_toolchain.how_to_guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 6a72e8ac0..3e98bcd4b 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -1,7 +1,7 @@ - [Notes Documentation Toolchain](#notes-documentation-toolchain) - * [1. Generate slides & PDFs — `notes_to_pdf.py`](#1-generate-slides--pdfs--notes_to_pdfpy) + * [1. Generate slides and PDFs — `notes_to_pdf.py`](#1-generate-slides-and-pdfs--notes_to_pdfpy) + [What it does](#what-it-does) + [Most used flags](#most-used-flags) + [Quickstart recipes](#quickstart-recipes) @@ -698,4 +698,4 @@ Supports full Prettier CLI flexibility via passthrough of additional options. | `--filename NAME.png` | Override default timestamped name. | | `--override` | Allow clobbering an existing file. | ---- \ No newline at end of file +--- From 2d9a9ec933f70841c883aedde24373d4eeb52ae7 Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 26 May 2025 19:08:16 -0500 Subject: [PATCH 10/26] Fixing TOC navigation in .md file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 3e98bcd4b..6a72e8ac0 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -1,7 +1,7 @@ - [Notes Documentation Toolchain](#notes-documentation-toolchain) - * [1. Generate slides and PDFs — `notes_to_pdf.py`](#1-generate-slides-and-pdfs--notes_to_pdfpy) + * [1. Generate slides & PDFs — `notes_to_pdf.py`](#1-generate-slides--pdfs--notes_to_pdfpy) + [What it does](#what-it-does) + [Most used flags](#most-used-flags) + [Quickstart recipes](#quickstart-recipes) @@ -698,4 +698,4 @@ Supports full Prettier CLI flexibility via passthrough of additional options. | `--filename NAME.png` | Override default timestamped name. | | `--override` | Allow clobbering an existing file. | ---- +--- \ No newline at end of file From bf39185582f3e6afb1bde4972c939b7131fb012b Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 26 May 2025 19:13:30 -0500 Subject: [PATCH 11/26] Fixing TOC navigation in .md file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 6a72e8ac0..463f72570 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -1,7 +1,7 @@ - [Notes Documentation Toolchain](#notes-documentation-toolchain) - * [1. Generate slides & PDFs — `notes_to_pdf.py`](#1-generate-slides--pdfs--notes_to_pdfpy) + * [1. Generate Slides and PDFs — `notes_to_pdf.py`](#1-generate-slides-and-pdfs--notes_to_pdfpy) + [What it does](#what-it-does) + [Most used flags](#most-used-flags) + [Quickstart recipes](#quickstart-recipes) @@ -10,7 +10,7 @@ - [Slides with navigation breadcrumbs](#slides-with-navigation-breadcrumbs) - [Focus on a subsection](#focus-on-a-subsection) - [Plain PDF article](#plain-pdf-article) - * [2. Auto render figures — `render_images.py`](#2-auto-render-figures--render_imagespy) + * [2. Auto Render Figures — `render_images.py`](#2-auto-render-figures--render_imagespy) + [Supported File types and Code blocks](#supported-file-types-and-code-blocks) + [Quick Start Recipes](#quick-start-recipes) - [Render to a new file](#render-to-a-new-file) @@ -18,14 +18,14 @@ - [HTML preview of already‑rendered images](#html-preview-of-already%E2%80%91rendered-images) - [Dry‑run (test parsing / comments only)](#dry%E2%80%91run-test-parsing--comments-only) + [Flags](#flags) - * [3. Lint & prettify — `lint_notes.py`](#3-lint--prettify--lint_notespy) + * [3. Lint and Prettify — `lint_notes.py`](#3-lint-and-prettify--lint_notespy) + [Quickstart recipes](#quickstart-recipes-1) - [Prettify with Dockerised Prettier and TOC rebuild](#prettify-with-dockerised-prettier-and-toc-rebuild) - [Custom print width and selective actions](#custom-print-width-and-selective-actions) + [Flags](#flags-1) - * [4. Notebook image scraping — `extract_notebook_images.py`](#4-notebook-image-scraping--extract_notebook_imagespy) + * [4. Notebook Image Scraping — `extract_notebook_images.py`](#4-notebook-image-scraping--extract_notebook_imagespy) + [Flag Options](#flag-options) - * [5. LLM powered transforms — `llm_transform.py`](#5-llm-powered-transforms--llm_transformpy) + * [5. LLM Powered Transforms — `llm_transform.py`](#5-llm-powered-transforms--llm_transformpy) + [Minimum viable command](#minimum-viable-command) + [Finding available prompts](#finding-available-prompts) + [Flags](#flags-2) @@ -83,7 +83,7 @@ notes into polished PDFs, slide decks, and more. --- -## 1. Generate slides & PDFs — `notes_to_pdf.py` +## 1. Generate Slides and PDFs — `notes_to_pdf.py` ### What it does @@ -223,7 +223,7 @@ render_images.py -i lesson.md -o /tmp/out.md --dry_run --- -## 3. Lint & prettify — `lint_notes.py` +## 3. Lint and Prettify — `lint_notes.py` A tidying‑up tool for Markdown/LaTeX notes: normalise G‑Doc artifacts, run Prettier, fix bullet/heading quirks, refresh the Table of Contents – all from a @@ -260,7 +260,7 @@ lint_notes.py -i draft.txt -o tidy.txt -w 100 \ --- -## 4. Notebook image scraping — `extract_notebook_images.py` +## 4. Notebook Image Scraping — `extract_notebook_images.py` Spins up a docker container and dumps every `png/svg` output cell into a folder. You can then publish or reuse the static plots/diagrams already rendered in a @@ -286,7 +286,7 @@ extract_notebook_images.py \ --- -## 5. LLM powered transforms — `llm_transform.py` +## 5. LLM Powered Transforms — `llm_transform.py` Apply a GPT‑style transformation (rewrite, summarise, critique code, convert to slides, etc.) to any text file _without_ leaving the terminal / editor. From ba588565e289c4d1539810aa9eaf66c323ebe6f7 Mon Sep 17 00:00:00 2001 From: GP Saggese Date: Wed, 28 May 2025 18:15:27 -0400 Subject: [PATCH 12/26] Improve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_to_pdf.explanation.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md b/docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md index b174dd1ff..1c43b8456 100644 --- a/docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md +++ b/docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md @@ -17,9 +17,9 @@ This document walks through the **architecture** of the `notes_to_pdf.py` ## Goal -Convert a lightweight, annotation‑heavy **plain‑text notes file** (usually -`*.txt`) into a share‑ready document – **PDF**, **HTML**, or **Beamer slide -deck** – while: +Convert a lightweight, annotated with markdown **plain‑text notes file** (usually +`*.txt`) into a share‑ready document, as **PDF**, **HTML**, or **Beamer slide +deck**, while: * honouring rich Markdown features and custom shorthand, * inlining auto‑generated diagrams (PlantUML, Mermaid, TikZ, Graphviz, LaTeX @@ -64,24 +64,22 @@ Rel(notes2pdf, style, "Injects LaTeX Style (.sty)") ## Dependencies -###   `preprocess_notes.py` +### `preprocess_notes.py` * **Input →** raw notes. * **Output →** Pandoc‑ready Markdown. * Handles banner frames, question formatting, colour commands (`\red{}` → `\textcolor{red}{...}`), TOC injection. -###   `render_images.py` +### `render_images.py` * Docker‑wrapper around _PlantUML_, _Mermaid CLI_, _TikZ → Ghostscript_, _Graphviz_. * Produces deterministic filenames: `figs/..png` → diff‑friendly. -###   `latex_abbrevs.sty` +### `latex_abbrevs.sty` * House style for math: bold‑underlined vectors (`\vv{x}`), matrices, colour presets, 9‑level `enumitem` lists, symbol shorthands. * Copied automatically; you rarely touch this unless you need new macros. - ---- \ No newline at end of file From 9538c368f74d20d00d6b4d35240d7505372b1040 Mon Sep 17 00:00:00 2001 From: GP Saggese Date: Wed, 28 May 2025 18:57:43 -0400 Subject: [PATCH 13/26] Improve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_to_pdf.explanation.md | 85 ------------- .../all.notes_toolchain.explanation.md | 112 ++++++++++++++++++ .../all.notes_toolchain.how_to_guide.md | 52 ++++---- linters/amp_lint_md.py | 2 +- 4 files changed, 138 insertions(+), 113 deletions(-) delete mode 100644 docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md create mode 100644 docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md diff --git a/docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md b/docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md deleted file mode 100644 index 1c43b8456..000000000 --- a/docs/tools/documentation_toolchain/all.notes_to_pdf.explanation.md +++ /dev/null @@ -1,85 +0,0 @@ - - -- [`notes_to_pdf.py` - Flow Explanation](#notes_to_pdfpy---flow-explanation) - * [Goal](#goal) - * [Architecture diagram](#architecture-diagram) - * [Steps](#steps) - * [Dependencies](#dependencies) - + [`preprocess_notes.py`](#preprocess_notespy) - + [`render_images.py`](#render_imagespy) - + [`latex_abbrevs.sty`](#latex_abbrevssty) - - - -# `notes_to_pdf.py` - Flow Explanation - -This document walks through the **architecture** of the `notes_to_pdf.py` - -## Goal - -Convert a lightweight, annotated with markdown **plain‑text notes file** (usually -`*.txt`) into a share‑ready document, as **PDF**, **HTML**, or **Beamer slide -deck**, while: - -* honouring rich Markdown features and custom shorthand, -* inlining auto‑generated diagrams (PlantUML, Mermaid, TikZ, Graphviz, LaTeX - tables). - ---- - -## Architecture diagram - -```mermaid -%%{init: {'theme':'default'}}%% -C4Context - -System_Ext(user, "User", "Author preparing course notes") -System(doc, "Documentation Toolchain", "Python & LaTeX") - -System_Boundary(doc_boundary, "") { - Container(notes2pdf, "notes_to_pdf.py", "Python CLI", "Orchestrates conversion: clean → images → Pandoc → LaTeX") - Container(render, "render_images.py", "Python module", "Renders diagram blocks to images. Caches results.") - Container(preproc, "preprocess_notes.py", "Python module", "Cleans notes & expands macros (pre-Pandoc)") - Container(style, "latex_abbrevs.sty", "LaTeX style", "Provides LaTeX math & formatting shortcuts") -} - -Rel(user, notes2pdf, "Invokes via CLI") -Rel(notes2pdf, preproc, "Uses for Note Pre-processing") -Rel(notes2pdf, render, "Calls to Render Images") -Rel(notes2pdf, style, "Injects LaTeX Style (.sty)") -``` - ---- - -## Steps - -| Stage | Performed by | Key idea | -| ------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **1 – Clean‑up & augmentation** | `preprocess_notes.py` | ‑ Normalises headers, expands arrow shorthand (`->` → `\rightarrow`), deals with comments, inserts Pandoc YAML front‑matter & optional navigation slides. | -| **2 – Diagram extraction** | `render_images.py` | ‑ Scans code blocks (` ```plantuml``, etc.), renders via Docker containers, replaces the code with `![](figs/...)\` include and comments out original block.
‑ Uses a SHA‑256 cache so unchanged diagrams are skipped. | -| **3 – Orchestration** | `notes_to_pdf.py` | ‑ Calls **1** & **2**, then Pandoc, then (for PDF) LaTeX. Flags control each sub‑action so you can skip, debug or re‑run steps individually. | -| **4 – Document synthesis** | **Pandoc + LaTeX** | ‑ Pandoc converts Markdown → LaTeX (or HTML / Beamer).
‑ `latex_abbrevs.sty` is copied next to the generated `.tex` so vector/matrix macros (`\vv{}`, `\mat{}`), deep lists and colour helpers compile correctly. | - ---- - -## Dependencies - -### `preprocess_notes.py` - -* **Input →** raw notes. -* **Output →** Pandoc‑ready Markdown. -* Handles banner frames, question formatting, colour commands (`\red{}` → - `\textcolor{red}{...}`), TOC injection. - -### `render_images.py` - -* Docker‑wrapper around _PlantUML_, _Mermaid CLI_, _TikZ → Ghostscript_, - _Graphviz_. -* Produces deterministic filenames: `figs/..png` → - diff‑friendly. - -### `latex_abbrevs.sty` - -* House style for math: bold‑underlined vectors (`\vv{x}`), matrices, colour - presets, 9‑level `enumitem` lists, symbol shorthands. -* Copied automatically; you rarely touch this unless you need new macros. diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md b/docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md new file mode 100644 index 000000000..14ffa9886 --- /dev/null +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md @@ -0,0 +1,112 @@ + + +- [`notes_to_pdf.py` - Flow Explanation](#notes_to_pdfpy---flow-explanation) + * [Goal](#goal) + * [Architecture diagram](#architecture-diagram) + * [Steps](#steps) + * [Dependencies](#dependencies) + + [`preprocess_notes.py`](#preprocess_notespy) + + [`render_images.py`](#render_imagespy) + + [`latex_abbrevs.sty`](#latex_abbrevssty) + + + +# Flow Explanation + +- This document walks through the architecture of the `notes_to_pdf.py` script + +## Goal + +- Convert a lightweight, annotated plain-text notes file (usually `*.txt`) into + a share-ready document + - Formats include PDF, HTML, or Beamer slide deck +- Ensure the following: + - Honor rich Markdown features and custom shorthand + - Inline auto-generated diagrams: + - PlantUML + - Mermaid + - TikZ + - Graphviz + - LaTeX tables + +## Architecture diagram + +```mermaid +%%{init: {'theme':'default'}}%% +C4Context + +System_Ext(user, "User", "Author preparing course notes") +System(doc, "Documentation Toolchain", "Python & LaTeX") + +System_Boundary(doc_boundary, "") { + Container(notes2pdf, "notes_to_pdf.py", "Python CLI", "Orchestrates conversion: clean -> images -> Pandoc -> LaTeX") + Container(render, "render_images.py", "Python module", "Renders diagram blocks to images. Caches results.") + Container(preproc, "preprocess_notes.py", "Python module", "Cleans notes & expands macros (pre-Pandoc)") + Container(style, "latex_abbrevs.sty", "LaTeX style", "Provides LaTeX math & formatting shortcuts") +} + +Rel(user, notes2pdf, "Invokes via CLI") +Rel(notes2pdf, preproc, "Uses for Note Pre-processing") +Rel(notes2pdf, render, "Calls to Render Images") +Rel(notes2pdf, style, "Injects LaTeX Style (.sty)") +``` + +## Steps + +- **Stage 1 – Clean-up & augmentation** + - Performed by: `preprocess_notes.py` + - Key ideas: + - Normalizes headers + - Expands arrow shorthand (`->` to `\rightarrow`) + - Deals with comments + - Inserts Pandoc YAML front-matter + - Inserts optional navigation slides +- **Stage 2 – Diagram extraction** + - Performed by: `render_images.py` + - Key ideas: + - Scans code blocks (e.g., ` plantuml) + - Renders diagrams via Docker containers + - Replaces the code with `![](figs/...)` include + - Comments out the original block + - Uses a SHA-256 cache to skip unchanged diagrams +- **Stage 3 – Orchestration** + - Performed by: `notes_to_pdf.py` + - Key ideas: + - Calls Stage 1 and Stage 2, then Pandoc, then (for PDF) LaTeX + - Flags control each sub-action to allow skipping, debugging, or re-running + steps individually +- **Stage 4 – Document synthesis** + - Performed by: Pandoc + LaTeX + - Key ideas: + - Pandoc converts Markdown to LaTeX (or HTML / Beamer) + - `latex_abbrevs.sty` is copied next to the generated `.tex` file + - Ensures vector/matrix macros (`\vv{}`, `\mat{}`), deep lists, and color + helpers compile correctly + +## Dependencies + +### `preprocess_notes.py` + +- **Input: ** raw notes. +- **Output: ** Pandoc‑ready Markdown. +- Handles + - formatting banner frames + - question formatting + - colour commands (`\red{}` -> `\textcolor{red}{...}`) + - TOC injection + +### `render_images.py` + +- Docker‑wrapper around PlantUML, Mermaid CLI, TikZ, Graphviz to convert image + description in a file, replacing the text with the picture + `figs/..png` + +### `latex_abbrevs.sty` + +- Custom style for Latex documents, including: + - Bold-underlined vectors (`\vv{x}`) + - Matrices + - Colour presets + - 9-level `enumitem` lists + - Symbol shorthands +- Copied automatically; you rarely touch this unless you need new macros. diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 463f72570..3937db94f 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -10,7 +10,7 @@ - [Slides with navigation breadcrumbs](#slides-with-navigation-breadcrumbs) - [Focus on a subsection](#focus-on-a-subsection) - [Plain PDF article](#plain-pdf-article) - * [2. Auto Render Figures — `render_images.py`](#2-auto-render-figures--render_imagespy) + * [2. Auto render figures — `render_images.py`](#2-auto-render-figures--render_imagespy) + [Supported File types and Code blocks](#supported-file-types-and-code-blocks) + [Quick Start Recipes](#quick-start-recipes) - [Render to a new file](#render-to-a-new-file) @@ -78,10 +78,8 @@ # Notes Documentation Toolchain -This is a high‑level guide to the helper scripts that turn raw `.txt` lecture -notes into polished PDFs, slide decks, and more. - ---- +- This is a high‑level guide to the helper scripts that turn raw `.txt` notes + into polished PDFs, slide decks, and more. ## 1. Generate Slides and PDFs — `notes_to_pdf.py` @@ -96,9 +94,9 @@ single command. ### Most used flags -* `--type {pdf|html|slides}` -* `--toc_type {none|pandoc_native|navigation}` -* `--debug_on_error`, `--skip_action ...`, `--filter_by_lines A:B` +- `--type {pdf|html|slides}` +- `--toc_type {none|pandoc_native|navigation}` +- `--debug_on_error`, `--skip_action ...`, `--filter_by_lines A:B` ### Quickstart recipes @@ -109,8 +107,8 @@ single command. | Build a **PDF article** (LaTeX) | `notes_to_pdf.py -i paper.txt -o paper.pdf --type pdf` | | Skip the final viewer **open** step | `... --skip_action open` | -_TIP  Run with `--preview_actions` to print the exact steps without executing -them._ +- **Tip**: Run with `--preview_actions` to print the exact steps without + executing them. ### CLI flags cheatsheet @@ -319,20 +317,20 @@ llm_transform.py -p list -i - -o - ### Example recipes -* **Turn a code file into a review checklist** +- **Turn a code file into a review checklist** ```bash llm_transform.py -i foo.py -o cfile -p code_review vim cfile ``` -* **Color‑accent the bold bullets for slides** +- **Color‑accent the bold bullets for slides** ```bash llm_transform.py -i deck.md -o - -p slide_colorize | tee deck.color.md ``` -* **Inline use in Vim** – visual‑select a block, then: +- **Inline use in Vim** – visual‑select a block, then: ```vim :'<,'>!llm_transform.py -p summarize -i - -o - @@ -344,9 +342,9 @@ llm_transform.py -p list -i - -o - ### What the script does -* Reads **Markdown** from _stdin_ or `--input` file. -* Dispatches to a named **action** (currently only `convert_md_to_latex`). -* Pushes the Pandoc output to _stdout_ or the `--output` file. +- Reads **Markdown** from _stdin_ or `--input` file. +- Dispatches to a named **action** (currently only `convert_md_to_latex`). +- Pushes the Pandoc output to _stdout_ or the `--output` file. ### Quickstart commands @@ -374,9 +372,9 @@ llm_transform.py -p list -i - -o - ### What it does -* Accepts a **text/Markdown** stream (file or `-`). -* Applies a named **action** (`-a/--action`). -* Writes the result to the given output (in‑place, file, or `-`). +- Accepts a **text/Markdown** stream (file or `-`). +- Applies a named **action** (`-a/--action`). +- Writes the result to the given output (in‑place, file, or `-`). ### Supported actions @@ -423,9 +421,9 @@ transform_notes.py -a toc -i notes/lecture.md -o - -l 2 Turn a Markdown document into either: -* a **plain list** of headers, -* a **nested header map**, or -* a \*_Vim_ quick‑fix\*\* (`cfile`) that lets you jump between sections with +- a **plain list** of headers, +- a **nested header map**, or +- a \*_Vim_ quick‑fix\*\* (`cfile`) that lets you jump between sections with `:cnext`. ### Examples @@ -510,7 +508,7 @@ rendering across systems. ### What it does Compiles a LaTeX `.tex` file into a PDF using `pdflatex` inside a Docker -container. +container. Automatically rebuilds the Docker image if needed. > ```bash @@ -595,7 +593,7 @@ Automatically sets output to match input name if `--output` is omitted. ### What it does -Converts documents using `pandoc` inside a Docker container. +Converts documents using `pandoc` inside a Docker container. Supports output to Beamer slides, PDFs, and more with custom CLI flags. > ```bash @@ -641,7 +639,7 @@ string. ### What it does Formats text files (`.md`, `.txt`, `.tex`, etc.) using Prettier within a Docker -container. +container. Avoids environment-specific issues and ensures consistent formatting. > ```bash @@ -688,7 +686,7 @@ Supports full Prettier CLI flexibility via passthrough of additional options. 1. Prompts you to select a screen region (`⌘ + Ctrl + 4`). 2. Saves it as `screenshot.YYYY‑MM‑DD_HH‑MM‑SS.png` (or your chosen name). -3. Prints and copies the Markdown embed `![](path/to/file.png)`. +3. Prints and copies the Markdown embed ``. ### Flags @@ -698,4 +696,4 @@ Supports full Prettier CLI flexibility via passthrough of additional options. | `--filename NAME.png` | Override default timestamped name. | | `--override` | Allow clobbering an existing file. | ---- \ No newline at end of file +--- diff --git a/linters/amp_lint_md.py b/linters/amp_lint_md.py index 43d3b642f..45594ba66 100644 --- a/linters/amp_lint_md.py +++ b/linters/amp_lint_md.py @@ -63,7 +63,7 @@ def _execute(self, file_name: str, pedantic: int) -> List[str]: cmd = [] cmd.append(self._executable) cmd.append(f"-i {file_name}") - cmd.append("--in_place") + cmd.append(f"-o {file_name}") cmd_as_str = " ".join(cmd) _, output = liutils.tee(cmd_as_str, self._executable, abort_on_error=True) # Check file name. From f5e00d4ce84a28b2abe5236c9d367a131dead3d8 Mon Sep 17 00:00:00 2001 From: GP Saggese Date: Wed, 28 May 2025 20:26:27 -0400 Subject: [PATCH 14/26] Improve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.explanation.md | 11 +- .../all.notes_toolchain.how_to_guide.md | 522 ++++++++++-------- 2 files changed, 313 insertions(+), 220 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md b/docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md index 14ffa9886..b4e62867c 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md @@ -53,7 +53,7 @@ Rel(notes2pdf, style, "Injects LaTeX Style (.sty)") ## Steps -- **Stage 1 – Clean-up & augmentation** +1. **Clean-up & augmentation** - Performed by: `preprocess_notes.py` - Key ideas: - Normalizes headers @@ -61,7 +61,8 @@ Rel(notes2pdf, style, "Injects LaTeX Style (.sty)") - Deals with comments - Inserts Pandoc YAML front-matter - Inserts optional navigation slides -- **Stage 2 – Diagram extraction** + +2. **Diagram extraction** - Performed by: `render_images.py` - Key ideas: - Scans code blocks (e.g., ` plantuml) @@ -69,13 +70,15 @@ Rel(notes2pdf, style, "Injects LaTeX Style (.sty)") - Replaces the code with `![](figs/...)` include - Comments out the original block - Uses a SHA-256 cache to skip unchanged diagrams -- **Stage 3 – Orchestration** + +3. **Orchestration** - Performed by: `notes_to_pdf.py` - Key ideas: - Calls Stage 1 and Stage 2, then Pandoc, then (for PDF) LaTeX - Flags control each sub-action to allow skipping, debugging, or re-running steps individually -- **Stage 4 – Document synthesis** + +4. **Document synthesis** - Performed by: Pandoc + LaTeX - Key ideas: - Pandoc converts Markdown to LaTeX (or HTML / Beamer) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 3937db94f..6fecb6873 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -81,206 +81,257 @@ - This is a high‑level guide to the helper scripts that turn raw `.txt` notes into polished PDFs, slide decks, and more. -## 1. Generate Slides and PDFs — `notes_to_pdf.py` +// TODO(*): Is it worth to report the flags? It's difficult to maintain -### What it does - -Convert plain‑text notes into polished **PDF, HTML, or Beamer slides** with a -single command. +## notes_to_pdf.py -> ```bash -> notes_to_pdf.py --input --output --type [pdf|html|slides] -> ``` +### What it does -### Most used flags +- Convert plain‑text notes into polished **PDF, HTML, or Beamer slides** with a + single command: + ```bash + > notes_to_pdf.py --input --output --type [pdf|html|slides] + ``` -- `--type {pdf|html|slides}` -- `--toc_type {none|pandoc_native|navigation}` -- `--debug_on_error`, `--skip_action ...`, `--filter_by_lines A:B` +- The most used flags are + - `--type {pdf|html|slides}` + - `--toc_type {none|pandoc_native|navigation}` + - `--debug_on_error`, `--skip_action ...`, `--filter_by_lines A:B` -### Quickstart recipes +### Quickstart recipes -| Goal | Command | -| ----------------------------------- | ------------------------------------------------------------------ | -| Compile to **Beamer slides** | `notes_to_pdf.py -i lesson.txt -o lesson.pdf --type slides` | -| Produce a **stand‑alone HTML** page | `notes_to_pdf.py -i cheatsheet.txt -o cheatsheet.html --type html` | -| Build a **PDF article** (LaTeX) | `notes_to_pdf.py -i paper.txt -o paper.pdf --type pdf` | -| Skip the final viewer **open** step | `... --skip_action open` | +- Compile to **Beamer slides** + ``` + > notes_to_pdf.py -i lesson.txt -o lesson.pdf --type slides + ``` +- Produce a **stand‑alone HTML** page + ``` + > notes_to_pdf.py -i cheatsheet.txt -o cheatsheet.html --type html + ``` +- Build a **PDF article** (LaTeX) + ``` + > notes_to_pdf.py -i paper.txt -o paper.pdf --type pdf + ``` +- Skip the final viewer **open** step + ``` + > ... --skip_action open` + ``` - **Tip**: Run with `--preview_actions` to print the exact steps without executing them. ### CLI flags cheatsheet -| Flag | Purpose | Notes | -| -------------------------------------------- | ------------------------------------------------------------------------- | ----------------------------------------------------- | -| `--type {pdf,html,slides}` | Output format | "slides" uses Beamer | -| `--toc_type {none,pandoc_native,navigation}` | TOC style | `navigation` inserts slide‑friendly breadcrumb frames | -| `--filter_by_header "# Intro"` | Build artefact from a _section subset_ | Useful for testing | -| `--filter_by_lines 120:250` | Compile only a range of lines | Accepts `None` sentinel | -| `--debug_on_error` | On Pandoc failure, generate _.tex_ and drop you a helpful log | | -| `--script myrun.sh` | Save every shell command executed | Repro build pipelines | -| Docker knobs | `--dockerized_force_rebuild`, `--dockerized_use_sudo`, `--use_host_tools` | Control container vs host pandoc/latex | - -_Run `notes_to_pdf.py -h` for the exhaustive list._ +- `--type {pdf,html,slides}` + - Purpose: Specifies the output format + - Notes: The "slides" option uses Beamer +- `--toc_type {none,pandoc_native,navigation}` + - Purpose: Determines the Table of Contents (TOC) style + - Notes: The `navigation` option inserts slide-friendly breadcrumb frames +- `--filter_by_header "# Intro"` + - Purpose: Builds an artefact from a section subset + - Notes: This is useful for testing +- `--filter_by_lines 120:250` + - Purpose: Compiles only a specified range of lines + - Notes: Accepts `None` as a sentinel value +- `--debug_on_error` + - Purpose: On Pandoc failure, generates a _.tex_ file and provides a helpful + log + - Notes: No additional notes +- `--script myrun.sh` + - Purpose: Saves every shell command executed + - Notes: Useful for reproducing build pipelines +- Docker knobs: + - Options: + - `--dockerized_force_rebuild` + - `--dockerized_use_sudo` + - `--use_host_tools` + - Purpose: Controls the use of container vs host for pandoc/latex + +- Run `notes_to_pdf.py -h` for the exhaustive list. ### Worked examples -#### Slides with navigation breadcrumbs - -TODO(indro): `--toc_type navigation` fails because of the preprocess step. - -```bash -notes_to_pdf.py \ - --input MSML610/Lesson5-Theory_Statistical_learning.txt \ - --output Lesson5.pdf \ - --type slides \ - --toc_type navigation \ - --debug_on_error \ - --skip_action cleanup_after -``` - -_Highlights_: adds breadcrumb navigation, keeps intermediates for inspection. - -#### Focus on a subsection - -```bash -notes_to_pdf.py \ - --input Lesson8-Reasoning_over_time.txt \ - --output Focus.pdf \ - --type slides \ - --filter_by_lines 362:None \ - --skip_action cleanup_after -``` +- Slides with navigation breadcrumbs, keeping intermediate files for inspection -Compiles only from line 362 to EOF—fast iteration when debugging slides. +// TODO(indro): `--toc_type navigation` fails because of the preprocess step. -#### Plain PDF article - -```bash -notes_to_pdf.py -i book_notes.txt -o book_notes.pdf --type pdf -``` + ```bash + > notes_to_pdf.py \ + --input MSML610/Lesson5-Theory_Statistical_learning.txt \ + --output Lesson5.pdf \ + --type slides \ + --toc_type navigation \ + --debug_on_error \ + --skip_action cleanup_after + ``` ---- +- Focus on a subsection, compiling only from line 362 to EOF for a fast iteration + when debugging slides + ```bash + > notes_to_pdf.py \ + --input Lesson8-Reasoning_over_time.txt \ + --output Focus.pdf \ + --type slides \ + --filter_by_lines 362:None \ + --skip_action cleanup_after + ``` -## 2. Auto render figures — `render_images.py` +- Plain PDF article + ```bash + > notes_to_pdf.py -i book_notes.txt -o book_notes.pdf --type pdf + ``` -Detects fenced code blocks (PlantUML, Mermaid, TikZ, Graphviz, ...), renders -them into images and swaps the block with `![](img)` markup. +## render_images.py -Example: +- This script auto renders figures by + - detecting fenced code blocks (PlantUML, Mermaid, TikZ, Graphviz, ...) + - rendering them into images calling the appropriate tool + - commenting them out the block + - inlining a `![](img)` markup -```bash -render_images.py -i notes/MSML610/Lesson9-Causal_inference.txt \ - -o lesson9.images.txt --run_dockerized -``` +- Render the images in a text file + ```bash + > render_images.py -i notes/MSML610/Lesson9-Causal_inference.txt \ + -o lesson9.images.txt --run_dockerized + ``` ### Supported File types and Code blocks -| File extension | Rendering syntax allowed | Output embeds as | -| -------------- | ---------------------------------------------- | -------------------------- | -| `.md`, `.txt` | `plantuml / mermaid / graphviz / tikz / latex` | `` | -| `.tex` | same tags (TikZ & LaTeX especially) | `\includegraphics{...}` | +- File extension: `.md`, `.txt` + - Rendering syntax allowed: + - `plantuml` + - `mermaid` + - `graphviz` + - `tikz` + - `latex` + - Output embeds as: `` +- File extension: `.tex` + - Rendering syntax allowed: + - same tags (TikZ & LaTeX especially) + - Output embeds as: `\includegraphics{...}` ### Quick Start Recipes -#### Render to a new file - -```bash -render_images.py -i lesson.md -o lesson.rendered.md --action render --run_dockerized -``` - -#### Render in‑place (Markdown or LaTeX) - -```bash -render_images.py -i lesson.md --action render --run_dockerized -``` - -#### HTML preview of already‑rendered images +- Render to a new file + ```bash + > render_images.py -i lesson.md -o lesson.rendered.md --action render --run_dockerized + ``` -```bash -render_images.py -i lesson.md --action open --run_dockerized -``` +- Render in‑place (Markdown or LaTeX) + ```bash + > render_images.py -i lesson.md --action render --run_dockerized + ``` -#### Dry‑run (test parsing / comments only) +- HTML preview of already‑rendered images + ```bash + > render_images.py -i lesson.md --action open --run_dockerized + ``` -```bash -render_images.py -i lesson.md -o /tmp/out.md --dry_run -``` +- Dry‑run (test parsing / comments only) + ```bash + > render_images.py -i lesson.md -o /tmp/out.md --dry_run + ``` ### Flags -| Flag | Default | Purpose | -| ----------------------------------- | ------------ | ---------------------------------------------------------- | -| `-i/--in_file_name` | **required** | Input `.md`, `.tex`, or `.txt` | -| `-o/--out_file_name` | `` | Output path (must share extension) | -| `--action` | `render` | `render` ↔ `open` | -| `--dry_run` | _False_ | Skip actual rendering, still rewrites markup | -| `--run_dockerized / --dockerized_*` | _False_ | Use pre‑built container images for PlantUML, Mermaid, etc. | -| `--verbosity/-v` | `INFO` | Logging verbosity | - ---- - -## 3. Lint and Prettify — `lint_notes.py` - -A tidying‑up tool for Markdown/LaTeX notes: normalise G‑Doc artifacts, run -Prettier, fix bullet/heading quirks, refresh the Table of Contents – all from a -single command or straight from vim. +- `-i/--in_file_name` + - Default: required + - Purpose: Input `.md`, `.tex`, or `.txt` +- `-o/--out_file_name` + - Default: `` + - Purpose: Output path (must share extension) +- `--action` + - Default: `render` + - Purpose: `render` ↔ `open` +- `--dry_run` + - Default: False + - Purpose: Skip actual rendering, still rewrites markup +- `--run_dockerized / --dockerized_*` + - Default: False + - Purpose: Use pre-built container images for PlantUML, Mermaid, etc +- `--verbosity/-v` + - Default: `INFO` + - Purpose: Logging verbosity + +## `lint_notes.py` + +- Tidy up Markdown/LaTeX/txt notes by: + - normalising G‑Doc artifacts + - running Prettier + - fixing bullet/heading quirks + - refreshing the Table of Contents ### Quickstart recipes -#### Prettify with Dockerised Prettier and TOC rebuild - -```bash -lint_notes.py -i Lesson10.md \ - --use_dockerized_prettier \ - --use_dockerized_markdown_toc -``` - -#### Custom print width and selective actions +- Prettify with Dockerised Prettier and TOC rebuild + ```bash + > lint_notes.py -i Lesson10.md \ + --use_dockerized_prettier \ + --use_dockerized_markdown_toc + ``` -```bash -lint_notes.py -i draft.txt -o tidy.txt -w 100 \ - --action preprocess,prettier,postprocess -``` +- Custom print width and selective actions + ```bash + > lint_notes.py -i draft.txt -o tidy.txt -w 100 \ + --action preprocess,prettier,postprocess + ``` ### Flags -| Flag | Default | Purpose | -| ------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------- | -| `-i/--infile` | **stdin** | Input `.txt` or `.md` (also via pipe) | -| `-o/--outfile` | **stdout** | Destination file (omit for pipe) | -| `-w/--print-width` | _None_ → Prettier default | Line wrap width | -| `--use_dockerized_prettier` | _False_ | Run Prettier inside helper container | -| `--use_dockerized_markdown_toc` | _False_ | Refresh TOC via containerised `markdown-toc` | -| `--action` | all five stages | Comma‑separated subset of: `preprocess`, `prettier`, `postprocess`, `frame_chapters`, `refresh_toc` | -| `-v/--verbosity` | `INFO` | Logging level | - ---- - -## 4. Notebook Image Scraping — `extract_notebook_images.py` - -Spins up a docker container and dumps every `png/svg` output cell into a folder. -You can then publish or reuse the static plots/diagrams already rendered in a -Jupyter notebook. - -Minimal call: - -```bash -extract_notebook_images.py \ - --in_notebook_filename notebooks/Lesson8.ipynb \ - --out_image_dir notebooks/screenshots -``` +- `-i/--infile` + - Default: stdin + - Purpose: Input `.txt` or `.md` (also via pipe) +- `-o/--outfile` + - Default: stdout + - Purpose: Destination file (omit for pipe) +- `-w/--print-width` + - Default: None $\rightarrow$ Prettier default + - Purpose: Line wrap width +- `--use_dockerized_prettier` + - Default: False + - Purpose: Run Prettier inside helper container +- `--use_dockerized_markdown_toc` + - Default: False + - Purpose: Refresh TOC via containerised `markdown-toc` +- `--action` + - Default: all five stages + - Purpose: Comma-separated subset of: `preprocess`, `prettier`, `postprocess`, + `frame_chapters`, `refresh_toc` +- `-v/--verbosity` + - Default: INFO + - Purpose: Logging level + +## `extract_notebook_images.py` + +- Spins up a docker container and dumps every `png/svg` output cell into a folder. +- You can then publish or reuse the static plots/diagrams already rendered in a + Jupyter notebook. + +- Minimal call: + ```bash + > extract_notebook_images.py \ + --in_notebook_filename notebooks/Lesson8.ipynb \ + --out_image_dir notebooks/screenshots + ``` ### Flag Options -| Flag | Purpose | Default | -| ---------------------------------- | ------------------------------------------------------------- | ------------ | -| `-i / --in_notebook_filename PATH` | Notebook to scan | **required** | -| `-o / --out_image_dir DIR` | Folder where images land | **required** | -| `--dockerized_force_rebuild` | Re‑build the Docker image (use if you changed extractor code) | _false_ | -| `--dockerized_use_sudo` | Prepend `sudo docker ...` | auto‑detects | -| `-v INFO/DEBUG` | Log verbosity | `INFO` | +- `-i / --in_notebook_filename PATH` + - Purpose: Notebook to scan + - Default: required +- `-o / --out_image_dir DIR` + - Purpose: Folder where images land + - Default: required +- `--dockerized_force_rebuild` + - Purpose: Re-build the Docker image (use if you changed extractor code) + - Default: false +- `--dockerized_use_sudo` + - Purpose: Prepend `sudo docker ...` + - Default: auto-detects +- `-v INFO/DEBUG` + - Purpose: Log verbosity + - Default: `INFO` --- @@ -305,29 +356,42 @@ llm_transform.py -p list -i - -o - ### Flags -| Flag | Role | Notes | -| -------------------------------------------------------------------- | ------------------------------------------------------------- | ---------------------- | -| `-i / --input` | Source text (`-` = stdin) | — | -| `-o / --output` | Destination (`-` = stdout) | — | -| `-p / --prompt` | **Prompt tag** (`list`, `code_review`, `slide_colorize`, ...) | required | -| `-c / --compare` | Print _both_ original & transformed blocks to stdout | helpful for quick diff | -| `-b / --bold_first_level_bullets` | Post‑format tweak for slide prompts | | -| `-s / --skip-post-transforms` | Return raw LLM output, skip prettier/cleanup | | -| Docker flags (`--dockerized_force_rebuild`, `--dockerized_use_sudo`) | Control container lifecycle | +- `-i / --input` + - Role: Source text (`-` = stdin) + - Notes: None +- `-o / --output` + - Role: Destination (`-` = stdout) + - Notes: None +- `-p / --prompt` + - Role: Prompt tag (`list`, `code_review`, `slide_colorize`, ...) + - Notes: Required +- `-c / --compare` + - Role: Print both original & transformed blocks to stdout + - Notes: Helpful for quick diff +- `-b / --bold_first_level_bullets` + - Role: Post-format tweak for slide prompts + - Notes: None +- `-s / --skip-post-transforms` + - Role: Return raw LLM output, skip prettier/cleanup + - Notes: None +- Docker flags + - Flags: `--dockerized_force_rebuild`, `--dockerized_use_sudo` + - Role: Control container lifecycle + - Notes: None ### Example recipes -- **Turn a code file into a review checklist** +- Turn a code file into a review checklist ```bash - llm_transform.py -i foo.py -o cfile -p code_review + > llm_transform.py -i foo.py -o cfile -p code_review vim cfile ``` - **Color‑accent the bold bullets for slides** ```bash - llm_transform.py -i deck.md -o - -p slide_colorize | tee deck.color.md + > llm_transform.py -i deck.md -o - -p slide_colorize | tee deck.color.md ``` - **Inline use in Vim** – visual‑select a block, then: @@ -336,9 +400,7 @@ llm_transform.py -p list -i - -o - :'<,'>!llm_transform.py -p summarize -i - -o - ``` ---- - -## 6. Pandoc Wrapper — `run_pandoc.py` +## `run_pandoc.py` ### What the script does @@ -348,27 +410,37 @@ llm_transform.py -p list -i - -o - ### Quickstart commands -| Goal | Command | -| ------------------------------------- | -------------------------------------------- | -| Convert a Markdown file to LaTeX | `run_pandoc.py -i note.md -o note.tex` | -| Same, but stream from STDIN to STDOUT | `cat note.md \| run_pandoc.py -i - -o -` | -| Inside **Vim** (visual range) | `:'<,'>!run_pandoc.py -i - -o - -v CRITICAL` | +- Convert a Markdown file to LaTeX + ``` + > run_pandoc.py -i note.md -o note.tex + ``` +- Same, but stream from STDIN to STDOUT + ``` + > cat note.md | run_pandoc.py -i - -o - + ``` +- Inside Vim (visual range) + ``` + > :<,'>!run_pandoc.py -i - -o - -v CRITICAL + ``` -> **Tip :** pass `-v CRITICAL` to silence helper logging when piping into -> editors. +**Tip :** pass `-v CRITICAL` to silence helper logging when piping into editors. ### Flags -| Flag | Default | Meaning | -| ------------------ | --------------------- | --------------------------------------------------------- | -| `-i / --input` | `-` | Source file or `-` for STDIN. | -| `-o / --output` | `-` | Destination file or `-` for STDOUT. | -| `--action` | `convert_md_to_latex` | Transformation to apply. Future‑proofed for more actions. | -| `-v / --log_level` | `INFO` | Standard helper‑library verbosity. | - ---- - -## 7. Automate notes transformations — `transform_notes.py` +- `-i / --input` + - Default: `-` + - Meaning: Source file or `-` for STDIN +- `-o / --output` + - Default: `-` + - Meaning: Destination file or `-` for STDOUT +- `--action` + - Default: `convert_md_to_latex` + - Meaning: Transformation to apply. Future-proofed for more actions +- `-v / --log_level` + - Default: `INFO` + - Meaning: Standard helper-library verbosity + +## `transform_notes.py` ### What it does @@ -376,42 +448,60 @@ llm_transform.py -p list -i - -o - - Applies a named **action** (`-a/--action`). - Writes the result to the given output (in‑place, file, or `-`). -### Supported actions - -| Run `-a list` to print. | Tag | Effect | Typical Vim one‑liner | | -| -------------------------------------------------------------- | -------------------------------------------------- | ------------------------ | --------------------- | --- | -| -------------------------------------------------------------- | -| -------------------------------------------------- | | `toc` | Generate a bullet | -| TOC (top‑level by default) | `:!transform_notes.py -a toc -i % -l 1` | | -| `format_headers` | Re‑flow / indent headers (≤ `--max_lev`) | -| `:%!transform_notes.py -a format -i - --max_lev 3` | | `increase_headers_level` | -| Bump all headers down one level | `:%!transform_notes.py -a increase -i -` | | -| `md_list_to_latex` | Convert a Markdown list to LaTeX `\begin{itemize}` | -| `:%!transform_notes.py -a md_list_to_latex -i -` | | `md_*` family | Formatting | -| clean‑ups (bold bullets, colourise bold text, etc.) | see `-a list` | +### Example of Supported Actions + +- Run `-a list` to print a list of the valid + +- `toc` + - Generate a bullet TOC (top-level by default) + - Typical Vim one-liner: `:!transform_notes.py -a toc -i % -l 1` +- `format_headers` + - Re-flow / indent headers (up to `--max_lev`) + - Typical Vim one-liner: `:%!transform_notes.py -a format -i - --max_lev 3` +- `increase_headers_level` + - Bump all headers down one level + - Typical Vim one-liner: `:%!transform_notes.py -a increase -i -` +- `md_list_to_latex` + - Convert a Markdown list to LaTeX `\begin{itemize}` + - Typical Vim one-liner: `:%!transform_notes.py -a md_list_to_latex -i -` +- `md_*` family + - Formatting clean-ups (bold bullets, colorize bold text, etc.) + - Additional Information: See `-a list` for more details ### Examples -```bash -# Re‑flow & clean a file in place -transform_notes.py -a md_format -i notes/lecture.txt --in_place +- Re‑flow & clean a file in place + ```bash + > transform_notes.py -a md_format -i notes/lecture.txt --in_place + ``` -# Generate a 2‑level TOC to STDOUT -transform_notes.py -a toc -i notes/lecture.md -o - -l 2 +- Generate a 2‑level TOC to STDOUT + ```bash + > transform_notes.py -a toc -i notes/lecture.md -o - -l 2 + ``` -# Tidy ChatGPT‑generated Markdown (visual mode in Vim) -:'<,'>!transform_notes.py -i - -o - -a md_fix_chatgpt_output -``` +- Tidy ChatGPT‑generated Markdown (visual mode in Vim) + ``` + :'<,'>!transform_notes.py -i - -o - -a md_fix_chatgpt_output + ``` ### Flags -| Flag | Default | Purpose | -| ---------------- | ------------ | -------------------------------------------------- | -| `-a / --action` | _(required)_ | Choose the transformation. | -| `-l / --max_lev` | `5` | Header depth for `format_headers`. | -| `-i / --input` | `-` | File path or `-` (STDIN). | -| `-o / --output` | `-` | File path or `-` (STDOUT). | -| `--in_place` | _False_ | Overwrite input file instead of writing elsewhere. | +- `-a / --action` + - Default: Required + - Purpose: Choose the transformation +- `-l / --max_lev` + - Default: 5 + - Purpose: Header depth for `format_headers` +- `-i / --input` + - Default: `-` + - Purpose: File path or `-` (STDIN) +- `-o / --output` + - Default: `-` + - Purpose: File path or `-` (STDOUT) +- `--in_place` + - Default: False + - Purpose: Overwrite input file instead of writing elsewhere --- From 869aec176cca030b767f4cd6ce4aca59395389a5 Mon Sep 17 00:00:00 2001 From: GP Saggese Date: Thu, 29 May 2025 16:17:04 -0400 Subject: [PATCH 15/26] Improve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.explanation.md | 62 +- .../all.notes_toolchain.how_to_guide.md | 880 ++++++++---------- 2 files changed, 444 insertions(+), 498 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md b/docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md index b4e62867c..446f0b5bc 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.explanation.md @@ -1,6 +1,6 @@ -- [`notes_to_pdf.py` - Flow Explanation](#notes_to_pdfpy---flow-explanation) +- [Flow Explanation](#flow-explanation) * [Goal](#goal) * [Architecture diagram](#architecture-diagram) * [Steps](#steps) @@ -54,37 +54,41 @@ Rel(notes2pdf, style, "Injects LaTeX Style (.sty)") ## Steps 1. **Clean-up & augmentation** - - Performed by: `preprocess_notes.py` - - Key ideas: - - Normalizes headers - - Expands arrow shorthand (`->` to `\rightarrow`) - - Deals with comments - - Inserts Pandoc YAML front-matter - - Inserts optional navigation slides + +- Performed by: `preprocess_notes.py` +- Key ideas: + - Normalizes headers + - Expands arrow shorthand (`->` to `\rightarrow`) + - Deals with comments + - Inserts Pandoc YAML front-matter + - Inserts optional navigation slides 2. **Diagram extraction** - - Performed by: `render_images.py` - - Key ideas: - - Scans code blocks (e.g., ` plantuml) - - Renders diagrams via Docker containers - - Replaces the code with `![](figs/...)` include - - Comments out the original block - - Uses a SHA-256 cache to skip unchanged diagrams + +- Performed by: `render_images.py` +- Key ideas: + - Scans code blocks (e.g., ` plantuml) + - Renders diagrams via Docker containers + - Replaces the code with `![](figs/...)` include + - Comments out the original block + - Uses a SHA-256 cache to skip unchanged diagrams 3. **Orchestration** - - Performed by: `notes_to_pdf.py` - - Key ideas: - - Calls Stage 1 and Stage 2, then Pandoc, then (for PDF) LaTeX - - Flags control each sub-action to allow skipping, debugging, or re-running - steps individually + +- Performed by: `notes_to_pdf.py` +- Key ideas: + - Calls Stage 1 and Stage 2, then Pandoc, then (for PDF) LaTeX + - Flags control each sub-action to allow skipping, debugging, or re-running + steps individually 4. **Document synthesis** - - Performed by: Pandoc + LaTeX - - Key ideas: - - Pandoc converts Markdown to LaTeX (or HTML / Beamer) - - `latex_abbrevs.sty` is copied next to the generated `.tex` file - - Ensures vector/matrix macros (`\vv{}`, `\mat{}`), deep lists, and color - helpers compile correctly + +- Performed by: Pandoc + LaTeX +- Key ideas: + - Pandoc converts Markdown to LaTeX (or HTML / Beamer) + - `latex_abbrevs.sty` is copied next to the generated `.tex` file + - Ensures vector/matrix macros (`\vv{}`, `\mat{}`), deep lists, and color + helpers compile correctly ## Dependencies @@ -93,9 +97,9 @@ Rel(notes2pdf, style, "Injects LaTeX Style (.sty)") - **Input: ** raw notes. - **Output: ** Pandoc‑ready Markdown. - Handles - - formatting banner frames - - question formatting - - colour commands (`\red{}` -> `\textcolor{red}{...}`) + - Formatting banner frames + - Question formatting + - Colour commands (`\red{}` -> `\textcolor{red}{...}`) - TOC injection ### `render_images.py` diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 6fecb6873..56155346c 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -1,78 +1,57 @@ - [Notes Documentation Toolchain](#notes-documentation-toolchain) - * [1. Generate Slides and PDFs — `notes_to_pdf.py`](#1-generate-slides-and-pdfs--notes_to_pdfpy) + * [notes_to_pdf.py](#notes_to_pdfpy) + [What it does](#what-it-does) - + [Most used flags](#most-used-flags) + [Quickstart recipes](#quickstart-recipes) + [CLI flags cheatsheet](#cli-flags-cheatsheet) + [Worked examples](#worked-examples) - - [Slides with navigation breadcrumbs](#slides-with-navigation-breadcrumbs) - - [Focus on a subsection](#focus-on-a-subsection) - - [Plain PDF article](#plain-pdf-article) - * [2. Auto render figures — `render_images.py`](#2-auto-render-figures--render_imagespy) + * [render_images.py](#render_imagespy) + [Supported File types and Code blocks](#supported-file-types-and-code-blocks) + [Quick Start Recipes](#quick-start-recipes) - - [Render to a new file](#render-to-a-new-file) - - [Render in‑place (Markdown or LaTeX)](#render-in%E2%80%91place-markdown-or-latex) - - [HTML preview of already‑rendered images](#html-preview-of-already%E2%80%91rendered-images) - - [Dry‑run (test parsing / comments only)](#dry%E2%80%91run-test-parsing--comments-only) + [Flags](#flags) - * [3. Lint and Prettify — `lint_notes.py`](#3-lint-and-prettify--lint_notespy) + * [`lint_notes.py`](#lint_notespy) + [Quickstart recipes](#quickstart-recipes-1) - - [Prettify with Dockerised Prettier and TOC rebuild](#prettify-with-dockerised-prettier-and-toc-rebuild) - - [Custom print width and selective actions](#custom-print-width-and-selective-actions) + [Flags](#flags-1) - * [4. Notebook Image Scraping — `extract_notebook_images.py`](#4-notebook-image-scraping--extract_notebook_imagespy) + * [`extract_notebook_images.py`](#extract_notebook_imagespy) + [Flag Options](#flag-options) * [5. LLM Powered Transforms — `llm_transform.py`](#5-llm-powered-transforms--llm_transformpy) + [Minimum viable command](#minimum-viable-command) + [Finding available prompts](#finding-available-prompts) + [Flags](#flags-2) + [Example recipes](#example-recipes) - * [6. Pandoc Wrapper — `run_pandoc.py`](#6-pandoc-wrapper--run_pandocpy) + * [`run_pandoc.py`](#run_pandocpy) + [What the script does](#what-the-script-does) + [Quickstart commands](#quickstart-commands) + [Flags](#flags-3) - * [7. Automate notes transformations — `transform_notes.py`](#7-automate-notes-transformations--transform_notespy) + * [`transform_notes.py`](#transform_notespy) + [What it does](#what-it-does-1) - + [Supported actions](#supported-actions) + + [Example of Supported Actions](#example-of-supported-actions) + [Examples](#examples) + [Flags](#flags-4) - * [8. Scrape headers from a markdown — `extract_headers_from_markdown.py`](#8-scrape-headers-from-a-markdown--extract_headers_from_markdownpy) + * [`extract_headers_from_markdown.py`](#extract_headers_from_markdownpy) + [Goal](#goal) + [Examples](#examples-1) - + [Flags](#flags-5) - * [9. TikZ to Bitmap — `dockerized_tikz_to_bitmap.py`](#9-tikz-to-bitmap--dockerized_tikz_to_bitmappy) + * [`dockerized_tikz_to_bitmap.py`](#dockerized_tikz_to_bitmappy) + [Examples](#examples-2) - * [10. Graphviz Renderer — `dockerized_graphviz.py`](#10-graphviz-renderer--dockerized_graphvizpy) + * [`dockerized_graphviz.py`](#dockerized_graphvizpy) + [What it does](#what-it-does-2) - + [Most used flags](#most-used-flags-1) + + [Most used flags](#most-used-flags) + [Quickstart recipes](#quickstart-recipes-2) - + [CLI flags cheatsheet](#cli-flags-cheatsheet-1) - * [11. LaTeX Renderer — `dockerized_latex.py`](#11-latex-renderer--dockerized_latexpy) + * [dockerized_latex.py](#dockerized_latexpy) + [What it does](#what-it-does-3) - + [Most used flags](#most-used-flags-2) + [Quickstart recipes](#quickstart-recipes-3) - + [CLI flags cheatsheet](#cli-flags-cheatsheet-2) - * [12. Mermaid Renderer — `dockerized_mermaid.py`](#12-mermaid-renderer--dockerized_mermaidpy) + * [dockerized_mermaid.py](#dockerized_mermaidpy) + [What it does](#what-it-does-4) - + [Most used flags](#most-used-flags-3) - + [Quickstart recipes](#quickstart-recipes-4) - + [CLI flags cheatsheet](#cli-flags-cheatsheet-3) - * [13. Pandoc Renderer — `dockerized_pandoc.py`](#13-pandoc-renderer--dockerized_pandocpy) + * [dockerized_pandoc.py](#dockerized_pandocpy) + [What it does](#what-it-does-5) - + [Most used flags](#most-used-flags-4) - + [Quickstart recipes](#quickstart-recipes-5) - + [CLI flags cheat‑sheet](#cli-flags-cheat%E2%80%91sheet) - * [14. Prettier Formatter — `dockerized_prettier.py`](#14-prettier-formatter--dockerized_prettierpy) + + [Quickstart recipes](#quickstart-recipes-4) + * [`dockerized_prettier.py`](#dockerized_prettierpy) + [What it does](#what-it-does-6) - + [Most used flags](#most-used-flags-5) - + [Quickstart recipes](#quickstart-recipes-6) - + [CLI flags cheatsheet](#cli-flags-cheatsheet-4) - * [15. MacOS screenshot helper — `save_screenshot.py`](#15-macos-screenshot-helper--save_screenshotpy) + + [Quickstart recipes](#quickstart-recipes-5) + + [Interface](#interface) + * [`save_screenshot.py`](#save_screenshotpy) + [What it does](#what-it-does-7) - + [Flags](#flags-6) @@ -81,7 +60,7 @@ - This is a high‑level guide to the helper scripts that turn raw `.txt` notes into polished PDFs, slide decks, and more. -// TODO(*): Is it worth to report the flags? It's difficult to maintain +// TODO(\*): Is it worth to report the flags? It's difficult to maintain ## notes_to_pdf.py @@ -93,12 +72,70 @@ > notes_to_pdf.py --input --output --type [pdf|html|slides] ``` -- The most used flags are - - `--type {pdf|html|slides}` - - `--toc_type {none|pandoc_native|navigation}` - - `--debug_on_error`, `--skip_action ...`, `--filter_by_lines A:B` +- The interface is: + ``` + > notes_to_pdf.py -h + usage: notes_to_pdf.py [-h] -i INPUT -o OUTPUT --type {pdf,html,slides} + [--filter_by_header FILTER_BY_HEADER] + [--filter_by_lines FILTER_BY_LINES] [--script SCRIPT] + [--preview_actions] + [--toc_type {none,pandoc_native,navigation}] + [--no_run_latex_again] [--debug_on_error] + [--gdrive_dir GDRIVE_DIR] [--use_host_tools] + [--action {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} | --skip_action {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after}] + [--all] [--dockerized_force_rebuild] + [--dockerized_use_sudo] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + + Convert a txt file into a PDF / HTML / slides using `pandoc`. + + # From scratch with TOC: + > notes_to_pdf.py -a pdf --input ... + + # For interactive mode: + > notes_to_pdf.py -a pdf --no_cleanup_before --no_cleanup --input ... + + # Check that can be compiled: + > notes_to_pdf.py -a pdf --no_toc --no_open_pdf --input ... + + > notes_to_pdf.py --input notes/IN_PROGRESS/math.The_hundred_page_ML_book.Burkov.2019.txt -t pdf --no_cleanup --no_cleanup_before --no_run_latex_again --no_open + + options: + -h, --help show this help message and exit + -i INPUT, --input INPUT + -o OUTPUT, --output OUTPUT + Output file + --type {pdf,html,slides} + Type of output to generate + --filter_by_header FILTER_BY_HEADER + Filter by header + --filter_by_lines FILTER_BY_LINES + Filter by lines (e.g., `0:10`, `1:None`, `None:10`) + --script SCRIPT Bash script to generate with all the executed sub- + commands + --preview_actions Print the actions and exit + --toc_type {none,pandoc_native,navigation} + --no_run_latex_again + --debug_on_error + --gdrive_dir GDRIVE_DIR + Directory where to save the output to share on Google + Drive + --use_host_tools Use the host tools instead of the dockerized ones + --action {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} + Actions to execute + --skip_action {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} + Actions to skip + --all Run all the actions (cleanup_before preprocess_notes + render_images run_pandoc open cleanup_after) + --dockerized_force_rebuild + Force to rebuild the Docker container + --dockerized_use_sudo + Use sudo inside the container + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level + ``` -### Quickstart recipes +### Examples - Compile to **Beamer slides** ``` @@ -120,42 +157,8 @@ - **Tip**: Run with `--preview_actions` to print the exact steps without executing them. -### CLI flags cheatsheet - -- `--type {pdf,html,slides}` - - Purpose: Specifies the output format - - Notes: The "slides" option uses Beamer -- `--toc_type {none,pandoc_native,navigation}` - - Purpose: Determines the Table of Contents (TOC) style - - Notes: The `navigation` option inserts slide-friendly breadcrumb frames -- `--filter_by_header "# Intro"` - - Purpose: Builds an artefact from a section subset - - Notes: This is useful for testing -- `--filter_by_lines 120:250` - - Purpose: Compiles only a specified range of lines - - Notes: Accepts `None` as a sentinel value -- `--debug_on_error` - - Purpose: On Pandoc failure, generates a _.tex_ file and provides a helpful - log - - Notes: No additional notes -- `--script myrun.sh` - - Purpose: Saves every shell command executed - - Notes: Useful for reproducing build pipelines -- Docker knobs: - - Options: - - `--dockerized_force_rebuild` - - `--dockerized_use_sudo` - - `--use_host_tools` - - Purpose: Controls the use of container vs host for pandoc/latex - -- Run `notes_to_pdf.py -h` for the exhaustive list. - -### Worked examples - - Slides with navigation breadcrumbs, keeping intermediate files for inspection - -// TODO(indro): `--toc_type navigation` fails because of the preprocess step. - + // TODO(indro): `--toc_type navigation` fails because of the preprocess step. ```bash > notes_to_pdf.py \ --input MSML610/Lesson5-Theory_Statistical_learning.txt \ @@ -166,8 +169,8 @@ --skip_action cleanup_after ``` -- Focus on a subsection, compiling only from line 362 to EOF for a fast iteration - when debugging slides +- Focus on a subsection, compiling only from line 362 to EOF for a fast + iteration when debugging slides ```bash > notes_to_pdf.py \ --input Lesson8-Reasoning_over_time.txt \ @@ -184,11 +187,13 @@ ## render_images.py +### What it does + - This script auto renders figures by - - detecting fenced code blocks (PlantUML, Mermaid, TikZ, Graphviz, ...) - - rendering them into images calling the appropriate tool - - commenting them out the block - - inlining a `![](img)` markup + - Detecting fenced code blocks (PlantUML, Mermaid, TikZ, Graphviz, ...) + - Rendering them into images calling the appropriate tool + - Commenting them out the block + - Inlining a `![](img)` markup - Render the images in a text file ```bash @@ -196,34 +201,36 @@ -o lesson9.images.txt --run_dockerized ``` -### Supported File types and Code blocks +The supported File types and code blocks are: + - File extension: `.md`, `.txt` + - Rendering syntax allowed: + - `plantuml` + - `mermaid` + - `graphviz` + - `tikz` + - `latex` + - Output embeds as: `` + - File extension: `.tex` + - Rendering syntax allowed: + - Same tags (TikZ & LaTeX especially) + - Output embeds as: `\includegraphics{...}` -- File extension: `.md`, `.txt` - - Rendering syntax allowed: - - `plantuml` - - `mermaid` - - `graphviz` - - `tikz` - - `latex` - - Output embeds as: `` -- File extension: `.tex` - - Rendering syntax allowed: - - same tags (TikZ & LaTeX especially) - - Output embeds as: `\includegraphics{...}` - -### Quick Start Recipes +### Examples - Render to a new file + ```bash > render_images.py -i lesson.md -o lesson.rendered.md --action render --run_dockerized ``` - Render in‑place (Markdown or LaTeX) + ```bash > render_images.py -i lesson.md --action render --run_dockerized ``` - HTML preview of already‑rendered images + ```bash > render_images.py -i lesson.md --action open --run_dockerized ``` @@ -233,38 +240,71 @@ > render_images.py -i lesson.md -o /tmp/out.md --dry_run ``` -### Flags +### Interface -- `-i/--in_file_name` - - Default: required - - Purpose: Input `.md`, `.tex`, or `.txt` -- `-o/--out_file_name` - - Default: `` - - Purpose: Output path (must share extension) -- `--action` - - Default: `render` - - Purpose: `render` ↔ `open` -- `--dry_run` - - Default: False - - Purpose: Skip actual rendering, still rewrites markup -- `--run_dockerized / --dockerized_*` - - Default: False - - Purpose: Use pre-built container images for PlantUML, Mermaid, etc -- `--verbosity/-v` - - Default: `INFO` - - Purpose: Logging verbosity +- The interface + + ```bash + > render_images.py -h + usage: render_images.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] + [--action {open,render} | --skip_action {open,render}] + [--all] [--dry_run] [--dockerized_force_rebuild] + [--dockerized_use_sudo] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + + Replace sections of image code with rendered images, commenting out the + original code, if needed. + + See `docs/work_tools/documentation_toolchain/all.render_images.explanation.md`. + + Usage: + + # Create a new Markdown file with rendered images: + > render_images.py -i ABC.md -o XYZ.md --action render --run_dockerized + + # Render images in place in the original Markdown file: + > render_images.py -i ABC.md --action render --run_dockerized + + # Render images in place in the original LaTeX file: + > render_images.py -i ABC.tex --action render --run_dockerized + + # Open rendered images from a Markdown file in HTML to preview: + > render_images.py -i ABC.md --action open --run_dockerized + + options: + -h, --help show this help message and exit + -i IN_FILE_NAME, --in_file_name IN_FILE_NAME + Path to the input file + -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME + Path to the output file + --action {open,render} + Actions to execute + --skip_action {open,render} + Actions to skip + --all Run all the actions () + --dry_run Update the file but do not render images + --dockerized_force_rebuild + Force to rebuild the Docker container + --dockerized_use_sudo + Use sudo inside the container + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level + ``` ## `lint_notes.py` +### What it does + - Tidy up Markdown/LaTeX/txt notes by: - - normalising G‑Doc artifacts - - running Prettier - - fixing bullet/heading quirks - - refreshing the Table of Contents + - Normalising G‑Doc artifacts + - Running Prettier + - Fixing bullet/heading quirks + - Refreshing the Table of Contents -### Quickstart recipes +### Examples - Prettify with Dockerised Prettier and TOC rebuild + ```bash > lint_notes.py -i Lesson10.md \ --use_dockerized_prettier \ @@ -277,37 +317,21 @@ --action preprocess,prettier,postprocess ``` -### Flags - -- `-i/--infile` - - Default: stdin - - Purpose: Input `.txt` or `.md` (also via pipe) -- `-o/--outfile` - - Default: stdout - - Purpose: Destination file (omit for pipe) -- `-w/--print-width` - - Default: None $\rightarrow$ Prettier default - - Purpose: Line wrap width -- `--use_dockerized_prettier` - - Default: False - - Purpose: Run Prettier inside helper container -- `--use_dockerized_markdown_toc` - - Default: False - - Purpose: Refresh TOC via containerised `markdown-toc` -- `--action` - - Default: all five stages - - Purpose: Comma-separated subset of: `preprocess`, `prettier`, `postprocess`, - `frame_chapters`, `refresh_toc` -- `-v/--verbosity` - - Default: INFO - - Purpose: Logging level +### Interface + +// TODO ## `extract_notebook_images.py` -- Spins up a docker container and dumps every `png/svg` output cell into a folder. +### What it does + +- Spins up a docker container and dumps every `png/svg` output cell into a + folder. - You can then publish or reuse the static plots/diagrams already rendered in a Jupyter notebook. +### Example + - Minimal call: ```bash > extract_notebook_images.py \ @@ -315,117 +339,79 @@ --out_image_dir notebooks/screenshots ``` -### Flag Options - -- `-i / --in_notebook_filename PATH` - - Purpose: Notebook to scan - - Default: required -- `-o / --out_image_dir DIR` - - Purpose: Folder where images land - - Default: required -- `--dockerized_force_rebuild` - - Purpose: Re-build the Docker image (use if you changed extractor code) - - Default: false -- `--dockerized_use_sudo` - - Purpose: Prepend `sudo docker ...` - - Default: auto-detects -- `-v INFO/DEBUG` - - Purpose: Log verbosity - - Default: `INFO` - ---- - -## 5. LLM Powered Transforms — `llm_transform.py` +### Interface -Apply a GPT‑style transformation (rewrite, summarise, critique code, convert to -slides, etc.) to any text file _without_ leaving the terminal / editor. +// TODO -> _Note: You need to have an `OPENAI_API_KEY` and an internet connection._ +## `llm_transform.py` -### Minimum viable command +### What it does -```bash -llm_transform.py -i draft.txt -o polished.txt -p rewrite_clearer -``` +- Apply a GPT‑style transformation (rewrite, summarise, critique code, convert to + slides, etc.) to any text file _without_ leaving the terminal / editor. -### Finding available prompts +- **Note**: You need to have an `OPENAI_API_KEY` and an internet connection. -```bash -llm_transform.py -p list -i - -o - -``` +### Examples -### Flags +- TODO + ```bash + llm_transform.py -i draft.txt -o polished.txt -p rewrite_clearer + ``` -- `-i / --input` - - Role: Source text (`-` = stdin) - - Notes: None -- `-o / --output` - - Role: Destination (`-` = stdout) - - Notes: None -- `-p / --prompt` - - Role: Prompt tag (`list`, `code_review`, `slide_colorize`, ...) - - Notes: Required -- `-c / --compare` - - Role: Print both original & transformed blocks to stdout - - Notes: Helpful for quick diff -- `-b / --bold_first_level_bullets` - - Role: Post-format tweak for slide prompts - - Notes: None -- `-s / --skip-post-transforms` - - Role: Return raw LLM output, skip prettier/cleanup - - Notes: None -- Docker flags - - Flags: `--dockerized_force_rebuild`, `--dockerized_use_sudo` - - Role: Control container lifecycle - - Notes: None - -### Example recipes +- Finding available prompts + ```bash + llm_transform.py -p list -i - -o - + ``` - Turn a code file into a review checklist - ```bash > llm_transform.py -i foo.py -o cfile -p code_review vim cfile ``` -- **Color‑accent the bold bullets for slides** +- Color‑accent the bold bullets for slides ```bash > llm_transform.py -i deck.md -o - -p slide_colorize | tee deck.color.md ``` -- **Inline use in Vim** – visual‑select a block, then: +- Inline use in Vim, visual‑select a block, then: ```vim :'<,'>!llm_transform.py -p summarize -i - -o - ``` +### Interface + +// TODO + ## `run_pandoc.py` -### What the script does +### What it does - Reads **Markdown** from _stdin_ or `--input` file. - Dispatches to a named **action** (currently only `convert_md_to_latex`). - Pushes the Pandoc output to _stdout_ or the `--output` file. -### Quickstart commands +### Example - Convert a Markdown file to LaTeX ``` > run_pandoc.py -i note.md -o note.tex ``` -- Same, but stream from STDIN to STDOUT +- Same, but stream from `stdin` to `stdout` ``` > cat note.md | run_pandoc.py -i - -o - ``` - Inside Vim (visual range) ``` - > :<,'>!run_pandoc.py -i - -o - -v CRITICAL + :<,'>!run_pandoc.py -i - -o - -v CRITICAL ``` -**Tip :** pass `-v CRITICAL` to silence helper logging when piping into editors. +- **Tip:** pass `-v CRITICAL` to silence helper logging when piping into editors. -### Flags +### Interface - `-i / --input` - Default: `-` @@ -448,34 +434,33 @@ llm_transform.py -p list -i - -o - - Applies a named **action** (`-a/--action`). - Writes the result to the given output (in‑place, file, or `-`). -### Example of Supported Actions +### Examples - Run `-a list` to print a list of the valid - -- `toc` - - Generate a bullet TOC (top-level by default) - - Typical Vim one-liner: `:!transform_notes.py -a toc -i % -l 1` -- `format_headers` - - Re-flow / indent headers (up to `--max_lev`) - - Typical Vim one-liner: `:%!transform_notes.py -a format -i - --max_lev 3` -- `increase_headers_level` - - Bump all headers down one level - - Typical Vim one-liner: `:%!transform_notes.py -a increase -i -` -- `md_list_to_latex` - - Convert a Markdown list to LaTeX `\begin{itemize}` - - Typical Vim one-liner: `:%!transform_notes.py -a md_list_to_latex -i -` -- `md_*` family - - Formatting clean-ups (bold bullets, colorize bold text, etc.) - - Additional Information: See `-a list` for more details - -### Examples + - `toc` + - Generate a bullet TOC (top-level by default) + - Typical Vim one-liner: `:!transform_notes.py -a toc -i % -l 1` + - `format_headers` + - Re-flow / indent headers (up to `--max_lev`) + - Typical Vim one-liner: `:%!transform_notes.py -a format -i - --max_lev 3` + - `increase_headers_level` + - Bump all headers down one level + - Typical Vim one-liner: `:%!transform_notes.py -a increase -i -` + - `md_list_to_latex` + - Convert a Markdown list to LaTeX `\begin{itemize}` + - Typical Vim one-liner: `:%!transform_notes.py -a md_list_to_latex -i -` + - `md_*` family + - Formatting clean-ups (bold bullets, colorize bold text, etc.) + - Additional Information: See `-a list` for more details - Re‑flow & clean a file in place + ```bash > transform_notes.py -a md_format -i notes/lecture.txt --in_place ``` - Generate a 2‑level TOC to STDOUT + ```bash > transform_notes.py -a toc -i notes/lecture.md -o - -l 2 ``` @@ -485,305 +470,262 @@ llm_transform.py -p list -i - -o - :'<,'>!transform_notes.py -i - -o - -a md_fix_chatgpt_output ``` -### Flags - -- `-a / --action` - - Default: Required - - Purpose: Choose the transformation -- `-l / --max_lev` - - Default: 5 - - Purpose: Header depth for `format_headers` -- `-i / --input` - - Default: `-` - - Purpose: File path or `-` (STDIN) -- `-o / --output` - - Default: `-` - - Purpose: File path or `-` (STDOUT) -- `--in_place` - - Default: False - - Purpose: Overwrite input file instead of writing elsewhere - ---- +### Interface -## 8. Scrape headers from a markdown — `extract_headers_from_markdown.py` +## `extract_headers_from_markdown.py` -### Goal - -Turn a Markdown document into either: +### What it does -- a **plain list** of headers, -- a **nested header map**, or -- a \*_Vim_ quick‑fix\*\* (`cfile`) that lets you jump between sections with - `:cnext`. +- Turn a Markdown document into either: + - A **plain list** of headers + - A **nested header map** + - A \*_Vim_ quick‑fix\*\* (`cfile`) that lets you jump between sections with + `:cnext`. ### Examples -```bash -# Human‑readable map (levels 1‑3) to STDOUT -extract_headers_from_markdown.py -i README.md -o - --mode list --max-level 3 - -# Build a quick‑fix file and open Vim on it -extract_headers_from_markdown.py -i README.md -o headers.cfile --mode cfile -vim -c "cfile headers.cfile" -``` +- Human‑readable map (levels 1‑3) to `stdout` -### Flags + ```bash + > extract_headers_from_markdown.py -i README.md -o - --mode list --max-level 3 + ``` -| Flag | Default | Meaning | -| ------------- | ------- | ------------------------------ | -| `--mode` | `list` | `list`, `headers`, or `cfile`. | -| `--max-level` | `3` | Maximum `#` depth to parse. | +- Build a quick‑fix file and open Vim on it + ```bash + > extract_headers_from_markdown.py -i README.md -o headers.cfile --mode cfile + > vim -c "cfile headers.cfile" + ``` ---- +## `dockerized_tikz_to_bitmap.py` -## 9. TikZ to Bitmap — `dockerized_tikz_to_bitmap.py` +- Converts ### Examples -```bash -# Plain 300 DPI conversion -./dockerized_tikz_to_bitmap.py -i figure.tikz -o figure.png +- Plain 300 DPI conversion -# Custom ImageMagick options (e.g. 600 DPI) -./dockerized_tikz_to_bitmap.py -i fig.tikz -o fig.png -- -density 600 -quality 90 -``` - -_Any extra tokens after `--` are passed verbatim to `convert`._ + ```bash + > dockerized_tikz_to_bitmap.py -i figure.tikz -o figure.png + ``` ---- +- Custom ImageMagick options (e.g. 600 DPI) + ```bash + > dockerized_tikz_to_bitmap.py -i fig.tikz -o fig.png -- -density 600 -quality 90 + ``` + - Any extra tokens after `--` are passed verbatim to `convert` -## 10. Graphviz Renderer — `dockerized_graphviz.py` +## `dockerized_graphviz.py` ### What it does -Converts a Graphviz `.dot` file into a `.png` image using a Dockerized -container. - -> ```bash -> graphviz_wrapper.py --input input.dot --output output.png -> ``` - -This script serves as a thin wrapper around Dockerized Graphviz for consistent -rendering across systems. - -### Most used flags +- Converts a Graphviz `.dot` file into a `.png` image using a Dockerized + container. + > ```bash + > graphviz_wrapper.py --input input.dot --output output.png + > ``` -- `--input`: path to the `.dot` file -- `--output`: destination `.png` image file -- `--dockerized_force_rebuild`: rebuild the container from scratch -- `--dockerized_use_sudo`: use `sudo` for Docker commands +- This script serves as a thin wrapper around Dockerized Graphviz for consistent + rendering across systems. -### Quickstart recipes +### Interface -| Goal | Command | -| --------------------- | ------------------------------------------------------------------------------ | -| Convert DOT to PNG | `graphviz_wrapper.py -i diagram.dot -o diagram.png` | -| Rebuild Docker image | `graphviz_wrapper.py -i diagram.dot -o diagram.png --dockerized_force_rebuild` | -| Use `sudo` for Docker | `graphviz_wrapper.py -i diagram.dot -o diagram.png --dockerized_use_sudo` | - -### CLI flags cheatsheet - -| Flag | Purpose | Notes | -| ---------------------------- | ---------------------------- | ------------- | -| `-i / --input` | Path to input `.dot` file | **required** | -| `-o / --output` | Output path for `.png` image | **required** | -| `--dockerized_force_rebuild` | Force Docker image rebuild | Optional | -| `--dockerized_use_sudo` | Run Docker with `sudo` | Optional | -| `-v / --verbosity` | Logging verbosity | Default: INFO | +### Examples ---- +- Convert DOT to PNG + ``` + > graphviz_wrapper.py -i diagram.dot -o diagram.png + ``` +- Rebuild Docker image + ``` + > graphviz_wrapper.py -i diagram.dot -o diagram.png --dockerized_force_rebuild + ``` +- Use `sudo` for Docker + ```bash + > graphviz_wrapper.py -i diagram.dot -o diagram.png --dockerized_use_sudo + ``` -## 11. LaTeX Renderer — `dockerized_latex.py` +## dockerized_latex.py ### What it does -Compiles a LaTeX `.tex` file into a PDF using `pdflatex` inside a Docker -container. -Automatically rebuilds the Docker image if needed. - -> ```bash -> latex_wrapper.py --input doc.tex --output doc.pdf -> ``` - -Supports optional rerun of LaTeX for proper references or table of contents -generation. - -### Most used flags - -- `--input`: LaTeX source file to compile -- `--output`: Output PDF path -- `--run_latex_again`: Compile the LaTeX file twice -- `--dockerized_force_rebuild`: Force container rebuild -- `--dockerized_use_sudo`: Run Docker with `sudo` - -### Quickstart recipes - -| Goal | Command | -| ------------------------ | ------------------------------------------------------------------------- | -| Compile `.tex` to `.pdf` | `latex_wrapper.py -i report.tex -o report.pdf` | -| Rebuild Docker image | `latex_wrapper.py -i report.tex -o report.pdf --dockerized_force_rebuild` | -| Use `sudo` for Docker | `latex_wrapper.py -i report.tex -o report.pdf --dockerized_use_sudo` | -| Run LaTeX twice | `latex_wrapper.py -i paper.tex -o paper.pdf --run_latex_again` | - -### CLI flags cheatsheet +- Compiles a LaTeX `.tex` file into a PDF using `pdflatex` inside a Docker + container. +- Automatically rebuilds the Docker image if needed. +- Supports optional rerun of LaTeX for proper references or table of contents + generation + ```bash + > latex_wrapper.py --input doc.tex --output doc.pdf + ``` -| Flag | Purpose | Notes | -| ---------------------------- | -------------------------- | ----------------------------- | -| `-i / --input` | Path to input `.tex` file | **required** | -| `-o / --output` | Output PDF file path | **required** | -| `--run_latex_again` | Run LaTeX a second time | Optional, useful for TOC/refs | -| `--dockerized_force_rebuild` | Force Docker image rebuild | Optional | -| `--dockerized_use_sudo` | Run Docker with `sudo` | Optional | -| `-v / --verbosity` | Logging verbosity | Default: INFO | +### Examples ---- +- Compile `.tex` to `.pdf` + ``` + > latex_wrapper.py -i report.tex -o report.pdf + ``` +- Rebuild Docker image + ``` + > latex_wrapper.py -i report.tex -o report.pdf --dockerized_force_rebuild + ``` +- Use `sudo` for Docker + ``` + > latex_wrapper.py -i report.tex -o report.pdf --dockerized_use_sudo + ``` +- Run LaTeX twice + ``` + > latex_wrapper.py -i paper.tex -o paper.pdf --run_latex_again + ``` -## 12. Mermaid Renderer — `dockerized_mermaid.py` +## dockerized_mermaid.py ### What it does -Renders Mermaid `.mmd` or `.md` diagrams into image files using a Dockerized -container. - -> ```bash -> mermaid_wrapper.py --input flowchart.mmd --output flowchart.png -> ``` - -Automatically sets output to match input name if `--output` is omitted. +- Renders Mermaid `.mmd` or `.md` diagrams into image files using a Dockerized + container. -### Most used flags - -- `--input`: Source Mermaid file -- `--output`: Destination image file (optional) -- `--dockerized_force_rebuild`: Rebuild Docker image -- `--dockerized_use_sudo`: Use `sudo` for Docker - -### Quickstart recipes - -| Goal | Command | -| ----------------------------- | ----------------------------------------------------------------------------- | -| Render Mermaid diagram | `mermaid_wrapper.py -i diagram.mmd -o diagram.png` | -| Use input as output (default) | `mermaid_wrapper.py -i diagram.mmd` | -| Rebuild container | `mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_force_rebuild` | -| Use `sudo` for Docker | `mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_use_sudo` | +### Examples -### CLI flags cheatsheet +- TODO + ```bash + > mermaid_wrapper.py --input flowchart.mmd --output flowchart.png + ``` -| Flag | Purpose | Notes | -| ---------------------------- | ---------------------------------- | -------------------------- | -| `-i / --input` | Path to input `.mmd` or `.md` file | **required** | -| `-o / --output` | Output image file | Defaults to input filename | -| `--dockerized_force_rebuild` | Force Docker image rebuild | Optional | -| `--dockerized_use_sudo` | Run Docker with `sudo` | Optional | -| `-v / --verbosity` | Logging verbosity | Default: INFO | +- Automatically sets output to match input name if `--output` is omitted ---- +- Mermaid diagram + ``` + > mermaid_wrapper.py -i diagram.mmd -o diagram.png + ``` +- Use input as output (default) + ``` + > mermaid_wrapper.py -i diagram.mmd + ``` +- Rebuild container + ``` + > mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_force_rebuild + ``` +- Use `sudo` for Docker + ``` + > mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_use_sudo + ``` -## 13. Pandoc Renderer — `dockerized_pandoc.py` +## dockerized_pandoc.py ### What it does -Converts documents using `pandoc` inside a Docker container. -Supports output to Beamer slides, PDFs, and more with custom CLI flags. +- Converts documents using `pandoc` inside a Docker container +- Supports output to Beamer slides, PDFs, and more with custom CLI flags. -> ```bash +```bash > pandoc_wrapper.py --input notes.md --output slides.pdf -- docker_args... -> ``` - -Internally builds a Docker container and passes the full `pandoc` command -string. +``` -### Most used flags +- Internally builds a Docker container and passes the full `pandoc` command + string. -- `--input`: source file (e.g., `.md`, `.txt`) -- `--output`: output file (e.g., `.pdf`, `.html`) -- `--container_type`: use `pandoc_only`, `pandoc_latex`, or `pandoc_texlive` -- `--dockerized_force_rebuild`: rebuild image from scratch -- `--dockerized_use_sudo`: run Docker with `sudo` +### Example -### Quickstart recipes +- Convert Markdown to PDF + ``` + > pandoc_wrapper.py --input notes.md --output notes.pdf --container_type pandoc_latex + ``` +- Convert to Beamer slides + ``` + > pandoc_wrapper.py --input slides.md --output slides.pdf --container_type pandoc_latex -- -t beamer + ``` +- Rebuild Docker image + ``` + > pandoc_wrapper.py --input notes.md --output notes.pdf --dockerized_force_rebuild + ``` +- Run with sudo + ``` + > pandoc_wrapper.py --input notes.md --output notes.pdf --dockerized_use_sudo + ``` -| Goal | Command | -| ------------------------ | ---------------------------------------------------------------------------------------------------- | -| Convert Markdown to PDF | `pandoc_wrapper.py --input notes.md --output notes.pdf --container_type pandoc_latex` | -| Convert to Beamer slides | `pandoc_wrapper.py --input slides.md --output slides.pdf --container_type pandoc_latex -- -t beamer` | -| Rebuild Docker image | `pandoc_wrapper.py --input notes.md --output notes.pdf --dockerized_force_rebuild` | -| Run with sudo | `pandoc_wrapper.py --input notes.md --output notes.pdf --dockerized_use_sudo` | +## `dockerized_prettier.py` -### CLI flags cheat‑sheet +### What it does -| Flag | Purpose | Notes | -| ---------------------------- | ------------------------------------------------------ | ---------------------- | -| `--input` | Input source file for Pandoc | **required** | -| `--output` | Output file path | Defaults to input name | -| `--data_dir` | Additional resource/data path | Optional | -| `--container_type` | Docker image type: `pandoc_only`, `pandoc_latex`, etc. | Default: `pandoc_only` | -| `--dockerized_force_rebuild` | Force rebuild of Docker image | Optional | -| `--dockerized_use_sudo` | Use `sudo` for Docker execution | Optional | -| `-v / --verbosity` | Logging level | Default: INFO | +- Formats text files (`.md`, `.txt`, `.tex`, etc.) using Prettier within a Docker + container +- Avoids environment-specific issues and ensures consistent formatting. +- Supports full Prettier CLI flexibility via passthrough of additional options. ---- + > ```bash + > dockerized_prettier.py --parser markdown --write test.md + > ``` -## 14. Prettier Formatter — `dockerized_prettier.py` -### What it does +### Examples -Formats text files (`.md`, `.txt`, `.tex`, etc.) using Prettier within a Docker -container. -Avoids environment-specific issues and ensures consistent formatting. +- Format a Markdown file + ``` + > dockerized_prettier.py --parser markdown --write test.md + ``` +- Use `sudo` for Docker execution + ``` + > dockerized_prettier.py --use_sudo --parser markdown --write test.md + ``` +- Rebuild the Docker image + ``` + > dockerized_prettier.py --dockerized_force_rebuild --parser markdown --write test.md + ``` +- Change indentation and wrap style + ``` + dockerized_prettier.py --parser markdown --tab-width 4 --prose-wrap always --write test.md + ``` -> ```bash -> dockerized_prettier.py --parser markdown --write test.md -> ``` +### Interface -Supports full Prettier CLI flexibility via passthrough of additional options. +- Interface + ``` + > dockerized_prettier.py -h + usage: dockerized_prettier.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] + [--dockerized_force_rebuild] + [--dockerized_use_sudo] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] -### Most used flags + Run `prettier` inside a Docker container to ensure consistent formatting across + different environments. -- `--parser`: Prettier parser (e.g. `markdown`) -- `--write`: Apply formatting in-place -- `--tab-width`: Number of spaces per indentation level -- `--dockerized_force_rebuild`: Force rebuild of Docker container -- `--dockerized_use_sudo`: Use `sudo` for Docker commands + This script builds the container dynamically if necessary and formats the + specified file using the provided `prettier` options. -### Quickstart recipes + Examples + # Basic usage: + > dockerized_prettier.py --parser markdown --prose-wrap always --write --tab-width 2 test.md -| Goal | Command | -| --------------------------------- | -------------------------------------------------------------------------------------------- | -| Format a Markdown file | `dockerized_prettier.py --parser markdown --write test.md` | -| Use `sudo` for Docker execution | `dockerized_prettier.py --use_sudo --parser markdown --write test.md` | -| Rebuild the Docker image | `dockerized_prettier.py --dockerized_force_rebuild --parser markdown --write test.md` | -| Change indentation and wrap style | `dockerized_prettier.py --parser markdown --tab-width 4 --prose-wrap always --write test.md` | + # Use sudo for Docker commands: + > dockerized_prettier.py --use_sudo --parser markdown --prose-wrap always --write --tab-width 2 test.md -### CLI flags cheatsheet + # Set logging verbosity: + > dockerized_prettier.py -v DEBUG --parser markdown --prose-wrap always --write --tab-width 2 test.md -| Flag | Purpose | Notes | -| ---------------------------- | ----------------------------------------------------- | ------------------------------------- | -| `-i / --input` | Input file path | Required | -| `-o / --output` | Output file path | Optional (defaults to input) | -| `--parser` | Prettier parser type (e.g. `markdown`, `babel`, etc.) | Required via passthrough | -| `--write` | Format and overwrite input file | Common usage flag | -| `--tab-width` | Number of spaces per tab | Optional, defaults to Prettier config | -| `--dockerized_force_rebuild` | Force Docker image rebuild | Optional | -| `--dockerized_use_sudo` | Use `sudo` for Docker commands | Optional | -| `-v / --verbosity` | Logging level | Default: INFO | + # Process a file: + > cat test.md + - a + - b + - c + > dockerized_prettier.py --parser markdown --prose-wrap always --write --tab-width 2 test.md ---- + options: + -h, --help show this help message and exit + -i IN_FILE_NAME, --in_file_name IN_FILE_NAME + Input file or `-` for stdin + -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME + Output file or `-` for stdout + --dockerized_force_rebuild + Force to rebuild the Docker container + --dockerized_use_sudo + Use sudo inside the container + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level + ``` -## 15. MacOS screenshot helper — `save_screenshot.py` +## `save_screenshot.py` ### What it does 1. Prompts you to select a screen region (`⌘ + Ctrl + 4`). 2. Saves it as `screenshot.YYYY‑MM‑DD_HH‑MM‑SS.png` (or your chosen name). 3. Prints and copies the Markdown embed ``. - -### Flags - -| Flag | Purpose | -| --------------------- | ---------------------------------------- | -| `--dst_dir DIR` | Target directory (e.g. `notes/figures`). | -| `--filename NAME.png` | Override default timestamped name. | -| `--override` | Allow clobbering an existing file. | - ---- From 729da51ec76e0592df8271c199333879ae828747 Mon Sep 17 00:00:00 2001 From: Aayush Date: Thu, 5 Jun 2025 17:30:26 -0500 Subject: [PATCH 16/26] HelpersTask741: Adding How-to-guide template MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.template_how_to_guide.md | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 docs/tools/documentation_toolchain/all.template_how_to_guide.md diff --git a/docs/tools/documentation_toolchain/all.template_how_to_guide.md b/docs/tools/documentation_toolchain/all.template_how_to_guide.md new file mode 100644 index 000000000..1697097b9 --- /dev/null +++ b/docs/tools/documentation_toolchain/all.template_how_to_guide.md @@ -0,0 +1,113 @@ +# Template for How-To-Guide +This is a high‑level ready-to-use template to document scripts and how to use them. + + +* [Tool/Script Name] + * [`script name.py`](#script-namepy) + + [What it does](#what-it-does) + + [Supported File types and Code Blocks](#supported-file-types-and-code-blocks) + + [Quickstart Recipes](#quickstart-recipes) + + [Flag Options](#flag-options) + + [Examples](#examples) + + * [`replicate above structure for multiple files`] + + + +### script name.py + +### What it does +- Explain in a few lines the objective of the document/script and what it aims to achieve. +Sample reference: +- This script auto renders figures by + - Detecting fenced code blocks (PlantUML, Mermaid, TikZ, Graphviz, ...) + - Rendering them into images calling the appropriate tool + - Commenting them out the block + - Inlining a `![](img)` markup + +### Supported File types and Code Blocks +List the file extensions or formats your tool/script can handle. Mention how each is processed. + +Reference Examples: +- Create a new Markdown file with rendered images: +> render_images.py -i ABC.md -o XYZ.md --action render --run_dockerized + +- Render images in place in the original Markdown file: +> render_images.py -i ABC.md --action render --run_dockerized + +- Render images in place in the original LaTeX file: +> render_images.py -i ABC.tex --action render --run_dockerized + +- Open rendered images from a Markdown file in HTML to preview: +> render_images.py -i ABC.md --action open --run_dockerized + +### Quickstart Recipes + +Show some example codes for running the scripts. Explain what the input and output will look like. + +Write all the example codes within ``` ``` blocks. + +Example: +- Render the images in a text file + ```bash + > render_images.py -i \ + -o lesson9.images.txt --run_dockerized + ``` + +### Flag Options +Describe each available flag, its purpose, and demonstrate how to use it in the CLI with examples. + +Reference: + + -h, --help show this help message and exit + + -i IN_FILE_NAME, --in_file_name IN_FILE_NAME + Path to the input file + + -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME + Path to the output file + + --action {open,render} + Actions to execute + + --skip_action {open,render} + Actions to skip + + --all Run all the actions () + + --dry_run Update the file but do not render images + + --dockerized_force_rebuild + Force to rebuild the Docker container + + --dockerized_use_sudo + Use sudo inside the container + + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level + +### Examples + +- Render to a new file + + ```bash + > render_images.py -i lesson.md -o lesson.rendered.md --action render --run_dockerized + ``` + +- Render in‑place (Markdown or LaTeX) + + ```bash + > render_images.py -i lesson.md --action render --run_dockerized + ``` + +- HTML preview of already‑rendered images + + ```bash + > render_images.py -i lesson.md --action open --run_dockerized + ``` + +- Dry‑run (test parsing / comments only) + ```bash + > render_images.py -i lesson.md -o /tmp/out.md --dry_run + ``` + From c7cd5816c706f649f222783bff747ee2530ba1d1 Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 30 Jun 2025 19:42:47 -0500 Subject: [PATCH 17/26] HelpersTask741: Modifying the toolchain documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 110 +++++++++--------- 1 file changed, 54 insertions(+), 56 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 56155346c..ba1c146cd 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -1,57 +1,57 @@ - -- [Notes Documentation Toolchain](#notes-documentation-toolchain) - * [notes_to_pdf.py](#notes_to_pdfpy) - + [What it does](#what-it-does) - + [Quickstart recipes](#quickstart-recipes) - + [CLI flags cheatsheet](#cli-flags-cheatsheet) - + [Worked examples](#worked-examples) - * [render_images.py](#render_imagespy) - + [Supported File types and Code blocks](#supported-file-types-and-code-blocks) - + [Quick Start Recipes](#quick-start-recipes) - + [Flags](#flags) - * [`lint_notes.py`](#lint_notespy) - + [Quickstart recipes](#quickstart-recipes-1) - + [Flags](#flags-1) - * [`extract_notebook_images.py`](#extract_notebook_imagespy) - + [Flag Options](#flag-options) - * [5. LLM Powered Transforms — `llm_transform.py`](#5-llm-powered-transforms--llm_transformpy) - + [Minimum viable command](#minimum-viable-command) - + [Finding available prompts](#finding-available-prompts) - + [Flags](#flags-2) - + [Example recipes](#example-recipes) - * [`run_pandoc.py`](#run_pandocpy) - + [What the script does](#what-the-script-does) - + [Quickstart commands](#quickstart-commands) - + [Flags](#flags-3) - * [`transform_notes.py`](#transform_notespy) - + [What it does](#what-it-does-1) - + [Example of Supported Actions](#example-of-supported-actions) - + [Examples](#examples) - + [Flags](#flags-4) - * [`extract_headers_from_markdown.py`](#extract_headers_from_markdownpy) - + [Goal](#goal) - + [Examples](#examples-1) - * [`dockerized_tikz_to_bitmap.py`](#dockerized_tikz_to_bitmappy) - + [Examples](#examples-2) - * [`dockerized_graphviz.py`](#dockerized_graphvizpy) - + [What it does](#what-it-does-2) - + [Most used flags](#most-used-flags) - + [Quickstart recipes](#quickstart-recipes-2) - * [dockerized_latex.py](#dockerized_latexpy) - + [What it does](#what-it-does-3) - + [Quickstart recipes](#quickstart-recipes-3) - * [dockerized_mermaid.py](#dockerized_mermaidpy) - + [What it does](#what-it-does-4) - * [dockerized_pandoc.py](#dockerized_pandocpy) - + [What it does](#what-it-does-5) - + [Quickstart recipes](#quickstart-recipes-4) - * [`dockerized_prettier.py`](#dockerized_prettierpy) - + [What it does](#what-it-does-6) - + [Quickstart recipes](#quickstart-recipes-5) - + [Interface](#interface) - * [`save_screenshot.py`](#save_screenshotpy) - + [What it does](#what-it-does-7) +# [Notes Documentation Toolchain] + + - [`notes_to_pdf.py`](#notes_to_pdfpy) + - [What it does](#what-it-does) + - [Quickstart recipes](#quickstart-recipes) + - [CLI flags cheatsheet](#cli-flags-cheatsheet) + - [Worked examples](#worked-examples) + - [`render_images.py`](#render_imagespy) + - [Supported File types and Code blocks](#supported-file-types-and-code-blocks) + - [Quick Start Recipes](#quick-start-recipes) + - [Flags](#flags) + - [`lint_notes.py`](#lint_notespy) + - [Quickstart recipes](#quickstart-recipes-1) + - [Flags](#flags-1) + - [`extract_notebook_images.py`](#extract_notebook_imagespy) + - [Flag Options](#flag-options) + - [LLM Powered Transforms — `llm_transform.py`](#llm-powered-transforms--llm_transformpy) + - [Minimum viable command](#minimum-viable-command) + - [Finding available prompts](#finding-available-prompts) + - [Flags](#flags-2) + - [Example recipes](#example-recipes) + - [`run_pandoc.py`](#run_pandocpy) + - [What the script does](#what-the-script-does) + - [Quickstart commands](#quickstart-commands) + - [Flags](#flags-3) + - [`transform_notes.py`](#transform_notespy) + - [What it does](#what-it-does-1) + - [Example of Supported Actions](#example-of-supported-actions) + - [Examples](#examples) + - [Flags](#flags-4) + - [`extract_headers_from_markdown.py`](#extract_headers_from_markdownpy) + - [Goal](#goal) + - [Examples](#examples-1) + - [`dockerized_tikz_to_bitmap.py`](#dockerized_tikz_to_bitmappy) + - [Examples](#examples-2) + - [`dockerized_graphviz.py`](#dockerized_graphvizpy) + - [What it does](#what-it-does-2) + - [Most used flags](#most-used-flags) + - [Quickstart recipes](#quickstart-recipes-2) + - [`dockerized_latex.py`](#dockerized_latexpy) + - [What it does](#what-it-does-3) + - [Quickstart recipes](#quickstart-recipes-3) + - [`dockerized_mermaid.py`](#dockerized_mermaidpy) + - [What it does](#what-it-does-4) + - [`dockerized_pandoc.py`](#dockerized_pandocpy) + - [What it does](#what-it-does-5) + - [Quickstart recipes](#quickstart-recipes-4) + - [`dockerized_prettier.py`](#dockerized_prettierpy) + - [What it does](#what-it-does-6) + - [Quickstart recipes](#quickstart-recipes-5) + - [Interface](#interface) + - [`save_screenshot.py`](#save_screenshotpy) + - [What it does](#what-it-does-7) @@ -60,8 +60,6 @@ - This is a high‑level guide to the helper scripts that turn raw `.txt` notes into polished PDFs, slide decks, and more. -// TODO(\*): Is it worth to report the flags? It's difficult to maintain - ## notes_to_pdf.py ### What it does @@ -726,6 +724,6 @@ The supported File types and code blocks are: ### What it does -1. Prompts you to select a screen region (`⌘ + Ctrl + 4`). +1. Prompts you to select a screen region (`⌘ - Ctrl - 4`). 2. Saves it as `screenshot.YYYY‑MM‑DD_HH‑MM‑SS.png` (or your chosen name). 3. Prints and copies the Markdown embed ``. From 671c37e74362851b34372551b8728119861fdaf1 Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 30 Jun 2025 19:49:13 -0500 Subject: [PATCH 18/26] HelpersTask741: Modifying the toolchain documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 283 ++++++++++-------- 1 file changed, 150 insertions(+), 133 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index ba1c146cd..b845d6cdf 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -1,57 +1,64 @@ -# [Notes Documentation Toolchain] - - - [`notes_to_pdf.py`](#notes_to_pdfpy) - - [What it does](#what-it-does) - - [Quickstart recipes](#quickstart-recipes) - - [CLI flags cheatsheet](#cli-flags-cheatsheet) - - [Worked examples](#worked-examples) - - [`render_images.py`](#render_imagespy) - - [Supported File types and Code blocks](#supported-file-types-and-code-blocks) - - [Quick Start Recipes](#quick-start-recipes) - - [Flags](#flags) - - [`lint_notes.py`](#lint_notespy) - - [Quickstart recipes](#quickstart-recipes-1) - - [Flags](#flags-1) - - [`extract_notebook_images.py`](#extract_notebook_imagespy) - - [Flag Options](#flag-options) - - [LLM Powered Transforms — `llm_transform.py`](#llm-powered-transforms--llm_transformpy) - - [Minimum viable command](#minimum-viable-command) - - [Finding available prompts](#finding-available-prompts) - - [Flags](#flags-2) - - [Example recipes](#example-recipes) - - [`run_pandoc.py`](#run_pandocpy) - - [What the script does](#what-the-script-does) - - [Quickstart commands](#quickstart-commands) - - [Flags](#flags-3) - - [`transform_notes.py`](#transform_notespy) - - [What it does](#what-it-does-1) - - [Example of Supported Actions](#example-of-supported-actions) - - [Examples](#examples) - - [Flags](#flags-4) - - [`extract_headers_from_markdown.py`](#extract_headers_from_markdownpy) - - [Goal](#goal) - - [Examples](#examples-1) - - [`dockerized_tikz_to_bitmap.py`](#dockerized_tikz_to_bitmappy) - - [Examples](#examples-2) - - [`dockerized_graphviz.py`](#dockerized_graphvizpy) - - [What it does](#what-it-does-2) - - [Most used flags](#most-used-flags) - - [Quickstart recipes](#quickstart-recipes-2) - - [`dockerized_latex.py`](#dockerized_latexpy) - - [What it does](#what-it-does-3) - - [Quickstart recipes](#quickstart-recipes-3) - - [`dockerized_mermaid.py`](#dockerized_mermaidpy) - - [What it does](#what-it-does-4) - - [`dockerized_pandoc.py`](#dockerized_pandocpy) - - [What it does](#what-it-does-5) - - [Quickstart recipes](#quickstart-recipes-4) - - [`dockerized_prettier.py`](#dockerized_prettierpy) - - [What it does](#what-it-does-6) - - [Quickstart recipes](#quickstart-recipes-5) - - [Interface](#interface) - - [`save_screenshot.py`](#save_screenshotpy) - - [What it does](#what-it-does-7) + +- [Notes Documentation Toolchain](#notes-documentation-toolchain) + * [notes_to_pdf.py](#notes_to_pdfpy) + + [What it does](#what-it-does) +- [From scratch with TOC:](#from-scratch-with-toc) +- [For interactive mode:](#for-interactive-mode) +- [Check that can be compiled:](#check-that-can-be-compiled) + + [Examples](#examples) + * [render_images.py](#render_imagespy) + + [What it does](#what-it-does-1) + + [Examples](#examples-1) + + [Interface](#interface) + * [`lint_notes.py`](#lint_notespy) + + [What it does](#what-it-does-2) + + [Examples](#examples-2) + + [Interface](#interface-1) + * [`extract_notebook_images.py`](#extract_notebook_imagespy) + + [What it does](#what-it-does-3) + + [Example](#example) + + [Interface](#interface-2) + * [`llm_transform.py`](#llm_transformpy) + + [What it does](#what-it-does-4) + + [Examples](#examples-3) + + [Interface](#interface-3) + * [`run_pandoc.py`](#run_pandocpy) + + [What it does](#what-it-does-5) + + [Example](#example-1) + + [Interface](#interface-4) + * [`transform_notes.py`](#transform_notespy) + + [What it does](#what-it-does-6) + + [Examples](#examples-4) + + [Interface](#interface-5) + * [`extract_headers_from_markdown.py`](#extract_headers_from_markdownpy) + + [What it does](#what-it-does-7) + + [Examples](#examples-5) + * [`dockerized_tikz_to_bitmap.py`](#dockerized_tikz_to_bitmappy) + + [Examples](#examples-6) + * [`dockerized_graphviz.py`](#dockerized_graphvizpy) + + [What it does](#what-it-does-8) + + [Interface](#interface-6) + + [Examples](#examples-7) + * [dockerized_latex.py](#dockerized_latexpy) + + [What it does](#what-it-does-9) + + [Examples](#examples-8) + * [dockerized_mermaid.py](#dockerized_mermaidpy) + + [What it does](#what-it-does-10) + + [Examples](#examples-9) + * [`dockerized_pandoc.py`](#dockerized_pandocpy) + + [What it does](#what-it-does-11) + + [Example](#example-2) + * [`dockerized_prettier.py`](#dockerized_prettierpy) + + [What it does](#what-it-does-12) + + [Examples](#examples-10) + + [Interface](#interface-7) +- [Basic usage:](#basic-usage) +- [Use sudo for Docker commands:](#use-sudo-for-docker-commands) +- [Set logging verbosity:](#set-logging-verbosity) +- [Process a file:](#process-a-file) + * [`save_screenshot.py`](#save_screenshotpy) + + [What it does](#what-it-does-13) @@ -66,6 +73,7 @@ - Convert plain‑text notes into polished **PDF, HTML, or Beamer slides** with a single command: + ```bash > notes_to_pdf.py --input --output --type [pdf|html|slides] ``` @@ -84,53 +92,44 @@ [--all] [--dockerized_force_rebuild] [--dockerized_use_sudo] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + ``` Convert a txt file into a PDF / HTML / slides using `pandoc`. # From scratch with TOC: + > notes_to_pdf.py -a pdf --input ... # For interactive mode: + > notes_to_pdf.py -a pdf --no_cleanup_before --no_cleanup --input ... # Check that can be compiled: - > notes_to_pdf.py -a pdf --no_toc --no_open_pdf --input ... - > notes_to_pdf.py --input notes/IN_PROGRESS/math.The_hundred_page_ML_book.Burkov.2019.txt -t pdf --no_cleanup --no_cleanup_before --no_run_latex_again --no_open + > notes_to_pdf.py -a pdf --no_toc --no_open_pdf --input ... - options: - -h, --help show this help message and exit - -i INPUT, --input INPUT - -o OUTPUT, --output OUTPUT - Output file - --type {pdf,html,slides} - Type of output to generate - --filter_by_header FILTER_BY_HEADER - Filter by header - --filter_by_lines FILTER_BY_LINES - Filter by lines (e.g., `0:10`, `1:None`, `None:10`) - --script SCRIPT Bash script to generate with all the executed sub- - commands - --preview_actions Print the actions and exit - --toc_type {none,pandoc_native,navigation} - --no_run_latex_again - --debug_on_error - --gdrive_dir GDRIVE_DIR - Directory where to save the output to share on Google - Drive - --use_host_tools Use the host tools instead of the dockerized ones - --action {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} - Actions to execute - --skip_action {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} - Actions to skip - --all Run all the actions (cleanup_before preprocess_notes - render_images run_pandoc open cleanup_after) - --dockerized_force_rebuild - Force to rebuild the Docker container - --dockerized_use_sudo - Use sudo inside the container - -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} - Set the logging level + > notes_to_pdf.py --input + > notes/IN_PROGRESS/math.The_hundred_page_ML_book.Burkov.2019.txt -t pdf + > --no_cleanup --no_cleanup_before --no_run_latex_again --no_open + + options: -h, --help show this help message and exit -i INPUT, --input INPUT -o + OUTPUT, --output OUTPUT Output file --type {pdf,html,slides} Type of output to + generate --filter_by_header FILTER_BY_HEADER Filter by header + --filter_by_lines FILTER_BY_LINES Filter by lines (e.g., `0:10`, `1:None`, + `None:10`) --script SCRIPT Bash script to generate with all the executed sub- + commands --preview_actions Print the actions and exit --toc_type + {none,pandoc_native,navigation} --no_run_latex_again --debug_on_error + --gdrive_dir GDRIVE_DIR Directory where to save the output to share on Google + Drive --use_host_tools Use the host tools instead of the dockerized ones + --action + {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} + Actions to execute --skip_action + {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} + Actions to skip --all Run all the actions (cleanup_before preprocess_notes + render_images run_pandoc open cleanup_after) --dockerized_force_rebuild Force + to rebuild the Docker container --dockerized_use_sudo Use sudo inside the + container -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level + ``` ``` ### Examples @@ -157,6 +156,7 @@ - Slides with navigation breadcrumbs, keeping intermediate files for inspection // TODO(indro): `--toc_type navigation` fails because of the preprocess step. + ```bash > notes_to_pdf.py \ --input MSML610/Lesson5-Theory_Statistical_learning.txt \ @@ -169,6 +169,7 @@ - Focus on a subsection, compiling only from line 362 to EOF for a fast iteration when debugging slides + ```bash > notes_to_pdf.py \ --input Lesson8-Reasoning_over_time.txt \ @@ -200,18 +201,19 @@ ``` The supported File types and code blocks are: - - File extension: `.md`, `.txt` - - Rendering syntax allowed: - - `plantuml` - - `mermaid` - - `graphviz` - - `tikz` - - `latex` - - Output embeds as: `` - - File extension: `.tex` - - Rendering syntax allowed: - - Same tags (TikZ & LaTeX especially) - - Output embeds as: `\includegraphics{...}` + +- File extension: `.md`, `.txt` + - Rendering syntax allowed: + - `plantuml` + - `mermaid` + - `graphviz` + - `tikz` + - `latex` + - Output embeds as: `` +- File extension: `.tex` + - Rendering syntax allowed: + - Same tags (TikZ & LaTeX especially) + - Output embeds as: `\includegraphics{...}` ### Examples @@ -287,7 +289,7 @@ The supported File types and code blocks are: Use sudo inside the container -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level - ``` + ``` ## `lint_notes.py` @@ -345,24 +347,27 @@ The supported File types and code blocks are: ### What it does -- Apply a GPT‑style transformation (rewrite, summarise, critique code, convert to - slides, etc.) to any text file _without_ leaving the terminal / editor. +- Apply a GPT‑style transformation (rewrite, summarise, critique code, convert + to slides, etc.) to any text file _without_ leaving the terminal / editor. - **Note**: You need to have an `OPENAI_API_KEY` and an internet connection. ### Examples - TODO + ```bash llm_transform.py -i draft.txt -o polished.txt -p rewrite_clearer ``` - Finding available prompts + ```bash llm_transform.py -p list -i - -o - ``` - Turn a code file into a review checklist + ```bash > llm_transform.py -i foo.py -o cfile -p code_review vim cfile @@ -407,7 +412,8 @@ The supported File types and code blocks are: :<,'>!run_pandoc.py -i - -o - -v CRITICAL ``` -- **Tip:** pass `-v CRITICAL` to silence helper logging when piping into editors. +- **Tip:** pass `-v CRITICAL` to silence helper logging when piping into + editors. ### Interface @@ -518,6 +524,7 @@ The supported File types and code blocks are: - Converts a Graphviz `.dot` file into a `.png` image using a Dockerized container. + > ```bash > graphviz_wrapper.py --input input.dot --output output.png > ``` @@ -583,7 +590,8 @@ The supported File types and code blocks are: ### Examples -- TODO +- TODO + ```bash > mermaid_wrapper.py --input flowchart.mmd --output flowchart.png ``` @@ -607,7 +615,7 @@ The supported File types and code blocks are: > mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_use_sudo ``` -## dockerized_pandoc.py +## `dockerized_pandoc.py` ### What it does @@ -627,14 +635,17 @@ The supported File types and code blocks are: ``` > pandoc_wrapper.py --input notes.md --output notes.pdf --container_type pandoc_latex ``` + - Convert to Beamer slides ``` > pandoc_wrapper.py --input slides.md --output slides.pdf --container_type pandoc_latex -- -t beamer ``` + - Rebuild Docker image ``` > pandoc_wrapper.py --input notes.md --output notes.pdf --dockerized_force_rebuild ``` + - Run with sudo ``` > pandoc_wrapper.py --input notes.md --output notes.pdf --dockerized_use_sudo @@ -644,15 +655,14 @@ The supported File types and code blocks are: ### What it does -- Formats text files (`.md`, `.txt`, `.tex`, etc.) using Prettier within a Docker - container +- Formats text files (`.md`, `.txt`, `.tex`, etc.) using Prettier within a + Docker container - Avoids environment-specific issues and ensures consistent formatting. - Supports full Prettier CLI flexibility via passthrough of additional options. - > ```bash + ```bash > dockerized_prettier.py --parser markdown --write test.md - > ``` - + ``` ### Examples @@ -660,14 +670,17 @@ The supported File types and code blocks are: ``` > dockerized_prettier.py --parser markdown --write test.md ``` + - Use `sudo` for Docker execution ``` > dockerized_prettier.py --use_sudo --parser markdown --write test.md ``` + - Rebuild the Docker image ``` > dockerized_prettier.py --dockerized_force_rebuild --parser markdown --write test.md ``` + - Change indentation and wrap style ``` dockerized_prettier.py --parser markdown --tab-width 4 --prose-wrap always --write test.md @@ -682,42 +695,46 @@ The supported File types and code blocks are: [--dockerized_force_rebuild] [--dockerized_use_sudo] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + ``` - Run `prettier` inside a Docker container to ensure consistent formatting across - different environments. + Run `prettier` inside a Docker container to ensure consistent formatting + across different environments. This script builds the container dynamically if necessary and formats the specified file using the provided `prettier` options. Examples + # Basic usage: - > dockerized_prettier.py --parser markdown --prose-wrap always --write --tab-width 2 test.md + + > dockerized_prettier.py --parser markdown --prose-wrap always --write + > --tab-width 2 test.md # Use sudo for Docker commands: - > dockerized_prettier.py --use_sudo --parser markdown --prose-wrap always --write --tab-width 2 test.md + + > dockerized_prettier.py --use_sudo --parser markdown --prose-wrap always + > --write --tab-width 2 test.md # Set logging verbosity: - > dockerized_prettier.py -v DEBUG --parser markdown --prose-wrap always --write --tab-width 2 test.md + + > dockerized_prettier.py -v DEBUG --parser markdown --prose-wrap always + > --write --tab-width 2 test.md # Process a file: - > cat test.md - - a - - b - - c - > dockerized_prettier.py --parser markdown --prose-wrap always --write --tab-width 2 test.md - options: - -h, --help show this help message and exit - -i IN_FILE_NAME, --in_file_name IN_FILE_NAME - Input file or `-` for stdin - -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME - Output file or `-` for stdout - --dockerized_force_rebuild - Force to rebuild the Docker container - --dockerized_use_sudo - Use sudo inside the container - -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} - Set the logging level + > cat test.md + - A + - B - c + > dockerized_prettier.py --parser markdown --prose-wrap always --write + > --tab-width 2 test.md + + options: -h, --help show this help message and exit -i IN_FILE_NAME, + --in_file_name IN_FILE_NAME Input file or `-` for stdin -o OUT_FILE_NAME, + --out_file_name OUT_FILE_NAME Output file or `-` for stdout + --dockerized_force_rebuild Force to rebuild the Docker container + --dockerized_use_sudo Use sudo inside the container -v + {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level + ``` ``` ## `save_screenshot.py` From 77252ff7bcb27891c503947d78030d78fa776518 Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 30 Jun 2025 20:58:02 -0500 Subject: [PATCH 19/26] HelpersTask741: Adding examples and interface DIY for documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 397 ++++++++++++------ 1 file changed, 279 insertions(+), 118 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index b845d6cdf..76df61e63 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -1,72 +1,70 @@ +# Documentation Toolchain - How To Guide +This is a high‑level guide to the helper scripts that turn raw `.txt` notes into polished PDFs, slide decks, and more. + -- [Notes Documentation Toolchain](#notes-documentation-toolchain) - * [notes_to_pdf.py](#notes_to_pdfpy) - + [What it does](#what-it-does) -- [From scratch with TOC:](#from-scratch-with-toc) -- [For interactive mode:](#for-interactive-mode) -- [Check that can be compiled:](#check-that-can-be-compiled) - + [Examples](#examples) - * [render_images.py](#render_imagespy) - + [What it does](#what-it-does-1) - + [Examples](#examples-1) - + [Interface](#interface) - * [`lint_notes.py`](#lint_notespy) - + [What it does](#what-it-does-2) - + [Examples](#examples-2) - + [Interface](#interface-1) - * [`extract_notebook_images.py`](#extract_notebook_imagespy) - + [What it does](#what-it-does-3) - + [Example](#example) - + [Interface](#interface-2) - * [`llm_transform.py`](#llm_transformpy) - + [What it does](#what-it-does-4) - + [Examples](#examples-3) - + [Interface](#interface-3) - * [`run_pandoc.py`](#run_pandocpy) - + [What it does](#what-it-does-5) - + [Example](#example-1) - + [Interface](#interface-4) - * [`transform_notes.py`](#transform_notespy) - + [What it does](#what-it-does-6) - + [Examples](#examples-4) - + [Interface](#interface-5) - * [`extract_headers_from_markdown.py`](#extract_headers_from_markdownpy) - + [What it does](#what-it-does-7) - + [Examples](#examples-5) - * [`dockerized_tikz_to_bitmap.py`](#dockerized_tikz_to_bitmappy) - + [Examples](#examples-6) - * [`dockerized_graphviz.py`](#dockerized_graphvizpy) - + [What it does](#what-it-does-8) - + [Interface](#interface-6) - + [Examples](#examples-7) - * [dockerized_latex.py](#dockerized_latexpy) - + [What it does](#what-it-does-9) - + [Examples](#examples-8) - * [dockerized_mermaid.py](#dockerized_mermaidpy) - + [What it does](#what-it-does-10) - + [Examples](#examples-9) - * [`dockerized_pandoc.py`](#dockerized_pandocpy) - + [What it does](#what-it-does-11) - + [Example](#example-2) - * [`dockerized_prettier.py`](#dockerized_prettierpy) - + [What it does](#what-it-does-12) - + [Examples](#examples-10) - + [Interface](#interface-7) -- [Basic usage:](#basic-usage) -- [Use sudo for Docker commands:](#use-sudo-for-docker-commands) -- [Set logging verbosity:](#set-logging-verbosity) -- [Process a file:](#process-a-file) - * [`save_screenshot.py`](#save_screenshotpy) - + [What it does](#what-it-does-13) +- [`notes_to_pdf.py`](#notes_to_pdfpy) + - [What it does](#what-it-does) +- [`render_images.py`](#render_imagespy) + - [What it does](#what-it-does-1) + - [Examples](#examples-1) + - [Interface](#interface) +- [`lint_notes.py`](#lint_notespy) + - [What it does](#what-it-does-2) + - [Examples](#examples-2) + - [Interface](#interface-1) +- [`extract_notebook_images.py`](#extract_notebook_imagespy) + - [What it does](#what-it-does-3) + - [Example](#example) + - [Interface](#interface-2) +- [`llm_transform.py`](#llm_transformpy) + - [What it does](#what-it-does-4) + - [Examples](#examples-3) + - [Interface](#interface-3) +- [`run_pandoc.py`](#run_pandocpy) + - [What it does](#what-it-does-5) + - [Example](#example-1) + - [Interface](#interface-4) +- [`transform_notes.py`](#transform_notespy) + - [What it does](#what-it-does-6) + - [Examples](#examples-4) + - [Interface](#interface-5) +- [`extract_headers_from_markdown.py`](#extract_headers_from_markdownpy) + - [What it does](#what-it-does-7) + - [Examples](#examples-5) +- [`dockerized_tikz_to_bitmap.py`](#dockerized_tikz_to_bitmappy) + - [Examples](#examples-6) +- [`dockerized_graphviz.py`](#dockerized_graphvizpy) + - [What it does](#what-it-does-8) + - [Interface](#interface-6) + - [Examples](#examples-7) +- [`dockerized_latex.py`](#dockerized_latexpy) + - [What it does](#what-it-does-9) + - [Examples](#examples-8) +- [`dockerized_mermaid.py`](#dockerized_mermaidpy) + - [What it does](#what-it-does-10) + - [Examples](#examples-9) +- [`dockerized_pandoc.py`](#dockerized_pandocpy) + - [What it does](#what-it-does-11) + - [Example](#example-2) +- [`dockerized_prettier.py`](#dockerized_prettierpy) + - [What it does](#what-it-does-12) + - [Examples](#examples-10) + - [Interface](#interface-7) +- [`save_screenshot.py`](#save_screenshotpy) + - [What it does](#what-it-does-13) + +- [Examples](#examples) + - [Basic usage](#basic-usage) + - [Use sudo for Docker commands](#use-sudo-for-docker-commands) + - [Set logging verbosity](#set-logging-verbosity) + - [Process a file:](#process-a-file) + - [From scratch with TOC:](#from-scratch-with-toc) + - [For interactive mode:](#for-interactive-mode) + - [Check that can be compiled:](#check-that-can-be-compiled) -# Notes Documentation Toolchain - -- This is a high‑level guide to the helper scripts that turn raw `.txt` notes - into polished PDFs, slide decks, and more. - ## notes_to_pdf.py ### What it does @@ -94,44 +92,6 @@ [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] ``` - Convert a txt file into a PDF / HTML / slides using `pandoc`. - - # From scratch with TOC: - - > notes_to_pdf.py -a pdf --input ... - - # For interactive mode: - - > notes_to_pdf.py -a pdf --no_cleanup_before --no_cleanup --input ... - - # Check that can be compiled: - - > notes_to_pdf.py -a pdf --no_toc --no_open_pdf --input ... - - > notes_to_pdf.py --input - > notes/IN_PROGRESS/math.The_hundred_page_ML_book.Burkov.2019.txt -t pdf - > --no_cleanup --no_cleanup_before --no_run_latex_again --no_open - - options: -h, --help show this help message and exit -i INPUT, --input INPUT -o - OUTPUT, --output OUTPUT Output file --type {pdf,html,slides} Type of output to - generate --filter_by_header FILTER_BY_HEADER Filter by header - --filter_by_lines FILTER_BY_LINES Filter by lines (e.g., `0:10`, `1:None`, - `None:10`) --script SCRIPT Bash script to generate with all the executed sub- - commands --preview_actions Print the actions and exit --toc_type - {none,pandoc_native,navigation} --no_run_latex_again --debug_on_error - --gdrive_dir GDRIVE_DIR Directory where to save the output to share on Google - Drive --use_host_tools Use the host tools instead of the dockerized ones - --action - {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} - Actions to execute --skip_action - {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} - Actions to skip --all Run all the actions (cleanup_before preprocess_notes - render_images run_pandoc open cleanup_after) --dockerized_force_rebuild Force - to rebuild the Docker container --dockerized_use_sudo Use sudo inside the - container -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level - ``` - ``` - ### Examples - Compile to **Beamer slides** @@ -195,6 +155,7 @@ - Inlining a `![](img)` markup - Render the images in a text file + ```bash > render_images.py -i notes/MSML610/Lesson9-Causal_inference.txt \ -o lesson9.images.txt --run_dockerized @@ -312,14 +273,73 @@ The supported File types and code blocks are: ``` - Custom print width and selective actions + ```bash > lint_notes.py -i draft.txt -o tidy.txt -w 100 \ --action preprocess,prettier,postprocess ``` +- Custom Line width + + ``` bash + > lint_notes.py -i test.txt -o tested.txt -w 10 + ``` + ### Interface -// TODO +```bash +> lint_notes.py -h +usage: lint_notes.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] + [--type TYPE] + [--print-width PRINT_WIDTH] + [--use_dockerized_prettier] + [--use_dockerized_markdown_toc] + [--action {preprocess,prettier,postprocess,frame_chapters,refresh_toc} + | --skip_action {preprocess,prettier,postprocess,frame_chapters,refresh_toc}] + [--all] [--dockerized_force_rebuild] + [--dockerized_use_sudo] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + +Lint and prettify Markdown, text, or LaTeX note files. It supports preprocessing, +formatting via Prettier, postprocessing, chapter framing, and automatic TOC insertion. + +Usage: + +# Format a Markdown file using all actions and Dockerized tools: +> lint_notes.py -i notes.md -o notes_clean.md --use_dockerized_prettier \ + --use_dockerized_markdown_toc --all + +# Run only preprocessing and prettification: +> lint_notes.py -i draft.md -o draft_clean.md --action preprocess prettier + +# Frame chapters and update TOC only: +> lint_notes.py -i notes.md --action frame_chapters refresh_toc + +options: + -h, --help Show this help message and exit + -i IN_FILE_NAME, --in_file_name IN_FILE_NAME + Path to the input file (use '-' for stdin) + -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME + Path to the output file (use '-' for stdout) + --type TYPE File type: 'md', 'tex', or 'txt' (required for stdin) + -w, --print-width PRINT_WIDTH + Maximum line width for Prettier (default: 80) + --use_dockerized_prettier + Use Dockerized Prettier for formatting + --use_dockerized_markdown_toc + Use Dockerized markdown-toc to generate TOC + --action {preprocess,prettier,postprocess,frame_chapters,refresh_toc} + Actions to execute + --skip_action {preprocess,prettier,postprocess,frame_chapters,refresh_toc} + Actions to skip + --all Run all actions (default) + --dockerized_force_rebuild + Force rebuild of the Docker container + --dockerized_use_sudo + Use sudo inside the container + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level +``` ## `extract_notebook_images.py` @@ -332,16 +352,53 @@ The supported File types and code blocks are: ### Example -- Minimal call: +- Minimal call ```bash > extract_notebook_images.py \ --in_notebook_filename notebooks/Lesson8.ipynb \ --out_image_dir notebooks/screenshots ``` +- Enforce rebuild and use sudo commands + + ``` bash + > extract_notebook_images.py \ + --in_notebook_filename notebooks/Lesson8.ipynb \ + --out_image_dir notebooks/screenshots --dockerized_force_rebuild --dockerized_use_sudo + ``` + + ### Interface -// TODO +``` bash +usage: extract_notebook_images.py [-h] --in_notebook_filename IN_NOTEBOOK_FILENAME --out_image_dir + OUT_IMAGE_DIR [--dockerized_force_rebuild] [--dockerized_use_sudo] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + +Extract images from a Jupyter notebook by running inside a Docker container. + +This script builds the container dynamically if necessary and extracts images +from the specified Jupyter notebook using the NotebookImageExtractor module. +``` +```bash +Extract images from notebook test_images.ipynb and save them to `screenshots` +directory. +> dev_scripts_helpers/notebooks/extract_notebook_images.py -i dev_scripts_helpers/notebooks/test_images.ipynb -o dev_scripts_helpers/notebooks/screenshots +``` +``` +options: + -h, --help show this help message and exit + --in_notebook_filename IN_NOTEBOOK_FILENAME + Input notebook filename + --out_image_dir OUT_IMAGE_DIR + Output image directory + --dockerized_force_rebuild + Force to rebuild the Docker container + --dockerized_use_sudo + Use sudo inside the container + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level +``` ## `llm_transform.py` @@ -386,8 +443,57 @@ The supported File types and code blocks are: ``` ### Interface +``` bash +usage: llm_transform.py [-h] [-i IN_FILE_NAME] [-o OUT_FILE_NAME] [--debug] -p PROMPT [-f] + [--dockerized_force_rebuild] [--dockerized_use_sudo] [-c] [-b] [-s] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + +Read input from either stdin or a file, apply a specified transformation using +an LLM, and then write the output to either stdout or a file. It is +particularly useful for integrating with editors like Vim. + +The script `dockerized_llm_transform.py` is executed within a Docker container to ensure +all dependencies are met. The Docker container is built dynamically if +necessary. The script requires an OpenAI API key to be set in the environment. +``` + +``` bash +Examples +# Basic Usage +> llm_transform.py -i input.txt -o output.txt -p uppercase + +# List of transforms +> llm_transform.py -i input.txt -o output.txt -p list -// TODO +# Code review +> llm_transform.py -i dev_scripts_helpers/documentation/render_images.py -o cfile -p code_review + +# Propose refactoring +> llm_transform.py -i dev_scripts_helpers/documentation/render_images.py -o cfile -p code_propose_refactoring +``` +``` bash +options: + -h, --help show this help message and exit + -i IN_FILE_NAME, --in_file_name IN_FILE_NAME + Input file or `-` for stdin + -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME + Output file or `-` for stdout + --debug Print before/after the transform + -p PROMPT, --prompt PROMPT + Prompt to apply + -f, --fast_model Use a fast LLM model vs a high-quality one + --dockerized_force_rebuild + Force to rebuild the Docker container + --dockerized_use_sudo + Use sudo inside the container + -c, --compare Print the original and transformed + -b, --bold_first_level_bullets + Bold the first level bullets + -s, --skip-post-transforms + Skip the post-transforms + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level +``` ## `run_pandoc.py` @@ -416,19 +522,36 @@ The supported File types and code blocks are: editors. ### Interface +``` bash +usage: run_pandoc.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] [--action ACTION] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + +Run pandoc on stdin/file to stdout/file. + +- Read value from stdin/file +- Transform it using Pandoc according to different transforms + (e.g., `convert_md_to_latex`) +- Write the result to stdout/file. + +To run in vim: +``` +:'<,'>!dev_scripts/documentation/run_pandoc.py -i - -o - -v CRITICAL +``` + +This script is derived from `dev_scripts/transform_template.py`. +``` -- `-i / --input` - - Default: `-` - - Meaning: Source file or `-` for STDIN -- `-o / --output` - - Default: `-` - - Meaning: Destination file or `-` for STDOUT -- `--action` - - Default: `convert_md_to_latex` - - Meaning: Transformation to apply. Future-proofed for more actions -- `-v / --log_level` - - Default: `INFO` - - Meaning: Standard helper-library verbosity +``` bash +options: + -h, --help show this help message and exit + -i IN_FILE_NAME, --in_file_name IN_FILE_NAME + Input file or `-` for stdin + -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME + Output file or `-` for stdout + --action ACTION Action to perform + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level +``` ## `transform_notes.py` @@ -744,3 +867,41 @@ The supported File types and code blocks are: 1. Prompts you to select a screen region (`⌘ - Ctrl - 4`). 2. Saves it as `screenshot.YYYY‑MM‑DD_HH‑MM‑SS.png` (or your chosen name). 3. Prints and copies the Markdown embed ``. + + + Convert a txt file into a PDF / HTML / slides using `pandoc`. + + # From scratch with TOC: + + > notes_to_pdf.py -a pdf --input ... + + # For interactive mode: + + > notes_to_pdf.py -a pdf --no_cleanup_before --no_cleanup --input ... + + # Check that can be compiled: + + > notes_to_pdf.py -a pdf --no_toc --no_open_pdf --input ... + + > notes_to_pdf.py --input + > notes/IN_PROGRESS/math.The_hundred_page_ML_book.Burkov.2019.txt -t pdf + > --no_cleanup --no_cleanup_before --no_run_latex_again --no_open + ```bash + options: -h, --help show this help message and exit -i INPUT, --input INPUT -o + OUTPUT, --output OUTPUT Output file --type {pdf,html,slides} Type of output to + generate --filter_by_header FILTER_BY_HEADER Filter by header + --filter_by_lines FILTER_BY_LINES Filter by lines (e.g., `0:10`, `1:None`, + `None:10`) --script SCRIPT Bash script to generate with all the executed sub- + commands --preview_actions Print the actions and exit --toc_type + {none,pandoc_native,navigation} --no_run_latex_again --debug_on_error + --gdrive_dir GDRIVE_DIR Directory where to save the output to share on Google + Drive --use_host_tools Use the host tools instead of the dockerized ones + --action + {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} + Actions to execute --skip_action + {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} + Actions to skip --all Run all the actions (cleanup_before preprocess_notes + render_images run_pandoc open cleanup_after) --dockerized_force_rebuild Force + to rebuild the Docker container --dockerized_use_sudo Use sudo inside the + container -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level + ``` From 4bcbcfdd10bbee3f8025521fff30306450afcb72 Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 30 Jun 2025 21:57:55 -0500 Subject: [PATCH 20/26] HelpersTask741: Adding examples and interface DIY for documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 50 +++++++++++++++++-- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 76df61e63..1d665a123 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -411,8 +411,6 @@ options: ### Examples -- TODO - ```bash llm_transform.py -i draft.txt -o polished.txt -p rewrite_clearer ``` @@ -471,6 +469,7 @@ Examples # Propose refactoring > llm_transform.py -i dev_scripts_helpers/documentation/render_images.py -o cfile -p code_propose_refactoring ``` + ``` bash options: -h, --help show this help message and exit @@ -503,7 +502,7 @@ options: - Dispatches to a named **action** (currently only `convert_md_to_latex`). - Pushes the Pandoc output to _stdout_ or the `--output` file. -### Example +### Examples - Convert a Markdown file to LaTeX ``` @@ -522,6 +521,7 @@ options: editors. ### Interface + ``` bash usage: run_pandoc.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] [--action ACTION] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] @@ -532,8 +532,9 @@ Run pandoc on stdin/file to stdout/file. - Transform it using Pandoc according to different transforms (e.g., `convert_md_to_latex`) - Write the result to stdout/file. +``` -To run in vim: +To run in VIM: ``` :'<,'>!dev_scripts/documentation/run_pandoc.py -i - -o - -v CRITICAL ``` @@ -599,6 +600,47 @@ options: ### Interface +```bash +usage: transform_notes.py [-h] [-i IN_FILE_NAME] [-o OUT_FILE_NAME] -a ACTION [-l MAX_LEV] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + +Perform one of several transformations on a txt file, e.g., + +1) `toc`: create table of context from the current file, with 1 level + > transform_notes.py -a toc -i % -l 1 + +2) `format`: format the current file with 3 levels + :!transform_notes.py -a format -i % --max_lev 3 + > transform_notes.py -a format -i notes/ABC.txt --max_lev 3 + + - In vim + :!transform_notes.py -a format -i % --max_lev 3 + :%!transform_notes.py -a format -i - --max_lev 3 + +3) `increase`: increase level + :!transform_notes.py -a increase -i % + :%!transform_notes.py -a increase -i - + +4) `md_list_to_latex`: convert a markdown list to a latex list + :!transform_notes.py -a md_list_to_latex -i % + :%!transform_notes.py -a md_list_to_latex -i - + +- The input or output can be filename or stdin (represented by '-') +- If output file is not specified then we assume that the output file is the + same as the input + +options: + -h, --help show this help message and exit + -i IN_FILE_NAME, --in_file_name IN_FILE_NAME + Input file or `-` for stdin + -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME + Output file or `-` for stdout + -a ACTION, --action ACTION + -l MAX_LEV, --max_lev MAX_LEV + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level + ``` + ## `extract_headers_from_markdown.py` ### What it does From ba4b0d75f8f3cb6b49c4ffab8b12665ea6a1dc62 Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 30 Jun 2025 21:59:35 -0500 Subject: [PATCH 21/26] HelpersTask741: Adding examples and interface DIY for documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 1d665a123..f1491a38a 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -665,6 +665,47 @@ options: > vim -c "cfile headers.cfile" ``` +### Interface + +```bash +usage: extract_headers_from_markdown.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] + [--mode {list,headers,cfile}] [--max-level MAX_LEVEL] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + +Extract headers from a Markdown file and generate a Vim cfile. + +The script: +- processes the input Markdown file +- extracts headers up to a specified maximum level +- prints a human-readable header map +- generates an output file in a format that can be used with Vim's quickfix + feature. + +# Extract headers up to level 3 from a Markdown file and save to an output file: +> extract_headers_from_markdown.py -i input.md -o cfile --mode cfile --max-level 3 + +# Extract headers up to level 2 and print to stdout: +> extract_headers_from_markdown.py -i input.md -o - --mode headers + +# To use the generated cfile in Vim: +- Open Vim and run `:cfile output.cfile` + > vim -c "cfile cfile" + +- Navigate between headers using `:cnext` and `:cprev` + +options: + -h, --help show this help message and exit + -i IN_FILE_NAME, --in_file_name IN_FILE_NAME + Input file or `-` for stdin + -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME + Output file or `-` for stdout + --mode {list,headers,cfile} + Output mode + --max-level MAX_LEVEL + Maximum header levels to parse + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level + ``` ## `dockerized_tikz_to_bitmap.py` - Converts From da6f40861b22e447a07639cc8c5fc31b25527b65 Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 30 Jun 2025 22:09:46 -0500 Subject: [PATCH 22/26] HelpersTask741: Adding examples and interface DIY for documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 188 ++++++++++++++---- 1 file changed, 149 insertions(+), 39 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index f1491a38a..bc3b3a1d5 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -708,7 +708,8 @@ options: ``` ## `dockerized_tikz_to_bitmap.py` -- Converts +- Convert a TikZ file to a PNG image using a dockerized version of `pdflatex` and +`imagemagick`. ### Examples @@ -724,6 +725,24 @@ options: ``` - Any extra tokens after `--` are passed verbatim to `convert` +### Interface + +``` bash +usage: dockerized_tikz_to_bitmap.py [-h] -i INPUT -o OUTPUT [--dockerized_force_rebuild] + [--dockerized_use_sudo] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + +options: + -h, --help show this help message and exit + -i INPUT, --input INPUT + -o OUTPUT, --output OUTPUT + --dockerized_force_rebuild + Force to rebuild the Docker container + --dockerized_use_sudo + Use sudo inside the container + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level +``` + ## `dockerized_graphviz.py` ### What it does @@ -738,8 +757,6 @@ options: - This script serves as a thin wrapper around Dockerized Graphviz for consistent rendering across systems. -### Interface - ### Examples - Convert DOT to PNG @@ -755,6 +772,26 @@ options: > graphviz_wrapper.py -i diagram.dot -o diagram.png --dockerized_use_sudo ``` +### Interface + +``` bash +usage: dockerized_graphviz.py [-h] -i INPUT -o OUTPUT [--dockerized_force_rebuild] [--dockerized_use_sudo] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + +Convert a Graphviz dot file to a PNG image. + +options: + -h, --help show this help message and exit + -i INPUT, --input INPUT + -o OUTPUT, --output OUTPUT + --dockerized_force_rebuild + Force to rebuild the Docker container + --dockerized_use_sudo + Use sudo inside the container + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level + ``` + ## dockerized_latex.py ### What it does @@ -787,6 +824,29 @@ options: > latex_wrapper.py -i paper.tex -o paper.pdf --run_latex_again ``` +## Interface + +``` bash +usage: dockerized_latex.py [-h] -i INPUT -o OUTPUT [--run_latex_again] [--dockerized_force_rebuild] + [--dockerized_use_sudo] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + +Run `latex` inside a Docker container. + +This script builds the container dynamically if necessary and formats the +specified file using the provided `prettier` options. + +options: + -h, --help show this help message and exit + -i INPUT, --input INPUT + -o OUTPUT, --output OUTPUT + --run_latex_again + --dockerized_force_rebuild + Force to rebuild the Docker container + --dockerized_use_sudo + Use sudo inside the container + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level + ``` ## dockerized_mermaid.py ### What it does @@ -796,8 +856,6 @@ options: ### Examples -- TODO - ```bash > mermaid_wrapper.py --input flowchart.mmd --output flowchart.png ``` @@ -821,6 +879,26 @@ options: > mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_use_sudo ``` +### Interface + +``` bash +usage: dockerized_mermaid.py [-h] -i INPUT [-o OUTPUT] [--dockerized_force_rebuild] [--dockerized_use_sudo] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + +Run `mermaid` inside a Docker container. + +options: + -h, --help show this help message and exit + -i INPUT, --input INPUT + -o OUTPUT, --output OUTPUT + --dockerized_force_rebuild + Force to rebuild the Docker container + --dockerized_use_sudo + Use sudo inside the container + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level + ``` + ## `dockerized_pandoc.py` ### What it does @@ -835,7 +913,7 @@ options: - Internally builds a Docker container and passes the full `pandoc` command string. -### Example +### Examples - Convert Markdown to PDF ``` @@ -857,6 +935,33 @@ options: > pandoc_wrapper.py --input notes.md --output notes.pdf --dockerized_use_sudo ``` +# Interface + +``` bash +usage: dockerized_pandoc.py [-h] [--dockerized_force_rebuild] [--dockerized_use_sudo] [--input INPUT] + [--output OUTPUT] [--data_dir DATA_DIR] [--container_type CONTAINER_TYPE] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + +Run `pandoc` inside a Docker container. + +This script builds the container dynamically if necessary. + +> pandoc tmp.pandoc.no_spaces.txt -t beamer --slide-level 4 -V theme:SimplePlus --include-in-header=latex_abbrevs.sty --toc --toc-depth 2 -o tmp.pandoc.no_spaces.pdf + +options: + -h, --help show this help message and exit + --dockerized_force_rebuild + Force to rebuild the Docker container + --dockerized_use_sudo + Use sudo inside the container + --input INPUT + --output OUTPUT + --data_dir DATA_DIR + --container_type CONTAINER_TYPE + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level +``` + ## `dockerized_prettier.py` ### What it does @@ -894,14 +999,13 @@ options: ### Interface -- Interface - ``` + ``` bash > dockerized_prettier.py -h usage: dockerized_prettier.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] [--dockerized_force_rebuild] [--dockerized_use_sudo] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] - ``` + Run `prettier` inside a Docker container to ensure consistent formatting across different environments. @@ -940,7 +1044,7 @@ options: --dockerized_force_rebuild Force to rebuild the Docker container --dockerized_use_sudo Use sudo inside the container -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level - ``` + ``` ## `save_screenshot.py` @@ -951,40 +1055,46 @@ options: 2. Saves it as `screenshot.YYYY‑MM‑DD_HH‑MM‑SS.png` (or your chosen name). 3. Prints and copies the Markdown embed ``. +### Examples - Convert a txt file into a PDF / HTML / slides using `pandoc`. +- Take screenshot and auto-name it with timestamp: - # From scratch with TOC: + ``` bash + > take_screenshot.py + ``` - > notes_to_pdf.py -a pdf --input ... +- Save to a specific directory with a custom filename: - # For interactive mode: + ```bash + > take_screenshot.py --dst_dir ./screenshots --filename example.png + ``` - > notes_to_pdf.py -a pdf --no_cleanup_before --no_cleanup --input ... +- Allow overwriting an existing file: - # Check that can be compiled: + ``` bash + > take_screenshot.py --filename my.png --override + ``` - > notes_to_pdf.py -a pdf --no_toc --no_open_pdf --input ... +### Interface - > notes_to_pdf.py --input - > notes/IN_PROGRESS/math.The_hundred_page_ML_book.Burkov.2019.txt -t pdf - > --no_cleanup --no_cleanup_before --no_run_latex_again --no_open - ```bash - options: -h, --help show this help message and exit -i INPUT, --input INPUT -o - OUTPUT, --output OUTPUT Output file --type {pdf,html,slides} Type of output to - generate --filter_by_header FILTER_BY_HEADER Filter by header - --filter_by_lines FILTER_BY_LINES Filter by lines (e.g., `0:10`, `1:None`, - `None:10`) --script SCRIPT Bash script to generate with all the executed sub- - commands --preview_actions Print the actions and exit --toc_type - {none,pandoc_native,navigation} --no_run_latex_again --debug_on_error - --gdrive_dir GDRIVE_DIR Directory where to save the output to share on Google - Drive --use_host_tools Use the host tools instead of the dockerized ones - --action - {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} - Actions to execute --skip_action - {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} - Actions to skip --all Run all the actions (cleanup_before preprocess_notes - render_images run_pandoc open cleanup_after) --dockerized_force_rebuild Force - to rebuild the Docker container --dockerized_use_sudo Use sudo inside the - container -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level - ``` +``` bash +usage: save_screenshot.py [-h] [--dst_dir DST_DIR] [--filename FILENAME] [--override] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] + [positional ...] + +Take a screenshot and save it to a file. + +The script should be run from the command line outside of a Docker +container. + +positional arguments: + positional ... + +options: + -h, --help show this help message and exit + --dst_dir DST_DIR Destination directory + --filename FILENAME File name + --override Override if file exists + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level + ``` \ No newline at end of file From 91718b6d6e21feb094408c52149bfc8e6e959166 Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 30 Jun 2025 22:54:22 -0500 Subject: [PATCH 23/26] HelpersTask741: Adding interface commands to documentation toolchain file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 510 ++++++++++-------- 1 file changed, 277 insertions(+), 233 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index bc3b3a1d5..6c3c17d89 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -1,73 +1,70 @@ -# Documentation Toolchain - How To Guide -This is a high‑level guide to the helper scripts that turn raw `.txt` notes into polished PDFs, slide decks, and more. - - [`notes_to_pdf.py`](#notes_to_pdfpy) - - [What it does](#what-it-does) -- [`render_images.py`](#render_imagespy) - - [What it does](#what-it-does-1) - - [Examples](#examples-1) - - [Interface](#interface) -- [`lint_notes.py`](#lint_notespy) - - [What it does](#what-it-does-2) - - [Examples](#examples-2) - - [Interface](#interface-1) -- [`extract_notebook_images.py`](#extract_notebook_imagespy) - - [What it does](#what-it-does-3) - - [Example](#example) - - [Interface](#interface-2) -- [`llm_transform.py`](#llm_transformpy) - - [What it does](#what-it-does-4) - - [Examples](#examples-3) - - [Interface](#interface-3) -- [`run_pandoc.py`](#run_pandocpy) - - [What it does](#what-it-does-5) - - [Example](#example-1) - - [Interface](#interface-4) -- [`transform_notes.py`](#transform_notespy) - - [What it does](#what-it-does-6) - - [Examples](#examples-4) - - [Interface](#interface-5) -- [`extract_headers_from_markdown.py`](#extract_headers_from_markdownpy) - - [What it does](#what-it-does-7) - - [Examples](#examples-5) -- [`dockerized_tikz_to_bitmap.py`](#dockerized_tikz_to_bitmappy) - - [Examples](#examples-6) -- [`dockerized_graphviz.py`](#dockerized_graphvizpy) - - [What it does](#what-it-does-8) - - [Interface](#interface-6) - - [Examples](#examples-7) -- [`dockerized_latex.py`](#dockerized_latexpy) - - [What it does](#what-it-does-9) - - [Examples](#examples-8) -- [`dockerized_mermaid.py`](#dockerized_mermaidpy) - - [What it does](#what-it-does-10) - - [Examples](#examples-9) -- [`dockerized_pandoc.py`](#dockerized_pandocpy) - - [What it does](#what-it-does-11) - - [Example](#example-2) -- [`dockerized_prettier.py`](#dockerized_prettierpy) - - [What it does](#what-it-does-12) - - [Examples](#examples-10) - - [Interface](#interface-7) -- [`save_screenshot.py`](#save_screenshotpy) - - [What it does](#what-it-does-13) - -- [Examples](#examples) - - [Basic usage](#basic-usage) - - [Use sudo for Docker commands](#use-sudo-for-docker-commands) - - [Set logging verbosity](#set-logging-verbosity) - - [Process a file:](#process-a-file) - - [From scratch with TOC:](#from-scratch-with-toc) - - [For interactive mode:](#for-interactive-mode) - - [Check that can be compiled:](#check-that-can-be-compiled) + * [What it does](#what-it-does) + + [Examples](#examples) + + [Interface](#interface) + * [`render_images.py`](#render_imagespy) + + [What it does](#what-it-does-1) + + [Examples](#examples-1) + + [Interface](#interface-1) + * [`lint_notes.py`](#lint_notespy) + + [What it does](#what-it-does-2) + + [Examples](#examples-2) + + [Interface](#interface-2) + * [`extract_notebook_images.py`](#extract_notebook_imagespy) + + [What it does](#what-it-does-3) + + [Example](#example) + + [Interface](#interface-3) + * [`llm_transform.py`](#llm_transformpy) + + [What it does](#what-it-does-4) + + [Examples](#examples-3) + + [Interface](#interface-4) + * [`run_pandoc.py`](#run_pandocpy) + + [What it does](#what-it-does-5) + + [Examples](#examples-4) + + [Interface](#interface-5) + * [`transform_notes.py`](#transform_notespy) + + [What it does](#what-it-does-6) + + [Examples](#examples-5) + + [Interface](#interface-6) + * [`extract_headers_from_markdown.py`](#extract_headers_from_markdownpy) + + [What it does](#what-it-does-7) + + [Examples](#examples-6) + + [Interface](#interface-7) + * [`dockerized_tikz_to_bitmap.py`](#dockerized_tikz_to_bitmappy) + + [Examples](#examples-7) + + [Interface](#interface-8) + * [`dockerized_graphviz.py`](#dockerized_graphvizpy) + + [What it does](#what-it-does-8) + + [Examples](#examples-8) + + [Interface](#interface-9) + * [dockerized_latex.py](#dockerized_latexpy) + + [What it does](#what-it-does-9) + + [Examples](#examples-9) + + [Interface](#interface-10) + * [dockerized_mermaid.py](#dockerized_mermaidpy) + + [What it does](#what-it-does-10) + + [Examples](#examples-10) + + [Interface](#interface-11) + * [`dockerized_pandoc.py`](#dockerized_pandocpy) + + [What it does](#what-it-does-11) + + [Examples](#examples-11) + + [Interface](#interface-12) + * [`dockerized_prettier.py`](#dockerized_prettierpy) + + [What it does](#what-it-does-12) + + [Examples](#examples-12) + + [Interface](#interface-13) + * [`save_screenshot.py`](#save_screenshotpy) + + [What it does](#what-it-does-13) + + [Examples](#examples-13) + + [Interface](#interface-14) -## notes_to_pdf.py +# notes_to_pdf.py -### What it does +## What it does - Convert plain‑text notes into polished **PDF, HTML, or Beamer slides** with a single command: @@ -76,38 +73,29 @@ This is a high‑level guide to the helper scripts that turn raw `.txt` notes in > notes_to_pdf.py --input --output --type [pdf|html|slides] ``` -- The interface is: - ``` - > notes_to_pdf.py -h - usage: notes_to_pdf.py [-h] -i INPUT -o OUTPUT --type {pdf,html,slides} - [--filter_by_header FILTER_BY_HEADER] - [--filter_by_lines FILTER_BY_LINES] [--script SCRIPT] - [--preview_actions] - [--toc_type {none,pandoc_native,navigation}] - [--no_run_latex_again] [--debug_on_error] - [--gdrive_dir GDRIVE_DIR] [--use_host_tools] - [--action {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} | --skip_action {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after}] - [--all] [--dockerized_force_rebuild] - [--dockerized_use_sudo] - [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] - ``` - ### Examples - Compile to **Beamer slides** - ``` + + ```bash > notes_to_pdf.py -i lesson.txt -o lesson.pdf --type slides ``` + - Produce a **stand‑alone HTML** page - ``` + + ```bash > notes_to_pdf.py -i cheatsheet.txt -o cheatsheet.html --type html ``` + - Build a **PDF article** (LaTeX) - ``` + + ```bash > notes_to_pdf.py -i paper.txt -o paper.pdf --type pdf ``` + - Skip the final viewer **open** step - ``` + + ```bash > ... --skip_action open` ``` @@ -140,10 +128,28 @@ This is a high‑level guide to the helper scripts that turn raw `.txt` notes in ``` - Plain PDF article + ```bash > notes_to_pdf.py -i book_notes.txt -o book_notes.pdf --type pdf ``` +### Interface + +```bash +> notes_to_pdf.py -h +usage: notes_to_pdf.py [-h] -i INPUT -o OUTPUT --type {pdf,html,slides} + [--filter_by_header FILTER_BY_HEADER] + [--filter_by_lines FILTER_BY_LINES] [--script SCRIPT] + [--preview_actions] + [--toc_type {none,pandoc_native,navigation}] + [--no_run_latex_again] [--debug_on_error] + [--gdrive_dir GDRIVE_DIR] [--use_host_tools] + [--action {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after} | --skip_action {cleanup_before,preprocess_notes,render_images,run_pandoc,copy_to_gdrive,open,cleanup_after}] + [--all] [--dockerized_force_rebuild] + [--dockerized_use_sudo] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] +``` + ## render_images.py ### What it does @@ -197,60 +203,59 @@ The supported File types and code blocks are: ``` - Dry‑run (test parsing / comments only) + ```bash > render_images.py -i lesson.md -o /tmp/out.md --dry_run ``` ### Interface -- The interface +```bash +> render_images.py -h +usage: render_images.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] + [--action {open,render} | --skip_action {open,render}] + [--all] [--dry_run] [--dockerized_force_rebuild] + [--dockerized_use_sudo] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] - ```bash - > render_images.py -h - usage: render_images.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] - [--action {open,render} | --skip_action {open,render}] - [--all] [--dry_run] [--dockerized_force_rebuild] - [--dockerized_use_sudo] - [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] +Replace sections of image code with rendered images, commenting out the +original code, if needed. - Replace sections of image code with rendered images, commenting out the - original code, if needed. - - See `docs/work_tools/documentation_toolchain/all.render_images.explanation.md`. - - Usage: - - # Create a new Markdown file with rendered images: - > render_images.py -i ABC.md -o XYZ.md --action render --run_dockerized - - # Render images in place in the original Markdown file: - > render_images.py -i ABC.md --action render --run_dockerized - - # Render images in place in the original LaTeX file: - > render_images.py -i ABC.tex --action render --run_dockerized - - # Open rendered images from a Markdown file in HTML to preview: - > render_images.py -i ABC.md --action open --run_dockerized - - options: - -h, --help show this help message and exit - -i IN_FILE_NAME, --in_file_name IN_FILE_NAME - Path to the input file - -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME - Path to the output file - --action {open,render} - Actions to execute - --skip_action {open,render} - Actions to skip - --all Run all the actions () - --dry_run Update the file but do not render images - --dockerized_force_rebuild - Force to rebuild the Docker container - --dockerized_use_sudo - Use sudo inside the container - -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} - Set the logging level - ``` +See `docs/work_tools/documentation_toolchain/all.render_images.explanation.md`. + +Usage: + +# Create a new Markdown file with rendered images: +> render_images.py -i ABC.md -o XYZ.md --action render --run_dockerized + +# Render images in place in the original Markdown file: +> render_images.py -i ABC.md --action render --run_dockerized + +# Render images in place in the original LaTeX file: +> render_images.py -i ABC.tex --action render --run_dockerized + +# Open rendered images from a Markdown file in HTML to preview: +> render_images.py -i ABC.md --action open --run_dockerized + +options: + -h, --help show this help message and exit + -i IN_FILE_NAME, --in_file_name IN_FILE_NAME + Path to the input file + -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME + Path to the output file + --action {open,render} + Actions to execute + --skip_action {open,render} + Actions to skip + --all Run all the actions () + --dry_run Update the file but do not render images + --dockerized_force_rebuild + Force to rebuild the Docker container + --dockerized_use_sudo + Use sudo inside the container + -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} + Set the logging level +``` ## `lint_notes.py` @@ -281,7 +286,7 @@ The supported File types and code blocks are: - Custom Line width - ``` bash + ```bash > lint_notes.py -i test.txt -o tested.txt -w 10 ``` @@ -294,7 +299,7 @@ usage: lint_notes.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] [--print-width PRINT_WIDTH] [--use_dockerized_prettier] [--use_dockerized_markdown_toc] - [--action {preprocess,prettier,postprocess,frame_chapters,refresh_toc} + [--action {preprocess,prettier,postprocess,frame_chapters,refresh_toc} | --skip_action {preprocess,prettier,postprocess,frame_chapters,refresh_toc}] [--all] [--dockerized_force_rebuild] [--dockerized_use_sudo] @@ -353,24 +358,24 @@ options: ### Example - Minimal call + ```bash > extract_notebook_images.py \ --in_notebook_filename notebooks/Lesson8.ipynb \ --out_image_dir notebooks/screenshots ``` -- Enforce rebuild and use sudo commands +- Enforce rebuild and use sudo commands - ``` bash + ```bash > extract_notebook_images.py \ --in_notebook_filename notebooks/Lesson8.ipynb \ --out_image_dir notebooks/screenshots --dockerized_force_rebuild --dockerized_use_sudo ``` - ### Interface -``` bash +```bash usage: extract_notebook_images.py [-h] --in_notebook_filename IN_NOTEBOOK_FILENAME --out_image_dir OUT_IMAGE_DIR [--dockerized_force_rebuild] [--dockerized_use_sudo] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] @@ -380,12 +385,14 @@ Extract images from a Jupyter notebook by running inside a Docker container. This script builds the container dynamically if necessary and extracts images from the specified Jupyter notebook using the NotebookImageExtractor module. ``` + ```bash Extract images from notebook test_images.ipynb and save them to `screenshots` directory. > dev_scripts_helpers/notebooks/extract_notebook_images.py -i dev_scripts_helpers/notebooks/test_images.ipynb -o dev_scripts_helpers/notebooks/screenshots ``` -``` + +```bash options: -h, --help show this help message and exit --in_notebook_filename IN_NOTEBOOK_FILENAME @@ -411,9 +418,9 @@ options: ### Examples - ```bash - llm_transform.py -i draft.txt -o polished.txt -p rewrite_clearer - ``` +```bash +llm_transform.py -i draft.txt -o polished.txt -p rewrite_clearer +``` - Finding available prompts @@ -441,7 +448,8 @@ options: ``` ### Interface -``` bash + +```bash usage: llm_transform.py [-h] [-i IN_FILE_NAME] [-o OUT_FILE_NAME] [--debug] -p PROMPT [-f] [--dockerized_force_rebuild] [--dockerized_use_sudo] [-c] [-b] [-s] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] @@ -455,7 +463,7 @@ all dependencies are met. The Docker container is built dynamically if necessary. The script requires an OpenAI API key to be set in the environment. ``` -``` bash +```bash Examples # Basic Usage > llm_transform.py -i input.txt -o output.txt -p uppercase @@ -470,7 +478,7 @@ Examples > llm_transform.py -i dev_scripts_helpers/documentation/render_images.py -o cfile -p code_propose_refactoring ``` -``` bash +```bash options: -h, --help show this help message and exit -i IN_FILE_NAME, --in_file_name IN_FILE_NAME @@ -505,15 +513,20 @@ options: ### Examples - Convert a Markdown file to LaTeX - ``` + + ```bash > run_pandoc.py -i note.md -o note.tex ``` + - Same, but stream from `stdin` to `stdout` - ``` + + ```bash > cat note.md | run_pandoc.py -i - -o - ``` + - Inside Vim (visual range) - ``` + + ```vim :<,'>!run_pandoc.py -i - -o - -v CRITICAL ``` @@ -522,7 +535,7 @@ options: ### Interface -``` bash +```bash usage: run_pandoc.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] [--action ACTION] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] @@ -535,14 +548,14 @@ Run pandoc on stdin/file to stdout/file. ``` To run in VIM: -``` + +```vim :'<,'>!dev_scripts/documentation/run_pandoc.py -i - -o - -v CRITICAL ``` This script is derived from `dev_scripts/transform_template.py`. -``` -``` bash +```bash options: -h, --help show this help message and exit -i IN_FILE_NAME, --in_file_name IN_FILE_NAME @@ -575,8 +588,9 @@ options: - Bump all headers down one level - Typical Vim one-liner: `:%!transform_notes.py -a increase -i -` - `md_list_to_latex` - - Convert a Markdown list to LaTeX `\begin{itemize}` - - Typical Vim one-liner: `:%!transform_notes.py -a md_list_to_latex -i -` + +* Convert a Markdown list to LaTeX `\begin{itemize}` + - Typical Vim one-liner: `:%!transform_notes.py -a md_list_to_latex -i -` - `md_*` family - Formatting clean-ups (bold bullets, colorize bold text, etc.) - Additional Information: See `-a list` for more details @@ -594,7 +608,8 @@ options: ``` - Tidy ChatGPT‑generated Markdown (visual mode in Vim) - ``` + + ```vim :'<,'>!transform_notes.py -i - -o - -a md_fix_chatgpt_output ``` @@ -612,7 +627,6 @@ Perform one of several transformations on a txt file, e.g., 2) `format`: format the current file with 3 levels :!transform_notes.py -a format -i % --max_lev 3 > transform_notes.py -a format -i notes/ABC.txt --max_lev 3 - - In vim :!transform_notes.py -a format -i % --max_lev 3 :%!transform_notes.py -a format -i - --max_lev 3 @@ -639,7 +653,7 @@ options: -l MAX_LEV, --max_lev MAX_LEV -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level - ``` +``` ## `extract_headers_from_markdown.py` @@ -660,12 +674,13 @@ options: ``` - Build a quick‑fix file and open Vim on it + ```bash > extract_headers_from_markdown.py -i README.md -o headers.cfile --mode cfile > vim -c "cfile headers.cfile" ``` -### Interface +### Interface ```bash usage: extract_headers_from_markdown.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] @@ -675,10 +690,10 @@ usage: extract_headers_from_markdown.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] Extract headers from a Markdown file and generate a Vim cfile. The script: -- processes the input Markdown file -- extracts headers up to a specified maximum level -- prints a human-readable header map -- generates an output file in a format that can be used with Vim's quickfix +- Processes the input Markdown file +- Extracts headers up to a specified maximum level +- Prints a human-readable header map +- Generates an output file in a format that can be used with Vim's quickfix feature. # Extract headers up to level 3 from a Markdown file and save to an output file: @@ -705,11 +720,12 @@ options: Maximum header levels to parse -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level - ``` +``` + ## `dockerized_tikz_to_bitmap.py` -- Convert a TikZ file to a PNG image using a dockerized version of `pdflatex` and -`imagemagick`. +- Convert a TikZ file to a PNG image using a dockerized version of `pdflatex` + and `imagemagick`. ### Examples @@ -720,6 +736,7 @@ options: ``` - Custom ImageMagick options (e.g. 600 DPI) + ```bash > dockerized_tikz_to_bitmap.py -i fig.tikz -o fig.png -- -density 600 -quality 90 ``` @@ -727,7 +744,7 @@ options: ### Interface -``` bash +```bash usage: dockerized_tikz_to_bitmap.py [-h] -i INPUT -o OUTPUT [--dockerized_force_rebuild] [--dockerized_use_sudo] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] @@ -760,21 +777,26 @@ options: ### Examples - Convert DOT to PNG - ``` + + ```bash > graphviz_wrapper.py -i diagram.dot -o diagram.png ``` + - Rebuild Docker image - ``` + + ```bash > graphviz_wrapper.py -i diagram.dot -o diagram.png --dockerized_force_rebuild ``` + - Use `sudo` for Docker + ```bash > graphviz_wrapper.py -i diagram.dot -o diagram.png --dockerized_use_sudo ``` ### Interface -``` bash +```bash usage: dockerized_graphviz.py [-h] -i INPUT -o OUTPUT [--dockerized_force_rebuild] [--dockerized_use_sudo] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] @@ -790,7 +812,7 @@ options: Use sudo inside the container -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level - ``` +``` ## dockerized_latex.py @@ -801,6 +823,7 @@ options: - Automatically rebuilds the Docker image if needed. - Supports optional rerun of LaTeX for proper references or table of contents generation + ```bash > latex_wrapper.py --input doc.tex --output doc.pdf ``` @@ -808,25 +831,32 @@ options: ### Examples - Compile `.tex` to `.pdf` - ``` + + ```bash > latex_wrapper.py -i report.tex -o report.pdf ``` + - Rebuild Docker image - ``` + + ```bash > latex_wrapper.py -i report.tex -o report.pdf --dockerized_force_rebuild ``` + - Use `sudo` for Docker - ``` + + ```bash > latex_wrapper.py -i report.tex -o report.pdf --dockerized_use_sudo ``` + - Run LaTeX twice - ``` + + ```bash > latex_wrapper.py -i paper.tex -o paper.pdf --run_latex_again ``` -## Interface +### Interface -``` bash +```bash usage: dockerized_latex.py [-h] -i INPUT -o OUTPUT [--run_latex_again] [--dockerized_force_rebuild] [--dockerized_use_sudo] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] @@ -846,7 +876,8 @@ options: Use sudo inside the container -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level - ``` +``` + ## dockerized_mermaid.py ### What it does @@ -856,32 +887,39 @@ options: ### Examples - ```bash - > mermaid_wrapper.py --input flowchart.mmd --output flowchart.png - ``` +```bash +> mermaid_wrapper.py --input flowchart.mmd --output flowchart.png +``` - Automatically sets output to match input name if `--output` is omitted - Mermaid diagram - ``` + + ```bash > mermaid_wrapper.py -i diagram.mmd -o diagram.png ``` + - Use input as output (default) - ``` + + ```bash > mermaid_wrapper.py -i diagram.mmd ``` + - Rebuild container - ``` + + ```bash > mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_force_rebuild ``` + - Use `sudo` for Docker - ``` + + ```bash > mermaid_wrapper.py -i diagram.mmd -o diagram.png --dockerized_use_sudo ``` ### Interface -``` bash +```bash usage: dockerized_mermaid.py [-h] -i INPUT [-o OUTPUT] [--dockerized_force_rebuild] [--dockerized_use_sudo] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] @@ -897,7 +935,7 @@ options: Use sudo inside the container -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level - ``` +``` ## `dockerized_pandoc.py` @@ -916,28 +954,32 @@ options: ### Examples - Convert Markdown to PDF - ``` + + ```bash > pandoc_wrapper.py --input notes.md --output notes.pdf --container_type pandoc_latex ``` - Convert to Beamer slides - ``` + + ```bash > pandoc_wrapper.py --input slides.md --output slides.pdf --container_type pandoc_latex -- -t beamer ``` - Rebuild Docker image - ``` + + ```bash > pandoc_wrapper.py --input notes.md --output notes.pdf --dockerized_force_rebuild ``` - Run with sudo - ``` + + ```bash > pandoc_wrapper.py --input notes.md --output notes.pdf --dockerized_use_sudo ``` -# Interface +### Interface -``` bash +```bash usage: dockerized_pandoc.py [-h] [--dockerized_force_rebuild] [--dockerized_use_sudo] [--input INPUT] [--output OUTPUT] [--data_dir DATA_DIR] [--container_type CONTAINER_TYPE] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] @@ -978,74 +1020,76 @@ options: ### Examples - Format a Markdown file - ``` + + ```bash > dockerized_prettier.py --parser markdown --write test.md ``` - Use `sudo` for Docker execution - ``` + + ```bash > dockerized_prettier.py --use_sudo --parser markdown --write test.md ``` - Rebuild the Docker image - ``` + + ```bash > dockerized_prettier.py --dockerized_force_rebuild --parser markdown --write test.md ``` - Change indentation and wrap style - ``` + + ```bash dockerized_prettier.py --parser markdown --tab-width 4 --prose-wrap always --write test.md ``` ### Interface - ``` bash - > dockerized_prettier.py -h - usage: dockerized_prettier.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] - [--dockerized_force_rebuild] - [--dockerized_use_sudo] - [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] - +```bash +> dockerized_prettier.py -h +usage: dockerized_prettier.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] + [--dockerized_force_rebuild] + [--dockerized_use_sudo] + [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] - Run `prettier` inside a Docker container to ensure consistent formatting - across different environments. +Run `prettier` inside a Docker container to ensure consistent formatting +across different environments. - This script builds the container dynamically if necessary and formats the - specified file using the provided `prettier` options. +This script builds the container dynamically if necessary and formats the +specified file using the provided `prettier` options. - Examples +Examples - # Basic usage: +# Basic usage: - > dockerized_prettier.py --parser markdown --prose-wrap always --write - > --tab-width 2 test.md +> dockerized_prettier.py --parser markdown --prose-wrap always --write +> --tab-width 2 test.md - # Use sudo for Docker commands: +# Use sudo for Docker commands: - > dockerized_prettier.py --use_sudo --parser markdown --prose-wrap always - > --write --tab-width 2 test.md +> dockerized_prettier.py --use_sudo --parser markdown --prose-wrap always +> --write --tab-width 2 test.md - # Set logging verbosity: +# Set logging verbosity: - > dockerized_prettier.py -v DEBUG --parser markdown --prose-wrap always - > --write --tab-width 2 test.md +> dockerized_prettier.py -v DEBUG --parser markdown --prose-wrap always +> --write --tab-width 2 test.md - # Process a file: +# Process a file: - > cat test.md - - A - - B - c - > dockerized_prettier.py --parser markdown --prose-wrap always --write - > --tab-width 2 test.md +> cat test.md +- A + - B - c + > dockerized_prettier.py --parser markdown --prose-wrap always --write + > --tab-width 2 test.md - options: -h, --help show this help message and exit -i IN_FILE_NAME, - --in_file_name IN_FILE_NAME Input file or `-` for stdin -o OUT_FILE_NAME, - --out_file_name OUT_FILE_NAME Output file or `-` for stdout - --dockerized_force_rebuild Force to rebuild the Docker container - --dockerized_use_sudo Use sudo inside the container -v - {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level - - ``` +options: -h, --help show this help message and exit -i IN_FILE_NAME, +--in_file_name IN_FILE_NAME Input file or `-` for stdin -o OUT_FILE_NAME, +--out_file_name OUT_FILE_NAME Output file or `-` for stdout +--dockerized_force_rebuild Force to rebuild the Docker container +--dockerized_use_sudo Use sudo inside the container +-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level +``` ## `save_screenshot.py` @@ -1055,11 +1099,11 @@ options: 2. Saves it as `screenshot.YYYY‑MM‑DD_HH‑MM‑SS.png` (or your chosen name). 3. Prints and copies the Markdown embed ``. -### Examples +### Examples - Take screenshot and auto-name it with timestamp: - ``` bash + ```bash > take_screenshot.py ``` @@ -1071,13 +1115,13 @@ options: - Allow overwriting an existing file: - ``` bash + ```bash > take_screenshot.py --filename my.png --override ``` ### Interface -``` bash +```bash usage: save_screenshot.py [-h] [--dst_dir DST_DIR] [--filename FILENAME] [--override] [-v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL}] [positional ...] @@ -1097,4 +1141,4 @@ options: --override Override if file exists -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} Set the logging level - ``` \ No newline at end of file +``` From 9cc5ab2e6ef758689616c24be881c8f05f3914ec Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 30 Jun 2025 23:11:48 -0500 Subject: [PATCH 24/26] HelpersTask741: Fixing the merge issues MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 6c3c17d89..491f14234 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -1,10 +1,10 @@ -- [`notes_to_pdf.py`](#notes_to_pdfpy) +- [notes_to_pdf.py](#notes_to_pdfpy) * [What it does](#what-it-does) + [Examples](#examples) + [Interface](#interface) - * [`render_images.py`](#render_imagespy) + * [render_images.py](#render_imagespy) + [What it does](#what-it-does-1) + [Examples](#examples-1) + [Interface](#interface-1) From 63a9df6990fb332d2fdc304df9833931e2fd8e31 Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 30 Jun 2025 23:29:51 -0500 Subject: [PATCH 25/26] HelpersTask741: Running Linter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 31 ++----------------- 1 file changed, 2 insertions(+), 29 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 072cc201a..205447692 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -221,51 +221,22 @@ usage: render_images.py [-h] -i IN_FILE_NAME [-o OUT_FILE_NAME] Replace sections of image code with rendered images, commenting out the original code, if needed. -See `docs/work_tools/documentation_toolchain/all.render_images.explanation.md`. See `docs/work_tools/documentation_toolchain/all.render_images.explanation.md`. -Usage: Usage: -# Create a new Markdown file with rendered images: -> render_images.py -i ABC.md -o XYZ.md --action render --run_dockerized # Create a new Markdown file with rendered images: > render_images.py -i ABC.md -o XYZ.md --action render --run_dockerized -# Render images in place in the original Markdown file: -> render_images.py -i ABC.md --action render --run_dockerized # Render images in place in the original Markdown file: > render_images.py -i ABC.md --action render --run_dockerized -# Render images in place in the original LaTeX file: -> render_images.py -i ABC.tex --action render --run_dockerized # Render images in place in the original LaTeX file: > render_images.py -i ABC.tex --action render --run_dockerized -# Open rendered images from a Markdown file in HTML to preview: -> render_images.py -i ABC.md --action open --run_dockerized # Open rendered images from a Markdown file in HTML to preview: > render_images.py -i ABC.md --action open --run_dockerized -options: - -h, --help show this help message and exit - -i IN_FILE_NAME, --in_file_name IN_FILE_NAME - Path to the input file - -o OUT_FILE_NAME, --out_file_name OUT_FILE_NAME - Path to the output file - --action {open,render} - Actions to execute - --skip_action {open,render} - Actions to skip - --all Run all the actions () - --dry_run Update the file but do not render images - --dockerized_force_rebuild - Force to rebuild the Docker container - --dockerized_use_sudo - Use sudo inside the container - -v {TRACE,DEBUG,INFO,WARNING,ERROR,CRITICAL} - Set the logging level -``` options: -h, --help show this help message and exit -i IN_FILE_NAME, --in_file_name IN_FILE_NAME @@ -450,6 +421,7 @@ options: ```bash llm_transform.py -i draft.txt -o polished.txt -p rewrite_clearer ``` + ```bash llm_transform.py -i draft.txt -o polished.txt -p rewrite_clearer ``` @@ -922,6 +894,7 @@ options: ```bash > mermaid_wrapper.py --input flowchart.mmd --output flowchart.png ``` + ```bash > mermaid_wrapper.py --input flowchart.mmd --output flowchart.png ``` From 692367fd2ef8f38226313aa483ab93f04d102d74 Mon Sep 17 00:00:00 2001 From: Aayush Date: Mon, 30 Jun 2025 23:41:04 -0500 Subject: [PATCH 26/26] HelpersTask741: Passing Linter checks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-commit checks: All checks passed ✅ --- .../all.notes_toolchain.how_to_guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md index 205447692..d0a2c3843 100644 --- a/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md +++ b/docs/tools/documentation_toolchain/all.notes_toolchain.how_to_guide.md @@ -176,7 +176,7 @@ The supported File types and code blocks are: - `graphviz` - `tikz` - `latex` - - Output embeds as: `` + - Output embeds as: `` - File extension: `.tex` - Rendering syntax allowed: - Same tags (TikZ & LaTeX especially) @@ -1105,7 +1105,7 @@ options: -h, --help show this help message and exit -i IN_FILE_NAME, 1. Prompts you to select a screen region (`⌘ - Ctrl - 4`). 2. Saves it as `screenshot.YYYY‑MM‑DD_HH‑MM‑SS.png` (or your chosen name). -3. Prints and copies the Markdown embed ``. +3. Prints and copies the Markdown embed ``. ### Examples