Add CI/CD workflows for automated notebook validation and branch-specific GitHub Pages deployment

[Copilot]

Implements automated building, testing, and publishing of Jupyter notebooks as branch-specific HTML pages on GitHub Pages.

Workflows

• .github/workflows/ci.yml - Validates code on pushes/PRs to main: runs flake8, pytest, and executes all notebooks with 600s timeout
• .github/workflows/deploy-notebooks-pages.yml - Builds and deploys HTML notebooks to gh-pages/<branch-name>/ on every push to any branch; generates index listing all published branches

Build Infrastructure

• scripts/build_notebooks.py - Discovers notebooks recursively, executes them via nbconvert, exports to HTML preserving folder structure, creates error placeholders for failed notebooks
• requirements.txt - Dependencies: jupyter, nbconvert, nbformat, pytest, flake8, black, pre-commit
• Dockerfile - Python 3.11 slim image with all dependencies
• .pre-commit-config.yaml - Hooks for black and flake8
• .gitignore - Excludes .venv, __pycache__, site/, gh-pages/, .ipynb_checkpoints/

Usage

Local build:

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python scripts/build_notebooks.py site/local-branch

Enable GitHub Pages in Settings → Pages (source: gh-pages branch). Branch pages available at https://<owner>.github.io/<repo>/<branch>/.

Original prompt

Add CI, notebook build and deployment workflows, supporting scripts, tooling, and documentation to enable building, testing, linting, and publishing Jupyter notebooks as branch-specific HTML pages on GitHub Pages.

Files to add (exact contents provided):

1. .github/workflows/ci.yml

name: CI

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Lint (flake8)
        run: |
          flake8 .

      - name: Run unit tests (pytest)
        run: |
          pytest -q --maxfail=1

      - name: Execute all notebooks (nbconvert)
        run: |
          python - <<'PY'
          import glob, subprocess, sys
          notebooks = glob.glob('**/*.ipynb', recursive=True)
          if not notebooks:
              print("No notebooks found.")
              sys.exit(0)
          for n in notebooks:
              print("Executing", n)
              subprocess.check_call([
                sys.executable, "-m", "jupyter", "nbconvert",
                "--to", "notebook", "--execute", "--inplace",
                "--ExecutePreprocessor.timeout=600", n
              ])
          PY

2. .github/workflows/deploy-notebooks-pages.yml

on:
  push:
    branches:
      - '**'

name: Build and deploy notebooks to GitHub Pages (per-branch)

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: write # needed to push gh-pages
    steps:
      - name: Checkout repository (current branch)
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Determine branch name
        id: branch
        run: |
          echo "BRANCH_NAME=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
          echo "branch=${GITHUB_REF#refs/heads/}" >> $GITHUB_OUTPUT

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install build dependencies
        run: |
          python -m pip install --upgrade pip
          if [ -f requirements.txt ]; then pip install -r requirements.txt; else pip install jupyter nbconvert nbformat; fi

      - name: Build notebooks to HTML for this branch
        run: |
          python -m pip install --upgrade pip
          mkdir -p site/${{ env.BRANCH_NAME }}
          python scripts/build_notebooks.py site/${{ env.BRANCH_NAME }}

      - name: Prepare gh-pages branch workspace
        run: |
          # If gh-pages exists, clone it, otherwise init an empty gh-pages branch
          REPO_URL="https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git"
          if git ls-remote --exit-code origin gh-pages; then
            git clone --depth 1 --branch gh-pages "$REPO_URL" gh-pages || git clone "$REPO_URL" gh-pages
          else
            # create temporary folder, init gh-pages and push
            mkdir gh-pages
            cd gh-pages
            git init
            git remote add origin "$REPO_URL"
            git checkout -b gh-pages || true
            touch .nojekyll
            git add .nojekyll
            git commit -m "Initialize gh-pages"
            git push origin gh-pages
            cd ..
            git clone --depth 1 --branch gh-pages "$REPO_URL" gh-pages
          fi

      - name: Copy built site into gh-pages under branch folder
        run: |
          set -e
          rsync -a --delete site/${{ env.BRANCH_NAME }}/ gh-pages/${{ env.BRANCH_NAME }}/
          # ensure a top-level index exists listing branches (optional)
          python3 - <<'PY'
import os, json
root='gh-pages'
items = sorted([d for d in os.listdir(root) if os.path.isdir(os.path.join(root,d))])
index_path = os.path.join(root, 'index.html')
with open(index_path, 'w') as f:
    f.write("<html><head><meta charset='utf-8'><title>Branches</title></head><body>")
    f.write("<h1>Published branches</h1><ul>")
    for d in items:
        f.write(f'<li><a href="./{d}/">{d}</a></li>')
    f.write("</ul></body></html>")
PY

      - name: Commit and push changes to gh-pages
        working-directory: gh-pages
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git add --all
          if git diff --quiet --cached; then
            echo "No changes to deploy"
          else
            git commit -m "Deploy notebooks for branch '${{ env.BRANCH_NAME }}' [ci skip]"
            git push origin gh-pages
          fi

3. scripts/build_notebooks.py

#!/usr/bin/env...

</details>

*This pull request was created as a result of the following prompt from Copilot chat.*
> Add CI, notebook build and deployment workflows, supporting scripts, tooling, and documentation to enable building, testing, linting, and publishing Jupyter notebooks as branch-specific HTML pages on GitHub Pages.
> 
> Files to add (exact contents provided):
> 
> 1) .github/workflows/ci.yml
> ```name=.github/workflows/ci.yml
> name: CI
> 
> on:
>   push:
>     branches:
>       - main
>   pull_request:
>     branches:
>       - main
> 
> jobs:
>   test:
>     runs-on: ubuntu-latest
>     steps:
>       - name: Checkout
>         uses: actions/checkout@v4
> 
>       - name: Set up Python
>         uses: actions/setup-python@v4
>         with:
>           python-version: '3.11'
> 
>       - name: Install dependencies
>         run: |
>           python -m pip install --upgrade pip
>           pip install -r requirements.txt
> 
>       - name: Lint (flake8)
>         run: |
>           flake8 .
> 
>       - name: Run unit tests (pytest)
>         run: |
>           pytest -q --maxfail=1
> 
>       - name: Execute all notebooks (nbconvert)
>         run: |
>           python - <<'PY'
>           import glob, subprocess, sys
>           notebooks = glob.glob('**/*.ipynb', recursive=True)
>           if not notebooks:
>               print("No notebooks found.")
>               sys.exit(0)
>           for n in notebooks:
>               print("Executing", n)
>               subprocess.check_call([
>                 sys.executable, "-m", "jupyter", "nbconvert",
>                 "--to", "notebook", "--execute", "--inplace",
>                 "--ExecutePreprocessor.timeout=600", n
>               ])
>           PY
> ```
> 
> 2) .github/workflows/deploy-notebooks-pages.yml
> ```name=.github/workflows/deploy-notebooks-pages.yml
> on:
>   push:
>     branches:
>       - '**'
> 
> name: Build and deploy notebooks to GitHub Pages (per-branch)
> 
> jobs:
>   build-and-deploy:
>     runs-on: ubuntu-latest
>     permissions:
>       contents: write # needed to push gh-pages
>     steps:
>       - name: Checkout repository (current branch)
>         uses: actions/checkout@v4
>         with:
>           fetch-depth: 0
> 
>       - name: Determine branch name
>         id: branch
>         run: |
>           echo "BRANCH_NAME=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV
>           echo "branch=${GITHUB_REF#refs/heads/}" >> $GITHUB_OUTPUT
> 
>       - name: Set up Python
>         uses: actions/setup-python@v4
>         with:
>           python-version: '3.11'
> 
>       - name: Install build dependencies
>         run: |
>           python -m pip install --upgrade pip
>           if [ -f requirements.txt ]; then pip install -r requirements.txt; else pip install jupyter nbconvert nbformat; fi
> 
>       - name: Build notebooks to HTML for this branch
>         run: |
>           python -m pip install --upgrade pip
>           mkdir -p site/${{ env.BRANCH_NAME }}
>           python scripts/build_notebooks.py site/${{ env.BRANCH_NAME }}
> 
>       - name: Prepare gh-pages branch workspace
>         run: |
>           # If gh-pages exists, clone it, otherwise init an empty gh-pages branch
>           REPO_URL="https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git"
>           if git ls-remote --exit-code origin gh-pages; then
>             git clone --depth 1 --branch gh-pages "$REPO_URL" gh-pages || git clone "$REPO_URL" gh-pages
>           else
>             # create temporary folder, init gh-pages and push
>             mkdir gh-pages
>             cd gh-pages
>             git init
>             git remote add origin "$REPO_URL"
>             git checkout -b gh-pages || true
>             touch .nojekyll
>             git add .nojekyll
>             git commit -m "Initialize gh-pages"
>             git push origin gh-pages
>             cd ..
>             git clone --depth 1 --branch gh-pages "$REPO_URL" gh-pages
>           fi
> 
>       - name: Copy built site into gh-pages under branch folder
>         run: |
>           set -e
>           rsync -a --delete site/${{ env.BRANCH_NAME }}/ gh-pages/${{ env.BRANCH_NAME }}/
>           # ensure a top-level index exists listing branches (optional)
>           python3 - <<'PY'
> import os, json
> root='gh-pages'
> items = sorted([d for d in os.listdir(root) if os.path.isdir(os.path.join(root,d))])
> index_path = os.path.join(root, 'index.html')
> with open(index_path, 'w') as f:
>     f.write("<html><head><meta charset='utf-8'><title>Branches</title></head><body>")
>     f.write("<h1>Published branches</h1><ul>")
>     for d in items:
>         f.write(f'<li><a href="./{d}/">{d}</a></li>')
>     f.write("</ul></body></html>")
> PY
> 
>       - name: Commit and push changes to gh-pages
>         working-directory: gh-pages
>         run: |
>           git config user.name "github-actions[bot]"
>           git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
>           git add --all
>           if git diff --quiet --cached; then
>             echo "No changes to deploy"
>           else
>             git commit -m "Deploy notebooks for branch '${{ env.BRANCH_NAME }}' [ci skip]"
>             git push origin gh-pages
>           fi
> ```
> 
> 3) scripts/build_notebooks.py
> ```name=scripts/build_notebooks.py
> #!/usr/bin/env python3
> """
> Execute all notebooks in the repository (recursive) and export them to HTML into an output folder.
> 
> Usage:
>   python scripts/build_notebooks.py <output_dir> [--timeout SECONDS]
> 
> By default this will search for all .ipynb files (excluding .ipynb_checkpoints) and:
> - execute them with a timeout
> - export resulting notebook to HTML and place the HTML in <output_dir> preserving folder structure
> """
> import sys
> import os
> import subprocess
> from pathlib import Path
> 
> def find_notebooks(root="."):
>     nbs = []
>     for p in Path(root).rglob("*.ipynb"):
>         # skip checkpoints and files inside .git or site output
>         if ".ipynb_checkpoints" in p.parts or "site" in p.parts or "gh-pages" in p.parts:
>             continue
>         nbs.append(p)
>     return nbs
> 
> def main():
>     if len(sys.argv) < 2:
>         print("Usage: build_notebooks.py <output_dir> [--timeout SECONDS]")
>         sys.exit(1)
>     outdir = Path(sys.argv[1])
>     timeout = 600
>     if "--timeout" in sys.argv:
>         try:
>             timeout = int(sys.argv[sys.argv.index("--timeout")+1])
>         except Exception:
>             pass
>     outdir.mkdir(parents=True, exist_ok=True)
> 
>     notebooks = find_notebooks(".")
>     if not notebooks:
>         print("No notebooks found.")
>         return
> 
>     print(f"Found {len(notebooks)} notebooks. Exporting to {outdir} ...")
>     for nb in notebooks:
>         rel = nb.relative_to(Path.cwd())
>         target_dir = outdir.joinpath(rel.parent)
>         target_dir.mkdir(parents=True, exist_ok=True)
>         print(f"Processing {nb} -> {target_dir}")
> 
>         # Execute notebook in place into a temp file and convert to HTML
>         # Use nbconvert CLI to execute and export; capture exit code
>         try:
>             subprocess.check_call([
>                 sys.executable, "-m", "jupyter", "nbconvert",
>                 "--to", "html",
>                 "--execute",
>                 "--ExecutePreprocessor.timeout={}".format(timeout),
>                 "--output-dir", str(target_dir),
>                 str(nb)
>             ])
>         except subprocess.CalledProcessError as e:
>             print(f"ERROR executing {nb}: {e}")
>             # Create a placeholder HTML with the failure message so CI pages report which notebooks failed
>             fail_html = target_dir.joinpath(nb.stem + ".html")
>             with open(fail_html, "w", encoding="utf-8") as fh:
>                 fh.write(f"<html><body><h1>Execution failed for {nb}</h1><pre>{e}</pre></body></html>")
>     print("Done.")
> 
> if __name__ == "__main__":
>     main()
> ```
> 
> 4) requirements.txt
> ```name=requirements.txt
> jupyter
> nbconvert
> nbformat
> pytest
> flake8
> black
> pre-commit
> ```
> 
> 5) Dockerfile
> ```name=Dockerfile
> FROM python:3.11-slim
> 
> ENV DEBIAN_FRONTEND=noninteractive
> WORKDIR /app
> COPY requirements.txt /app/requirements.txt
> RUN python -m pip install --upgrade pip && \
>     pip install --no-cache-dir -r /app/requirements.txt
> COPY . /app
> CMD ["bash"]
> ```
> 
> 6) .pre-commit-config.yaml
> ```name=.pre-commit-config.yaml
> repos:
>   - repo: https://github.com/psf/black
>     rev: 24.1.0
>     hooks:
>       - id: black
>         language_version: python3.11
> 
>   - repo: https://gitlab.com/pycqa/flake8
>     rev: 6.0.0
>     hooks:
>       - id: flake8
> ```
> 
> 7) .gitignore
> ```name=.gitignore
> .env
> .venv
> __pycache__/
> site/
> gh-pages/
> *.pyc
> .ipynb_checkpoints/
> ```
> 
> 8) README.md (updated top-level README explaining builds and Pages)
> ````markdown name=README.md
> ```markdown
> # Repository: Notebook pages (branch-per-branch)
> 
> This repository contains many Jupyter notebooks. This change adds CI and an automated build-and-deploy workflow that runs on pushes to any branch and publishes rendered HTML pages for that branch under gh-pages/<branch-name>.
> 
> Files added:
> - .github/workflows/ci.yml — runs lint/tests and executes notebooks on pushes/PRs to main.
> - .github/workflows/deploy-notebooks-pages.yml — builds and deploys per-branch HTML pages to gh-pages when any branch is pushed.
> - scripts/build_notebooks.py — script that executes notebooks and exports them to HTML.
> - requirements.txt, Dockerfile, .pre-commit-config.yaml, .gitignore — tooling and environment files.
> 
> How to run locally:
> 1. python -m venv .venv
> 2. source .venv/bin/activate
> 3. pip install -r requirements.txt
> 4. python scripts/build_notebooks.py site/local-branch
> 
> Enable GitHub Pages to serve the gh-pages branch in repository Settings -> Pages. The per-branch pages will be available at https://<owner>.github.io/<repo>/<branch>/ .
> ```
> ````
> 
> Please create a feature branch (name it `ci/add-notebook-pages`) from `main`, add these files, commit them, and open a pull request titled "Add CI, notebook build scripts, and branch-per-branch GitHub Pages deployment" targeting `main`. The PR description should summarize the above problem_statement and list the files added.
> 
> Do not modify other repository files. If any file path already exists, update it to match the content above.
> 
> After creating the PR, please return the PR URL and the list of files added or changed with permalinks to the branch blobs in the PR branch.

<!-- START COPILOT CODING AGENT TIPS -->
---

✨ Let Copilot coding agent [set things up for you](https://github.com/solveforceapp/-/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pytest@​9.0.1
	pre-commit@​4.4.0
	nbconvert@​7.16.6
	black@​25.11.0
	jupyter@​1.1.1
	nbformat@​5.10.4
	flake8@​7.3.0

View full report