Pandoc package requires explicit installation

[nblettner]

I encountered an issue when getting started to contribute. Following the workflow for contributing an error was raised when executing nox -s docs -- --serve (same for nox -s docs).  Part of the error message was:

nbconvert.utils.pandoc.PandocMissing: Pandoc wasn't found.

That is, pandoc is not installed automatically based on the current setup. Instead, pandoc needs to be installed separately, and this installation is essentially only possible via conda (see pandoc project description). It might not be important in which environment it is installed, and the issue may go unnoticed if you happen to have had it installed before. Also, when executed online, no error is raised, presumably  because pandoc is part of the image that is used for the container in which the code runs for the build of the docs on readthedocs.org.

For developing, it seems like the use of conda and an explicit installation of pandoc is required. The workflow for contributing needs to be adjusted accordingly.

[cchwala]

Thanks for finding and raising this issue @nblettner.

I just found that there is a pypandoc version called pypandoc_binary which installs also the pandoc binaries via pip, see the info here.

Hence, one easy fix right now, without requiring everyone to use a conda env as basis, could be to add pypandoc_binary as dependency for the doc build process.

[cchwala]

After uninstalling pandoc from my conda base env I can reproduce this error.

[cchwala]

The good news is that pip install pypandoc_binary results in having a pandoc binary in the env it is installed to. The bad news is that the pandoc binary is not added to the PATH or to the /bin directory of the env.

Making the env aware of the pandoc binary is complicated since we have to do that inside the nox session that runs the build of the docs.

My current solution is to use a small script that creates a symbolic link form the env's /bin dir to the pandoc binary that lives somwhere in e.g. ......./lib/python3.10/site-packages/pypandoc/files/pandoc.

[bwalraven]

I get the following error message when trying to build the docs using nox -s docs -- --serve:
FileNotFoundError: [WinError 3] The system cannot find the path specified: '../lib/python3.10/site-packages/pypandoc/files/pandoc' -> '..\\.nox\\docs\\bin\\pandoc'
and then
nox > Command python set_pandoc_symlink.py failed with exit code 1.

This seems to make sense because there is no bin folder inside my .nox\docs folder. Any ideas how to fix this?

[cchwala]

...seems that my fix from #19 did only partially solve this issue. It worked for me and @nblettner, but we both use MacOS. Hence, this is maybe a Windows-related problem because of different paths where things are stored.

[cchwala]

@bwalraven Can you please check if the following file is present in your poligrain directory

.nox/docs/lib/python3.10/site-packages/pypandoc/files/pandoc 

The problem is that this script, which I added to fix the initial pandoc issue, does not run on your machine, as stated by this output that you got Command python set_pandoc_symlink.py failed with exit code 1.

[cchwala]

...ah now I see.... 🤦

The path to the pandoc binary is hardcoded in the script, not accounting for the different slashes that Windows uses for directories... 😬

I guess the solution will be to do

p.symlink_to(
        pathlib.Path(f"../lib/python{python_version[0]}.{python_version[1]}/site-packages/pypandoc/files/pandoc")
)

instead of just passing the path as string.

@bwalraven Would be great if you could test that out on your machine by modifying the script.

But I would suggest to not include that in your current PR. Either open a new one from a new branch, or just confirm here that it works and I can do a small PR.

[bwalraven]

Thanks for looking into this @cchwala. Unfortunately, it doesn't work yet with your suggestion. I also noticed my .nox folder structure might also be slightly different.

I have a pandoc.exe file located at .nox/docs/lib/site-packages/pypandoc/files/pandoc, so without the python version in there. site-packages is the only folder inside my lib folder.

So I also tried

p.symlink_to(
        pathlib.Path(f"../lib/site-packages/pypandoc/files/pandoc")
)

but without succes.

The full Traceback call I get is:

Traceback (most recent call last):
  File "C:\Users\baswalraven\OneDrive - Delft University of Technology\Documents\OpenSense\WG2\poligrain\docs\set_pandoc_symlink.py", line 14, in <module>
    p.symlink_to(
  File "C:\Users\baswalraven\Miniconda3\envs\test_env\lib\pathlib.py", line 1255, in symlink_to
    self._accessor.symlink(target, self, target_is_directory)
FileNotFoundError: [WinError 3] The system cannot find the path specified: '..\\lib\\site-packages\\pypandoc\\files\\pandoc' -> '..\\.nox\\docs\\bin\\pandoc'
nox > Command python set_pandoc_symlink.py failed with exit code 1
nox > Session docs failed.

So it indeed seems something related to Windows paths, but it might be easier debugging this f2f..

[cchwala]

Okay. Since this is not urgent, let's discuss it f2f next week.