instructions for running versions of problemtools

[godmar]

Right now, the pipx and most recent release of problemtools won't work with Kattis, so we need to use older versions. The home page doesn't describe a way of doing so.

I've been pursuing the third supported way: Method 3: Running directly from the repository. However, it assumes that I have a virtualenv in ~/venv. The patch below fixes this:

diff --git a/bin/.run_in_venv.sh b/bin/.run_in_venv.sh
index dce93af..1f9746f 100755
--- a/bin/.run_in_venv.sh
+++ b/bin/.run_in_venv.sh
@@ -3,7 +3,11 @@
 # Helper script for the other wrapper scripts to check that a venv exists, or
 # give a helpful message if it doesn't.
 
-VENVPATH="$(dirname "$(dirname "$(readlink -f "$0")")")/venv"
+if [ -d "$VIRTUAL_ENV" ]; then
+    VENVPATH="$VIRTUAL_ENV"
+else
+    VENVPATH="$(dirname "$(dirname "$(readlink -f "$0")")")/venv"
+fi
 
 if [ ! -x "$VENVPATH/bin/python" ]; then
     echo "I could not find a python venv at $VENVPATH."

[gkreitz]

Right now, the pipx and most recent release of problemtools won't work with Kattis

Why do you believe the current release version of porblemtools don't work with Kattis? The current release version has been merged to master on Kattis and will be deployed very soon, but there are no real incompatibilities with the previous release version, which is what we run in production currently.

However, it assumes that I have a virtualenv in ~/venv.

No, it does not. It assumes you have a virtualenv in your problemtools checkout, as the error message also tells you:

gkreitz@norn:~/problemtools$ ./bin/verifyproblem.sh 
I could not find a python venv at /home/gkreitz/problemtools/venv.
To use these wrapper scripts, please set up a venv by:
 cd /home/gkreitz/problemtools
 python3 -m venv venv
 venv/bin/pip install -r requirements.txt

Your diff is a useful addition - if the user has activated a venv, it makes sense to use that instead of insisting on it sitting in a particular directory. If you make a PR with it, I'll happily merge it.

[godmar]

Right now, the pipx and most recent release of problemtools won't work with Kattis

Why do you believe the current release version of porblemtools don't work with Kattis?

Because Greg and Fredrik told me so. Perhaps I misunderstood.

In about September, when we were working to get the NAQ problemset ready, we tried the then current version of problemtools, but it was unusable (flagged files and directories in .gitignore, etc.). I'm currently using v1.20250606 which appears to work.

FWIW, here is my Dockerfile (for a github action runner) which may be of interest to others:

FROM myoung34/github-runner:ubuntu-noble

LABEL maintainer="godmar@gmail.com"
ENV DEBIAN_FRONTEND=noninteractive

# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199#23
RUN mkdir -p /usr/share/man/man1
RUN apt update && \
    apt install -y \
        ghostscript \
        libgmpxx4ldbl \
        python3-pkg-resources \
        python3-yaml \
        pandoc \
        python3-yaml \
        texlive-luatex \
        dvisvgm \
        texlive-fonts-recommended \
        texlive-lang-cyrillic \
        texlive-latex-extra \
        texlive-plain-generic \
        tidy \
        g++ \
        python3 \
        pypy3 \
        mono-complete \
        ghc \
        gccgo \
        fp-compiler \
        php-cli \
        swi-prolog \
        scala \
        sbcl \
        ocaml-nox \
        rustc \
        python3-virtualenv \
        python3-pip \
        libboost-regex-dev \
        openjdk-21-jdk

# now install problem tools

WORKDIR /home/runner
RUN git clone https://github.com/Kattis/problemtools.git
WORKDIR /home/runner/problemtools
RUN git checkout v1.20250606

RUN virtualenv /home/runner/problemtools/venv
ENV PATH="/home/runner/problemtools/venv/bin:$PATH"
RUN pip install -r requirements.txt
RUN make

# set homedir back to where github runner expects it
WORKDIR /actions-runner

[godmar]

Right now, the pipx and most recent release of problemtools won't work with Kattis

Why do you believe the current release version of porblemtools don't work with Kattis? The current release version has been merged to master on Kattis and will be deployed very soon, but there are no real incompatibilities with the previous release version, which is what we run in production currently.

However, it assumes that I have a virtualenv in ~/venv.

No, it does not. It assumes you have a virtualenv in your problemtools checkout, as the error message also tells you:

Here's a chance to improve the documentation. It does not mention that the venv directory must be created inside the problemtools directory. I discovered this later.

Why not say:

cd problemtools
python3 -m venv venv
venv/bin/pip install -r requirements.txt
make

(I also forgot to run make, which leads to a really mysterious "CRITICAL - no default validator found" error.)

On that note, you could add that it requires Python 3.12 (or at least 3.10 or later) and won't work with Python 3.10 that's installed by default on Ubuntu 22.04.

[gkreitz]

In about September, when we were working to get the NAQ problemset ready, we tried the then current version of problemtools, but it was unusable (flagged files and directories in .gitignore, etc.). I'm currently using v1.20250606 which appears to work.

This has me even more confused, as 1.20250606 has a regexp which forbids the existence of .gitignore. No version of problemtool has ever parsed a .gitignore to ignore files. v1.20250606 was also the current release version up until November 28th.

Regardless of any confusion, you should be safe to run the latest release version of problemtools when preparing problems for Kattis (which will very closely track problemtool releases). If I were to guess, Greg and Fredrik may have cautioned about using the 2023-07-draft problem format, which is still valid (and which will trigger a warning in problemtools, as problemtools support is also very incomplete).

When updating your dockerfile to the latest version of problemtools, do note the latex package dependencies have changed slightly, the problemtools README is up to date (from memory, texlive-lang-cyrillic is gone, you've already added texlive-luatex, and fonts-cmu fonts-noto-color-emoji are recommended). I'm not quite sure what is in the base image myoung34/github-runner:ubuntu-noble, but aside from that, what you're installing is also very close to what you'd get from the problemtools/full:v1.20250606 image we provide.

[gkreitz]

Here's a chance to improve the documentation. [...]

Excellent suggestions, thank you!

[godmar]

In about September, when we were working to get the NAQ problemset ready, we tried the then current version of problemtools, but it was unusable (flagged files and directories in .gitignore, etc.). I'm currently using v1.20250606 which appears to work.

This has me even more confused, as 1.20250606 has a regexp which forbids the existence of .gitignore. No version of problemtool has ever parsed a .gitignore to ignore files. v1.20250606 was also the current release version up until November 28th.

The version we used flagged, for instance, the existence of .class files in the submissions/accepted directory as an error, even though we had a .gitignore file that ignored those .class files. This made it very annoying (given that we commonly have .class files in the accepted directory during development.) Older versions of problemtools would not flag .class files in accepted. (I'm not saying that problemtools ever looked at .gitignore files, but you should before enforcing silly rules like this one.) It also flagged .gitkeep files intentionally placed in directories, same issue.

Those were all reasons why we reverted to older versions of problemtools which we still happened to have installed.

[gkreitz]

The version we used flagged, for instance, the existence of .class files in the submissions/accepted directory as an error,

I assume you mean that you have a java solution foo.java directly in submissions/accepted, and for manual testing, you've compiled it to create a foo.class directly in submissions/accepted? The code that looks at that hasn't changed in a long time, and I cannot recreate the issue you're describing at all (I tested the two latest releases, and a couple of commits in between in case you were using master). Or is there some tool which drops a file called only .class?

I'm not saying that problemtools ever looked at .gitignore files, but you should before enforcing silly rules like this one.

We do try to follow the problem package format, which is where this "silly rule" comes from (and I have a PR, Kattis/problem-package-format#526 to make the standard more permissive for legacy problems). I disagree that we should attempt to parse .gitignore files.

Note that problemtools is intended as a checker/linter for problem packages. It's job is to be nit-picky and complain about potential problems. That said, we should of course still offer good ergonomics for developing problems. Would you mind describing your workflow a bit more in detail? You've posted a dockerfile for a github CI runner, but you've also mentioned having .class files and things in .gitignore complained about, so I assume you're using problemtools both in CI and locally?

[godmar]

The version we used flagged, for instance, the existence of .class files in the submissions/accepted directory as an error,

I assume you mean that you have a java solution foo.java directly in submissions/accepted, and for manual testing, you've compiled it to create a foo.class directly in submissions/accepted? The code that looks at that hasn't changed in a long time, and I cannot recreate the issue you're describing at all (I tested the two latest releases, and a couple of commits in between in case you were using master). Or is there some tool which drops a file called only .class?

Sure. A typical workflow: I work on a sample solution in the accepted directory using the editor that allows me to type at the speed at which I think (I'm slow, so I'm using vim). I suspend the editor and run verifyproblem, which with the latest version gives me:

(cd ../../..; ~/problemtools/bin/verifyproblem.sh busiestantlevel)
Loading problem busiestantlevel
ERROR Invalid file name '.BusiestAntLevel.java.swp' in busiestantlevel/submissions/accepted (should match ^[a-z0-9][a-z0-9_.-]{0,253}[a-z0-9]$ ignoring case)
ERROR Invalid file name 'BusiestAntLevel$DFSEntry.class' in busiestantlevel/submissions/accepted (should match ^[a-z0-9][a-z0-9_.-]{0,253}[a-z0-9]$ ignoring case)
ERROR Invalid file name '.2.in.swp' in busiestantlevel/data/sample (should match ^[a-z0-9][a-z0-9_.-]{0,253}[a-z0-9]$ ignoring case)
ERROR Invalid file name '.1.in.swp' in busiestantlevel/data/sample (should match ^[a-z0-9][a-z0-9_.-]{0,253}[a-z0-9]$ ignoring case)

(I have files 1.in and 2.in open in vim, I have BusiestAntLevel.java open (that's the .swp) files, and from prior testing I have a .class file. Interestingly, it seems to be complained about the nested class name with the dollar ($) sign, since it's not complaining about BusiestAntLevel.class which is also there.)

On the other hand, we have maketest.py in data/secret about which it doesn't complain.

I'm not saying that problemtools ever looked at .gitignore files, but you should before enforcing silly rules like this one.

We do try to follow the problem package format, which is where this "silly rule" comes from (and I have a PR, Kattis/problem-package-format#526 to make the standard more permissive for legacy problems). I disagree that we should attempt to parse .gitignore files.

Note that problemtools is intended as a checker/linter for problem packages. It's job is to be nit-picky and complain about potential problems.

We managed just fine the last 12 years or so without such checks. Has there ever been an issue caused by extra files?

PS: if you want to warn about something useful, warn if somebody calls wrong_answer wrong-answer - this is right now completely ignored, and the WA are not run.

[Matistjati]

PS: if you want to warn about something useful, warn if somebody calls wrong_answer wrong-answer - this is right now completely ignored, and the WA are not run.

Good idea #377

[gkreitz]

Thanks for outlining your workflow, it's very valuable input.

The .sw? files from vim should be ignored according to the just-updated spec, so that will be fixed in a future version of problemtools.

The *$*.class file, however, would still be an illegal file name. Flagging it as an error is to some degree a matter of perspective. I fully understand why it's there during development, but I also feel it would be an error if you sent us (or someone else) a problem package littered with compiled programs, swp files, et.c. Now, I'm gonna assume you use git to ship problems, and have this covered via a .gitignore, so that would never happen, but we also have much less experienced problem authors who occasionally ship things as zip files.

Taking a step back from the filename regex, longer term, I would want to add at least a warning for files and directories problemtools does not know what to do with (for basically the same reason as your excellent suggestion to warn about wrong-answer). That would warn about dropping .class files in submissions/<status>, where each file is supposed to be a submission.

I've given this a bit of thought, but I'm still not sure I have a good solution. The options I'm weighing are:

• Leave it as-is - it is expected behavior that we flag clutter as errors
• Add some sort of --development or --allow-unused-files flag to disable this warning specifically
• Decide that it is unclear what problem was actually solved by the standard adding a regex, and back that out from problemtools (or degrade it to a warning)
• Allow a configuration file to more generally enable/disable warning (similar to most linters)

We managed just fine the last 12 years or so without such checks. Has there ever been an issue caused by extra files?

Not that I'm aware of. If you do have the time and energy to participate in discussions on the format on https://github.com/Kattis/problem-package-format , I'm sure it would be very valuable input there. I fully understand if you don't - I must admit I don't really have the time and energy for it myself, so I mostly try to follow the spec.

open in vim

Excellent editor choice. :)

[godmar]

More feedback:

• I often do gcc -o w program.c; ./w < ../../data/sample/1.in - this now becomes an error.
• If, on the other hand, I do gcc program.c; ./a.out < ../../data/sample/1.in - it's not an error.
• A student left a CUDA solution solution.cu in the accepted directory. It's not an error, just silently ignored.