feat: add %log start|off|on|state|level|stop for kernel logging

[maxandersen]

its notoriously hard to get kernel output so added ipython inspired %log commands magic
to enable better inspection.

with this the kernel can start adding logging more widespread for easier debugging.

Usage: %log <command>

Commands:
  start [filename] - Start logging to file (default: <kernelname>.log)
  stop             - Stop logging and close log file
  on               - Resume logging (if paused)
  off              - Pause logging (but keep file open)
  state            - Show current logging status
  level <LEVEL>    - Set logging level (SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST)

Examples:
  %log start                   - Start logging to <kernelname>.log
  %log start myapp.log         - Start logging to myapp.log
  %log off                     - Pause logging
  %log on                      - Resume logging
  %log state                   - Check status
  %log level WARNING           - Set logging level
  %log stop                    - Stop and close log file

[maxandersen]

any feedback/comments?

[andrus]

I was able to play with it locally, and this is pretty neat, and I am +1 on the feature in general. The overall logging architecture was something we needed to revisit, and I think this magic will work much more nicely once we do it:

SLF4J vs JUL (Java built-in logging)

My preference is generally SLF4J, but maybe this is just because JUL outputs 2 lines by default, and I am annoyed by it 🙂 (I know it can be customized with a formatter for a 1-line output). FWIW, SLF4J + Logback would allow lots of advanced configuration of output files, rotation and such.

Various log origins

Logs can come from the notebook, the libraries used in the notebook, and the kernel itself. All of them can be using different loggers:

• System.out
• System.err
• JUL
• SLF4J
• (Log4J maybe?)

The suggested magic would only do log routing for JUL loggers:

// this would get rerouted to file
 java.util.logging.Logger.getLogger("test").info("JUL log");

// this will be ignored
org.slf4j.LoggerFactory.getLogger(DataFrame.class).info("DFLib log");

// this will be output within the notebook
System.err.println("Error!")

So, how do we decide which one goes where? Does the ipython %log take over stdout / stderr?

[maxandersen]

I would refrain from using/forcing log4j by default - can always use and wire it up specifically if you really need it. log4j should just use log4j-jul and things should be aligned?

[maxandersen]

about stdout/stderr - then those are "lost". stdout goes to notebook anyways - and stderr is not visible afaik (haven't tested/verified). I wouldn't include them in any log control.

[andrus]

I would refrain from using/forcing log4j by default - can always use and wire it up specifically if you really need it. log4j should just use log4j-jul and things should be aligned?

I think 90% of common third-party libs are using SLF4J for logging (as it can be easily bridged to anything). Including SLF4J in the kernel would've solved a lot of problems. But of course this is a slippery slope and we want to stay away from dependencies. I guess we can declare JUL to be the "logger of JJava", and provide simple recipes for bridging SLF4J and others. Need to experiment with that.

Ok, this turned out to be a non-issue once we fixed the underlying bugs and did proper dependency shading.

about stdout/stderr - then those are "lost". stdout goes to notebook anyways - and stderr is not visible afaik (haven't tested/verified).

Stderr goes into notebook as well (with red background). FWIW, we control both those streams and can send them anywhere.

I wouldn't include them in any log control.

Fair enough. Let me flip the question then - most Java loggers by default (with no explicit config) send logs to stdout. So following that logic, the output of java.util.logging.Logger.getLogger("").info("Hi!") should go into the notebook itself. Currently, it is sent to the Jupyter server console.

I hope this is not a big deal, and just some prior IJava decision inherited by JJava that we can easily deal with. Just outlining the scope of work 🙂

And this takes us to another Jupyter-specific thing: there two logical "consoles" - the notebook itself, and the Jupyter server console (logging in the latter is a hot mess now with a mix of python, 2-line JUL, and SLF4J warnings).

So should the %log magic differentiate between these log targets:

1. Log to notebook
2. Log to Jupyter console (I think this is the default now? Maybe the notebook should be the default instead)
3. Log to file

[andrus]

Talking of SLF4J, somehow it managed to get into our main distro jar through a transitive dependency - #99 It is just everywhere these days. Anyways, after fixing that task, I experimented with it in a notebook, and it looks pretty good...

Sending logs to notebook

%maven org.slf4j:slf4j-simple:2.0.15
    
import org.slf4j.*;
Logger logger = LoggerFactory.getLogger("nl");
logger.info("Hi!")

With slf4j-simple, the logs end in the notebook stderr.

Sending logs to JUL

%maven org.slf4j:slf4j-jdk14:2.0.15
    
import org.slf4j.*;
Logger logger = LoggerFactory.getLogger("nl");
logger.info("Hi!")

With slf4j-jdk14, the logs are printed in the Jupyter console:

Oct 19, 2025 11:42:18 AM REPL.$JShell$16 do_it$
INFO: Hi!

So:

1. Shading SLF4J used by the kernel dependencies worked. The internal SLF4J is now effectively hidden from the notebook users and is not a factor in our discussion.
2. Using your own SLF4J in a notebook is possible and is rather trivial
3. SLF4J assumes the notebook is the default console (something I think we should replicate for JUL)