Pre-PLIP: maintenance/recurrent/one-off scripts support

[gforcada]

PLIP (Plone Improvement Proposal)

Responsible Persons

Proposer: Gil Forcada

Seconder:

Abstract

Get a way to run scripts and show some proper output.

Motivation

My use case: I have a slew of scripts +50, that I need to run every now and then on our Plone instance, either to gather some data, to generate something, etc.

My workflow is:

• create the python script and test it locally
• scp the script to one Plone servers
• ./bin/instance -O Plone run XXX.py

That's perfect to debug and iterate with the script, but once it's stable, having a control panel where it lists the scripts available and pressing a button is way better 😄

Assumptions

Running maintenance scripts on production/staging/testing servers is something that might appeal not only to me.

Using upgrade steps means having to re-create over and over the same upgrade step, which makes it not really fit for the job.

We have a way to register scripts, with Generic Setup, python entry points, etc. I'm open to the exact implementation.

We use some convention like that each script (a single python module) has to have a specific function name (def run(app):) and that its output should be a generator of either plain text lines (to display the progress) or something like JSON tables so some nice formatting can be generated.

Proposal & Implementation

• a way to define scripts
• a way to see the available scripts (a control panel I would say)
• a way to run those scripts (within the control panel)
• a convention on how those scripts are run, which parameters (if any) they get, the output format, etc.

One design goal should probably be that scripts should be on the filesystem, i.e. delivered with a python distribution or to be found on a specific folder within the Zope instance, but nothing that should be uploaded on Plone. There is this python scripts thing that might be somehow what I'm proposing here but has its drawbacks (I never used it, but that's the wisdom I keep getting from them).

Deliverables

• documentation on how to create scripts
• documentation on how the control panel works
• some example scripts on CMFPlone (or somewhere else) as ready to use examples
• the control panel that shows and runs the scripts
• the discovery system to find the scripts

Risks

Keeping random scripts everywhere without any structure and easy to forget, not being easy to share with coworkers, not being easy to share upstream in Plone, etc.

Participants

[mauritsvanrees]

Could be useful.

But couldn't this also be done by just creating a browser view for each script, with essentially the same code? And then you could create a control panel with a simple page template behind it, that you manually update whenever you create a new browser view. Or you automatically show a list of all browser views of which the name starts with cp_scripts_ or something like that. And show the docstring of each view as a help text. In that case the control panel could be done in a collective add-on, with some docs on how to register views in a way to have them show up there.

Alternatively, from the old days I remember that you can go to the ZMI and add an External Method. An External Method is also where we used to have installation code for the QuickInstaller, at a time when GenericSetup was not available yet. You could create a folder scripts on the filesystem, make sure it ends up in sys.path, and put your scripts in there. As module name you would then use scripts.scriptname and as function name probably run. And then in a control panel I suppose you could automatically get a list of all external methods that have been added in the Plone Site root.

Those are just some ideas that might work already, without extra functionality in core Plone.

[davisagli]

Some benefits to running a script in a separate process are:

• it won't hit request timeouts in proxies in front of Plone
• it won't get killed due the host process getting restarted when it fails a healthcheck
• it doesn't use a ZODB connection from the same pool as a process that is serving requests

To solve this well requires a framework for running asynchronous tasks and sending notifications when they have been interrupted.