Added replacement facility for page content.

[jhogstrom]

By using either command line replacement pairs or specifying a file with json defintions of replacements an arbitrary regexp ("pattern") can be detected and replaced with another string, including expanding captured groups in the pattern.

The replacement phase is taking place just before upsert, so all other textual manipulations are done by that time.

Replacements happen in a deterministic sequence. There are ample opportunities to get unexpected (but logically consistent) results by inadvertently result of a previous replacement.

Format of json file:

{
    "environment": [
        {
            "import": "<module name>",
            "path": "<source file>"
        }
    ],
    "replacements":[
        {
            "name": "<name - optional>",
            "pattern": "<regexp>",
            "new_value": "<string with optional group expansions>"
            "evaluate": <true|false - optional>
        },
    ]
}

The environment block is optional and used for very dynamic replacements. By specifying a python source file, it will be dynamically imported at run time. The new_value field can then specify a <module>.<func> that returns a string value. As an example, the following adds a replacement of "TODAY" to an iso-formatted datetime.

{
    "environment": [
        {
            "import": "funcs",
            "path": "funcs.py"
        }
    ],
    "replacements":[
        {
            "name": "Todays date",
            "pattern": "TODAY",
            "new_value": "funcs.today"
            "evaluate": true
        },
    ]
}

Funcs.py:

import datetime

def today(term):
    return datetime.datetime.now().isoformat()

The parameter term is a Match object as per using https://docs.python.org/3/library/re.html#re.subn.

[iamjackg]

Sorry about the slew of comments -- this is really cool functionality (although it borders the edge of what I would consider appropriate in a tool that's "simply" uploading documents to Confluence) so I'd love for it to be polished for inclusion!

[iamjackg]

I would probably recommend doing this in YAML instead of JSON. Most other configuration files in the documentation-adjacent space tend to be YAML, and JSON is a valid subset of YAML anyway.

[iamjackg]

This is not particularly intuitive. I would suggest using two different fields instead of making the behaviour dependent on this flag (e.g. replacement_text and replacement_function).

[iamjackg]

Since this is implemented with an eval, the field can actually specify any valid python code at all, which can be a security issue. Granted, once you're importing a custom module all security concerns go out the window anyway, but I'd still prefer something a bit smarter, like only allowing module.function here, parsing it, and using getattr to get it from the module.

[iamjackg]

The name is not particularly intuitive. I would maybe use something like dynamic_replacement_modules

[iamjackg]

Suggested change

	parser.add_argument(
	"--replacements",
	dest="replacementfile",
	help="Filename with replacement definition in json format",
	)
	parser.add_argument(
	"--replacement-file",
	dest="replacement_file",
	help="Filename with replacement definition in YAML format",
	)

[iamjackg]

This can go wrong pretty easily if somebody forgets the =: it will just result in a non-obvious exception about the number of parameters when calling the constructor. I'd love to have this function receive data that's already well-structured. For example, you could parse the command line arguments in __main.py__ (with error handling if the format is wrong) and give create_replacement a well-formed list of lists, e.g.

[["pattern", "replacement], ["another pattern", "another replacement"]]

[iamjackg]

Suggested change

	file_replacements = json.load(open(replacementfile))
	with open(replacement_file) as fp:
	file_replacements = json.load(fp)

[iamjackg]

Suggested change

	"pattern": "<regexp>",
	"regex": "<regex>",

[iamjackg]

This behaviour is not documented anywhere. Is it really necessary? What's a use case for this vs. a string with newlines?

[iamjackg]

There are no tests at all for this :(