Overview

Overview of codega

The basics of codega are simple: some source (like a file, a directory, a Python module, etc.) needs to be read and with the help of a generator (later on this) converted to a file. It sounds simple, but in reality it's also simple. But a few tools can make it simpler.

If the source is an XML file, there is no need to write a special parser. The XML format - so to speak - is codegas native source format. Other sources may need to have an additional source class that can parse some source and create an appropriate structure that can be rendered by a generator.

codega uses the lxml etree library for representing the XML nodes. Fortunately lxml has proper documentation at http://lxml.de/tutorial.html.

The cgx script

codega comes with a script named cgx. cgx has multiple commands. The list of commands along with their purpose can be listed with the help command:

$ cgx help
codega run-time script

 help     Display list of commands with their meaning
 make     Build codega targets listed in the make file
 clean    Clean up codega targets and any additional files listed in the make file
 build    Build the source with specified generator
 pack     Create a script containing the compressed codega module and the main script

• help displays a short help message.
• make builds targets specified in an XML config file. The format of this file is described in detail in ConfigFormat along with the build system.
• Using the makefile, clean will remove generated files.
• build takes a source and a target and generates the output. This is different from make because the source and target is specified on the command line.
• pack creates a self-contained script (this can be distributed without installing codega).

There is one option which is not specified in the helps (it will be fixed): -v. -v enables logging on and above a specified log level. The log levels are the following:

• debug or 4 - information usefull only for debugging
• info or 3 - informative but not problematic information
• warning or 2 - some problem occured but the execution could continue
• error or 1 - some problem occured where the current operation was terminated
• critical or 0 - some critical problem occured that terminated the program

The -v option can be specified anywhere (even before the command). It won't show up for the other commands.

Codega comes with several examples, so we will use the books example to demonstrate how to use the cgx script.

cgx make

cgx make also has a help:

$ cgx make --help
Usage: cgx [options]

Options:
  -c CONFIG, --config=CONFIG
                        Specify config file (default: codega.xml)
  -t TARGET, --target=TARGET
                        Specify targets (default: all)
  -f, --force           Force rebuild
  -h, --help            show this help message and exit

Running this to build the books example is easy: just go into the examples/books path and invoke cgx make without arguments

$ cd examples/books
$ cgx make

or directly specify the config file with -c

$ cgx make -c examples/books/codega.xml

codega will generate all files relative to the destination defined in makefile (see the makefile description for further information on path lists) which in turn is relative to the config file. So the books.c and books.html files in the above example will always be generated in the examples/books directory.

The -t option can be used to specify the targets to be built.

$ cgx make -c examples/books/codega.xml -t books.c

This will only build the target books.c. Multiple targets can be given, in which case the specified targets will be built (in the order they were specified).

Files whose source didn't change since the last generation will not be generated by default. To force the rebuild add the -f option. This will cause the build process to run even if the inputs and outputs weren't changed. If -f is not specified, the source file, the codega module and the destination file modification times will be compared to determine if rebuilding the targets is necessary.

cgx build

codega can build a file even if there is no config file. This is the build command:

$ cgx build --help
Usage: cgx [options]

Options:
  -s SOURCE, --source=SOURCE
                        Specify the source (file, directory, etc., depending
                        on the parser)
  -p PARSER, --parser=PARSER
                        Specify the parser in <module>:<source class> format
                        (default: codega.source:XmlSource)
  -t TARGET, --target=TARGET
                        Specify the target file (default: codega.out)
  -g GENERATOR, --generator=GENERATOR
                        Specify the generator in <module>:<source class>
                        format
  -I INCLUDE, --include=INCLUDE
                        Add a new search path (for modules, sources, etc.)
  -c CONFIG, --config=CONFIG
                        Write the equivalent config to the given file
  -h, --help            show this help message and exit

In order to generate a file, -s, -t and -g must be specified (-p also if the source is not an XML). We can build the books example with this command:

$ cgx build -s examples/books/books.xml -t books.c -g bookgen:CBookGenerator -I examples/books

In this case (unlike with make) the target is generated relative to the current working directory. If no -I is specified, the current working directory is also used for searching for modules. Any number of search paths can be specified with -I, they'll be scanned in the order they were given.

With the -c option we can get the equivalent config file of this build command.

cgx clean

This command can be used to clean up generated files. The clean command uses the same configuration file as make does.

$ cgx clean --help
Usage: cgx [options]

Options:
  -c CONFIG, --config=CONFIG
                        Specify config file (default: codega.xml)
  -h, --help            show this help message and exit

If we need to clean up the results of the make in build, simply run the command:

$ cgx clean -c examples/books/codega.xml

Codega modules

• codega: The main API module
  • build: The builder that is used to control the build process
  • config: Configuration classes, the parser and saver. See Configuration
  • context: Generation context class
  • commands: Commands for the CLI.
  • decorators: Utility function decorators
  • error: Exception definitions specific to codega
  • generator: Generator base classes, see Generators
  • logger: Makes using Pythons logging module easier
  • ordereddict: Implementation of an ordered dictionary
  • rsclocator: Resource locator classes
  • source: Source base classes.
  • stringio: Wrapper module for conveinently importing StringIO or cStringIO
  • template: Template base classes.
  • version: Module for the Version class.
  • visitor: Generic visitor classes.
• cgextra: Extra modules
  • dicttools: Dictionary manipulation tools
  • flavor: Flavor tool.
  • indent: Helper classes for handling intentation, commenting, etc.
  • makowrapper: Template extension for the mako templating engine
  • matcher: Objects for making matching easier
  • scope: Scope handling utility.
• tests: Unit- and functional tests
• examples: Small example codega projects

Configuration

Every command creates a codega.config.Config instance (even build does this internally). This contains everything specified in the ConfigFormat document.

The API documentation is in codega.config, please refer to this.

Generators

A generator object is used to generate a target from the source. There are no restrictions on the structure of the source (by default the source is an lxml.etree._Element object). A generator object has a generate() method that is called with two arguments:

• The source argument refers to the source.
• The context argument is a codega.context.Context instance containing information about the current target and source from the configuration and the configuration itself.

Each generator object needs to be descendent of the codega.generator.base.GeneratorBase class.

Object generators

One generator for a task is usually insufficient: the source object may have some sub-items which should have their own generators. So a lot of generators need to be organized into one, bigger generator.

Object generators are the easiest way to do this. Object generators descend from the codega.generator.object.ObjectGenerator class. The class, uppon instantiation collects specially decorated functions from the class. These functions are turned into a sub-generator.

from codega.generator.object import *
from codega.generator.function import *

from cgextra import matcher

class ExpressionGenerator(ObjectGenerator):

    @matcher(match.tag('operator'))
    @generator(FunctionGenerator.factory)
    def generate_operator(self, source, context):
        return source.attrib['operator'].join(context.map(self, source))

    @matcher(match.tag('value'))
    @generator(FunctionGenerator.factory)
    def generate_value(self, source, context):
        return source.text.strip()

    @priority(PRI_FALLBACK)
    @generator(FunctionGenerator.factory)
    def generate_fallback(self, source, context):
        return '/* Unknown source %r */' % source

    ...

In the above example ExpressionGenerator is the main generator. generate_operator, generate_value and generate_fallback are subgenerators. We use the generator decorator to denote that the function is a sub-generator.

The following decorators can be used:

• generator as noted above denotes that the function is a sub-generator. The argument to the decorator is a factory function: this will be called to turn the function into a generator. This decorator is mandatory.
• matcher can be specified any number of times. The matcher takes a callable argument which will determine if the sub-generator can handle the source. If more than one are specified, each one will have to return True for a generator to match the source.
• priority needs to be specified if two generators can match the same source. The one with the highest priority (i.e. lowest value) will be checked first.

Other generators

There are several other generator classes, some of which are useful tools in generation tasks.

• codega.generator.template.TemplateGenerator: instead of directly returning a rendered string, only a dictionary with bindings is returned, and a codega.template.TemplateBase-descendent instance is used to create the rendered string.

from codega.generator.object import *
from codega.generator.template import *

template = SomeTemplate()

class ExpressionGenerator(ObjectGenerator):
    @matcher(matcher.tag('reference'))
    @generator(TemplateGenerator.factory(template))
    def generate_reference(self, source, context):

        ...

        return dict(name = source.attrib['name'])

• cgextra.makowrapper.DocstringMakoTemplate this is a template generator which uses the functions docstring as a mako template. The generator factory to use is cgextra.makowrapper.inline(). The template is then rendered with the resulting dictionary.

from codega.generator.object import *
from cgextra.makowrapper import *

class ExpressionGenerator(ObjectGenerator):
    @matcher(matcher.tag('reference'))
    @generator(inline)
    def generate_reference(self, source, context):
        '''
        &${name}
        '''

        return dict(name = source.attrib['name'])