Kettingar is a micro-framework for building Python microservices that expose an HTTP/1.1 interface. The motivation was to solve the folowing two use cases:
- Split a complex application into multiple cooperating processes
- Control, inspect and manage simple Python background services
Some features:
- Fully async
- Authenticated and unauthenticated API methods
- Allows generator functions to incrementally provide results
- Supports passing open file descriptors to or from the microservice
- Supports msgpack (preferred) or JSON for RPC request/response
- Python type annotations are used to validate/convert arguments
- Serve over TCP/IP and over a local unix domain socket
- Built in CLI for configuring, running and interacting with the service
- Tested on Python 3.10, 3.13 and 3.14
See below for a bit more discussion about these features.
Status: Useful!
TODO:
- Improve and document how we do logging
- Document metrics and internal stats
- Document the
public_raw_andraw_magic prefixes - Document error handling / exception propogation
- Add websocket support
- Use types and annotations to bring more clarity to return values
If you think you can help with any of those, feel free to ping me!
git clone https://github.com/mailpile/kettlingar.git
cd kettlingar
pip install .git clone https://github.com/mailpile/kettlingar.git
cd kettlingar
pip install -e ".[dev]"See the examples/
for fully documented versions of the snippets below,
as well as other helpful examples.
This is a kettlingar microservice named MyKitten:
import asyncio
from kettlingar import RPCKitten, HttpResult
class MyKitten(RPCKitten):
class Configuration(RPCKitten.Configuration):
APP_NAME = 'mykitten'
WORKER_NAME = 'Kitty'
async def public_api_meow(self, _request_info):
return HttpResult(
'text/plain', # Fixed MIME type of `text/plain`
'Meow world, meow!\n') # Meow!
async def api_purr(self, _request_info, count:int=1, purr:str='purr'):
_format = self.config.worker_name + ' says %(purr)s'
for i in range(count):
yield {
'purr': purr * (i + 1), # Purring!
'_format': _format} # Formatting rule for CLI interface
await asyncio.sleep(1)
if __name__ == '__main__':
MyKitten.Main()This (or something very similar) can be found in the examples folder, and run like so:
$ python3 -m examples.kitten help
...
$ python3 -m examples.kitten start --worker-listen-port=12345
...
$ python3 -m examples.kitten ping
Pong via /path/to/mykitten/worker.sock!
$ python3 -m examples.kitten meow
{'mimetype': 'text/plain', 'data': bytearray(b'Meow world, meow!\n')}
$ curl http://127.0.0.1:12345/meow
Meow world, meowThis is an app that uses the microservice:
import asyncio
import sys
from .kitten import MyKitten
async def test_function():
kitty = await MyKitten(args=sys.argv[1:]).connect(auto_start=True)
print('%s' % kitty)
print('Our first meow: %s' % (await kitty.meow()))
async for result in kitty.purr(10):
print(result['purr'])
await kitty.quitquitquit()
asyncio.run(test_function())If you prefer to write synchronous code (instead of async), that is possible as well, but will be somewhat less efficient. See test_kitten_sync for an example.
As illustrated above,
an API endpoint is simply a Python function named using one of the following prefixes:
public_api_, or api_ -
to understand the difference between them, see Access controls below.
These functions must always take at least one positional argument
(_request_info),
and must either return a result, or an HttpResult(mimetype, data).
Instead of returning a single value it is also supported to implement a generator which yields multiple results,
(the first of which may be an HttpResult).
Arguments after _request_info (both positional and keyword arguments) are part of the exposed API.
If the arguments have type hints,
kettlingar will automatically convert/validate incoming data accordingly.
The functions should also have docstrings which explain what they do and how to use them.
The functions can access (or modify!) the microservice configuration,
as self.config.* (see below for details).
That's all;
kettlingar will automatically:
- define HTTP/1.1 paths within the microservice for each method,
- create Pythonic client functions which send requests to the microservice and deserialize the responses, and
- create a command-line (CLI) interface for asking for
helpor invoking the function directly
Note that kettlingar microservices are single-threaded,
but achieve concurrency using asyncio.
Any API method could block the entire service,
so write them carefully!
In the above example,
kettlingar uses introspection (the inspect module) to auto-generate client functions for each of the two API methods:
MyKitten.meow() and
MyKitten.purr(...).
When invoked,
each client function will use MyKitten.call(...)
and then either return the result directly or implement a generator.
Digging even deeper,
MyKitten.call(...) makes an HTTP/1.1 POST request to the running microservice,
using msgpack to bundle up all the arguments and deserialized the response.
(JSON is also supported for interfacing with humans or systems lacking msgpack.)
In the case of the generator function, the HTTP response uses chunked encoding, with one chunk per yielded purr. Kettlingar also supports Server Sent Events in generator functions, as illustrated in the htmx example.
Kinda. Kettlingar implements a subset of the HTTP/1.1 specification.
This allows us to use tools like curl or even a web browser for debugging and should facilitate use of non-Python clients.
But the emphasis was on simplicity (and hopefully performance),
rather than a complete implementation.
If you want to implement a public landing page at /
(the root of the web server)
you can do so by creating an API endpoint named public_api_web_root.
If you want something a bit more simlar to a modern web framework -
with an URL routing table,
Jinja templating,
and static resources,
you can use the WebKitten mix-in as demonstrated in
examples/htmx.py.
And if you want a TLS-enabled server,
there is also a TLSKitten mix-in
(see: test_tlskitten).
For lower-level applications the routing mechanism can be customzied by subclassing and overriding or extending
RPCKitten.get_method_name(...) and/or
RPCKitten.get_default_methods(...)
(see example in examples/kitten.py).
A kettlingar microservice offers two levels of access control:
- "Public", or unauthenticated methods
- Private methods
Access to private methods is granted by checking for a special token's presence anywhere in the HTTP header. The common case is for the token to be included in the path part of the HTTP URL, but you can use any (standard or made up) HTTP header if you prefer.
In the case where the client is running on the same machine, and is running using the same user-id as the microservice, credentials (and the unix domain socket, if it exists) are automatically found at a well defined location in the user's home directory. Access to them is restricted using Unix file system permissions.
The kettlingar command-line interface aims to solve these tasks:
- Administering the microservice (start, stop, get help)
- Configuring the microservice (including defining a config file format)
- Running functions on the microservice (testing, or real use from the shell)
A kettlingar microservice will out-of-the-box support these CLI commands:
start- Start the microservice (if it isn't already running)stop- Stop the microservicerestart- Stop and Startping- Check whether the microservice is runninghelp- Get help on which commands exist and how to use themconfig- Display the configuration of the running microservice
Each RPCKitten subclass contains a Configuration class which defines a set of constants in all-caps.
This defines
a) the names of the microservice configuration variables and
b) their default values.
For each VARIABLE_NAME defined in the configuration class,
the instanciated configuration objects will have a variable_name (lower-case) attribute with the current setting.
This can be accessed from within API methods as self.config.variable_name.
The value can be set on the command-line
(when starting the service) by passing an argument --variable-name=value.
Examples:
## Tweak the configuration on the command-line
$ python3 -m examples.kitten start --app-name=Doggos --worker-name=Spot
...
## Load settings from a file
$ python3 -m examples.kitten start --worker-config=examples/kitten.cfg
...
## View the current configuration as JSON
$ python3 -m examples.kitten --json config
...Note that the app_name and worker_name settings influence the location of the authentication files (see Access controls above),
so changing either one will allow you to run multiple instances of the same microservice side-by-side.
The arguments and keyword arguments specified on the api_* functions translate in the "obvious" way to the command-line:
positional arguments are positional,
keyword arguments can be specified using --key=val.
Example:
$ python3 -m examples.kitten purr 2 --purr=rarr
Kitty says rarr
Kitty says rarrrarrNote that if you expect to use your microservice from the command line,
you will need to either handle any type conversions (from string representations) yourself within the function,
or add type annotations to the arguments -
kettlingar knows how to automatically convert basic types
(ints, floats, bools, ...)
from common string representations.
For custom classes/types,
if your class can be constructed from a single string argument,
that should work as well,
but beware that msgpack may also need help with the de/serialization.
A kettlingar microservice will by default listen on both a TCP/IP socket,
and a unix domain socket.
Processes located on the same machine should use the unix domain socket by default.
This theoretically has lower overhead (better performance), but also allows the microservice to send and receive open file descriptors (see filecat.py and test_filecat.py for a few demos).
This can be used in a few ways:
- A kettlingar RPC method can delegate responding to another microservice,
without needing to copy the response (or even wait for the call to complete).
This is supported in the Python client by passing
call_reply_to=...as an argument to the autogenerated client helper function (orRPCKitten.call()directly). - A server process can listen for incoming TCP/IP client connections, and then pass the connection to a worker process to finish the job.
- Seamless upgrades of running servers; a graceful shutdown command could send all open sockets/file descriptors to the replacement process before terminating.
It's also just neat and I wanted to play with it!
- Clone the repository
git clone https://github.com/mailpile/kettlingar.git
cd kettlingar- Create a virtual environment
python -m venv venv
source venv/bin/activate # On Windows, use `venv\Scripts\activate`- Install development dependencies
pip install -e ".[dev]"Before opening any pull requests, please ensure tests pass:
pytest tests... and you haven't made the linter unhappy:
pylint src tests examplesSee also pre-push.githook for client-side automation to prevent pushing code that breaks tests or fails the linter.
Contributions are always welcome! Here's how you can help:
- Fork the repository
- Create a new branch (
git checkout -b feature/amazing-feature) - Make your changes
- Commit your changes (
git commit -m 'Add some amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
Please make sure to update tests as appropriate and follow the existing coding style.
Kettlingar means "kittens" in Icelandic. This is a spin-off project from moggie (a moggie is a cat) and the author is Icelandic.
MIT, have fun!
Created by Bjarni R. Einarsson for use with moggie and probably some other things besides.
Thanks to Aniket Maurya for the handy Python project template.