Skip to content

Migrate template system to Jinja2 #125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
anticomputer wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Migrate template system to Jinja2 #125

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
799ab18
Add Jinja2 templating foundation and core integration
anticomputer Dec 29, 2025
8188c35
Add migration script and migrate YAML files to Jinja2 syntax
anticomputer Dec 29, 2025
c00dc11
Update YAML version scheme to v2 with deprecation warning
anticomputer Dec 29, 2025
23eb693
Reject v1 YAML files and add migration documentation
anticomputer Dec 29, 2025
98874d0
Update remaining YAML files to v2 and fix repeat_prompt rendering
anticomputer Dec 29, 2025
7807de4
Bump version to 0.1.0 for breaking YAML API change
anticomputer Dec 29, 2025
cff0449
Update documentation to use new env template syntax
anticomputer Dec 29, 2025
6be0c03
Add test script and fix example_reusable_taskflows model
anticomputer Dec 29, 2025
57e49cc
Detect rate limit errors in test script
anticomputer Dec 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 26 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,24 @@ The Taskflow Agent is built on top of the [OpenAI Agents SDK](https://openai.git

While the Taskflow Agent does not integrate into the GitHub Dotcom Copilot UX, it does operate using the Copilot API (CAPI) as its backend, similar to Copilot IDE extensions.

## Template Syntax Migration (v2)

**Breaking Change:** Taskflow YAML files now use Jinja2 templating (version 2). Version 1 files are no longer supported and will be rejected at load time.

**New Jinja2 syntax:**
- `{{ globals.key }}` instead of `{{ GLOBALS_key }}`
- `{{ inputs.key }}` instead of `{{ INPUTS_key }}`
- `{{ result }}` / `{{ result.key }}` instead of `{{ RESULT }}` / `{{ RESULT_key }}`
- `{{ env('VAR') }}` instead of `{{ env VAR }}`
- `{% include 'path' %}` instead of `{{ PROMPTS_path }}`

**To migrate existing taskflows:**
```bash
python scripts/migrate_to_jinja2.py /path/to/your/taskflows
```

See [doc/MIGRATION.md](doc/MIGRATION.md) for detailed migration instructions and new Jinja2 features.

## Core Concepts

The Taskflow Agent leverages a GitHub Workflow-esque YAML based grammar to perform a series of tasks using a set of Agents.
Expand Down Expand Up @@ -296,10 +314,10 @@ server_params:
url: https://api.githubcopilot.com/mcp/
#See https://github.com/github/github-mcp-server/blob/main/docs/remote-server.md
headers:
Authorization: "{{ env GITHUB_AUTH_HEADER }}"
Authorization: "{{ env('GITHUB_AUTH_HEADER') }}"
optional_headers:
X-MCP-Toolsets: "{{ env GITHUB_MCP_TOOLSETS }}"
X-MCP-Readonly: "{{ env GITHUB_MCP_READONLY }}"
X-MCP-Toolsets: "{{ env('GITHUB_MCP_TOOLSETS') }}"
X-MCP-Readonly: "{{ env('GITHUB_MCP_READONLY') }}"
```

You can force certain tools within a `toolbox` to require user confirmation to run. This can be helpful if a tool may perform irreversible actions and should require user approval prior to its use. This is done by including the name of the tool (function) in the MCP server in the `confirm` section:
Expand Down Expand Up @@ -347,7 +365,7 @@ taskflow:
Finally, why are apples and oranges healthy to eat?

# taskflows can set temporary environment variables, these support the general
# "{{ env FROM_EXISTING_ENVIRONMENT }" pattern we use elsewhere as well
# "{{ env('FROM_EXISTING_ENVIRONMENT') }}" pattern we use elsewhere as well
# these environment variables can then be made available to any stdio mcp server
# through its respective yaml configuration, see memcache.yaml for an example
# you can use these to override top-level environment variables on a per-task basis
Expand Down Expand Up @@ -494,12 +512,12 @@ Files of types `taskflow` and `toolbox` allow environment variables to be passed
server_params:
...
env:
CODEQL_DBS_BASE_PATH: "{{ env CODEQL_DBS_BASE_PATH }}"
CODEQL_DBS_BASE_PATH: "{{ env('CODEQL_DBS_BASE_PATH') }}"
# prevent git repo operations on gh codeql executions
GH_NO_UPDATE_NOTIFIER: "disable"
```

For `toolbox`, `env` can be used inside `server_params`. A template of the form `{{ env ENV_VARIABLE_NAME }}` can be used to pass values of the environment variable from the current process to the MCP server. So in the above, the MCP server is run with `GH_NO_UPDATE_NOTIFIER=disable` and passes the value of `CODEQL_DBS_BASE_PATH` from the current process to the MCP server. The templated paramater `{{ env CODEQL_DBS_BASE_PATH }}` is replaced by the value of the environment variable `CODEQL_DBS_BASE_PATH` in the current process.
For `toolbox`, `env` can be used inside `server_params`. A template of the form `{{ env('ENV_VARIABLE_NAME') }}` can be used to pass values of the environment variable from the current process to the MCP server. So in the above, the MCP server is run with `GH_NO_UPDATE_NOTIFIER=disable` and passes the value of `CODEQL_DBS_BASE_PATH` from the current process to the MCP server. The templated parameter `{{ env('CODEQL_DBS_BASE_PATH') }}` is replaced by the value of the environment variable `CODEQL_DBS_BASE_PATH` in the current process.

Similarly, environment variables can be passed to a `task` in a `taskflow`:

Expand All @@ -516,9 +534,9 @@ taskflow:
MEMCACHE_BACKEND: "dictionary_file"
```

This overwrites the environment variables `MEMCACHE_STATE_DIR` and `MEMCACHE_BACKEND` for the task only. A template `{{ env ENV_VARIABLE_NAME }}` can also be used.
This overwrites the environment variables `MEMCACHE_STATE_DIR` and `MEMCACHE_BACKEND` for the task only. A template `{{ env('ENV_VARIABLE_NAME') }}` can also be used.

Note that when using the template `{{ env ENV_VARIABLE_NAME }}`, `ENV_VARIABLE_NAME` must be the name of an environment variable in the current process.
Note that when using the template `{{ env('ENV_VARIABLE_NAME') }}`, `ENV_VARIABLE_NAME` must be the name of an environment variable in the current process.

## Import paths

Expand Down
30 changes: 15 additions & 15 deletions doc/GRAMMAR.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,10 +133,10 @@ Often we may want to iterate through the same tasks with different inputs. For e
agents:
- seclab_taskflow_agent.personalities.c_auditer
user_prompt: |
The function has name {{ RESULT_name }} and body {{ RESULT_body }} analyze the function.
The function has name {{ result.name }} and body {{ result.body }} analyze the function.
```

In the above, the first task fetches functions in the code base and creates a json list object, with each entry having a `name` and `body` field. In the next task, `repeat_prompt` is set to true, meaning that a task is created for each individual object in the list and the object fields are referenced in the templated prompt using `{{ RESULT_<fieldname> }}`. In other words, `{{ RESULT_name }}` in the prompt is replaced with the value of the `name` field of the object etc. For example, if the list of functions fetched from the first task is:
In the above, the first task fetches functions in the code base and creates a json list object, with each entry having a `name` and `body` field. In the next task, `repeat_prompt` is set to true, meaning that a task is created for each individual object in the list and the object fields are referenced in the templated prompt using `{{ result.fieldname }}`. In other words, `{{ result.name }}` in the prompt is replaced with the value of the `name` field of the object etc. For example, if the list of functions fetched from the first task is:

```javascript
[{'name' : foo, 'body' : foo(){return 1;}}, {'name' : bar, 'body' : bar(a) {return a + 1;}}]
Expand All @@ -152,7 +152,7 @@ etc.

Note that when using `repeat_prompt`, the last tool call result of the previous task is used as the iterable. It is recommended to keep the task that creates the iterable short and simple (e.g. just make one tool call to fetch a list of results) to avoid wrong results being passed to the repeat prompt.

The iterable can also contain a list of primitives like string or number, in which case, the template `{{ RESULT }}` can be used in the `repeat_prompt` prompt to parse the results instead:
The iterable can also contain a list of primitives like string or number, in which case, the template `{{ result }}` can be used in the `repeat_prompt` prompt to parse the results instead:

```yaml
- task:
Expand All @@ -173,7 +173,7 @@ The iterable can also contain a list of primitives like string or number, in whi
agents:
- seclab_taskflow_agent.personalities.assistant
user_prompt: |
What is the integer value of {{ RESULT }}?
What is the integer value of {{ result }}?
```

Repeat prompt can be run in parallel by setting the `async` field to `true`:
Expand All @@ -185,7 +185,7 @@ Repeat prompt can be run in parallel by setting the `async` field to `true`:
agents:
- seclab_taskflow_agent.personalities.c_auditer
user_prompt: |
The function has name {{ RESULT_name }} and body {{ RESULT_body }} analyze the function.
The function has name {{ result.name }} and body {{ result.body }} analyze the function.
```

An optional limit can be set to limit the number of asynchronous tasks via `async_limit`. If not set, the default value (5) is used.
Expand All @@ -198,7 +198,7 @@ An optional limit can be set to limit the number of asynchronous tasks via `asyn
agents:
- seclab_taskflow_agent.personalities.c_auditer
user_prompt: |
The function has name {{ RESULT_name }} and body {{ RESULT_body }} analyze the function.
The function has name {{ result.name }} and body {{ result.body }} analyze the function.
```

Both `async` and `async_limit` have no effect when used outside of a `repeat_prompt`.
Expand All @@ -211,7 +211,7 @@ At the moment, we do not support nested `repeat_prompt`. So the following is not
agents:
- seclab_taskflow_agent.personalities.c_auditer
user_prompt: |
The function has name {{ RESULT_name }} and body {{ RESULT_body }} analyze the function.
The function has name {{ result.name }} and body {{ result.body }} analyze the function.
- task:
repeat_prompt: true
...
Expand All @@ -233,7 +233,7 @@ For example:
agents:
- seclab_taskflow_agent.personalities.assistant
user_prompt: |
What kind of fruit is {{ RESULT }}?
What kind of fruit is {{ result }}?
```

The string `["apple", "banana", "orange"]` is then passed directly to the next task.
Expand Down Expand Up @@ -349,7 +349,7 @@ taskflow:
agents:
- examples.personalities.fruit_expert
user_prompt: |
Tell me more about {{ GLOBALS_fruit }}.
Tell me more about {{ globals.fruit }}.
```

Global variables can also be set or overridden from the command line using the `-g` or `--global` flag:
Expand Down Expand Up @@ -422,10 +422,10 @@ A reusable taskflow can also have a templated prompt that takes inputs from its
agents:
- examples.personalities.fruit_expert
user_prompt: |
Tell me more about {{ INPUTS_fruit }}.
Tell me more about {{ inputs.fruit }}.
```

In this case, the template parameter `{{ INPUTS_fruit }}` is replaced by the value of `fruit` from the `inputs` of the user, which is apples in this case:
In this case, the template parameter `{{ inputs.fruit }}` is replaced by the value of `fruit` from the `inputs` of the user, which is apples in this case:

```yaml
- task:
Expand All @@ -437,9 +437,9 @@ In this case, the template parameter `{{ INPUTS_fruit }}` is replaced by the val

### Reusable Prompts

Reusable prompts are defined in files of `filetype` `prompts`. These are like macros that get replaced when a templated parameter of the form `{{ PROMPTS_<import-path> }}` is encountered.
Reusable prompts are defined in files of `filetype` `prompts`. These are like macros that get included using Jinja2's `{% include %}` directive.

Tasks can incorporate templated prompts which are then replaced by the actual prompt. For example:
Tasks can incorporate reusable prompts using the include directive. For example:

Example:

Expand All @@ -449,8 +449,8 @@ Example:
- examples.personalities.fruit_expert
user_prompt: |
Tell me more about apples.
{{ PROMPTS_examples.prompts.example_prompt }}

{% include 'examples.prompts.example_prompt' %}
```
and `examples.prompts.example_prompt` is the following:

Expand Down
Loading
Loading