Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 2 additions & 15 deletions .github/workflows/docs-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,8 @@
name: Docs publishing

on:
push:
branches: [ "main" ]
paths:
- docs/**
- gloe/**
- .github/workflows/docs-publish.yml
- README.md
pull_request:
branches: [ "main" ]
types: [opened, reopened]
paths:
- docs/**
- gloe/**
- .github/workflows/docs-publish.yml
- README.md
release:
types: [released]

jobs:
build:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]

steps:
- uses: actions/checkout@v3
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,9 @@

# Your Code as a Flow

Gloe (pronounced /ɡloʊ/, like "glow") is a general-purpose library designed to guide developers in expressing their code as a **flow**.
Gloe (pronounced like "glow") is a general-purpose library designed to guide developers in expressing their code as a **flow**.

**Why follow this approach?** Because it ensures that Gloe can keep your Python code easy to maintain, document, and test. Gloe guides you to write code in the form of small, safely connected units, rather than relying on scattered functions and classes with no clear relationship.
**Why follow this approach?** Because it ensures that you can keep your Python code easy to maintain, document, and test. Gloe guides you to write code in the form of small, safely connected units, rather than relying on scattered functions and classes with no clear relationship.

**What is a flow?** Formally, a flow is defined as a DAG (Directed Acyclic Graph) with one source and one sink, meaning it has a beginning and an end. In practice, it is a sequence of steps that transform data from one form to another.

Expand Down
3 changes: 1 addition & 2 deletions docs/source/examples/gloe-and-fastapi.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,10 +118,9 @@ The fourth transformer creates the final processed order object:

@transformer
def create_processed_order(
data: tuple[Order, User, list[OrderItem]]
order: Order, user: User, items: list[OrderItem]
) -> ProcessedOrder:
"""Creates the final processed order object."""
order, user, items = data
total_amount = sum(item.price * item.quantity for item in items)
processed_order = ProcessedOrder(
order_id=order.id, user=user, items=items, total_amount=total_amount
Expand Down
3 changes: 1 addition & 2 deletions docs/source/examples/gloe-and-pytorch.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,8 +161,7 @@ The evaluation transformer evaluates the model's performance using same MSE loss

```python
@transformer
def evaluate_model(entry: tuple[SimpleNN, Data]) -> float:
model, data = entry
def evaluate_model(model: SimpleNN, data: Data) -> float:
X_train, X_test, y_train, y_test = data
model.eval()
with torch.no_grad():
Expand Down
3 changes: 1 addition & 2 deletions docs/source/examples/gloe-and-scikit-learn.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,7 @@ def train_model(entry: tuple[LogisticRegression, Data], max_iter: int = 100) ->
Finally, evaluates the trained model on the test data and returns the accuracy score.
```python
@transformer
def evaluate_model(entry: tuple[LogisticRegression, Data]) -> float:
model, data = entry
def evaluate_model(model: LogisticRegression, data: Data) -> float:
X_train, X_test, y_train, y_test = data
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
Expand Down
17 changes: 6 additions & 11 deletions docs/source/getting-started/collection.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,17 +109,12 @@ from gloe.collection import MapOver
roles: list[Role] = [admin_role, member_role, manager_role]

@transformer
def format_user(entry: tuple[User, Role]) -> str:
user, role = entry
def format_user(user: User, role: Role) -> str:
return f'{user.name} is {role.name}'

format_users = MapOver(roles, format_user) # Transformer[Iterable[User], Iterable[str]]
format_user_roles = MapOver(roles, format_user) # Transformer[User, Iterable[str]]

format_users([
User(name='Anny', age=16),
User(name='Alice', age=25),
User(name='Bob', age=30)
]) # returns ['Anny is admin', 'Alice is member', 'Bob is manager']
format_user_roles(User(name='Anny', age=16)) # returns ['Anny is admin', 'Anny is member', 'Anny is manager']
```

## MapOverAsync
Expand All @@ -133,9 +128,9 @@ from gloe.collection import MapOverAsync
roles: list[Role] = [admin_role, member_role, manager_role]

@async_transformer
async def update_user_role(entry: tuple[User, Role]) -> User:
return await UserService.update_role(*entry)
async def update_user_role(user: User, role: Role) -> User:
return await UserService.update_role(user, role)

update_users_roles = MapOverAsync(roles, update_user_role) # AsyncTransformer[Iterable[User], Iterable[User]]
update_users_roles = MapOverAsync(roles, update_user_role) # AsyncTransformer[User, Iterable[User]]
```

5 changes: 4 additions & 1 deletion docs/source/getting-started/partial-transformers.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
- {func}`gloe.partial_transformer`
```

The single parameter of a transformer represents the input data during the execution and this input will be the return of the previous transformer in the pipeline. That being said, it doesn't make sense to allow transformers to have multiple parameters, because functions can't return multiple things (only a tuple of multiple things). However, sometimes we need some accessory data to perform the desired transformation.
The parameters of a transformer represents the input data during the execution and this input is the return of the previous transformer in the pipeline. However, sometimes we need some accessory data to perform the desired transformation.

For example, suppose we have a [Pandas](https://pandas.pydata.org/) dataframe of people with a numeric column "age". We want to filter people older than a specific age:

Expand Down Expand Up @@ -43,6 +43,9 @@ Gloe provides a much easier way to implement this behavior using the `@partial_t
def filter_older_than(people_df: pd.DataFrame, min_age: int) -> pd.DataFrame:
return people_df[people_df['age'] >= min_age]
```
```{important}
If the decorator used in the previous example was `@transformer`, the transformer would be created with two parameters: `people_df` and `min_age`, meaning that the previous transformer must return a tuple with these two elements.
```

It is possible to instantiate many transformers with different ages as well:

Expand Down
4 changes: 3 additions & 1 deletion docs/source/getting-started/plotting.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,9 @@ def get_user_network_data(users: User) -> UserNetwork: ...

@transformer
def get_recommended_items(
items: tuple[list[Product], list[Product], UserNetwork]
last_seen_items: list[Product],
last_ordered_items: list[Product],
user_network: UserNetwork
) -> list[Product]: ...

@transformer
Expand Down
21 changes: 18 additions & 3 deletions docs/source/getting-started/transformers.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ Transformers work like functions, so you can create a function and then apply th

- We **strongly recommend** you to type the transformers. Because of Python, it is not mandatory, but Gloe was designed to be used with typed code. Take a look at the [Python typing
library](https://docs.python.org/3/library/typing.html) to learn more about the Python type notation.
- Transformers must have only one parameter. Any complex data you need to use in its code must be passed in a complex structure like a [tuple](https://docs.python.org/3/tutorial/datastructures.html#tuples-and-sequences), a [dict](https://docs.python.org/3/tutorial/datastructures.html#dictionaries), a [TypedDict](https://docs.python.org/3/library/typing.html#typing.TypedDict), a [dataclass](https://docs.python.org/3/library/dataclasses.html), a [namedtuple](https://docs.python.org/3/library/collections.html#collections.namedtuple) or any other. We will see later {ref}`why it is necessary <partial-transformers>`.
- Transformers can have multiple parameters, but, under the hoods, they will be treated as a tuple.
- Transformers can have no parameters.
- Documentations with pydoc will be preserved in transformers.
- After applying the `@transformer` decorator to a function, it becomes an instance of the `Transformer` class.
```
Expand Down Expand Up @@ -123,7 +124,7 @@ send_promotion = get_users >> (
This example makes it clear how easy it is to understand and refactor the code when using Gloe to express the process as a graph, with each node (transformer) having an atomic and well-defined responsibility.

```{important}
You should not assume any **order of execution** between branches.
You should not assume any **order of execution** between parallel branches.
```

The right shift operator can receive a transformer or a tuple of transformers as an argument. In the second case, the transformer returned will be as described bellow (pay attention to the types).
Expand Down Expand Up @@ -167,7 +168,21 @@ graph: Transformer[In, FinalOut] = begin >> (
We call this last connection **convergent**.
```

The `end` transformer can be implemented in two ways. Or using a single argument being a tuple:
```python
@transformer
def end(data: tuple[Out1, Out2, ..., OutN]) -> FinalOut:
...
```

Either using multiple arguments, one for each branch:
```python
@transformer
def end(arg1: Out1, arg2: Out2, ..., argN: OutN) -> FinalOut:
...
```

```{attention}
Python doesn't provide a generic way to map the outcome type of an arbitrary number of branches on a tuple of arbitrary size. Due to this, the overload of possible sizes was treated one by one until the size 7, it means, considering the typing notation, it is possible to have at most 7 branches currently.
Python doesn't provide a generic way to map the outcome type of an arbitrary number of branches on a tuple of arbitrary size. Due to this, the overload of possible sizes was treated one by one until the size 7, it means, considering the typing notation, it is possible to have at most 7 branches currently. That's is not supposed to be a problem in practice.
```

6 changes: 0 additions & 6 deletions docs/source/theory.md

This file was deleted.

2 changes: 1 addition & 1 deletion gloe/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from gloe.base_transformer import TransformerException
from gloe.async_transformer import AsyncTransformer, MultiArgsAsyncTransformer

__version__ = "0.7.0-rc0"
__version__ = "0.7.0-rc1"

__all__ = [
"transformer",
Expand Down
6 changes: 4 additions & 2 deletions gloe/transformers.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,11 +110,11 @@ def _safe_transform(self, data: _I) -> _O:

@overload
def __call__(self: "Transformer[None, _O]") -> _O:
return _execute_flow(self._flow, None)
pass

@overload
def __call__(self, data: _I) -> _O:
return _execute_flow(self._flow, data)
pass

def __call__(self, data=None):
return _execute_flow(self._flow, data)
Expand Down Expand Up @@ -238,6 +238,8 @@ class MultiArgsTransformer(
def __call__( # type: ignore[override]
self: "MultiArgsTransformer[Unpack[Args], _O]", *data: Unpack[Args]
) -> _O:
if len(data) == 1 and type(data[0]) is tuple: # type: ignore
data = data[0] # type: ignore
return _execute_flow(self._flow, data)

@overload # type: ignore[override]
Expand Down
34 changes: 28 additions & 6 deletions tests/basic/test_transformer_basic.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import asyncio
import re
import unittest
from typing import cast

Expand All @@ -17,17 +18,37 @@
minus1,
natural_logarithm,
LnOfNegativeNumber,
times2,
tuplicate,
)


class TestTransformerBasic(unittest.TestCase):
def test_transformer_multiple_args(self):
def test_transformer_multiargs(self):
@transformer
def many_args(arg1: str, arg2: int) -> str:
return arg1 + str(arg2)

self.assertEqual(many_args("hello", 1), "hello1")

@transformer
def many_args2(arg1: str, arg2: str) -> str:
return arg1 + arg2

graph = tuplicate >> many_args2
self.assertEqual("hellohello", graph("hello"))

def test_transformer_multiargs_complex(self):
@transformer
def many_args(arg1: tuple[float, float], arg2: float) -> float:
return sum(arg1) + arg2

self.assertEqual(many_args((1.0, 2), 3), 6.0)

graph = plus1 >> (times2 >> plus1 >> (square, minus1), square) >> many_args

self.assertEqual(graph(3), 81 + 8 + 16.0)

def test_transformer_hash(self):
self.assertEqual(hash(square.id), square.__hash__())

Expand Down Expand Up @@ -187,11 +208,12 @@ def to_string(num: int) -> str:
as a string"""
return str(num)

self.assertEqual(
to_string.__doc__,
"""This transformer receives a number as input and return its representation
as a string""",
)
if to_string.__doc__ is not None:
self.assertEqual(
re.sub(r"\s+", " ", to_string.__doc__),
"""This transformer receives a number as input and return its """
"""representation as a string""",
)

def test_transformer_signature_representation(self):
signature = square.signature()
Expand Down
33 changes: 31 additions & 2 deletions tests/collection/test_transformer_collection.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,20 @@


class TestTransformerCollection(unittest.TestCase):
def test_transformer_multiargs_inside_map(self):
@transformer
def many_args(arg1: str, arg2: int) -> str:
return arg1 + str(arg2)

mapping = Map(many_args)

result = list(mapping([("hello", 1), ("world", 2)]))

self.assertListEqual(result, ["hello1", "world2"])

def test_transformer_map(self):
"""
Test the mapping transformer
Test the map transformer
"""

mapping1 = Map(square) >> Map(plus1)
Expand Down Expand Up @@ -41,7 +52,7 @@ def is_even(num: int) -> bool:

def test_transformer_map_over(self):
"""
Test the mapping transformer
Test the map over transformer
"""

data = [10.0, 9.0, 3.0, 2.0, -1.0]
Expand All @@ -50,3 +61,21 @@ def test_transformer_map_over(self):
result = list(mapping(-1.0))

self.assertListEqual(result, data)

def test_transformer_multiargs_inside_map_over(self):
roles: list[str] = ["admin_role", "member_role", "manager_role"]

@transformer
def format_user(user: str, role: str) -> str:
return f"User {user} has the role {role}."

format_users = MapOver(roles, format_user)

self.assertListEqual(
format_users("Alice"),
[
"User Alice has the role admin_role.",
"User Alice has the role member_role.",
"User Alice has the role manager_role.",
],
)
5 changes: 5 additions & 0 deletions tests/lib/transformers.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,11 @@ def duplicate(string: str) -> str:
return string + string


@transformer
def tuplicate(string: str) -> tuple[str, str]:
return string, string


@transformer
def natural_logarithm(num: float) -> float:
if num < 0:
Expand Down
Loading