Skip to content

docs(idempotency): use tab navigation, improves custom serializer example, and additional explanations #3067

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 45 additions & 21 deletions docs/utilities/idempotency.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,43 +154,67 @@ When using `idempotent_function`, you must tell us which keyword parameter in yo

#### Output serialization

The default return of the `idempotent_function` decorator is a JSON object, but you can customize the function's return type by utilizing the `output_serializer` parameter. The output serializer supports any JSON serializable data, **Python Dataclasses** and **Pydantic Models**.
By default, `idempotent_function` serializes, stores, and returns your annotated function's result as a JSON object. You can change this behavior using `output_serializer` parameter.

The output serializer supports any JSON serializable data, **Python Dataclasses** and **Pydantic Models**.

!!! info "When using the `output_serializer` parameter, the data will continue to be stored in DynamoDB as a JSON object."

Working with Pydantic Models:
=== "Pydantic"

=== "Explicitly passing the Pydantic model type"
You can use `PydanticSerializer` to automatically serialize what's retrieved from the persistent storage based on the return type annotated.

```python hl_lines="6 24 25 32 35 44"
--8<-- "examples/idempotency/src/working_with_pydantic_explicitly_output_serializer.py"
```
=== "Deducing the Pydantic model type from the return type annotation"
=== "Inferring via the return type"

```python hl_lines="6 24 25 32 36 45"
--8<-- "examples/idempotency/src/working_with_pydantic_deduced_output_serializer.py"
```
```python hl_lines="6 24 25 32 36 45"
--8<-- "examples/idempotency/src/working_with_pydantic_deduced_output_serializer.py"
```

Working with Python Dataclasses:
1. We'll use `OrderOutput` to instantiate a new object using the data retrieved from persistent storage as input. <br><br> This ensures the return of the function is not impacted when `@idempotent_function` is used.

=== "Explicitly passing the model type"
=== "Explicit model type"

```python hl_lines="8 27-29 36 39 48"
--8<-- "examples/idempotency/src/working_with_dataclass_explicitly_output_serializer.py"
```
Alternatively, you can provide an explicit model as an input to `PydanticSerializer`.

=== "Deducing the model type from the return type annotation"
```python hl_lines="6 24 25 32 35 44"
--8<-- "examples/idempotency/src/working_with_pydantic_explicitly_output_serializer.py"
```

```python hl_lines="8 27-29 36 40 49"
--8<-- "examples/idempotency/src/working_with_dataclass_deduced_output_serializer.py"
```
=== "Dataclasses"

You can use `DataclassSerializer` to automatically serialize what's retrieved from the persistent storage based on the return type annotated.

=== "Inferring via the return type"

```python hl_lines="8 27-29 36 40 49"
--8<-- "examples/idempotency/src/working_with_dataclass_deduced_output_serializer.py"
```

=== "Using A Custom Type (Dataclasses)"
1. We'll use `OrderOutput` to instantiate a new object using the data retrieved from persistent storage as input. <br><br> This ensures the return of the function is not impacted when `@idempotent_function` is used.

```python hl_lines="9 33 37 41-44 51 54"
=== "Explicit model type"

Alternatively, you can provide an explicit model as an input to `DataclassSerializer`.

```python hl_lines="8 27-29 36 39 48"
--8<-- "examples/idempotency/src/working_with_dataclass_explicitly_output_serializer.py"
```

=== "Any type"

You can use `CustomDictSerializer` to have full control over the serialization process for any type. It expects two functions:

* **to_dict**. Function to convert any type to a JSON serializable dictionary before it saves into the persistent storage.
* **from_dict**. Function to convert from a dictionary retrieved from persistent storage and serialize in its original form.

```python hl_lines="8 32 36 40 50 53"
--8<-- "examples/idempotency/src/working_with_idempotent_function_custom_output_serializer.py"
```

1. This function does the following <br><br>**1**. Receives the return from `process_order`<br> **2**. Converts to dictionary before it can be saved into the persistent storage.
2. This function does the following <br><br>**1**. Receives the dictionary saved into the persistent storage <br>**1** Serializes to `OrderOutput` before `@idempotent` returns back to the caller.
3. This serializer receives both functions so it knows who to call when to serialize to and from dictionary.

#### Batch integration

You can can easily integrate with [Batch utility](batch.md){target="_blank"} via context manager. This ensures that you process each record in an idempotent manner, and guard against a [Lambda timeout](#lambda-timeouts) idempotent situation.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ class OrderOutput:
persistence_store=dynamodb,
output_serializer=DataclassSerializer,
)
# order output is deduced from return type
def deduced_order_output_serializer(order: Order) -> OrderOutput:
# order output is inferred from return type
def process_order(order: Order) -> OrderOutput: # (1)!
return OrderOutput(order_id=order.order_id)


Expand All @@ -46,4 +46,4 @@ def lambda_handler(event: dict, context: LambdaContext):
order = Order(item=order_item, order_id=1)

# `order` parameter must be called as a keyword argument to work
deduced_order_output_serializer(order=order)
process_order(order=order)
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ class OrderOutput:
persistence_store=dynamodb,
output_serializer=DataclassSerializer(model=OrderOutput),
)
def explicit_order_output_serializer(order: Order):
def process_order(order: Order):
return OrderOutput(order_id=order.order_id)


Expand All @@ -45,4 +45,4 @@ def lambda_handler(event: dict, context: LambdaContext):
order = Order(item=order_item, order_id=1)

# `order` parameter must be called as a keyword argument to work
explicit_order_output_serializer(order=order)
process_order(order=order)
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
from dataclasses import asdict, dataclass
from typing import Any, Dict
from typing import Dict, Type

from aws_lambda_powertools.utilities.idempotency import (
DynamoDBPersistenceLayer,
Expand All @@ -13,34 +12,34 @@
config = IdempotencyConfig(event_key_jmespath="order_id") # see Choosing a payload subset section


@dataclass
class OrderItem:
sku: str
description: str
def __init__(self, sku: str, description: str):
self.sku = sku
self.description = description


@dataclass
class Order:
item: OrderItem
order_id: int
def __init__(self, item: OrderItem, order_id: int):
self.item = item
self.order_id = order_id


@dataclass
class OrderOutput:
order_id: int
def __init__(self, order_id: int):
self.order_id = order_id


def custom_to_dict(x: Any) -> Dict:
return asdict(x)
def order_to_dict(x: Type[OrderOutput]) -> Dict: # (1)!
return x.__dict__


def custom_from_dict(x: Dict) -> Any:
def dict_to_order(x: Dict) -> OrderOutput: # (2)!
return OrderOutput(**x)


order_output_serializer = CustomDictSerializer(
to_dict=custom_to_dict,
from_dict=custom_from_dict,
order_output_serializer = CustomDictSerializer( # (3)!
to_dict=order_to_dict,
from_dict=dict_to_order,
)


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ class OrderOutput(BaseModel):
persistence_store=dynamodb,
output_serializer=PydanticSerializer,
)
# order output is deduced from return type
def deduced_order_output_serializer(order: Order) -> OrderOutput:
# order output is inferred from return type
def process_order(order: Order) -> OrderOutput: # (1)!
return OrderOutput(order_id=order.order_id)


Expand All @@ -42,4 +42,4 @@ def lambda_handler(event: dict, context: LambdaContext):
order = Order(item=order_item, order_id=1)

# `order` parameter must be called as a keyword argument to work
deduced_order_output_serializer(order=order)
process_order(order=order)
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ class OrderOutput(BaseModel):
persistence_store=dynamodb,
output_serializer=PydanticSerializer(model=OrderOutput),
)
def explicit_order_output_serializer(order: Order):
def process_order(order: Order):
return OrderOutput(order_id=order.order_id)


Expand All @@ -41,4 +41,4 @@ def lambda_handler(event: dict, context: LambdaContext):
order = Order(item=order_item, order_id=1)

# `order` parameter must be called as a keyword argument to work
explicit_order_output_serializer(order=order)
process_order(order=order)