Feature request: For idempotency feature, allow no return value / undefined from lambda handlers.

[kuber-]

Use case

To make functions with side effects and no return value idempotent, undefined /no return value needs to be handled. Currently, the library fails with error message: Failed to update success record to idempotency store. This error was caused by: Pass options.removeUndefinedValues=true to remove undefined values from map/array/set..

Solution/User Experience

Update DynamoDBPersistenceLayer to handle undefined with appropriate Marshall options while updating the records in the persistence store.

Alternative solutions

No response

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

[dreamorosi]

Hi @kuber-, thank you for taking the time to open this issue.

As we discussed on Discord, the short-term solution while we work on a more permanent fix would be to simply return something. This will allow the utility to work normally and especially if you don't care about the result,

I have just tested this with Powertools for AWS Lambda (Python) and they handle this by marshalling the None value correctly, for example:

from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    idempotent,
)
import boto3
from aws_lambda_powertools.utilities.typing import LambdaContext

persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
dynamodb_local_client = boto3.client("dynamodb", endpoint_url="http://localhost:8000")
persistence_layer.client = dynamodb_local_client


@idempotent(persistence_store=persistence_layer)
def lambda_handler(event: dict, context: LambdaContext):
    print("expensive operation")
    return None

if __name__ == "__main__":
    print(lambda_handler({
        "key": "value"
    }, LambdaContext()) )

Results in this item being added to DynamoDB:

ConsumedCapacity: null
Count: 1
Items:
- data:
    S: 'null'
  expiration:
    N: '1715351592'
  id:
    S: test-func.__main__.lambda_handler#88bac95f31528d13a072c05f2a1cf371
  status:
    S: COMPLETED
ScannedCount: 1

Which when retried, results in the value being unmarshalled back to None.

---

In this version of Powertools we use @util-dynamodb from the AWS SDK v3 and we call the marshall/unmarshall helpers with default options.

Both utilities accept an options object (example) that allows you to customize the serialization operation.

I will need to look into it and run some tests as I don't know if we should use the convertEmptyValues or removeUndefinedValues option in this case, but I think we should work on this and so I am adding it to the backlog.

[kuber-]

Thanks for your help @dreamorosi 🙏 ! I look forward to its implementation. In the meantime, returning a value works perfectly fine for me.

[dreamorosi]

After some tests I think the removeUndefinedValues option is kind of useless and won't help in this case.

Setting it to true does remove the undefined value from the object being marshalled, however the SDK at that point complains because now one of the of the expression attributes used in the expression is not defined/not present:

await client.send(
  new UpdateItemCommand({
    TableName: 'IdempotencyTable',
    Key: marshall({ id: '123' }),
    UpdateExpression: 'SET #expiration = :expiration, #data = :data',
    ExpressionAttributeNames: { '#expiration': 'expiration', '#data': 'data' },
    ExpressionAttributeValues: marshall(
      {
        ':expiration': 123,
        ':data': undefined,
      },
      { removeUndefinedValues: true }
    ),
  })
);

// ValidationException: Invalid UpdateExpression: An expression attribute value used in expression is not defined; attribute value: :data

This was brought up to the AWS SDK team in aws/aws-sdk-js-v3#4280 but apparently it was then closed as working as intended (🫠). Likewise, the convertEmptyValues option I mentioned in my previous comment is also useless in this instance.

Even if the SDK was smart enough to remove the attribute from the UpdateExpression and ExpressionAttributeNames when the corresponding ExpressionAttributeValues entry is undefined, I'm not convinced removing the value is the right option in this case and I agree much more with the approach used by boto3 of converting this to null instead.

I will try to work on a PR to handle this aligning with boto3. If that's not possible, I will instead implement logic that handles this on our side - something along these lines:

const updateExpressionOps = ['#expiration = :expiration'];
const expressionAttributeNames: Record<string, string> = {
  '#expiration': 'expiration',
};
const expressionAttributeValues: Record<string, unknown> = {
  ':expiration': 123,
};

const data = undefined;

if (data !== undefined) {
  updateExpressionOps.push('#data = :data');
  expressionAttributeNames['#data'] = 'data';
  expressionAttributeValues[':data'] = data;
}

await client.send(
  new UpdateItemCommand({
    TableName: 'IdempotencyTable',
    Key: marshall({ id: '123' }),
    UpdateExpression: `SET ${updateExpressionOps.join(', ')}`,
    ExpressionAttributeNames: expressionAttributeNames,
    ExpressionAttributeValues: marshall(expressionAttributeValues),
  })
);

[dreamorosi]

PR is up, at the end I went the way of avoiding saving the result when it's undefined. That was the only way I could think in order to still be able to handle other cases. Saving 'null' like Python does, or setting the value to Null (DynamoDB type) are not an option respectively due to customers maybe wanting to return 'null' (string) and to JS having actual null.

See more details in the linked PR.

[ptlls]

I'm sorry that it's not directly related, but I have a very similar issue also caused by marshalling. Whenever the response to be stored contains nested objects, I get the following error when executing the Lambda function:

Failed to update success record to idempotency store. This error was caused by: Unsupported type passed: [object Object]. Pass options.convertClassInstanceToMap=true to marshall typeof object as map attribute..`

It would be really helpful to be able to pass in the optional marshallOptions configuration object as part of the DynamoDBPersistenceLayer client configuration, so that the marshall calls being made here take the options into account.

Thanks!

[dreamorosi]

Hi @ptlls thank you for chiming in.

I don't think that option is meant for plain objects but rather class instances.

With that said, I'm open to explore adding it, could you please open a feature request issue - ideally with a code example of what you're looking to serialize?

This way we can run some tests and track the activity if we decide to add it to the backlog.

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under v2.2.0 version!