utilities.parser.models.sns.SnsNotificationModel has incorrect field names

[zlalvani]

What were you trying to accomplish?
Using the @event_parser decorator with an SNS -> SQS -> Lambda invocation

Expected Behavior

The parser's validation succeeds and the lambda invokes successfully

Current Behavior

Pydantic validation on the SnsNotificationModel raises the following errors:

[ERROR] ValidationError: 3 validation errors for SnsNotificationModel
UnsubscribeUrl
  field required (type=value_error.missing)
MessageAttributes
  field required (type=value_error.missing)
SigningCertUrl
  field required (type=value_error.missing)

This is because the URL fields should have URL capitalized, e.g. UnsubscribeURL and the MessageAttributes field isn't always present.

Possible Solution

Update the field names, or use a pydantic alias for backwards compatibility with the current name scheme.

Steps to Reproduce (for bugs)

I defined my own envelope class for SNS -> SQS -> Lambda messages, as seen below:

class SnsSqsEnvelope(BaseEnvelope):

    def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: Model) -> List[Optional[Model]]:
        parsed_envelope = _SqsModel.parse_obj(data)
        output = []
        for record in parsed_envelope.Records:
            sns_notification = SnsNotificationModel.parse_raw(record.body)
            output.append(self._parse(data=sns_notification.Message, model=model))
        return output

1. Use the above envelope with @event_parser on a lambda handler function
2. Subscribe an SQS queue to an SNS topic, and have the queue invoke the lambda
3. Publish a message to the topic

Environment

• Powertools version used: 1.9.0
• Packaging format (Layers, PyPi): pypi
• AWS Lambda function runtime: 3.8

[heitorlessa]

hey @zlalvani thank you for raising that! I've just looked into the SNS docs and while that confirms Message attributes should've been optional in our model, the UnsubscribeUrl key is correct.

Could you share a sample payload with a request ID? redact any sensitive data if there's any

This will help us validate with the SNS team whether the contract was broken, or if the error is misleading.

I've created a new environment with a hello-world Lambda function subscribed to a SNS topic, and confirmed the payload[1] follows the model structure for UnsubscribeUrl key - I've double checked Lambda docs, SAM CLI, and Lambda Console for testing events.

We're gonna release 1.10 tomorrow - As soon as we have a sample payload and receive confirmation from SNS team we'll release a fast follow patch version 1.10.1.

Note: If you don't need SNS message metadata in your Lambda consumer, you can enable raw message delivery when using SQS as a SNS subscriber. This will simplify your model and thus requiring SQS model only too, since the payload would be sent to SQS as-is.

Thanks!

---

Sample payload received by SNS

{
    "Records": [
        {
            "EventSource": "aws:sns",
            "EventVersion": "1.0",
            "EventSubscriptionArn": "arn:aws:sns:eu-west-1:231436140809:powertools265:5e4c8d5d-383c-4aa1-90aa-d849f068e7dc", 
            "Sns": {
                "Type": "Notification",
                "MessageId": "195dbd83-fb19-5cc5-886c-e5fda7f69b25", 
                "TopicArn": "arn:aws:sns:eu-west-1: 231436140809:powertools265",
                "Subject": null,
                "Message": "{\n  \"issue\": \"https://github.com/awslabs/aws-lambda-powertools-python/issues/265\"\n}",
                "Timestamp": "2021-01-17T15:41:35.421Z",
                "SignatureVersion": "1",
                "Signature": "Tb9Y/BZnBDbDwqj3yEOnjHuuIulXDHKZgHo3VT8bClqHwglWMBABnWOfUFFsJWaQLWRE8MIDPWSr0IR0gVHpTicmA9XWpMGhLy0KBnWuRF/FEQU886SdZz3TJW94lX1vGDZFJX6LA8ZpwQFJ69bVf2WemSCcPnzPFaVnSp+2W4fTMRIreRwGvAQW1HSHObowmRqtSz7ZBmDFRjyF9wuPte0KyWwJh9m9Z/zhIFkJfNUBMM+hSafxVqVUrHKcSO1vnAi7eFeIzJzpQHiXaZEKCmPMD4NVAeuqF3CHjfbd6n4Jwe1dcVP+o69fuXyADqrV1kGtHpLBAlGk9MRyIeWWoQ==", "SigningCertUrl": "https: //sns.eu-west-1.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd                93083a.pem", "UnsubscribeUrl": "https: //sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:231436140809:powertools265: 5e4c8d5d-383c-4aa1-90aa-d849f068e7dc",
                "MessageAttributes": {}
            }
        }
    ]
}

[heitorlessa]

I've just created a test for this and all passed without the Optional, since MessageAttributes key is always presented except its value can be an empty dict (expected in the model) - I think the issue here is the handling of SQS + SNS.

While we don't have your payload, I'll create a test infra for SQS + SNS and a custom envelope to confirm this theory - I suspect record.body might not contain what we think it does

[heitorlessa]

@zlalvani SNS team confirmed yesterday that the payload is slightly different when SQS is the subscriber vs Lambda.

This explains why UnsubscribeUrl became UnsubscribeURL, and those other keys weren't even present.

As it doesn't seem to be documented, we'll work on a patch this week on our end, since the SNS team can't make this correction.

For now, using Raw Delivery is the best course since you're already stripping all metadata to get the payload only in your envelope - we'll provide a built-in envelope either way and put a note in the documentation about the raw delivery feature for best results too.

---

SNS -> SQS -> Lambda payload serialized as JSON

Note for historical reasons that it's (as described in the issue description):

• Missing MessageAttributes key
• UnsubscribeUrl became UnsubscribeURL
• SigningCertUrl became SigningCertURL

{
    "Records": [
        {
            "messageId": "0314e520-0a6a-493b-b8bb-2981629f9597",
            "receiptHandle": "AQEBXln06tGQ5sT0ctGBrIlwMRhkqzZhNO7kO9qfx0SwoBljuZGE8WJmaZKuguB3qvKswtVlmmHCGxADWKQF52PUVlL3POG/R/KRrorG2RAGKHoQpi4V9+GsDoJ2OMwAGeNMAXbes/q1An3b17W6HRVCQiGZxOo9QXLfd522zRyrMgc3IFLQcVXCtbsPnwxziYMHhtmzSjFpjcWUXTKFpXRERszrlSrMXuHBOS3AAnIlgaVVqSslAn1IYxVWCuanF9h7Qj/Qwrn1bOa4TtDqdoKMxdF852mvPpbw34eIgt9eppGwd0l29pOCFhe0Lr6BEC1C9MA7Fhf9kLL2Qwkikv1IlzC6xIpqqmTpyDiTzIFRSFmk+AdFjacH/5p3Lo3NOp9bAobGtMraISao2zAivm6V5w==",
            "body": {
                "Type": "Notification",
                "MessageId": "056c5c0e-f77b-5e31-aee6-048e65f5f45f",
                "TopicArn": "arn:aws:sns:eu-west-1:231436140809:powertools265",
                "Message": {
                    "issue": "https://github.com/awslabs/aws-lambda-powertools-python/issues/265"
                },
                "Timestamp": "2021-01-19T07:54:54.353Z",
                "SignatureVersion": "1",
                "Signature": "oFFcYb29wYnL7sWGkASkw61jHWaW4Hllwk+rudspBjks3pZVsx1COjbShVg/Y6RzmzK98r65LMaiF1OhrElGYNK3TV9kZKuD4BBqZohs2guuq4bXa1DYn7jnBlVNJcmhTttNzLRvgIOJcQz8XJBcvvuramukdMEVPgw5qrNnC/Jhw0V4lGlAmuQplM2Mm0D4xF/tp3pDVCrQnvs8cZUfqJ1Ol+I3MEuUE1eg8SI4lugnZzN4aJJRb1rvSqa4khWlyk09BdIQRd93JY0n0wjbfHhxwaahL/k16/OhAmmh3rK5zUrZXX3RLO+cY3B8XC07XZ4iDfMc4zUXiOMeE2k6Bw==",
                "SigningCertURL": "https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
                "UnsubscribeURL": "https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:231436140809:powertools265:15189ad7-870e-40e5-a7dd-a48898cd9f86"
            },
            "attributes": {
                "ApproximateReceiveCount": "2",
                "SentTimestamp": "1611042894394",
                "SenderId": "AIDAISMY7JYY5F7RTT6AO",
                "ApproximateFirstReceiveTimestamp": "1611042916135"
            },
            "messageAttributes": {},
            "md5OfBody": "5c0dc7d673c946e9ebe4f3ceccede0e1",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:eu-west-1:231436140809:powertools265",
            "awsRegion": "eu-west-1"
        }
    ]
}

[heitorlessa]

This is now released in 1.10.1 - You can use SnsSqsEnvelope and it'll dynamically rename those keys without any extra line of code for you.

From

class SnsSqsEnvelope(BaseEnvelope):

    def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: Model) -> List[Optional[Model]]:
        parsed_envelope = _SqsModel.parse_obj(data)
        output = []
        for record in parsed_envelope.Records:
            sns_notification = SnsNotificationModel.parse_raw(record.body)
            output.append(self._parse(data=sns_notification.Message, model=model))
        return output

To

from aws_lambda_powertools.utilities.parser import envelopes, event_parser

@event_parser(model=YourModel envelope=envelopes.SnsSqsEnvelope)
def handle_sns_sqs_json_body(event: List[YourModel] _: LambdaContext):
    ...
    # example
    #    assert event[0].message == "hello world"
    #    assert event[0].username == "lessa"

Thank you @zlalvani for raising this once again; closing this now

[zlalvani]

Cool, thanks for the investigation @heitorlessa!