Replacing Amazon S3 Events with Amazon S3 Data Events

When AWS released suffix filtering for Amazon EventBridge rules in Nov 2022, EventBridge rules for Amazon S3 events reached feature parity with the original Amazon S3 Event Notifications feature except for the cost.¹

Of course, the event formats aren’t the same which, while understandable, is very unfortunate.

There are situations where you might want to use the flexibility of the newly styled CloudTrail events but with the payload format of the old SNS/SQS events. For instance, you might:

• Want to consume events from an S3 Bucket defined in a different stack (or just want to avoid a circular dependency in a single stack)
• Not care about the cost (or you already have S3 data events enabled for all your buckets)
• Have to forward the events to something that only expects the old event format, e.g. a 3rd party integration you don’t have control over

If you ever find yourself in a situation like this you can use an EventBridge Rule Input Transformation to synthesize the “old style” event. I’ll show you how in this post.

Here is an S3 Notification Event and an S3 Event for the same PutObject operation for comparison.

Old Style S3 Event:

{
  "Records": [
    {
      "eventVersion": "2.1",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventTime": "2023-09-02T02:10:23.087Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "AWS:AROAX7U2EUGEQA7NG7PQI:ts"
      },
      "requestParameters": {
        "sourceIPAddress": "52.2.140.103"
      },
      "responseElements": {
        "x-amz-request-id": "TQS46783602DMSZC",
        "x-amz-id-2": "6YvW5sWDx/RpLX+g31EPoctLVaEtuxbXVaXoBRmQHjwjL31gwHW1nSn9X31W98CbGZ7cfri42Y/z5ThTu7Sy1gWtnnyMdMeD"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "e41037ac-1090-4900-996c-c3fd5a8e7428",
        "bucket": {
          "name": "trek10-joel-s3-event",
          "ownerIdentity": {
            "principalId": "A37I0H6V1UEIUZ"
          },
          "arn": "arn:aws:s3:::trek10-joel-s3-event"
        },
        "object": {
          "key": "test.txt",
          "size": 2451,
          "eTag": "94137ea995d2c4dc3d0936e3c142d4f8",
          "sequencer": "0064F2998F02963CC8"
        }
      }
    }
  ]
}

New Style S3 Event:

{
  "version": "0",
  "id": "4455b4bd-3eb0-9c2f-622c-465a95c07b65",
  "detail-type": "AWS API Call via CloudTrail",
  "source": "aws.s3",
  "account": "549005336969",
  "time": "2023-09-02T02:10:23Z",
  "region": "us-east-1",
  "resources": [],
  "detail": {
    "eventVersion": "1.09",
    "userIdentity": {
      "type": "AssumedRole",
      "principalId": "AROAX7U2EUGEQA7NG7PQI:ts",
      "arn": "arn:aws:sts::549005336969:assumed-role/trek10-kernel/ts",
      "accountId": "549005336969",
      "accessKeyId": "ASIAX7U2EUGERE23RNSU",
      "sessionContext": {
        "sessionIssuer": {
          "type": "Role",
          "principalId": "AROAX7U2EUGEQA7NG7PQI",
          "arn": "arn:aws:iam::549005336969:role/trek10-kernel",
          "accountId": "549005336969",
          "userName": "trek10-kernel"
        },
        "attributes": {
          "creationDate": "2023-09-02T02:07:40Z",
          "mfaAuthenticated": "true"
        }
      }
    },
    "eventTime": "2023-09-02T02:10:23Z",
    "eventSource": "s3.amazonaws.com",
    "eventName": "PutObject",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "52.2.140.103",
    "userAgent": "[aws-cli/1.27.153 Python/3.10.5 Darwin/22.6.0 botocore/1.29.153]",
    "requestParameters": {
      "bucketName": "trek10-joel-s3-event",
      "Host": "trek10-joel-s3-event.s3.amazonaws.com",
      "key": "test.txt"
    },
    "responseElements": {
      "x-amz-server-side-encryption": "AES256"
    },
    "additionalEventData": {
      "SignatureVersion": "SigV4",
      "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
      "bytesTransferredIn": 2451,
      "SSEApplied": "Default_SSE_S3",
      "AuthenticationMethod": "AuthHeader",
      "x-amz-id-2": "BzIPOMEZzd0m43KS9YZEbxY5BsXKwaUbaPL2FjMLtnDkDWO37FJLn8q2CtdKHTnftSD5/mFmApM=",
      "bytesTransferredOut": 0
    },
    "requestID": "TQS46783602DMSZC",
    "eventID": "4215be81-289b-456a-b322-856449bc1029",
    "readOnly": false,
    "resources": [
      {
        "type": "AWS::S3::Object",
        "ARN": "arn:aws:s3:::trek10-joel-s3-event/test.txt"
      },
      {
        "accountId": "549005336969",
        "type": "AWS::S3::Bucket",
        "ARN": "arn:aws:s3:::trek10-joel-s3-event"
      }
    ],
    "eventType": "AwsApiCall",
    "managementEvent": false,
    "recipientAccountId": "549005336969",
    "eventCategory": "Data",
    "tlsDetails": {
      "tlsVersion": "TLSv1.2",
      "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
      "clientProvidedHostHeader": "trek10-joel-s3-event.s3.amazonaws.com"
    }
  }
}

The newer event is a CloudTrail event. The older version follows the pattern of many other events such as SNS, SQS, etc.

Event Bridge Rule Input Transformations let you select attributes of the incoming event using JSONPath expressions and then substitute these values into a string to create a new event (the new string can also be a stringified JSON object).

Not all of the attributes from the new event style are present in the old, but the important ones are. Here is my best attempt to create this transformation:

The EventBridge Rule defined by CloudFormation:

LambdaFunctionNewEventS3Eb:
    Type: AWS::Events::Rule
    Properties:
      EventPattern:
        source:
          - aws.s3
        detail:
          eventSource:
            - s3.amazonaws.com
          eventName:
            - CopyObject
            - PutObject
            - CompleteMultipartUpload
          requestParameters:
            bucketName:
              - !Ref 'S3Bucket'
      State: ENABLED
      Targets:
        - Arn: !GetAtt 'LambdaFunctionNewEvent.Arn'
          Id: LambdaFunctionNewEventS3EbLambdaTarget
          InputTransformer:
              InputTemplate: |
                {
                    "Records": [
                        {
                            "eventVersion": "2.1",
                            "eventSource": "aws:s3",
                            "awsRegion": "<awsRegion>",
                            "eventTime": "<time>",
                            "userIdentity": {
                              "principalId": "AWS:<principalId>"
                            },
                            "requestParameters": {
                              "sourceIPAddress": "<sourceIpAddress>"
                            },
                            "responseElements": {
                              "x-amz-request-id": "<requestId>",
                              "x-amz-id-2": "<xAmzId2>"
                            },
                            "s3": {
                              "s3SchemaVersion": "1.0",
                              "bucket": {
                                  "name": "<bucketName>",
                                  "arn": "arn:aws:s3:::<bucketName>"
                              },
                              "object": {
                                  "key": "<s3Key>",
                                  "size": <size>
                              }
                            }
                        }
                    ]
                  }
              InputPathsMap:
                  awsRegion:  '$.detail.awsRegion'
                  time: '$.time'
                  principalId: '$.detail.userIdentity.principalId'
                  sourceIpAddress: $.detail.sourceIPAddress
                  requestId: $.detail.requestID
                  xAmzId2: $.detail.additionalEventData.x-amz-id-2
                  bucketName: $.detail.requestParameters.bucketName
                  s3Key: $.detail.requestParameters.key
                  size: $.detail.additionalEventData.bytesTransferredIn

Here is a comparison of the old-style event for the same S3 action and the synthesized event:

A few attributes are present in the old style event that are missing in the new event, namely the bucket ownerIdentity, configurationId, the eventTime are slightly different, and the eventName is also different. Unfortunately, there is currently no way to do more complicated logic directly in an Event Rule Input Transformation.

I hope you’ve found this example demonstrating some of the power of the Event Rule Input Transforms to be useful.

References and additional reading:

• https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-transform-target-input.html
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html