Understanding EventBridge Retry Behavior
When an EventBridge rule triggers a Lambda function, the service manages the execution lifecycle. If the Lambda function returns a non-200 status code or throws an exception, EventBridge considers the invocation failed. By default, EventBridge will retry the invocation up to two times with exponential backoff. However, this standard behavior does not change the payload sent to the target. The same static payload defined in the rule's target configuration is used for both the initial attempt and any subsequent retries.
This means that if you need different data in the payload for a retry scenario, you cannot rely on EventBridge to automatically swap out the JSON object. You must implement logic within the Lambda function itself to detect that it is being called again or to process the payload in a way that accounts for potential duplicates. This approach ensures that your system remains robust against transient failures without requiring complex orchestration layers.
Configuring Retry Policies
To enable retries, you must explicitly configure the AWS::Events::Rule resource or use the AWS Management Console to set the maximum event age and number of attempts. When setting these parameters, keep in mind that the payload remains constant across all attempts. If your Lambda function is sensitive to duplicate inputs, you must implement idempotency keys or check internal state variables before processing the request.
Using Input Transformers for Custom Payloads
An effective strategy to differentiate between initial calls and retries involves using EventBridge's input transformation features. Although you cannot dynamically change the payload based on the retry count directly in the rule configuration, you can create a structured payload that includes fields useful for tracking. For example, you can use an input transformer to inject a timestamp or a generated UUID into the payload sent to Lambda.
By ensuring each invocation has a unique identifier, your Lambda function can track whether it has already processed a specific request ID. If a retry occurs, the function will see the same ID (if you control the generation) or a new one (if EventBridge generates it). To truly distinguish retries, consider adding a header or field that indicates the attempt number. Since EventBridge does not provide a built-in "retry count" variable in its path patterns, you may need to handle this logic inside the Lambda code by checking CloudWatch logs or using a distributed state store like DynamoDB to record previous invocations.
Implementing Idempotency in Lambda
The most reliable way to handle retries with special payloads is to make your Lambda function idempotent. This means that calling the function multiple times with the same input produces the same result. You can achieve this by storing the outcome of the first successful invocation in a database. When a retry arrives, the Lambda function checks the database. If the result exists, it returns immediately without reprocessing the business logic. This approach simplifies the EventBridge configuration, allowing you to send a consistent payload while handling the complexity of retries at the application layer.
Advanced Patterns: Step Functions Integration
If your use case requires significantly different payloads for retries, consider moving away from direct EventBridge-to-Lambda integrations. Instead, trigger an AWS Step Function state machine from EventBridge. Step Functions offer more granular control over error handling and retry logic. You can define specific states for initial processing and separate states for retry logic, each with their own input path and transformation rules. This architecture allows you to modify the payload structure entirely between the first attempt and subsequent retries, providing greater flexibility than EventBridge alone.
While this adds complexity, it solves the problem of needing distinct payloads for different retry stages. You can map errors to specific retry branches and inject different context information into the payload for each branch. This pattern is particularly useful for long-running processes where the state of the system changes significantly between attempts.
Monitoring and Debugging
To verify that your special payload logic is working correctly, monitor the invocations in CloudWatch Logs. Look for patterns in the log entries that indicate retries, such as repeated invocation IDs or specific error messages. Ensure that your input transformers are correctly formatting the JSON output. Misconfigured transformers can lead to malformed payloads that cause Lambda execution failures, creating a loop of retries that never resolves.
Frequently Asked Questions
Can I pass a variable retry count to the Lambda payload?
No, EventBridge does not expose a native variable for the current retry attempt number in its input transformers. You must implement tracking logic within your Lambda function or use an external state store to determine if a request is a retry.
What happens if the Lambda function fails all retry attempts?
If the Lambda function fails after the configured maximum number of retries, EventBridge discards the event unless you have configured a dead-letter queue (DLQ). The DLQ can be an SQS queue or SNS topic, allowing you to inspect the original payload and retry manually later.
Is there a way to change the payload only on the second retry?
Not directly through EventBridge rule settings. You would need to use a Step Function workflow to manage the state and inject different payloads for each step, or handle the logic entirely within the Lambda function by checking previous execution history.
How do I prevent duplicate processing when retries occur?
Implement idempotency by using unique identifiers in your payload. Store the result of the operation in a persistent store like DynamoDB. Before processing, check if the identifier already exists. If it does, return the cached result instead of executing the main logic again.
Does EventBridge resend the original event or a modified one?
EventBridge resends the exact same event payload that triggered the rule. It does not modify the content based on retry status. Any changes to the payload must be done via input transformers before the event reaches the target, but these transformations are static and apply to all invocations equally.