INFRA-41: reconcile async-invoke DLQs + error alarms into IaC

Bring the interim CLI-created dead-letter queues, error alarms, and
SendMessage role policies for payments-processPaymentCsv and
payments-processPayrollEmail under CloudFormation control (H-8 drift).

- Add per-function async-invoke OnFailure SQS DLQs (CFN-named
  payments-<fn>-async-dlq, 14d retention to match payments-payroll-batch-dlq)
- Wire EventInvokeConfig OnFailure on both functions (SAM auto-generates the
  scoped sqs:SendMessage policy on each execution role); explicit retry/age
  defaults locked in
- Add ALARM-only Lambda Errors alarms (Sum, threshold>0) -> site-alerts

Interim CLI resources (queues, alarms, role policies, event-invoke-configs)
removed post-deploy after the CFN-managed versions were confirmed live.
This commit is contained in:
Adam Moussa 2026-06-08 15:52:02 -04:00
parent ef2dcdccbf
commit 3b6cc32fd3

View file

@ -250,6 +250,65 @@ Resources:
AlarmActions:
- !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
# Async-invoke OnFailure DLQs (INFRA-41 / H-8). Reconciles the interim
# CLI-created queues into CloudFormation. Names are CFN-generated to avoid
# colliding with the live interim payments-<fn>-dlq queues (deleted after
# this deploy). 14-day retention mirrors the payments-payroll-batch DLQ so a
# Friday failure survives the weekend.
# Distinct -async-dlq name (not the live interim payments-<fn>-dlq) so this
# CFN queue does not collide with the queue being deleted post-deploy.
ProcessPaymentCsvDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: payments-processPaymentCsv-async-dlq
MessageRetentionPeriod: 1209600 # 14d, matches payments-payroll-batch-dlq
ProcessPayrollEmailDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: payments-processPayrollEmail-async-dlq
MessageRetentionPeriod: 1209600 # 14d, matches payments-payroll-batch-dlq
# ALARM-only Lambda Errors alarms (INFRA-41 / H-8). Threshold > 0 on the
# Errors Sum, no OK/recovery action by convention.
ProcessPaymentCsvErrorsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: payments-processPaymentCsv-errors
AlarmDescription: payments-processPaymentCsv invocation errors
Namespace: AWS/Lambda
MetricName: Errors
Dimensions:
- Name: FunctionName
Value: !Ref ProcessPaymentCsvFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
ProcessPayrollEmailErrorsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: payments-processPayrollEmail-errors
AlarmDescription: payments-processPayrollEmail invocation errors
Namespace: AWS/Lambda
MetricName: Errors
Dimensions:
- Name: FunctionName
Value: !Ref ProcessPayrollEmailFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
ProcessPayrollEmailLogGroup:
Type: AWS::Logs::LogGroup
Properties:
@ -292,6 +351,13 @@ Resources:
FunctionName: payments-processPayrollEmail
Handler: src/processPayrollEmail.handler
Timeout: 60
EventInvokeConfig:
MaximumRetryAttempts: 2
MaximumEventAgeInSeconds: 21600
DestinationConfig:
OnFailure:
Type: SQS
Destination: !GetAtt ProcessPayrollEmailDLQ.Arn
Environment:
Variables:
SLACK_BOT_TOKEN_SECRET_NAME: payments-dashboard/slack-bot-token
@ -334,6 +400,13 @@ Resources:
FunctionName: payments-processPaymentCsv
Handler: src/processPaymentCsv.handler
Timeout: 120
EventInvokeConfig:
MaximumRetryAttempts: 2
MaximumEventAgeInSeconds: 21600
DestinationConfig:
OnFailure:
Type: SQS
Destination: !GetAtt ProcessPaymentCsvDLQ.Arn
Environment:
Variables:
BOA_BASE_URL: https://api.bofa.com