Add messages-present alarms on async-invoke DLQs

The two async-invoke OnFailure DLQs (payments-processPaymentCsv-async-dlq and payments-processPayrollEmail-async-dlq) had no CloudWatch alarm, so a failed async invocation could sit in the DLQ unnoticed. Add a messages-present alarm for each, mirroring PayrollBatchDLQAlarm exactly: AWS/SQS ApproximateNumberOfMessagesVisible, Maximum, threshold > 0, Period 300, EvaluationPeriods 1, TreatMissingData notBreaching, ALARM-only to the site-alerts SNS topic.
Add CloudWatch alarm coverage (payments-dashboard) (#51 )
2026-10-06 21:42:08 +00:00 · 2026-06-17 15:04:09 -04:00 · 2026-06-17 14:51:59 -04:00 · 2026-06-17 14:43:41 -04:00
2 changed files with 499 additions and 0 deletions
--- a/README.md
+++ b/README.md
@ -79,6 +79,32 @@ All BoA and Slack credentials are stored in AWS Secrets Manager (per `engineerin

 `boa-account-number` is duplicated into both BoA secrets. Each Lambda is granted `secretsmanager:GetSecretValue` scoped to only the secret it needs. The Expense Approval Bot uses two additional secrets (`payments-dashboard/expense-slack-token`, `payments-dashboard/expense-slack-signing-secret`).

+## Monitoring & Alarms
+
+All CloudWatch alarms publish to the shared `site-alerts` SNS topic (`arn:aws:sns:us-east-1:328440206208:site-alerts`). Alarms are ALARM-only by convention (no OK/recovery action) and treat missing data as `notBreaching`. Each alarm evaluates a single 5-minute period.
+
+**SQS dead-letter queues** (messages-present, Maximum > 0):
+
+| Alarm | Source |
+|-------|--------|
+| `payments-payroll-batch-dlq-messages` | `payments-payroll-batch-dlq` |
+| `payments-processPaymentCsv-async-dlq-messages` | async-invoke OnFailure DLQ |
+| `payments-processPayrollEmail-async-dlq-messages` | async-invoke OnFailure DLQ |
+
+**Lambda** (per function — `payments-<fn>-...`):
+
+| Type | Metric / Statistic | Threshold |
+|------|--------------------|-----------|
+| `-errors` (all 6) | `Errors` / Sum | > 0 |
+| `-throttles` (all 6) | `Throttles` / Sum | > 0 |
+| `-duration` (all 6) | `Duration` / Maximum | ~80% of each function's timeout |
+
+Duration thresholds (ms): processPaymentCsv 96000, processPayrollEmail 48000, fetchBoaTransactions 48000, slackAppHome 24000, expenseProcessor 12000, expenseReceiver 4000.
+
+**DynamoDB** (`PaymentsDashboard` table, `TableName` dimension, Sum > 0): `payments-dashboard-table-read-throttle` (`ReadThrottleEvents`), `payments-dashboard-table-write-throttle` (`WriteThrottleEvents`). The table is PAY_PER_REQUEST; these metrics emit only when a throttle occurs. `SystemErrors` is intentionally not alarmed because it does not emit at the `TableName`-only dimension.
+
+**API Gateway** (implicit HTTP API v2 `ServerlessHttpApi`, `ApiId` dimension): `payments-dashboard-api-5xx` (`5xx` Sum > 0), `payments-dashboard-api-4xx` (`4xx` Sum > 10, client-error noise floor), `payments-dashboard-api-latency-p99` (`Latency` p99 > 3000 ms).
+
 ## Scripts

 | Script | Purpose |
--- a/template.yaml
+++ b/template.yaml
@ -162,6 +162,11 @@ Resources:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub seahaven-payments-csv-${AWS::AccountId}
+      PublicAccessBlockConfiguration:
+        BlockPublicAcls: true
+        IgnorePublicAcls: true
+        BlockPublicPolicy: true
+        RestrictPublicBuckets: true

  DashboardTable:
    Type: AWS::DynamoDB::Table
@ -190,6 +195,11 @@ Resources:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub seahaven-payroll-emails-${AWS::AccountId}
+      PublicAccessBlockConfiguration:
+        BlockPublicAcls: true
+        IgnorePublicAcls: true
+        BlockPublicPolicy: true
+        RestrictPublicBuckets: true
      LifecycleConfiguration:
        Rules:
          - Id: ExpireEmails
@ -328,6 +338,469 @@ Resources:
      AlarmActions:
        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts

+  # ── Lambda Errors alarms (Wave 1) ──────────────────────────────────────────
+  # Clone of ProcessPaymentCsvErrorsAlarm for the remaining functions. AWS/Lambda
+  # Errors, Sum over 5m, threshold > 0, ALARM-only by convention.
+  SlackAppHomeErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-slackAppHome-errors
+      AlarmDescription: payments-slackAppHome invocation errors
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SlackAppHomeFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  FetchBoaTransactionsErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-fetchBoaTransactions-errors
+      AlarmDescription: payments-fetchBoaTransactions invocation errors
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref FetchBoaTransactionsFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseReceiverErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseReceiver-errors
+      AlarmDescription: payments-expenseReceiver invocation errors
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseReceiverFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseProcessorErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseProcessor-errors
+      AlarmDescription: payments-expenseProcessor invocation errors
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseProcessorFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  # ── Lambda Throttles alarms (Wave 1) ───────────────────────────────────────
+  # AWS/Lambda Throttles, Sum over 5m, threshold > 0, ALARM-only. Throttling
+  # signals concurrency exhaustion / reserved-concurrency starvation.
+  ProcessPaymentCsvThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-processPaymentCsv-throttles
+      AlarmDescription: payments-processPaymentCsv invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ProcessPaymentCsvFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ProcessPayrollEmailThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-processPayrollEmail-throttles
+      AlarmDescription: payments-processPayrollEmail invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ProcessPayrollEmailFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  FetchBoaTransactionsThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-fetchBoaTransactions-throttles
+      AlarmDescription: payments-fetchBoaTransactions invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref FetchBoaTransactionsFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  SlackAppHomeThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-slackAppHome-throttles
+      AlarmDescription: payments-slackAppHome invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SlackAppHomeFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseReceiverThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseReceiver-throttles
+      AlarmDescription: payments-expenseReceiver invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseReceiverFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseProcessorThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseProcessor-throttles
+      AlarmDescription: payments-expenseProcessor invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseProcessorFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  # ── Lambda Duration alarms (Wave 1) ────────────────────────────────────────
+  # AWS/Lambda Duration (ms), Statistic Maximum over 5m. Thresholds are ~80% of
+  # each function's configured timeout — early warning before timeout-kills.
+  ProcessPaymentCsvDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-processPaymentCsv-duration
+      AlarmDescription: payments-processPaymentCsv approaching timeout (~80% of 120s)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ProcessPaymentCsvFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 96000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ProcessPayrollEmailDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-processPayrollEmail-duration
+      AlarmDescription: payments-processPayrollEmail approaching timeout (~80% of 60s)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ProcessPayrollEmailFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 48000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  FetchBoaTransactionsDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-fetchBoaTransactions-duration
+      AlarmDescription: payments-fetchBoaTransactions approaching timeout (~80% of 60s)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref FetchBoaTransactionsFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 48000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseProcessorDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseProcessor-duration
+      AlarmDescription: payments-expenseProcessor approaching timeout (~80% of 15s)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseProcessorFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 12000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseReceiverDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseReceiver-duration
+      AlarmDescription: payments-expenseReceiver approaching timeout (~80% of 5s)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseReceiverFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 4000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  SlackAppHomeDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-slackAppHome-duration
+      AlarmDescription: payments-slackAppHome approaching timeout (~80% of 30s default)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SlackAppHomeFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 24000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  # ── DynamoDB alarms (Wave 1, SCOPE-CONFIRM) ────────────────────────────────
+  # AWS/DynamoDB throttle metrics for the PaymentsDashboard table. These metrics
+  # emit at the TableName dimension and only on the occurrence of a throttle
+  # event — none are currently present in CloudWatch (the table is
+  # PAY_PER_REQUEST, so sustained throttling is unlikely but possible during
+  # burst-capacity ramp). SystemErrors is intentionally not alarmed: AWS/DynamoDB
+  # SystemErrors does not emit at the TableName-only dimension, so it can never
+  # fire. Threshold > 0, Sum over 5m, ALARM-only.
+  DashboardTableReadThrottleAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-dashboard-table-read-throttle
+      AlarmDescription: PaymentsDashboard table read requests throttled
+      Namespace: AWS/DynamoDB
+      MetricName: ReadThrottleEvents
+      Dimensions:
+        - Name: TableName
+          Value: !Ref DashboardTable
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  DashboardTableWriteThrottleAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-dashboard-table-write-throttle
+      AlarmDescription: PaymentsDashboard table write requests throttled
+      Namespace: AWS/DynamoDB
+      MetricName: WriteThrottleEvents
+      Dimensions:
+        - Name: TableName
+          Value: !Ref DashboardTable
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  # ── API Gateway (HTTP API v2) alarms (Wave 1, SCOPE-CONFIRM) ────────────────
+  # AWS/ApiGateway v2 metrics on the implicit ServerlessHttpApi (ApiId dim).
+  # v2 metric names are 4xx/5xx/Latency (not 4XXError/5XXError). 5xx and Latency
+  # alarm on the API itself; 4xx is mostly client-driven so its threshold is
+  # set above zero to avoid noise (Slack URL-verification / bad requests).
+  ApiGateway5xxAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-dashboard-api-5xx
+      AlarmDescription: payments-dashboard HTTP API returned 5xx responses
+      Namespace: AWS/ApiGateway
+      MetricName: 5xx
+      Dimensions:
+        - Name: ApiId
+          Value: !Ref ServerlessHttpApi
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ApiGateway4xxAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-dashboard-api-4xx
+      AlarmDescription: payments-dashboard HTTP API elevated 4xx responses
+      Namespace: AWS/ApiGateway
+      MetricName: 4xx
+      Dimensions:
+        - Name: ApiId
+          Value: !Ref ServerlessHttpApi
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 10
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ApiGatewayLatencyAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-dashboard-api-latency-p99
+      AlarmDescription: payments-dashboard HTTP API p99 latency elevated (>3s)
+      Namespace: AWS/ApiGateway
+      MetricName: Latency
+      Dimensions:
+        - Name: ApiId
+          Value: !Ref ServerlessHttpApi
+      ExtendedStatistic: p99
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 3000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  # Messages-present alarms on the async-invoke OnFailure DLQs, mirroring
+  # PayrollBatchDLQAlarm. Threshold > 0 on the visible-message count, ALARM-only.
+  ProcessPaymentCsvDLQAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-processPaymentCsv-async-dlq-messages
+      AlarmDescription: Failed processPaymentCsv async invocations landed in the DLQ
+      Namespace: AWS/SQS
+      MetricName: ApproximateNumberOfMessagesVisible
+      Dimensions:
+        - Name: QueueName
+          Value: !GetAtt ProcessPaymentCsvDLQ.QueueName
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      # ALARM-only notification by convention — no OK/recovery action
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ProcessPayrollEmailDLQAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-processPayrollEmail-async-dlq-messages
+      AlarmDescription: Failed processPayrollEmail async invocations landed in the DLQ
+      Namespace: AWS/SQS
+      MetricName: ApproximateNumberOfMessagesVisible
+      Dimensions:
+        - Name: QueueName
+          Value: !GetAtt ProcessPayrollEmailDLQ.QueueName
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      # ALARM-only notification by convention — no OK/recovery action
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
  ProcessPayrollEmailLogGroup:
    Type: AWS::Logs::LogGroup
    Properties: