diff --git a/README.md b/README.md index a68600c..ae4d1c4 100644 --- a/README.md +++ b/README.md @@ -79,6 +79,32 @@ All BoA and Slack credentials are stored in AWS Secrets Manager (per `engineerin `boa-account-number` is duplicated into both BoA secrets. Each Lambda is granted `secretsmanager:GetSecretValue` scoped to only the secret it needs. The Expense Approval Bot uses two additional secrets (`payments-dashboard/expense-slack-token`, `payments-dashboard/expense-slack-signing-secret`). +## Monitoring & Alarms + +All CloudWatch alarms publish to the shared `site-alerts` SNS topic (`arn:aws:sns:us-east-1:328440206208:site-alerts`). Alarms are ALARM-only by convention (no OK/recovery action) and treat missing data as `notBreaching`. Each alarm evaluates a single 5-minute period. + +**SQS dead-letter queues** (messages-present, Maximum > 0): + +| Alarm | Source | +|-------|--------| +| `payments-payroll-batch-dlq-messages` | `payments-payroll-batch-dlq` | +| `payments-processPaymentCsv-async-dlq-messages` | async-invoke OnFailure DLQ | +| `payments-processPayrollEmail-async-dlq-messages` | async-invoke OnFailure DLQ | + +**Lambda** (per function — `payments--...`): + +| Type | Metric / Statistic | Threshold | +|------|--------------------|-----------| +| `-errors` (all 6) | `Errors` / Sum | > 0 | +| `-throttles` (all 6) | `Throttles` / Sum | > 0 | +| `-duration` (all 6) | `Duration` / Maximum | ~80% of each function's timeout | + +Duration thresholds (ms): processPaymentCsv 96000, processPayrollEmail 48000, fetchBoaTransactions 48000, slackAppHome 24000, expenseProcessor 12000, expenseReceiver 4000. + +**DynamoDB** (`PaymentsDashboard` table, `TableName` dimension, Sum > 0): `payments-dashboard-table-read-throttle` (`ReadThrottleEvents`), `payments-dashboard-table-write-throttle` (`WriteThrottleEvents`). The table is PAY_PER_REQUEST; these metrics emit only when a throttle occurs. `SystemErrors` is intentionally not alarmed because it does not emit at the `TableName`-only dimension. + +**API Gateway** (implicit HTTP API v2 `ServerlessHttpApi`, `ApiId` dimension): `payments-dashboard-api-5xx` (`5xx` Sum > 0), `payments-dashboard-api-4xx` (`4xx` Sum > 10, client-error noise floor), `payments-dashboard-api-latency-p99` (`Latency` p99 > 3000 ms). + ## Scripts | Script | Purpose | diff --git a/template.yaml b/template.yaml index c351ba8..e7acb0f 100644 --- a/template.yaml +++ b/template.yaml @@ -338,6 +338,427 @@ Resources: AlarmActions: - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + # ── Lambda Errors alarms (Wave 1) ────────────────────────────────────────── + # Clone of ProcessPaymentCsvErrorsAlarm for the remaining functions. AWS/Lambda + # Errors, Sum over 5m, threshold > 0, ALARM-only by convention. + SlackAppHomeErrorsAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-slackAppHome-errors + AlarmDescription: payments-slackAppHome invocation errors + Namespace: AWS/Lambda + MetricName: Errors + Dimensions: + - Name: FunctionName + Value: !Ref SlackAppHomeFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + FetchBoaTransactionsErrorsAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-fetchBoaTransactions-errors + AlarmDescription: payments-fetchBoaTransactions invocation errors + Namespace: AWS/Lambda + MetricName: Errors + Dimensions: + - Name: FunctionName + Value: !Ref FetchBoaTransactionsFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + ExpenseReceiverErrorsAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-expenseReceiver-errors + AlarmDescription: payments-expenseReceiver invocation errors + Namespace: AWS/Lambda + MetricName: Errors + Dimensions: + - Name: FunctionName + Value: !Ref ExpenseReceiverFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + ExpenseProcessorErrorsAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-expenseProcessor-errors + AlarmDescription: payments-expenseProcessor invocation errors + Namespace: AWS/Lambda + MetricName: Errors + Dimensions: + - Name: FunctionName + Value: !Ref ExpenseProcessorFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + # ── Lambda Throttles alarms (Wave 1) ─────────────────────────────────────── + # AWS/Lambda Throttles, Sum over 5m, threshold > 0, ALARM-only. Throttling + # signals concurrency exhaustion / reserved-concurrency starvation. + ProcessPaymentCsvThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-processPaymentCsv-throttles + AlarmDescription: payments-processPaymentCsv invocations throttled + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref ProcessPaymentCsvFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + ProcessPayrollEmailThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-processPayrollEmail-throttles + AlarmDescription: payments-processPayrollEmail invocations throttled + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref ProcessPayrollEmailFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + FetchBoaTransactionsThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-fetchBoaTransactions-throttles + AlarmDescription: payments-fetchBoaTransactions invocations throttled + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref FetchBoaTransactionsFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + SlackAppHomeThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-slackAppHome-throttles + AlarmDescription: payments-slackAppHome invocations throttled + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref SlackAppHomeFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + ExpenseReceiverThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-expenseReceiver-throttles + AlarmDescription: payments-expenseReceiver invocations throttled + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref ExpenseReceiverFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + ExpenseProcessorThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-expenseProcessor-throttles + AlarmDescription: payments-expenseProcessor invocations throttled + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref ExpenseProcessorFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + # ── Lambda Duration alarms (Wave 1) ──────────────────────────────────────── + # AWS/Lambda Duration (ms), Statistic Maximum over 5m. Thresholds are ~80% of + # each function's configured timeout — early warning before timeout-kills. + ProcessPaymentCsvDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-processPaymentCsv-duration + AlarmDescription: payments-processPaymentCsv approaching timeout (~80% of 120s) + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref ProcessPaymentCsvFunction + Statistic: Maximum + Period: 300 + EvaluationPeriods: 1 + Threshold: 96000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + ProcessPayrollEmailDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-processPayrollEmail-duration + AlarmDescription: payments-processPayrollEmail approaching timeout (~80% of 60s) + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref ProcessPayrollEmailFunction + Statistic: Maximum + Period: 300 + EvaluationPeriods: 1 + Threshold: 48000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + FetchBoaTransactionsDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-fetchBoaTransactions-duration + AlarmDescription: payments-fetchBoaTransactions approaching timeout (~80% of 60s) + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref FetchBoaTransactionsFunction + Statistic: Maximum + Period: 300 + EvaluationPeriods: 1 + Threshold: 48000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + ExpenseProcessorDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-expenseProcessor-duration + AlarmDescription: payments-expenseProcessor approaching timeout (~80% of 15s) + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref ExpenseProcessorFunction + Statistic: Maximum + Period: 300 + EvaluationPeriods: 1 + Threshold: 12000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + ExpenseReceiverDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-expenseReceiver-duration + AlarmDescription: payments-expenseReceiver approaching timeout (~80% of 5s) + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref ExpenseReceiverFunction + Statistic: Maximum + Period: 300 + EvaluationPeriods: 1 + Threshold: 4000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + SlackAppHomeDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-slackAppHome-duration + AlarmDescription: payments-slackAppHome approaching timeout (~80% of 30s default) + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref SlackAppHomeFunction + Statistic: Maximum + Period: 300 + EvaluationPeriods: 1 + Threshold: 24000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + # ── DynamoDB alarms (Wave 1, SCOPE-CONFIRM) ──────────────────────────────── + # AWS/DynamoDB throttle metrics for the PaymentsDashboard table. These metrics + # emit at the TableName dimension and only on the occurrence of a throttle + # event — none are currently present in CloudWatch (the table is + # PAY_PER_REQUEST, so sustained throttling is unlikely but possible during + # burst-capacity ramp). SystemErrors is intentionally not alarmed: AWS/DynamoDB + # SystemErrors does not emit at the TableName-only dimension, so it can never + # fire. Threshold > 0, Sum over 5m, ALARM-only. + DashboardTableReadThrottleAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-dashboard-table-read-throttle + AlarmDescription: PaymentsDashboard table read requests throttled + Namespace: AWS/DynamoDB + MetricName: ReadThrottleEvents + Dimensions: + - Name: TableName + Value: !Ref DashboardTable + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + DashboardTableWriteThrottleAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-dashboard-table-write-throttle + AlarmDescription: PaymentsDashboard table write requests throttled + Namespace: AWS/DynamoDB + MetricName: WriteThrottleEvents + Dimensions: + - Name: TableName + Value: !Ref DashboardTable + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + # ── API Gateway (HTTP API v2) alarms (Wave 1, SCOPE-CONFIRM) ──────────────── + # AWS/ApiGateway v2 metrics on the implicit ServerlessHttpApi (ApiId dim). + # v2 metric names are 4xx/5xx/Latency (not 4XXError/5XXError). 5xx and Latency + # alarm on the API itself; 4xx is mostly client-driven so its threshold is + # set above zero to avoid noise (Slack URL-verification / bad requests). + ApiGateway5xxAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-dashboard-api-5xx + AlarmDescription: payments-dashboard HTTP API returned 5xx responses + Namespace: AWS/ApiGateway + MetricName: 5xx + Dimensions: + - Name: ApiId + Value: !Ref ServerlessHttpApi + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + ApiGateway4xxAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-dashboard-api-4xx + AlarmDescription: payments-dashboard HTTP API elevated 4xx responses + Namespace: AWS/ApiGateway + MetricName: 4xx + Dimensions: + - Name: ApiId + Value: !Ref ServerlessHttpApi + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 10 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + + ApiGatewayLatencyAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: payments-dashboard-api-latency-p99 + AlarmDescription: payments-dashboard HTTP API p99 latency elevated (>3s) + Namespace: AWS/ApiGateway + MetricName: Latency + Dimensions: + - Name: ApiId + Value: !Ref ServerlessHttpApi + ExtendedStatistic: p99 + Period: 300 + EvaluationPeriods: 1 + Threshold: 3000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts + ProcessPayrollEmailLogGroup: Type: AWS::Logs::LogGroup Properties: