mirror of
https://github.com/Sea-Haven-Industries/front-sla-monitor.git
synced 2026-05-18 20:20:14 +00:00
Add business hours gating, start date, and morning summary
Restrict alerts to 8am-5pm ET Mon-Fri. Lambda skips before START_DATE (2026-05-14). First run of day with 2+ breaches sends consolidated summary to #front-sla-alerts. Cron narrowed to UTC 12-22 for DST coverage. README updated with new config and architecture.
This commit is contained in:
parent
1aabc6a425
commit
fa2397cb5d
3 changed files with 122 additions and 31 deletions
24
README.md
24
README.md
|
|
@ -1,6 +1,6 @@
|
|||
# front-sla-monitor
|
||||
|
||||
Scheduled Lambda that monitors Front conversations for SLA breaches and sends tiered Slack alerts. Runs every 15 minutes on weekdays.
|
||||
Scheduled Lambda that monitors Front conversations for SLA breaches and sends tiered Slack alerts. Runs every 15 minutes during business hours (8 AM–5 PM ET, Mon-Fri).
|
||||
|
||||
## SLA Rules
|
||||
|
||||
|
|
@ -11,22 +11,28 @@ Scheduled Lambda that monitors Front conversations for SLA breaches and sends ti
|
|||
|
||||
Business time counts weekday hours only (Mon-Fri, Eastern time). The SLA clock pauses on Saturday and Sunday.
|
||||
|
||||
Alerts are only sent during business hours (8 AM–5 PM ET). If breaches accumulate overnight, the first morning run sends a single summary message to #front-sla-alerts instead of individual alerts.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
EventBridge (every 15 min, Mon-Fri)
|
||||
EventBridge (every 15 min, 8 AM–5 PM ET, Mon-Fri)
|
||||
│
|
||||
▼
|
||||
Lambda (Python 3.12, arm64)
|
||||
│
|
||||
├── Start date gate → skip before START_DATE
|
||||
├── Business hours gate → skip outside 8 AM–5 PM ET
|
||||
│
|
||||
├── Secrets Manager → front-sla-monitor/front-api-token
|
||||
├── Secrets Manager → front-sla-monitor/slack-bot-token
|
||||
│
|
||||
├── GET Front API /inboxes → list shared inboxes
|
||||
├── GET Front API /inboxes/{id}/conversations → open conversations
|
||||
├── GET Front API /inboxes → filter to MONITOR_INBOXES
|
||||
├── GET Front API /inboxes/{id}/conversations → open conversations (max 10 pages)
|
||||
│
|
||||
├── DynamoDB (front-sla-alerts) → dedup: skip already-alerted conversations
|
||||
├── DynamoDB (front-sla-alerts) → dedup + first-run-of-day detection
|
||||
│
|
||||
├── Morning (first run) → summary message to #front-sla-alerts
|
||||
├── Tier 1 → Slack DM assignee or #front-sla-alerts
|
||||
└── Tier 2 → Slack DM Adam
|
||||
```
|
||||
|
|
@ -34,9 +40,9 @@ Lambda (Python 3.12, arm64)
|
|||
## AWS Resources
|
||||
|
||||
- **Stack:** `front-sla-monitor` (SAM, us-east-1)
|
||||
- **Lambda:** `front-sla-monitor` — Python 3.12, arm64, 128 MB, 120s timeout, 60-day log retention
|
||||
- **DynamoDB:** `front-sla-alerts` — tracks alert history per conversation, 7-day TTL
|
||||
- **EventBridge:** `cron(0/15 * ? * MON-FRI *)` — every 15 min on weekdays
|
||||
- **Lambda:** `front-sla-monitor` — Python 3.12, arm64, 128 MB, 300s timeout, 60-day log retention
|
||||
- **DynamoDB:** `front-sla-alerts` — tracks alert history per conversation + monitor state, 7-day TTL
|
||||
- **EventBridge:** `cron(0/15 12-22 ? * MON-FRI *)` — every 15 min during business hours (UTC range covers EDT/EST)
|
||||
|
||||
## Setup
|
||||
|
||||
|
|
@ -104,3 +110,5 @@ aws logs tail /aws/lambda/front-sla-monitor --follow --region us-east-1
|
|||
| `ACTION_SLA_MINUTES` | 1440 | Business minutes before Tier 2 alert |
|
||||
| `ADAM_EMAIL` | adam@seahavenind.com | Tier 2 escalation recipient |
|
||||
| `SLACK_ALERT_CHANNEL` | — | Channel ID for broadcast alerts |
|
||||
| `MONITOR_INBOXES` | Triage,California,West Coast,Central,East Coast,Vendors | Comma-separated inbox names to monitor (empty = all shared) |
|
||||
| `START_DATE` | 2026-05-14 | Date when monitoring begins (YYYY-MM-DD, Eastern time) |
|
||||
|
|
|
|||
|
|
@ -15,6 +15,8 @@ EASTERN = ZoneInfo("America/New_York")
|
|||
FRONT_BASE = "https://api2.frontapp.com"
|
||||
SLACK_BASE = "https://slack.com/api"
|
||||
RATE_LIMIT_DELAY = 0.6
|
||||
BH_START = 8
|
||||
BH_END = 17
|
||||
|
||||
_sm = boto3.client("secretsmanager")
|
||||
_ddb = boto3.resource("dynamodb")
|
||||
|
|
@ -210,6 +212,19 @@ def _record_alert(conv_id, tier):
|
|||
)
|
||||
|
||||
|
||||
def _is_first_run_today(today_str):
|
||||
resp = _get_table().get_item(Key={"conversationId": "_monitor_state"})
|
||||
return resp.get("Item", {}).get("lastRunDate", "") != today_str
|
||||
|
||||
|
||||
def _record_run(today_str):
|
||||
_get_table().update_item(
|
||||
Key={"conversationId": "_monitor_state"},
|
||||
UpdateExpression="SET lastRunDate = :d",
|
||||
ExpressionAttributeValues={":d": today_str},
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Slack message blocks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -253,6 +268,34 @@ def _tier2_blocks(conv):
|
|||
]
|
||||
|
||||
|
||||
def _summary_blocks(tier1_breaches, tier2_breaches):
|
||||
lines = []
|
||||
|
||||
if tier2_breaches:
|
||||
lines.append("*:rotating_light: Over 1 Business Day Without Reply*")
|
||||
for conv in tier2_breaches:
|
||||
subject = conv.get("subject", "No subject")
|
||||
link = f"https://app.frontapp.com/open/{conv.get('id', '')}"
|
||||
assignee = conv.get("assignee")
|
||||
who = assignee["email"] if assignee else "Unassigned"
|
||||
lines.append(f"• <{link}|{subject}> — {who}")
|
||||
lines.append("")
|
||||
|
||||
if tier1_breaches:
|
||||
lines.append("*:warning: Over 1 Business Hour Without Reply*")
|
||||
for conv, email in tier1_breaches:
|
||||
subject = conv.get("subject", "No subject")
|
||||
link = f"https://app.frontapp.com/open/{conv.get('id', '')}"
|
||||
who = email or "Unassigned"
|
||||
lines.append(f"• <{link}|{subject}> — {who}")
|
||||
|
||||
total = len(tier1_breaches) + len(tier2_breaches)
|
||||
return [
|
||||
{"type": "header", "text": {"type": "plain_text", "text": f":sunrise: Morning SLA Summary — {total} breach{'es' if total != 1 else ''}"}},
|
||||
{"type": "section", "text": {"type": "mrkdwn", "text": "\n".join(lines)}},
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Handler
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -262,8 +305,24 @@ def handler(event, context):
|
|||
action_threshold = int(os.environ["ACTION_SLA_MINUTES"])
|
||||
alert_channel = os.environ["SLACK_ALERT_CHANNEL"]
|
||||
adam_email = os.environ["ADAM_EMAIL"]
|
||||
start_date = os.environ.get("START_DATE", "")
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
now_et = now.astimezone(EASTERN)
|
||||
|
||||
if start_date:
|
||||
start = datetime.strptime(start_date, "%Y-%m-%d").date()
|
||||
if now_et.date() < start:
|
||||
logger.info("Before start date %s, skipping", start_date)
|
||||
return {"skipped": True, "reason": "before_start_date"}
|
||||
|
||||
if now_et.hour < BH_START or now_et.hour >= BH_END:
|
||||
logger.info("Outside business hours (%s ET), skipping", now_et.strftime("%H:%M"))
|
||||
return {"skipped": True, "reason": "outside_business_hours"}
|
||||
|
||||
today_str = now_et.strftime("%Y-%m-%d")
|
||||
first_run = _is_first_run_today(today_str)
|
||||
_record_run(today_str)
|
||||
|
||||
monitor_names = os.environ.get("MONITOR_INBOXES", "").strip()
|
||||
if monitor_names:
|
||||
|
|
@ -280,8 +339,8 @@ def handler(event, context):
|
|||
logger.info("Monitoring %d inboxes: %s", len(shared),
|
||||
", ".join(i.get("name", "?") for i in shared))
|
||||
|
||||
tier1_count = 0
|
||||
tier2_count = 0
|
||||
tier1_breaches = []
|
||||
tier2_breaches = []
|
||||
seen = set()
|
||||
|
||||
for inbox in shared:
|
||||
|
|
@ -314,30 +373,49 @@ def handler(event, context):
|
|||
assignee = conv.get("assignee")
|
||||
|
||||
if elapsed >= action_threshold and not _already_alerted(conv_id, 2):
|
||||
adam_uid = _resolve_slack_user(adam_email)
|
||||
target = adam_uid or alert_channel
|
||||
_send_slack(target, _tier2_blocks(conv),
|
||||
f"SLA Breach: {conv.get('subject', '')} - 1 day without reply")
|
||||
_record_alert(conv_id, 2)
|
||||
tier2_count += 1
|
||||
logger.info("Tier 2 alert: %s", conv_id)
|
||||
tier2_breaches.append(conv)
|
||||
logger.info("Tier 2 breach: %s", conv_id)
|
||||
|
||||
elif elapsed >= ack_threshold and not _already_alerted(conv_id, 1):
|
||||
if assignee and assignee.get("email"):
|
||||
uid = _resolve_slack_user(assignee["email"])
|
||||
target = uid or alert_channel
|
||||
_send_slack(target, _tier1_blocks(conv, assignee["email"]),
|
||||
f"SLA Breach: {conv.get('subject', '')} - 1 hour without reply")
|
||||
else:
|
||||
_send_slack(alert_channel, _tier1_blocks(conv),
|
||||
f"SLA Breach: {conv.get('subject', '')} - unassigned, 1 hour without reply")
|
||||
_record_alert(conv_id, 1)
|
||||
tier1_count += 1
|
||||
logger.info("Tier 1 alert: %s", conv_id)
|
||||
email = assignee["email"] if assignee and assignee.get("email") else None
|
||||
tier1_breaches.append((conv, email))
|
||||
logger.info("Tier 1 breach: %s", conv_id)
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error processing conversation %s", conv.get("id", "?"))
|
||||
|
||||
result = {"tier1_alerts": tier1_count, "tier2_alerts": tier2_count}
|
||||
total = len(tier1_breaches) + len(tier2_breaches)
|
||||
|
||||
if first_run and total > 1:
|
||||
logger.info("Morning summary: %d breaches accumulated overnight", total)
|
||||
blocks = _summary_blocks(tier1_breaches, tier2_breaches)
|
||||
_send_slack(alert_channel, blocks, f"Morning SLA Summary: {total} breaches")
|
||||
if tier2_breaches:
|
||||
adam_uid = _resolve_slack_user(adam_email)
|
||||
if adam_uid:
|
||||
_send_slack(adam_uid, blocks, f"Morning SLA Summary: {total} breaches")
|
||||
else:
|
||||
for conv in tier2_breaches:
|
||||
adam_uid = _resolve_slack_user(adam_email)
|
||||
target = adam_uid or alert_channel
|
||||
_send_slack(target, _tier2_blocks(conv),
|
||||
f"SLA Breach: {conv.get('subject', '')} - 1 day without reply")
|
||||
|
||||
for conv, email in tier1_breaches:
|
||||
if email:
|
||||
uid = _resolve_slack_user(email)
|
||||
target = uid or alert_channel
|
||||
_send_slack(target, _tier1_blocks(conv, email),
|
||||
f"SLA Breach: {conv.get('subject', '')} - 1 hour without reply")
|
||||
else:
|
||||
_send_slack(alert_channel, _tier1_blocks(conv),
|
||||
f"SLA Breach: {conv.get('subject', '')} - unassigned, 1 hour without reply")
|
||||
|
||||
for conv in tier2_breaches:
|
||||
_record_alert(conv["id"], 2)
|
||||
for conv, _ in tier1_breaches:
|
||||
_record_alert(conv["id"], 1)
|
||||
|
||||
result = {"tier1_alerts": len(tier1_breaches), "tier2_alerts": len(tier2_breaches)}
|
||||
logger.info("Run complete: %s", result)
|
||||
return result
|
||||
|
|
|
|||
|
|
@ -28,6 +28,10 @@ Parameters:
|
|||
Type: String
|
||||
Default: "Triage,California,West Coast,Central,East Coast,Vendors"
|
||||
Description: Comma-separated inbox names to monitor (empty = all shared)
|
||||
StartDate:
|
||||
Type: String
|
||||
Default: "2026-05-14"
|
||||
Description: Date when monitoring begins (YYYY-MM-DD, Eastern time)
|
||||
|
||||
Globals:
|
||||
Function:
|
||||
|
|
@ -76,6 +80,7 @@ Resources:
|
|||
ACK_SLA_MINUTES: !Ref AckSlaMinutes
|
||||
ACTION_SLA_MINUTES: !Ref ActionSlaMinutes
|
||||
MONITOR_INBOXES: !Ref MonitorInboxes
|
||||
START_DATE: !Ref StartDate
|
||||
Policies:
|
||||
- DynamoDBCrudPolicy:
|
||||
TableName: !Ref AlertsTable
|
||||
|
|
@ -91,8 +96,8 @@ Resources:
|
|||
SlaCheck:
|
||||
Type: Schedule
|
||||
Properties:
|
||||
Schedule: cron(0/15 * ? * MON-FRI *)
|
||||
Description: Check Front conversations for SLA breaches every 15 min on weekdays
|
||||
Schedule: cron(0/15 12-22 ? * MON-FRI *)
|
||||
Description: Check Front conversations for SLA breaches every 15 min during business hours
|
||||
Enabled: true
|
||||
|
||||
Outputs:
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue