After-Hours Shift Manager

Slack bot for managing after-hours on-call shifts at Sea Haven Industries. Employees can pick up, drop, and swap shifts directly from Slack. Changes automatically update 3CX queue routing via the integrated ring scheduler.

How It Works

A recurring weekly schedule assigns employees to after-hours phone duty. Weekend shifts are split into Day (8am-5pm) and Night (5pm-8am). Any unassigned shift shows as Available in Slack with a pickup button. When someone picks up or drops a shift for today, the 3CX queue is updated immediately. Future changes take effect when the ring scheduler runs at 8am daily and 5pm on weekends.

The work week runs Sunday through Saturday, matching payroll. A Saturday night shift (5pm Saturday through 8am Sunday) stays in the Saturday week. Sunday day and Sunday night open the next week. The Monday 7am post shows the week that started the day before, plus the following week, and the pay summary covers the previous Sunday through Saturday.

The weekly schedule post is updated live when shifts change, and the previous week's post is automatically deleted when the new one goes out.

The bot also has an About page: open the bot in Slack and click its Home tab to see what it does, the full command list, and the latest "What's New" (see Releases & Versioning).

Employees can do the same pick, drop, swap, and admin work from the internal portal After Hours pages. Slack DMs for swaps and late pickups still go out. App Home admin modals stay as they are.

Slack Commands

Command	Description
/oncall	Show this week's schedule
/oncall next	Show next week's schedule
/oncall pick <date>	Pick up an available shift — instant if the shift hasn't started; if it's already underway (but not ended) it needs admin approval (see Late-pickup approval)
/oncall drop <date>	Drop your shift (marks it available) — blocked within 24h of shift start; swap or ask an admin instead
/oncall swap <date> @person	Request a swap — the other person gets an Accept/Decline DM and the shift only moves once they accept
/oncall register <ext>	Link your Slack account to your phone extension
/oncall roster	Show all employees and their link status
/oncall pay	Show last week's bonus pay summary (Sunday through Saturday)
/oncall rate	Show current shift pay rates
/oncall rate default <amount>	Set the default per-shift rate
/oncall rate <ext> <amount>	Set a per-person shift rate
/oncall help	Show help

Admin Commands

Available to users listed in admin_users in the CONFIG record:

Command	Description
/oncall admin override <date> <ext>	Assign a shift to an extension
/oncall admin open <date>	Mark a shift as open
/oncall admin clear <date>	Remove override (revert to weekly)
/oncall admin roster add <ext> <name>	Add an employee to the roster
/oncall admin roster remove <ext>	Remove an employee
/oncall admin roster rename <ext> <name>	Rename an employee
/oncall admin holiday add <date> <slots> [x<mult>] <label>	Schedule a holiday day shift (8am-5pm ET) with N slots, optional pay multiplier override (e.g. x2), and a label
/oncall admin holiday remove <date>	Remove a scheduled holiday and its activate/deactivate schedules
/oncall admin holiday list	List today-and-future scheduled holidays

Dates accept: today, tomorrow, monday-sunday, 4/5, 2026-04-05

Example: /oncall admin holiday add 2026-07-04 2 x2 Independence Day schedules a 2-slot holiday paying 2x. Omitting the x<mult> token uses the default holiday_multiplier from CONFIG (1.5). See Holidays for the full flow.

Architecture

• Runtime: Python 3.12 on ECS Fargate (arm64) plus dual-run Lambdas until cutover. seahaven-prod 011934824531, seahaven-dev 710827005802
• VPC: This stack owns 10.70.0.0/16. Meals and portal Fargate attach with HCP existing_vpc_id / existing_public_subnet_ids from outputs vpc_id and public_subnet_ids.
• HTTP contract: openapi.yaml, linted in CI with npm run openapi:lint (Redocly extends: recommended, same as internal-portal and meal-order-manager).
• Data: DynamoDB single-table (afterhours-shifts)
• IaC: HCP Terraform workspaces tagged app:afterhours-shift-manager (afterhours-shift-manager-dev / -prod) plus GitHub Actions deploy-api.yaml (image). Terraform does not package src/.
• Slack: Slack Bolt on POST /slack/events
• 3CX Integration: Queue routing updated directly via 3CX Queue XAPI
• Secrets: AWS Secrets Manager (afterhours-shift-manager/*)

Leftover zip handlers (packaging only)

Fargate is the live path. These src/*/app.py zip sources remain for scripts/package_lambdas.py only. Leftover Lambda IAM roles were removed after Paychex dropped AfterhoursWeeklyPostSend (PLAT-218).

Handler	Former trigger	Purpose
afterhours-shift-manager	API Gateway (POST /slack/events)	Slack bot — handles /oncall commands and interactive buttons
afterhours-weekly-post	EventBridge (Monday 7am ET)	Posts the Sunday-Saturday schedule and the previous week's pay summary
afterhours-roster-sync	EventBridge (daily 6am ET)	Syncs employee roster from 3CX
afterhours-portal-api	API Gateway (ANY /api/shifts, ANY /api/shifts/{proxy+})	Cognito-authenticated employee/admin shift API for the internal portal
afterhours-ring-scheduler	EventBridge (daily 8am ET + weekend 5pm ET)	Updates 3CX queue routing based on who's on shift
afterhours-holiday-router	EventBridge Scheduler (per-holiday one-off: 8am activate / 5pm deactivate ET)	Repoints the IVR to the holiday queue and sets queue agents for a holiday day shift; reverts at 5pm (see Holidays)

Project Layout

src/
  server/            Flask + gunicorn + SQS worker (Fargate)
  slack-bot/         Slack Bolt app; leftover zip source for packaging; ships CHANGELOG.md for App Home
  weekly-post/       Monday 7am schedule post and prior Sunday-Saturday pay (leftover zip source)
  roster-sync/       Daily 3CX roster sync (leftover zip source for packaging)
  roster-api/        HTTP PUT/DELETE /roster for identity hire/offboard (leftover zip source for packaging)
  portal-api/        Cognito employee/admin shift API for the internal portal (leftover zip source for packaging)
  ring-scheduler/    3CX queue routing updates (leftover zip source for packaging)
  holiday-router/    3CX IVR/queue repoint for holiday day shifts (leftover zip source for packaging)
  shared/            Bundled into leftover function zips and the Fargate image
terraform/           HCP Terraform (ECS/ALB, API, DDB, IAM, leftover zip packaging locals, schedules)
scripts/             in-package changelog copy sync + cutover
tests/               pytest suite (mirrors src/, one dir per leftover zip handler + shared + server)

DynamoDB Schema

Single table with PK / SK keys:

PK	SK	Description
ROSTER	<extension>	Employee: name, extension, slack_user_id, optional email
WEEKLY	<DayName>	Default weekly schedule: extension, name
OVERRIDE	<YYYY-MM-DD>	Date override from pickup/drop (or OPEN)
SWAP	<YYYY-MM-DD>	Pending/verified swap request: requester, target, status, expires_at (TTL)
HOLIDAY	<YYYY-MM-DD>	Holiday day shift (one per date): slots (int), assignees (MAP keyed by extension — {"114": {name, claimed_at}}), multiplier (Decimal, defaults to CONFIG.holiday_multiplier = 1.5, overridable per holiday), label, created_at, created_by, activated (bool), schedule_names (list)
PICKUP_REQUEST	<YYYY-MM-DD>[-DAY]#<ext>	Pending late-pickup awaiting admin approval: requester_ext, requester_name, requester_slack, shift_type, is_holiday, status, created_at, expires_at (TTL = shift end)
SCHEDULE_POST	<channel_id>	Current schedule message timestamp
PAY	<YYYY-MM-DD>	Weekly pay record (Sunday date key; older rows may use Monday)
CONFIG	CONFIG	Settings: shift_rate, fallback_extension, admin_users, ring_group, holiday_multiplier (default holiday pay multiplier, 1.5), holiday_queue (3CX queue repointed during holidays, default 802), ivr_number (3CX IVR repointed during holidays, default 800), captured_ivr_routes (original IVR routes saved at holiday activation, restored at deactivation)

Weekend day-shift rows use a -DAY suffix on the SK (e.g. OVERRIDE / 2026-04-05-DAY). The table has TTL enabled on expires_at so abandoned pending swaps and pickup requests self-clean.

Shift priority for any date is HOLIDAY > OVERRIDE > WEEKLY — a holiday record wins over a regular override, which wins over the standing weekly schedule.

Holiday slot claims are atomic Map updates so concurrent pickers can't oversubscribe:

• Claim — SET assignees.#ext guarded by attribute_not_exists(assignees.#ext) AND size(assignees) < :slots.
• Release — REMOVE assignees.#ext guarded by attribute_exists.
• Swap — REMOVE #from SET #to guarded by attribute_exists(#from) AND attribute_not_exists(#to).

Swap flow: /oncall swap writes a pending SWAP record and DMs the target Accept/Decline buttons; it does not reassign the shift. On Accept, the override is written, 3CX is repointed if it's the active shift, and the record is marked verified. On Decline (or once the shift has started) the request is dropped and the shift stays with the original owner.

Holidays

A holiday is a single day-only shift (08:00-17:00 ET), one HOLIDAY record per date, that can hold multiple people (slots). Admins manage holidays with /oncall admin holiday add|remove|list. A holiday takes priority over a regular override and the weekly schedule for that date, and pays at its multiplier (per-holiday override, else CONFIG.holiday_multiplier, default 1.5). Open slots show in the schedule with a pickup button; claims, releases, and swaps are atomic Map updates on the record (see above) so the slot count can't be oversubscribed.

Holiday-router + Scheduler flow. When an admin adds a holiday, the slack-bot creates two one-off EventBridge Scheduler schedules for that date — holiday-activate-<YYYYMMDD> at 08:00 ET and holiday-deactivate-<YYYYMMDD> at 17:00 ET — whose names are stored on the record's schedule_names. Scheduler assumes afterhours-shift-manager-holiday-scheduler to invoke afterhours-holiday-router:

• Activate (08:00): capture both IVR ivr_number (800) routes — key-0 and no-input/timeout — into CONFIG.captured_ivr_routes (skipped if they already point at the holiday queue, so re-runs don't clobber the originals), set holiday_queue (802) agents to the holiday's assignees (or [fallback_extension] = [100] when no slots are filled — set once here), repoint both IVR 800 routes to queue 802, and mark activated = True. Idempotent.
• Deactivate (17:00): restore both IVR routes from captured_ivr_routes (only routes still pointing at the queue, defensive against manual changes), clear them, empty queue 802's agents, and mark activated = False. Idempotent.

Queue 801 (the daily ring-scheduler queue) is left untouched. If an admin adds a holiday whose 08:00-17:00 window is already open, the slack-bot inline-activates it immediately (invoking the router) rather than waiting for the 08:00 schedule. Removing a holiday deletes the record and any outstanding schedules.

Late-pickup approval

Picking up a shift behaves differently depending on timing, for both regular and holiday shifts:

• Before the shift starts — immediate pickup (the prior behaviour, unchanged).
• After the shift has started but before it ends (08:00 for a day/holiday shift, 17:00 for a night shift) — the shift is not claimed yet. The bot writes a pending PICKUP_REQUEST and DMs every admin Approve/Deny buttons (mirroring the verified-swap flow). The first admin to approve wins (the claim is conditional, so a second approval is a safe no-op). On approve, the shift is claimed (regular: set_override; holiday: atomic claim_holiday_slot), the requester and channel are notified, and 3CX is fired if the window is live — regular shifts call _update_3cx_routing(picker) when it's today's active shift; holidays refresh queue 802's agents to the current assignees. On deny, the request is cleared and the requester is told.
• After the shift has ended — rejected outright; it's too late to pick up.

A slot claimed after the shift has started always needs an admin to approve it.

Secrets Manager

Secret	Description
afterhours-shift-manager/slack-bot-token	Slack bot OAuth token (xoxb-...)
afterhours-shift-manager/slack-signing-secret	Slack app signing secret
afterhours-shift-manager/3cx-domain	3CX FQDN (e.g. company.3cx.us)
afterhours-shift-manager/3cx-client-id	3CX OAuth2 client ID
afterhours-shift-manager/3cx-client-secret	3CX OAuth2 client secret
afterhours-shift-manager/roster-api-token	Bearer token for PUT/DELETE /roster. Duplicate the same value into the seahaven-prod secret paychex-integrations/afterhours-roster-token.

Roster HTTP API

Identity hire/offboard in paychex-integrations calls this API. It is a separate Lambda on the same HTTP API as Slack (POST /slack/events is unchanged).

Method	Path	Body	Success
PUT	/roster	{"name","extension","slack_user_id"} plus optional "email"	200 {"ok":true}
DELETE	/roster/{extension}	none	204 empty body, including when the row is already gone

Header: Authorization: Bearer {token}. Missing or wrong token is 401. Invalid JSON or fields is 400. A secret-read failure is 503.

email is optional so existing Paychex syncs keep working. Portal identity looks up that email (case-insensitive) against the signed-in Google account. Without it, the portal shows an unlinked roster message. Admin access is still the Slack IDs in admin_users after the email lookup.

Portal HTTP API

The internal portal SPA calls this API with the Cognito ID token from GET /api/auth/meals-token. CORS allows https://internal.seahaven.com, https://internal.dev.seahaven.com, and local Vite.

Method	Path	Who
GET	/api/shifts?week=this|next	Linked employee; unlinked Google accounts get {linked:false}
POST	/api/shifts/pick	Employee
POST	/api/shifts/drop	Employee
POST	/api/shifts/swap	Employee
POST	/api/shifts/swaps/{date}/{shiftType}/accept|decline	Swap target
POST	/api/shifts/admin/override	Admin
POST	/api/shifts/admin/open	Admin
POST	/api/shifts/admin/clear	Admin
POST	/api/shifts/admin/holidays	Admin
DELETE	/api/shifts/admin/holidays/{date}	Admin
POST	/api/shifts/admin/pickups/{date}/{shiftType}/{extension}/approve|deny	Admin

Set portal VITE_SHIFTS_API_BASE to Terraform output api_origin. Set HCP variables portal_cognito_issuer and portal_cognito_audience (and portal_cognito_extra_trust if the portal has separate dev and prod pools).

Set processor AFTERHOURS_BASE_URL to the Terraform output api_origin (HCP variable afterhours_base_url on paychex-integrations-prod). That value is the API origin only. Do not append /roster. Flip it at cutover after DynamoDB is copied, not before.

Daily afterhours-roster-sync still owns the 3CX DEFAULT group at 6am ET: rows absent from that group are deleted. Hire is safe because 3CX create (into DEFAULT) happens before the roster PUT. An HTTP-only row that is not in that group will be removed on the next sync. Sync preserves slack_user_id on existing rows and does not overwrite a just-created API row's Slack id.

Token rotation is a maintenance-window action. The Lambda caches the token per execution environment. Update afterhours-shift-manager/roster-api-token and paychex-integrations/afterhours-roster-token together, then recycle afterhours-roster-api. Updating only one copy, or recycling environments out of order, causes 401s until both sides match.

Documentation

The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's afterhours-shift-manager stack is represented there as a Mermaid subgraph.

• AWS Architecture Map (Confluence, IT space, page 1540098)

Deployment

Infrastructure is applied by HCP Terraform workspace afterhours-shift-manager-prod (VCS on main, working directory terraform/, file trigger terraform/** only). The Flask image is shipped by .github/workflows/deploy-api.yaml.

Monitoring & Alarms

All CloudWatch alarms are defined in terraform/alarms.tf and notify the shared site-alerts SNS topic (→ AWS Chatbot → Slack). None set OKActions — recovery is not paged. Alarm names follow ALB-<Metric>-afterhours-shift-manager and DDB-<Metric>-afterhours-shifts.

ALB alarms:

Alarm	Metric	Condition
ALB-5xx-afterhours-shift-manager	HTTPCode_Target_5XX_Count (Sum)	> 0 over one 5-min period
ALB-Latency-afterhours-shift-manager	TargetResponseTime (p99, s)	>= 3 s, 2 of 3 5-min periods
ALB-UnhealthyHost-afterhours-shift-manager	UnHealthyHostCount (Maximum)	> 0, 3 of 3 1-min periods

DynamoDB alarm (afterhours-shifts table):

Alarm	Metric	Condition	Notes
DDB-ReadThrottle-afterhours-shifts	ReadThrottleEvents (Sum)	>= 1 over one 5-min period	TreatMissingData: notBreaching
DDB-WriteThrottle-afterhours-shifts	WriteThrottleEvents (Sum)	>= 1 over one 5-min period	TreatMissingData: notBreaching

ReadThrottleEvents / WriteThrottleEvents are the table-level throttle signals: AWS/DynamoDB emits them at the TableName dimension, so these alarms transition normally. ThrottledRequests and SystemErrors are intentionally not alarmed: AWS emits them only at TableName+Operation granularity, so a TableName-only alarm would sit permanently in INSUFFICIENT_DATA.

API Gateway alarms were removed with the Fargate cutover (PLAT-216).

Releases & Versioning

Prod is a human GitHub Release:

gh release create vX.Y.Z --target main --generate-notes

That tag applies prod infra in HCP and queues deploy-api.yaml behind the prod Environment. Merge to main deploys dev. Nothing in GitHub Actions creates the Release.

App Home "What's New" still reads CHANGELOG.md. That file is product copy, not the prod tag driver. After editing the root file, run python scripts/sync_changelog.py so src/slack-bot/CHANGELOG.md stays in sync. Pytest fails if the two copies drift.

Testing

Unit tests use pytest with all external boundaries mocked — DynamoDB / SES / Secrets Manager via moto, 3CX HTTP via responses, Slack via fakes, and time via freezegun. No test touches the network or real AWS.

python -m venv .venv && source .venv/bin/activate
pip install -r tests/requirements.txt              # test-only deps
pip install -r src/slack-bot/requirements.txt \
            -r src/weekly-post/requirements.txt \
            -r src/shared/requirements.txt \
            -r src/portal-api/requirements.txt \
            -r requirements-api.txt
pytest

Each Lambda has its own app.py, so the per-package conftest.py loads each one under a unique module name (importlib mode) to avoid collisions. CI runs the same suite on every PR via pytest plus terraform fmt / init -backend=false / validate and npm run openapi:lint.

Local Fargate process (needs the same DynamoDB table and Secrets Manager names the Lambdas use, plus JOBS_QUEUE_URL to consume jobs):

export PYTHONPATH=src:src/shared:src/slack-bot
export STAGE=local GIT_SHA=dev
python -m server.entrypoint

Health is GET /api/health on port 8080.

See SETUP.md for full deployment and Slack app creation instructions.