After-Hours Shift Manager

Slack bot for managing after-hours on-call shifts at Sea Haven Industries. Employees can pick up, drop, and swap shifts directly from Slack. Changes automatically update 3CX queue routing via the integrated ring scheduler.

How It Works

A recurring weekly schedule assigns employees to after-hours phone duty. Weekend shifts are split into Day (8am-5pm) and Night (5pm-8am). Any unassigned shift shows as Available in Slack with a pickup button. When someone picks up or drops a shift for today, the 3CX queue is updated immediately. Future changes take effect when the ring scheduler runs at 8am daily and 5pm on weekends.

The weekly schedule post is updated live when shifts change, and the previous week's post is automatically deleted when the new one goes out.

The bot also has an About page: open the bot in Slack and click its Home tab to see what it does, the full command list, and the latest "What's New" (see Releases & Versioning).

Slack Commands

Command	Description
/oncall	Show this week's schedule
/oncall next	Show next week's schedule
/oncall pick <date>	Pick up an available shift — instant if the shift hasn't started; if it's already underway (but not ended) it needs admin approval (see Late-pickup approval)
/oncall drop <date>	Drop your shift (marks it available) — blocked within 24h of shift start; swap or ask an admin instead
/oncall swap <date> @person	Request a swap — the other person gets an Accept/Decline DM and the shift only moves once they accept
/oncall register <ext>	Link your Slack account to your phone extension
/oncall roster	Show all employees and their link status
/oncall pay	Show last week's bonus pay summary
/oncall rate	Show current shift pay rates
/oncall rate default <amount>	Set the default per-shift rate
/oncall rate <ext> <amount>	Set a per-person shift rate
/oncall help	Show help

Admin Commands

Available to users listed in admin_users in the CONFIG record:

Command	Description
/oncall admin override <date> <ext>	Assign a shift to an extension
/oncall admin open <date>	Mark a shift as open
/oncall admin clear <date>	Remove override (revert to weekly)
/oncall admin roster add <ext> <name>	Add an employee to the roster
/oncall admin roster remove <ext>	Remove an employee
/oncall admin roster rename <ext> <name>	Rename an employee
/oncall admin holiday add <date> <slots> [x<mult>] <label>	Schedule a holiday day shift (8am-5pm ET) with N slots, optional pay multiplier override (e.g. x2), and a label
/oncall admin holiday remove <date>	Remove a scheduled holiday and its activate/deactivate schedules
/oncall admin holiday list	List today-and-future scheduled holidays

Dates accept: today, tomorrow, monday-sunday, 4/5, 2026-04-05

Example: /oncall admin holiday add 2026-07-04 2 x2 Independence Day schedules a 2-slot holiday paying 2x. Omitting the x<mult> token uses the default holiday_multiplier from CONFIG (1.5). See Holidays for the full flow.

Architecture

• Runtime: Python 3.12 on AWS Lambda (arm64), seahaven-prod 011934824531
• Data: DynamoDB single-table (afterhours-shifts)
• IaC: HCP Terraform workspace afterhours-shift-manager-prod (containers) plus GitHub Actions deploy.yaml (zips). src/shared is bundled into each function zip. Terraform does not package src/.
• Slack: Slack Bolt framework with /oncall slash command
• 3CX Integration: Queue routing updated directly via 3CX Queue XAPI
• Secrets: AWS Secrets Manager (afterhours-shift-manager/*)

Lambda Functions

Function	Trigger	Purpose
afterhours-shift-manager	API Gateway (POST /slack/events)	Slack bot — handles /oncall commands and interactive buttons
afterhours-weekly-post	EventBridge (Monday 7am ET)	Posts weekly schedule to Slack, sends pay report email
afterhours-roster-sync	EventBridge (daily 6am ET)	Syncs employee roster from 3CX
afterhours-roster-api	API Gateway (PUT /roster, DELETE /roster/{extension})	Bearer-authenticated roster upsert/delete for the identity processor
afterhours-ring-scheduler	EventBridge (daily 8am ET + weekend 5pm ET)	Updates 3CX queue routing based on who's on shift
afterhours-holiday-router	EventBridge Scheduler (per-holiday one-off: 8am activate / 5pm deactivate ET)	Repoints the IVR to the holiday queue and sets queue agents for a holiday day shift; reverts at 5pm (see Holidays)
afterhours-release-notifier	Skeleton only until tagging exists	Posts a "What's New" announcement to the shift channel

Project Layout

src/
  slack-bot/         Slack Bolt Lambda (handler + app); ships CHANGELOG.md for App Home
  weekly-post/       Monday schedule + pay post
  roster-sync/       Daily 3CX roster sync
  roster-api/        HTTP PUT/DELETE /roster for identity hire/offboard
  ring-scheduler/    3CX queue routing updates
  holiday-router/    3CX IVR/queue repoint for holiday day shifts (activate/deactivate)
  release-notifier/  Posts release announcements to Slack
  shared/            Bundled into each function zip (schedule, blocks, changelog, 3CX client, secrets)
terraform/           HCP Terraform (function skeletons, API, DDB, IAM, schedules)
scripts/             changelog CLI + CI guard + in-package copy sync
tests/               pytest suite (mirrors src/, one dir per Lambda + shared)

DynamoDB Schema

Single table with PK / SK keys:

PK	SK	Description
ROSTER	<extension>	Employee: name, extension, slack_user_id
WEEKLY	<DayName>	Default weekly schedule: extension, name
OVERRIDE	<YYYY-MM-DD>	Date override from pickup/drop (or OPEN)
SWAP	<YYYY-MM-DD>	Pending/verified swap request: requester, target, status, expires_at (TTL)
HOLIDAY	<YYYY-MM-DD>	Holiday day shift (one per date): slots (int), assignees (MAP keyed by extension — {"114": {name, claimed_at}}), multiplier (Decimal, defaults to CONFIG.holiday_multiplier = 1.5, overridable per holiday), label, created_at, created_by, activated (bool), schedule_names (list)
PICKUP_REQUEST	<YYYY-MM-DD>[-DAY]#<ext>	Pending late-pickup awaiting admin approval: requester_ext, requester_name, requester_slack, shift_type, is_holiday, status, created_at, expires_at (TTL = shift end)
SCHEDULE_POST	<channel_id>	Current schedule message timestamp
PAY	<YYYY-MM-DD>	Weekly pay record (Monday date key)
CONFIG	CONFIG	Settings: shift_rate, fallback_extension, admin_users, ring_group, holiday_multiplier (default holiday pay multiplier, 1.5), holiday_queue (3CX queue repointed during holidays, default 802), ivr_number (3CX IVR repointed during holidays, default 800), captured_ivr_routes (original IVR routes saved at holiday activation, restored at deactivation)

Weekend day-shift rows use a -DAY suffix on the SK (e.g. OVERRIDE / 2026-04-05-DAY). The table has TTL enabled on expires_at so abandoned pending swaps and pickup requests self-clean.

Shift priority for any date is HOLIDAY > OVERRIDE > WEEKLY — a holiday record wins over a regular override, which wins over the standing weekly schedule.

Holiday slot claims are atomic Map updates so concurrent pickers can't oversubscribe:

• Claim — SET assignees.#ext guarded by attribute_not_exists(assignees.#ext) AND size(assignees) < :slots.
• Release — REMOVE assignees.#ext guarded by attribute_exists.
• Swap — REMOVE #from SET #to guarded by attribute_exists(#from) AND attribute_not_exists(#to).

Swap flow: /oncall swap writes a pending SWAP record and DMs the target Accept/Decline buttons; it does not reassign the shift. On Accept, the override is written, 3CX is repointed if it's the active shift, and the record is marked verified. On Decline (or once the shift has started) the request is dropped and the shift stays with the original owner.

Holidays

A holiday is a single day-only shift (08:00-17:00 ET), one HOLIDAY record per date, that can hold multiple people (slots). Admins manage holidays with /oncall admin holiday add|remove|list. A holiday takes priority over a regular override and the weekly schedule for that date, and pays at its multiplier (per-holiday override, else CONFIG.holiday_multiplier, default 1.5). Open slots show in the schedule with a pickup button; claims, releases, and swaps are atomic Map updates on the record (see above) so the slot count can't be oversubscribed.

Holiday-router + Scheduler flow. When an admin adds a holiday, the slack-bot creates two one-off EventBridge Scheduler schedules for that date — holiday-activate-<YYYYMMDD> at 08:00 ET and holiday-deactivate-<YYYYMMDD> at 17:00 ET — whose names are stored on the record's schedule_names. Scheduler assumes afterhours-shift-manager-holiday-scheduler to invoke afterhours-holiday-router:

• Activate (08:00): capture both IVR ivr_number (800) routes — key-0 and no-input/timeout — into CONFIG.captured_ivr_routes (skipped if they already point at the holiday queue, so re-runs don't clobber the originals), set holiday_queue (802) agents to the holiday's assignees (or [fallback_extension] = [100] when no slots are filled — set once here), repoint both IVR 800 routes to queue 802, and mark activated = True. Idempotent.
• Deactivate (17:00): restore both IVR routes from captured_ivr_routes (only routes still pointing at the queue, defensive against manual changes), clear them, empty queue 802's agents, and mark activated = False. Idempotent.

Queue 801 (the daily ring-scheduler queue) is left untouched. If an admin adds a holiday whose 08:00-17:00 window is already open, the slack-bot inline-activates it immediately (invoking the router) rather than waiting for the 08:00 schedule. Removing a holiday deletes the record and any outstanding schedules.

Late-pickup approval

Picking up a shift behaves differently depending on timing, for both regular and holiday shifts:

• Before the shift starts — immediate pickup (the prior behaviour, unchanged).
• After the shift has started but before it ends (08:00 for a day/holiday shift, 17:00 for a night shift) — the shift is not claimed yet. The bot writes a pending PICKUP_REQUEST and DMs every admin Approve/Deny buttons (mirroring the verified-swap flow). The first admin to approve wins (the claim is conditional, so a second approval is a safe no-op). On approve, the shift is claimed (regular: set_override; holiday: atomic claim_holiday_slot), the requester and channel are notified, and 3CX is fired if the window is live — regular shifts call _update_3cx_routing(picker) when it's today's active shift; holidays refresh queue 802's agents to the current assignees. On deny, the request is cleared and the requester is told.
• After the shift has ended — rejected outright; it's too late to pick up.

A slot claimed after the shift has started always needs an admin to approve it.

Secrets Manager

Secret	Description
afterhours-shift-manager/slack-bot-token	Slack bot OAuth token (xoxb-...)
afterhours-shift-manager/slack-signing-secret	Slack app signing secret
afterhours-shift-manager/3cx-domain	3CX FQDN (e.g. company.3cx.us)
afterhours-shift-manager/3cx-client-id	3CX OAuth2 client ID
afterhours-shift-manager/3cx-client-secret	3CX OAuth2 client secret
afterhours-shift-manager/roster-api-token	Bearer token for PUT/DELETE /roster. Duplicate the same value into the seahaven-prod secret paychex-integrations/afterhours-roster-token.

Roster HTTP API

Identity hire/offboard in paychex-integrations calls this API. It is a separate Lambda on the same HTTP API as Slack (POST /slack/events is unchanged).

Method	Path	Body	Success
PUT	/roster	{"name","extension","slack_user_id"} (all required strings)	200 {"ok":true}
DELETE	/roster/{extension}	none	204 empty body, including when the row is already gone

Header: Authorization: Bearer {token}. Missing or wrong token is 401. Invalid JSON or fields is 400. A secret-read failure is 503.

Set processor AFTERHOURS_BASE_URL to the Terraform output api_origin (HCP variable afterhours_base_url on paychex-integrations-prod). That value is the API origin only. Do not append /roster. Flip it at cutover after DynamoDB is copied, not before.

Daily afterhours-roster-sync still owns the 3CX DEFAULT group at 6am ET: rows absent from that group are deleted. Hire is safe because 3CX create (into DEFAULT) happens before the roster PUT. An HTTP-only row that is not in that group will be removed on the next sync. Sync preserves slack_user_id on existing rows and does not overwrite a just-created API row's Slack id.

Token rotation is a maintenance-window action. The Lambda caches the token per execution environment. Update afterhours-shift-manager/roster-api-token and paychex-integrations/afterhours-roster-token together, then recycle afterhours-roster-api. Updating only one copy, or recycling environments out of order, causes 401s until both sides match.

Documentation

The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's afterhours-shift-manager stack is represented there as a Mermaid subgraph.

• AWS Architecture Map (Confluence, IT space, page 1540098)

Deployment

Infrastructure is applied by HCP Terraform workspace afterhours-shift-manager-prod (VCS on main, working directory terraform/, file trigger terraform/** only). Function code is shipped by .github/workflows/deploy.yaml on push to main (environment: prod). A terraform-only merge does not run the zip deploy. A mixed app+terraform merge may race the apply; re-run the deploy job if the functions are still stubs.

Manual zip redeploy: Actions → Deploy → Run workflow (workflow_dispatch, always prod). Do not terraform apply locally to prod.

Monitoring & Alarms

All CloudWatch alarms are defined in terraform/alarms.tf and notify the shared site-alerts SNS topic (→ AWS Chatbot → Slack). None set OKActions — recovery is not paged. Alarm names follow Lambda-<Metric>-<fn> (e.g. Lambda-Errors-afterhours-ring-scheduler).

Lambda alarms (all seven functions: afterhours-shift-manager, afterhours-weekly-post, afterhours-roster-sync, afterhours-roster-api, afterhours-ring-scheduler, afterhours-holiday-router, afterhours-release-notifier):

Alarm	Metric	Condition	Notes
Lambda-Errors-<fn>	Errors (Sum)	>= 1 over one 5-min period	TreatMissingData: notBreaching
Lambda-Duration-<fn>	Duration (Maximum, ms)	>= ~80% of timeout, 2 of 3 5-min periods	Thresholds: 24000 ms (30s-timeout fns) / 48000 ms (60s-timeout fns)
Lambda-Throttles-<fn>	Throttles (Sum)	>= 1 over one 5-min period	TreatMissingData: notBreaching

DynamoDB alarm (afterhours-shifts table):

Alarm	Metric	Condition	Notes
DDB-ReadThrottle-afterhours-shifts	ReadThrottleEvents (Sum)	>= 1 over one 5-min period	TreatMissingData: notBreaching
DDB-WriteThrottle-afterhours-shifts	WriteThrottleEvents (Sum)	>= 1 over one 5-min period	TreatMissingData: notBreaching

ReadThrottleEvents / WriteThrottleEvents are the table-level throttle signals: AWS/DynamoDB emits them at the TableName dimension, so these alarms transition normally. ThrottledRequests and SystemErrors are intentionally not alarmed: AWS emits them only at TableName+Operation granularity, so a TableName-only alarm would sit permanently in INSUFFICIENT_DATA.

API Gateway alarms (HTTP API, AWS/ApiGateway v2 metrics, ApiId dimension):

Alarm	Metric	Condition
ApiGateway-4xx-<apiId>	4xx (Sum)	>= 5 over one 5-min period
ApiGateway-5xx-<apiId>	5xx (Sum)	>= 1 over one 5-min period
ApiGateway-Latency-<apiId>	Latency (p99, ms)	>= 3000 ms, 2 of 3 5-min periods

Duration and API latency thresholds are starting points and may be tuned after observing real traffic.

Releases & Versioning

The bot is versioned with SemVer, driven entirely by CHANGELOG.md. The App Home tab reads the copy that ships in the slack-bot zip. Run python scripts/sync_changelog.py after editing the root file. Changelog Guard enforces that the in-package copy matches.

GitHub Releases and git tags are not cut by deploy.yaml. afterhours-release-notifier exists as a function skeleton; CD does not invoke it until tagging exists.

Testing

Unit tests use pytest with all external boundaries mocked — DynamoDB / SES / Secrets Manager via moto, 3CX HTTP via responses, Slack via fakes, and time via freezegun. No test touches the network or real AWS.

python -m venv .venv && source .venv/bin/activate
pip install -r tests/requirements.txt              # test-only deps
pip install -r src/slack-bot/requirements.txt \
            -r src/weekly-post/requirements.txt \
            -r src/shared/requirements.txt          # runtime deps the imports need
pytest

Each Lambda has its own app.py, so the per-package conftest.py loads each one under a unique module name (importlib mode) to avoid collisions. CI runs the same suite on every PR via pytest plus terraform fmt / init -backend=false / validate.

See SETUP.md for full deployment and Slack app creation instructions.