sh-mcp/docs/g16-spike-design.md

G16 Spike — Does an Agentforce Employee-Agent-in-Slack callout forward the invoking user's identity?

Status: Spike design, ready to run (2026-07-07). Unblocked by the 2026-07-07 resolution of the Agentforce org-wide execution wall (Salesforce backend misconfig, fixed). This is the critical-path experiment the /sh-plan-review gate (GPT-4.1) demanded be run before committing to the M2M pivot in docs/agentforce-auth-decision.md. Throwaway kit — not production, not gated by the IAM/auth cross-review.

---

0. Why this settles the whole auth-model question

The auth-model decision (docs/agentforce-auth-decision.md) hinges on one unproven fact (G16): when a Slack user talks to an Agentforce Employee Agent and the agent fires an External Service callout, does the outbound request carry that human's identity, or a single service identity?

• The 0a spike already proved the per-user OAuth chain (Named Credential → Per-User Browser-Flow External Credential → Cognito → endpoint delivers a valid per-user access JWT). But every 0a probe hit was a direct External Service test as one user (logs show the same sub 34f8a4e8-… and the sfdc_stack_depth header) — never an actual agent invocation, because the execution wall threw before any agent callout fired.
• So the only open question is the agent-invocation hop: per-user or service. That is now directly testable. Its answer decides the architecture (see §7):
  • PASS (per-user forwarded) → keep the simpler, already-proven per-user D4; shelve the M2M pivot and its Apex-signer machinery.
  • FAIL (service identity) → execute the M2M + tool-band plan.

Run this before deleting any per-user substrate.

---

1. Hypotheses (pre-registered)

• H-PASS — an agent callout invoked by user U arrives at the probe with sub == U's own consented identity; ≥2 distinct users → ≥2 distinct correct subs.
• H-FAIL — all users' agent callouts arrive with the same principal (a service/Named identity), i.e. user identity is lost inside the agent orchestration.
• H-HEADER — identity is not in the bearer token but is in a Salesforce-added request header (a per-user id / signed assertion). Would open a third architecture option (service transport + a trustworthy SF user header, simpler than the custom Apex-signer).
• H-PARTIAL — per-user, but sub is a Salesforce/Cognito-internal id, not the Google Workspace sub (the CR-2 concern) — per-user still viable, add a sub↔Workspace validation step.

2. Decision matrix (decide the action BEFORE running, to avoid post-hoc rationalizing)

Observed at the probe across ≥2 distinct invoking users	Verdict	Architecture decision
Distinct, correct per-user sub each time	G16 PASS	Keep per-user D4 (proven + simpler). Shelve agentforce-auth-decision.md; finance audit falls out for free (real sub).
Same sub / same principal for every user	G16 FAIL	Execute the M2M + tool-band plan; the Apex-signer becomes the finance-audit mechanism.
Per-user identity present, but in a header not the token	PASS-via-header	New option: service transport + verified SF user header. Re-evaluate both docs.
Per-user sub, but not the Google Workspace sub	PARTIAL	Per-user viable; add sub↔Workspace mapping/validation (CR-2) before build.
Principal reverts to "Configured" / 401 after ~1h	(orthogonal)	Not a G16 verdict — token-survival bug; fixed by offline_access (controlled pre-spike, §4).

3. The kit (reuse the live 0a kit — verified present 2026-07-07)

Piece	Value
Cognito pool	us-east-1_GsDbGe0pa (sh-mcp-0a-spike, ESSENTIALS), IdP = Cognito-native
Hosted-UI domain / token endpoint	sh-mcp-0a-spike-19568 → https://sh-mcp-0a-spike-19568.auth.us-east-1.amazoncognito.com
App client	7kavkhm97etk00bfk7sjua2rfq — flow code; scopes openid email sh-mcp-ops/ops:read sh-mcp-ops/ops:tasks; no offline_access (fix in §4)
Probe API	ex1v1x1hqc → https://ex1v1x1hqc.execute-api.us-east-1.amazonaws.com/probe
Probe Lambda	sh-mcp-0a-spike-probe (py3.12) — already logs PROBE_RESULT {claims_of_interest, all_claims, seen_header_keys}
Salesforce	Prod speed-speed-957 (Adam's choice 2026-07-07 — no data to lose there, and it's the real Enterprise Grid, so the result is definitive; see venue note below). Objects sh_mcp_0a: Auth Provider / External Credential (Per-User) / Named Credential; External Service shMcp0aProbe → operation identityProbe. The prod callback URL is already on the app client.

Venue = prod speed-speed-957 (definitive). The Gemini-Q caveat (plan §0.3) was that a free Slack workspace may forward identity differently than production Enterprise Grid + real licenses, so a sandbox result would need a Grid re-confirm. Running in the prod org on the real Grid removes that caveat — a prod PASS or FAIL is definitive. Requirement to keep it clean: wire the agent action to the External Service operation identityProbe directly (or via Flow), NOT an Apex @InvocableMethod, so the "no Apex authoring on an active prod org" gotcha never applies. If an Apex wrapper is unavoidable, deploy it via metadata/change set (don't author in the prod Setup UI).

4. Pre-spike changes — AWS side DONE 2026-07-07

1. Token survival past 1h — resolved at the AWS layer (no offline_access needed). Cognito rejects offline_access as a scope (ScopeDoesNotExistException); it issues a refresh token automatically for the authorization-code grant. The app client already has RefreshTokenValidity=30d, ALLOW_REFRESH_TOKEN_AUTH, and token revocation on — so the refresh token IS returned to Salesforce. The 0a "reverts after 1h" symptom is therefore Salesforce-side (the Per-User External Credential must store + use that refresh token), not an AWS gap. Verified by test E3; if it still reverts, the fix is in the SF External Credential config, not here.
2. Probe enhanced + deployed. sh-mcp-0a-spike-probe now logs full header names+values (seen_headers, with the bearer redacted to length only) and echoes an X-Probe-Tag header — so a Salesforce-injected identity header (H-HEADER) can't hide and every invocation is correlatable to a user. Smoke-tested green (custom headers captured, bearer shown as Bearer <redacted len=N>).
3. Two labelled test users ready. Both Cognito-native users have known permanent passwords, and because each username == its sub, the mapping is self-checking (sign in as X → probe must show sub X). Credentials handed to Adam separately (§5 / the summary).
4. (Pass-2, optional, deferred) Re-federate the pool to Google to confirm the arriving sub is the real Workspace sub (settles H-PARTIAL / CR-2). Not needed for the primary PASS/FAIL — two distinct Cognito-native users already discriminate per-user vs service.

4b. Salesforce build from scratch (prod speed-speed-957)

Assumes no sh_mcp_0a objects exist yet. Values to paste:

Field	Value
Consumer Key (Cognito app client)	2oqnqep996kvrh9u12jr774til (confidential client sh-mcp-0a-spike-sf)
Consumer Secret	t2vhiu7nsjvrecm65v12hhmfqiblac03rrv2rbggb680ktuqvu9
Authorize URL	https://sh-mcp-0a-spike-19568.auth.us-east-1.amazoncognito.com/oauth2/authorize
Token URL	https://sh-mcp-0a-spike-19568.auth.us-east-1.amazoncognito.com/oauth2/token
User Info URL	https://sh-mcp-0a-spike-19568.auth.us-east-1.amazoncognito.com/oauth2/userInfo
Token Issuer (for validation)	https://cognito-idp.us-east-1.amazonaws.com/us-east-1_GsDbGe0pa
Scopes	openid email sh-mcp-ops/ops:read sh-mcp-ops/ops:tasks
MCP base URL (Named Credential)	https://ex1v1x1hqc.execute-api.us-east-1.amazonaws.com
OpenAPI schema for the External Service	docs/g16-probe.openapi.json (path /probe, operation identityProbe)

Cognito wrinkle: the OAuth endpoints live on the hosted-UI domain (…-19568.auth…), but the token issuer is the cognito-idp…/<poolId> URL. Enter the explicit authorize/token/userinfo URLs above (don't rely on OIDC discovery, which points at the issuer host, not the domain host).

A. Auth. Provider — Setup → Auth. Providers → New. Type OpenID Connect; Name/URL-Suffix sh_mcp_0a; Consumer Key/Secret + Authorize/Token/UserInfo URLs + Default Scopes from the table; Token Issuer as above. Save, then copy the generated Callback URL — it should be https://speed-speed-957.my.salesforce.com/services/authcallback/sh_mcp_0a (already registered on the Cognito client). If it differs, send it to me and I'll add it to Cognito.

B. External Credential — Setup → Named Credentials → External Credentials → New. Label/Name sh_mcp_0a; Authentication Protocol OAuth 2.0; Authentication Flow Type Browser Flow (reference the sh_mcp_0a Auth Provider); add the scopes. Add a Principal with Identity Type = Per User.

C. Named Credential — Setup → Named Credentials → Named Credentials → New. Label/Name sh_mcp_0a; URL https://ex1v1x1hqc.execute-api.us-east-1.amazonaws.com; External Credential sh_mcp_0a; enable Generate Authorization Header.

D. Permission Set — create/choose a PS → External Credential Principal Access → add sh_mcp_0a — Per User. Assign it to both test users. (Without this, users can't use the credential.)

E. External Service — Setup → External Services → New → From API Specification. Named Credential sh_mcp_0a; paste docs/g16-probe.openapi.json. Confirm it registers operation identityProbe. This creates the invocable action for Agentforce/Flow.

F. Agent — Agent Builder → open/create an Employee Agent (Slack). Add a Topic + Action that calls the identityProbe External Service operation (keep the default "Agent Router" subagent). Activate and connect to Slack.

G. Per-user consent — each user: personal Settings → Authentication Settings for External Systems (or the External Credential principal → Authenticate) → at the Cognito hosted-UI login, sign in as User A (user 1) / User B (user 2). Principal must read Authenticated.

5. Procedure (ordered; each step has a check)

Setup

• S1. Apply §4.1–§4.2. Check: curl -H "Authorization: Bearer <hand-minted token>" the /probe endpoint → PROBE_RESULT shows the decoded claims + full headers + your X-Probe-Tag.
• S2. Ensure two distinct test identities, each able to (a) log into the pool as a distinct Cognito user (distinct sub) and (b) invoke the Slack agent as a distinct Slack/Salesforce user: create Cognito user B (user A = the existing 34f8a4e8-…); a 2nd sandbox Salesforce user mapped to a 2nd Slack user. Check: each can independently complete the per-user Browser-Flow consent for the sh_mcp_0a External Credential (principal shows Authenticated).
• S3. In the sandbox, wire shMcp0aProbe.identityProbe as an Agentforce action on a deployed Employee Agent in Slack; keep the default "Agent Router" subagent (deleting it breaks compile — memory gotcha). Check: the agent is live in the Slack workspace and lists the probe action.

Experiment

• E1 (User A). In Slack, prompt the agent so it invokes the probe action (e.g. "run the identity probe"). Capture: the PROBE_RESULT log line + the agent's returned body.
• E2 (User B). Same prompt, as User B. Capture: same.
• E3 (token survival). Re-run E1 >1h after A's consent → confirm the callout still succeeds (validates offline_access).
• E4 (autonomous, if feasible). Trigger the same action from a non-interactive context (scheduled/automated) → record whether the arriving principal is distinguishable from a human (review QUESTION-6: a service context must not masquerade as a human).

Read-out — pull the results in one shot:

aws logs tail /aws/lambda/sh-mcp-0a-spike-probe --region us-east-1 --since 1h --format short \
  | grep PROBE_RESULT

Tabulate per invocation: invoking user, sub, client_id, token_use, scope, any user-identifying header, principal-still-Authenticated-after-1h. Compare against §2.

6. What we're measuring (the observables the probe already emits)

sub (the identity signal — access token has no aud, no email, memory-confirmed), client_id, token_use, scope, iss, iat/exp; all request headers (post-§4.2) to catch header-borne identity; whether the per-user principal survived >1h.

7a. Run outcome (2026-07-09) — programmatic Agent-API attempt, INCONCLUSIVE (leans FAIL)

Drove this entirely via SF CLI + AWS CLI (no browser, no 2nd human), using the Agent API instead of Slack:

• Provisioning works (/agentforce/bootstrap/nameduser → sfap_api JWT → einstein/ai-agent/v1). Built an activated agent (sh_mcp_g16c, id 0XxhR0000000UnhSAE) purely via Metadata API because CLI agent publish/agent create and aiAuthoringBundle deploy are all broken on this trial org.
• Wall: the planner returns result: [] — narrates running the probe but never executes the Apex (Task-DML sink + unauth beacon + per-user callout all confirm zero execution). The fix — GenAiFunction input/output/schema.json — server-errors on deploy (ErrorId 44568769-691223). Same degraded-runtime signature as the broken publish/create APIs.
• Direct callout still carries the real per-user sub (baseline good), and found a live token-refresh bug: the Per-User External Credential served a 44h-expired token (test E3, live).
• Verdict: LEANS FAIL (service identity) on architecture (agent mandates default_agent_user; human rides as context vars only), but the decisive UserInfo/Task.CreatedById observable was never captured because the action wouldn't fire. Committed M2M plan already assumes FAIL. Definitive confirm still needs the Slack path (§4b F–G, 2 real users) OR a SF-support fix for GenAiFunction schema deploy on this org. Instrumentation is deployed and ready (IdentityProbeAction logs PROBE_RUNAS, inserts G16RUNAS Task, beacons UserInfo).

7. Outcome → next action (closes the loop with the plan)

• PASS → update project_sh_mcp memory + mark agentforce-auth-decision.md superseded/shelved; proceed on per-user D4; the M2M pivot is unnecessary.
• FAIL → that doc becomes the committed plan; run its Phase A–F (which the /sh-plan-review FIXes still apply to: provisioning order, rollback branch, cross-review gating, docs).
• PASS-via-header / PARTIAL → record the third option / CR-2 follow-up; re-decide.
• Either way: write the result + the exact probe evidence into project_sh_mcp memory the same day.

8. Time-box, ownership, teardown

• Time-box: 1–2 days; most cost is the Salesforce agent + 2nd-user setup, not AWS.
• I can do: the §4 AWS-side changes (add offline_access, enhance the probe) and the read-out. Adam/hands-on: the Salesforce sandbox agent wiring, the two real Slack test users, the consents, and the Enterprise-Grid re-confirm on a PASS.
• Teardown: re-authored as docs/g16-spike-teardown.sh (bash docs/g16-spike-teardown.sh --yes) — tears down the Cognito pool/domain/users, the HTTP API, and the probe Lambda/role/log group; the Salesforce sh_mcp_0a objects are a manual delete in both orgs. Keep the kit live until the verdict is recorded in project_sh_mcp memory.