sh-mcp/docs/g16-spike-design.md

# G16 Spike — Does an Agentforce Employee-Agent-in-Slack callout forward the *invoking user's* identity?

**Status:** Spike design, ready to run (2026-07-07). Unblocked by the 2026-07-07 resolution of the
Agentforce org-wide execution wall (Salesforce backend misconfig, fixed). This is the critical-path
experiment the `/sh-plan-review` gate (GPT-4.1) demanded be run **before** committing to the M2M pivot
in `docs/agentforce-auth-decision.md`. Throwaway kit — not production, not gated by the IAM/auth
cross-review.

---

## 0. Why this settles the whole auth-model question

The auth-model decision (`docs/agentforce-auth-decision.md`) hinges on **one unproven fact (G16)**:
when a Slack user talks to an Agentforce **Employee Agent** and the agent fires an External Service
callout, does the outbound request carry **that human's** identity, or a single **service** identity?

- The 0a spike already **proved the per-user OAuth chain** (Named Credential → Per-User Browser-Flow
  External Credential → Cognito → endpoint delivers a valid per-user access JWT). **But every 0a probe
  hit was a *direct* External Service test as *one* user** (logs show the same `sub`
  `34f8a4e8-…` and the `sfdc_stack_depth` header) — **never an actual agent invocation**, because the
  execution wall threw before any agent callout fired.
- So the *only* open question is the **agent-invocation hop**: per-user or service. That is now
  directly testable. Its answer decides the architecture (see §7):
  - **PASS (per-user forwarded)** → keep the simpler, already-proven **per-user D4**; shelve the M2M
    pivot and its Apex-signer machinery.
  - **FAIL (service identity)** → execute the **M2M + tool-band** plan.

Run this before deleting any per-user substrate.

---

## 1. Hypotheses (pre-registered)

- **H-PASS** — an agent callout invoked by user *U* arrives at the probe with `sub` == *U*'s own
  consented identity; **≥2 distinct users → ≥2 distinct correct subs**.
- **H-FAIL** — all users' agent callouts arrive with the **same** principal (a service/Named identity),
  i.e. user identity is lost inside the agent orchestration.
- **H-HEADER** — identity is **not** in the bearer token but **is** in a Salesforce-added request header
  (a per-user id / signed assertion). Would open a *third* architecture option (service transport + a
  trustworthy SF user header, simpler than the custom Apex-signer).
- **H-PARTIAL** — per-user, but `sub` is a Salesforce/Cognito-internal id, **not the Google Workspace
  `sub`** (the CR-2 concern) — per-user still viable, add a `sub`↔Workspace validation step.

## 2. Decision matrix (decide the action BEFORE running, to avoid post-hoc rationalizing)

| Observed at the probe across ≥2 distinct invoking users | Verdict | Architecture decision |
|---|---|---|
| Distinct, correct per-user `sub` each time | **G16 PASS** | Keep **per-user D4** (proven + simpler). Shelve `agentforce-auth-decision.md`; finance audit falls out for free (real `sub`). |
| Same `sub` / same principal for every user | **G16 FAIL** | Execute the **M2M + tool-band** plan; the Apex-signer becomes the finance-audit mechanism. |
| Per-user identity present, but in a **header** not the token | **PASS-via-header** | New option: service transport + verified SF user header. Re-evaluate both docs. |
| Per-user `sub`, but not the Google Workspace `sub` | **PARTIAL** | Per-user viable; add `sub`↔Workspace mapping/validation (CR-2) before build. |
| Principal reverts to "Configured" / 401 after ~1h | (orthogonal) | Not a G16 verdict — token-survival bug; fixed by `offline_access` (controlled pre-spike, §4). |

## 3. The kit (reuse the live 0a kit — verified present 2026-07-07)

| Piece | Value |
|---|---|
| Cognito pool | `us-east-1_GsDbGe0pa` (`sh-mcp-0a-spike`, ESSENTIALS), IdP = **Cognito-native** |
| Hosted-UI domain / token endpoint | `sh-mcp-0a-spike-19568` → `https://sh-mcp-0a-spike-19568.auth.us-east-1.amazoncognito.com` |
| App client | `7kavkhm97etk00bfk7sjua2rfq` — flow `code`; scopes `openid email sh-mcp-ops/ops:read sh-mcp-ops/ops:tasks`; **no `offline_access`** (fix in §4) |
| Probe API | `ex1v1x1hqc` → `https://ex1v1x1hqc.execute-api.us-east-1.amazonaws.com/probe` |
| Probe Lambda | `sh-mcp-0a-spike-probe` (py3.12) — already logs `PROBE_RESULT {claims_of_interest, all_claims, seen_header_keys}` |
| Salesforce | **Prod `speed-speed-957`** (Adam's choice 2026-07-07 — no data to lose there, and it's the real Enterprise Grid, so the result is definitive; see venue note below). Objects `sh_mcp_0a`: Auth Provider / External Credential (Per-User) / Named Credential; External Service `shMcp0aProbe` → operation `identityProbe`. The prod callback URL is already on the app client. |

**Venue = prod `speed-speed-957` (definitive).** The Gemini-Q caveat (plan §0.3) was that a *free* Slack
workspace may forward identity differently than production Enterprise Grid + real licenses, so a sandbox
result would need a Grid re-confirm. Running in the **prod org on the real Grid removes that caveat** — a
prod PASS *or* FAIL is definitive. Requirement to keep it clean: wire the agent action to the **External
Service operation `identityProbe` directly (or via Flow), NOT an Apex `@InvocableMethod`**, so the "no Apex
authoring on an active prod org" gotcha never applies. If an Apex wrapper is unavoidable, deploy it via
metadata/change set (don't author in the prod Setup UI).

## 4. Pre-spike changes — AWS side **DONE 2026-07-07**

1. **Token survival past 1h — resolved at the AWS layer (no `offline_access` needed).** Cognito
   **rejects `offline_access`** as a scope (`ScopeDoesNotExistException`); it issues a refresh token
   automatically for the authorization-code grant. The app client already has `RefreshTokenValidity=30d`,
   `ALLOW_REFRESH_TOKEN_AUTH`, and token revocation on — so the refresh token IS returned to Salesforce.
   The 0a "reverts after 1h" symptom is therefore **Salesforce-side** (the Per-User External Credential
   must store + use that refresh token), **not** an AWS gap. Verified by test **E3**; if it still reverts,
   the fix is in the SF External Credential config, not here.
2. **Probe enhanced + deployed.** `sh-mcp-0a-spike-probe` now logs **full header names+values**
   (`seen_headers`, with the bearer redacted to length only) and echoes an **`X-Probe-Tag`** header — so
   a Salesforce-injected identity header (H-HEADER) can't hide and every invocation is correlatable to a
   user. Smoke-tested green (custom headers captured, bearer shown as `Bearer <redacted len=N>`).
3. **Two labelled test users ready.** Both Cognito-native users have **known permanent passwords**, and
   because each **username == its `sub`**, the mapping is self-checking (sign in as X → probe must show
   `sub` X). Credentials handed to Adam separately (§5 / the summary).
4. **(Pass-2, optional, deferred) Re-federate the pool to Google** to confirm the arriving `sub` is the
   real Workspace `sub` (settles H-PARTIAL / CR-2). Not needed for the primary PASS/FAIL — two distinct
   Cognito-native users already discriminate per-user vs service.

## 4b. Salesforce build from scratch (prod `speed-speed-957`)

Assumes **no** `sh_mcp_0a` objects exist yet. Values to paste:

| Field | Value |
|---|---|
| Consumer Key (Cognito app client) | `2oqnqep996kvrh9u12jr774til` (confidential client `sh-mcp-0a-spike-sf`) |
| Consumer Secret | `t2vhiu7nsjvrecm65v12hhmfqiblac03rrv2rbggb680ktuqvu9` |
| Authorize URL | `https://sh-mcp-0a-spike-19568.auth.us-east-1.amazoncognito.com/oauth2/authorize` |
| Token URL | `https://sh-mcp-0a-spike-19568.auth.us-east-1.amazoncognito.com/oauth2/token` |
| User Info URL | `https://sh-mcp-0a-spike-19568.auth.us-east-1.amazoncognito.com/oauth2/userInfo` |
| Token Issuer (for validation) | `https://cognito-idp.us-east-1.amazonaws.com/us-east-1_GsDbGe0pa` |
| Scopes | `openid email sh-mcp-ops/ops:read sh-mcp-ops/ops:tasks` |
| MCP base URL (Named Credential) | `https://ex1v1x1hqc.execute-api.us-east-1.amazonaws.com` |
| OpenAPI schema for the External Service | `docs/g16-probe.openapi.json` (path `/probe`, operation `identityProbe`) |

> **Cognito wrinkle:** the OAuth endpoints live on the hosted-UI **domain** (`…-19568.auth…`), but the
> token **issuer** is the `cognito-idp…/<poolId>` URL. Enter the explicit authorize/token/userinfo URLs
> above (don't rely on OIDC discovery, which points at the issuer host, not the domain host).

**A. Auth. Provider** — Setup → Auth. Providers → New. Type **OpenID Connect**; Name/URL-Suffix `sh_mcp_0a`;
Consumer Key/Secret + Authorize/Token/UserInfo URLs + Default Scopes from the table; Token Issuer as above.
Save, then **copy the generated Callback URL** — it should be
`https://speed-speed-957.my.salesforce.com/services/authcallback/sh_mcp_0a` (already registered on the
Cognito client). **If it differs, send it to me and I'll add it to Cognito.**

**B. External Credential** — Setup → Named Credentials → **External Credentials** → New. Label/Name
`sh_mcp_0a`; Authentication Protocol **OAuth 2.0**; Authentication Flow Type **Browser Flow** (reference the
`sh_mcp_0a` Auth Provider); add the scopes. Add a **Principal** with **Identity Type = Per User**.

**C. Named Credential** — Setup → Named Credentials → **Named Credentials** → New. Label/Name `sh_mcp_0a`;
URL `https://ex1v1x1hqc.execute-api.us-east-1.amazonaws.com`; External Credential `sh_mcp_0a`; enable
**Generate Authorization Header**.

**D. Permission Set** — create/choose a PS → **External Credential Principal Access** → add
`sh_mcp_0a — Per User`. Assign it to both test users. (Without this, users can't use the credential.)

**E. External Service** — Setup → **External Services** → New → From API Specification. Named Credential
`sh_mcp_0a`; paste `docs/g16-probe.openapi.json`. Confirm it registers operation **`identityProbe`**. This
creates the invocable action for Agentforce/Flow.

**F. Agent** — Agent Builder → open/create an **Employee Agent** (Slack). Add a **Topic + Action** that
calls the `identityProbe` External Service operation (keep the default **"Agent Router"** subagent).
Activate and **connect to Slack**.

**G. Per-user consent** — each user: personal **Settings → Authentication Settings for External Systems**
(or the External Credential principal → **Authenticate**) → at the Cognito hosted-UI login, sign in as
**User A** (user 1) / **User B** (user 2). Principal must read **Authenticated**.

## 5. Procedure (ordered; each step has a check)

**Setup**
- **S1.** Apply §4.1–§4.2. *Check:* `curl -H "Authorization: Bearer <hand-minted token>"` the `/probe`
  endpoint → `PROBE_RESULT` shows the decoded claims + full headers + your `X-Probe-Tag`.
- **S2.** Ensure **two distinct test identities**, each able to (a) log into the pool as a distinct
  Cognito user (distinct `sub`) and (b) invoke the Slack agent as a distinct Slack/Salesforce user:
  create Cognito user **B** (user **A** = the existing `34f8a4e8-…`); a 2nd sandbox Salesforce user
  mapped to a 2nd Slack user. *Check:* each can independently complete the per-user Browser-Flow consent
  for the `sh_mcp_0a` External Credential (principal shows **Authenticated**).
- **S3.** In the sandbox, wire `shMcp0aProbe.identityProbe` as an **Agentforce action** on a deployed
  **Employee Agent** in Slack; **keep the default "Agent Router" subagent** (deleting it breaks compile —
  memory gotcha). *Check:* the agent is live in the Slack workspace and lists the probe action.

**Experiment**
- **E1 (User A).** In Slack, prompt the agent so it invokes the probe action (e.g. "run the identity
  probe"). *Capture:* the `PROBE_RESULT` log line + the agent's returned body.
- **E2 (User B).** Same prompt, as User B. *Capture:* same.
- **E3 (token survival).** Re-run E1 **>1h** after A's consent → confirm the callout still succeeds
  (validates `offline_access`).
- **E4 (autonomous, if feasible).** Trigger the same action from a **non-interactive** context
  (scheduled/automated) → record whether the arriving principal is **distinguishable** from a human
  (review QUESTION-6: a service context must not masquerade as a human).

**Read-out** — pull the results in one shot:
```
aws logs tail /aws/lambda/sh-mcp-0a-spike-probe --region us-east-1 --since 1h --format short \
  | grep PROBE_RESULT
```
Tabulate per invocation: invoking user, `sub`, `client_id`, `token_use`, `scope`, any user-identifying
header, principal-still-Authenticated-after-1h. Compare against §2.

## 6. What we're measuring (the observables the probe already emits)
`sub` (the identity signal — access token has **no `aud`, no `email`**, memory-confirmed), `client_id`,
`token_use`, `scope`, `iss`, `iat`/`exp`; **all request headers** (post-§4.2) to catch header-borne
identity; whether the per-user principal survived >1h.

## 7a. Run outcome (2026-07-09) — programmatic Agent-API attempt, INCONCLUSIVE (leans FAIL)
Drove this entirely via SF CLI + AWS CLI (no browser, no 2nd human), using the Agent API instead of Slack:
- **Provisioning works** (`/agentforce/bootstrap/nameduser` → `sfap_api` JWT → `einstein/ai-agent/v1`). Built an
  activated agent (`sh_mcp_g16c`, id `0XxhR0000000UnhSAE`) **purely via Metadata API** because CLI
  `agent publish`/`agent create` and `aiAuthoringBundle` deploy are all broken on this trial org.
- **Wall:** the planner returns `result: []` — narrates running the probe but never executes the Apex
  (Task-DML sink + unauth beacon + per-user callout all confirm zero execution). The fix — `GenAiFunction`
  `input/output/schema.json` — **server-errors on deploy** (`ErrorId 44568769-691223`). Same degraded-runtime
  signature as the broken publish/create APIs.
- **Direct callout still carries the real per-user `sub`** (baseline good), and found a **live token-refresh
  bug**: the Per-User External Credential served a **44h-expired** token (test E3, live).
- **Verdict: LEANS FAIL (service identity)** on architecture (agent mandates `default_agent_user`; human rides
  as context vars only), but the decisive `UserInfo`/`Task.CreatedById` observable was never captured because
  the action wouldn't fire. Committed M2M plan already assumes FAIL. **Definitive confirm still needs the Slack
  path (§4b F–G, 2 real users) OR a SF-support fix for GenAiFunction schema deploy on this org.** Instrumentation
  is deployed and ready (`IdentityProbeAction` logs `PROBE_RUNAS`, inserts `G16RUNAS` Task, beacons UserInfo).

## 7. Outcome → next action (closes the loop with the plan)
- **PASS** → update `project_sh_mcp` memory + mark `agentforce-auth-decision.md` **superseded/shelved**;
  proceed on **per-user D4**; the M2M pivot is unnecessary.
- **FAIL** → **that doc becomes the committed plan**; run its Phase A–F (which the `/sh-plan-review`
  FIXes still apply to: provisioning order, rollback branch, cross-review gating, docs).
- **PASS-via-header / PARTIAL** → record the third option / CR-2 follow-up; re-decide.
- Either way: write the result + the exact probe evidence into `project_sh_mcp` memory the same day.

## 8. Time-box, ownership, teardown
- **Time-box:** 1–2 days; most cost is the Salesforce agent + 2nd-user setup, not AWS.
- **I can do:** the §4 AWS-side changes (add `offline_access`, enhance the probe) and the read-out.
  **Adam/hands-on:** the Salesforce sandbox agent wiring, the two real Slack test users, the consents,
  and the Enterprise-Grid re-confirm on a PASS.
- **Teardown:** re-authored as **`docs/g16-spike-teardown.sh`** (`bash docs/g16-spike-teardown.sh --yes`)
  — tears down the Cognito pool/domain/users, the HTTP API, and the probe Lambda/role/log group; the
  Salesforce `sh_mcp_0a` objects are a manual delete in both orgs. Keep the kit live until the verdict is
  recorded in `project_sh_mcp` memory.