orchestrator/agent-team/agent_team/transport/checker_intake.py

"""Plane-1 checker-finding INTAKE: a confirmed finding becomes a pipeline task.

This is the Plane-2 **P5 cross-plane loop**: it closes the gap between the
Plane-1 read-only *checkers*
(``security-review/checkers/compliance-drift.sh`` and
``security-review/checkers/dependency-cve.sh``) and the Plane-2 human-gated
SDLC *pipeline*. Where :mod:`agent_team.transport.github_intake` turns a
labeled GitHub issue into one pipeline task, this leaf turns a confirmed,
at-or-above-threshold checker *finding* into one pipeline remediation task by
calling the same committed coordinator intake entry,
:meth:`agent_team.coordinator.Coordinator.start_task`
(``task_text=<rendered finding>``, ``transport_name=<configured>``).

It deliberately mirrors the ``github_intake`` seam so the two front doors stay
consistent and equally testable:

* Input is **plain data**, not network. The poller reads checker *report* JSON
  (the exact shape the bash checkers already emit — a top-level object with a
  ``findings`` array) from one or more files / a directory. No SDK, no socket,
  no token: a checker run already wrote the report; this only reads it.
* ``coordinator`` is anything exposing ``start_task(task_text=...,
  transport_name=...)``: the live :class:`~agent_team.coordinator.Coordinator`
  in production, a stub in tests. No model or transport is touched here.

Selection contract:
    A finding is ingested only when it is both ``status == "confirmed"`` AND its
    ``severity`` is at or above the configured threshold (default ``high``).
    Unconfirmed / suppressed findings and below-threshold severities are
    skipped. An ``unverified`` severity (the schema's auto-downgrade marker) is
    treated as below every real threshold and never ingested.

De-duplication (P5 scope note — matches the github_intake discipline):
    The poller tracks already-ingested findings in an **in-memory** set keyed by
    a stable content identity (the finding ``id`` when present, else a
    content-hash of checker+repo+title+severity). So re-reading the same nightly
    report — or two reports that both carry the same finding — does not start a
    second task within one process. This is deliberately simple and mirrors
    ``github_intake``: it does NOT survive a process restart. Durable de-dup (a
    ledger table of ingested finding ids, mirroring the ``pending_questions``
    discipline) is the known FOLLOW-UP and is intentionally not shipped here.
    After a restart an already-ingested finding still present in a fresh report
    would be re-ingested; treat the in-memory set as a best-effort guard, not a
    durable contract.

Untrusted-input hygiene:
    Checker findings carry **repo-controlled strings** (titles, proofs) — a
    repo name, a dependency advisory summary, a PR title. Those must never reach
    operator logs or the rendered task text raw, or a forged multi-line value
    could spoof log lines / pipeline-task framing (log injection). Every such
    string is sanitized via :func:`_sanitize` (newline/control-char neutralised,
    length-bounded) before it is logged or rendered, mirroring the coordinator's
    ``start_task`` log-injection hardening.

Design constraints (pre-deployment scaffolding):
    * **No live infrastructure.** Nothing is provisioned or called at import.
      The poller only reads local JSON and calls the injected coordinator.
    * **P2 stays the production default; this is OPT-IN and INERT.** It is wired
      ONLY behind the ``intake-checker`` run-team subcommand, never into the
      always-on ``serve`` path. It does no CI, OIDC, git/patch apply, or
      network; it only reads a report file and calls the existing intake entry.
"""

from __future__ import annotations

import hashlib
import json
import logging
from pathlib import Path
from typing import Any

__all__ = [
    "CHECKER_TRANSPORT_NAME",
    "DEFAULT_SEVERITY_THRESHOLD",
    "SEVERITY_RANK",
    "CheckerFindingIntake",
    "finding_identity",
    "finding_task_text",
    "load_report_findings",
    "select_findings",
]

_LOG = logging.getLogger(__name__)

# Transport name handed to the coordinator's intake entry so the resulting
# remediation task's clarifier question-sets route over a real channel. Findings
# are about org repos, so GitHub is the natural default (mirrors github_intake).
CHECKER_TRANSPORT_NAME = "github"

# Severity ordering, matching the finding.schema.json enum. Higher rank = more
# severe. ``info`` / ``unverified`` sit below every real remediation threshold.
SEVERITY_RANK: dict[str, int] = {
    "unverified": -1,
    "info": 0,
    "low": 1,
    "medium": 2,
    "high": 3,
    "critical": 4,
}

# Default minimum severity a confirmed finding must reach to spawn a task.
DEFAULT_SEVERITY_THRESHOLD = "high"

# Hard cap on any single rendered/logged untrusted string. Generous enough for a
# real title/proof, tight enough that a forged megastring cannot flood the log or
# the task text.
_MAX_FIELD_LEN = 500


def _sanitize(value: Any, *, max_len: int = _MAX_FIELD_LEN) -> str:
    """Neutralise an untrusted, repo-controlled string for logs / task text.

    Coerces ``value`` to ``str``, strips surrounding whitespace, replaces every
    control character (newlines, carriage returns, tabs, and other C0/C1
    controls) with a visible escape so a forged multi-line value cannot spoof a
    log line or the framing of the rendered task text, and bounds the length so a
    megastring cannot flood the sink. Mirrors the coordinator's ``start_task``
    log-injection hardening, generalised to every untrusted field.
    """
    text = str(value if value is not None else "").strip()
    if len(text) > max_len:
        text = text[:max_len] + "…(truncated)"
    out: list[str] = []
    for ch in text:
        if ch == "\n":
            out.append("\\n")
        elif ch == "\r":
            out.append("\\r")
        elif ch == "\t":
            out.append("\\t")
        elif ord(ch) < 0x20 or ord(ch) == 0x7F:
            # Any other C0 control (and DEL) -> visible escape.
            out.append(f"\\x{ord(ch):02x}")
        else:
            out.append(ch)
    return "".join(out)


def _proof_hint(proof: Any) -> str:
    """Render the remediation hint from a finding's ``proof`` object, sanitized.

    Both checkers nest the actionable detail under ``proof``:

    * compliance-drift: ``{"outcome": "<rule / what to fix>"}``
    * dependency-cve: ``{"package", "version", "advisory_id", "summary",
      "fixed_version", ...}``

    so this renders whichever keys are present into one compact, sanitized line.
    A non-mapping or empty ``proof`` yields an empty hint (the caller omits the
    line). Every value is run through :func:`_sanitize` because proofs are
    repo-controlled (e.g. an advisory summary copied from an upstream feed).
    """
    if not isinstance(proof, dict):
        return ""
    # Order keys for a stable, readable hint; unknown keys are appended after.
    preferred = (
        "outcome",
        "summary",
        "package",
        "version",
        "fixed_version",
        "advisory_id",
    )
    parts: list[str] = []
    seen: set[str] = set()
    for key in preferred:
        if key in proof and proof[key] not in (None, ""):
            parts.append(f"{key}={_sanitize(proof[key])}")
            seen.add(key)
    for key, val in proof.items():
        if key in seen or val in (None, ""):
            continue
        parts.append(f"{_sanitize(key, max_len=80)}={_sanitize(val)}")
    return "; ".join(parts)


def finding_identity(finding: dict[str, Any]) -> str:
    """Return the stable de-dup identity for ``finding`` as a string.

    Prefers the finding ``id`` (the checkers mint a stable ``<repo>-<slug>``),
    which keeps the same finding from spawning two tasks across nightly runs.
    When ``id`` is absent (a malformed/partial report), falls back to a content
    hash of checker+repo+title+severity so two structurally identical findings
    still collapse to one task rather than slipping the de-dup.
    """
    raw_id = finding.get("id")
    if raw_id not in (None, ""):
        return str(raw_id)
    payload = "\x1f".join(
        str(finding.get(key) or "") for key in ("check", "repo", "title", "severity")
    )
    return "sha256:" + hashlib.sha256(payload.encode("utf-8")).hexdigest()


def finding_task_text(finding: dict[str, Any], *, checker: str = "") -> str:
    """Render one confirmed finding into the pipeline task's ``task_text``.

    Produces a compact, fully-sanitized remediation brief: a headline line with
    the checker, repo, and severity; the finding title; and a remediation hint
    drawn from ``proof``. Every interpolated value is repo-controlled and so is
    passed through :func:`_sanitize` first (no raw newline / control char reaches
    the task text or, downstream, the operator log). ``checker`` falls back to
    the finding's own ``check`` field when not supplied by the report header.
    """
    repo = _sanitize(finding.get("repo") or "<unknown-repo>", max_len=120)
    severity = _sanitize(finding.get("severity") or "<unknown>", max_len=40)
    title = _sanitize(finding.get("title") or "<untitled finding>")
    checker_name = _sanitize(checker or finding.get("check") or "checker", max_len=80)

    lines = [
        f"[Plane-1 {checker_name}] remediation for {repo} (severity={severity})",
        f"Finding: {title}",
    ]
    hint = _proof_hint(finding.get("proof"))
    if hint:
        lines.append(f"Remediation hint: {hint}")
    return "\n".join(lines)


def _meets_threshold(severity: Any, *, threshold_rank: int) -> bool:
    """True when ``severity`` is a known level at or above ``threshold_rank``.

    Unknown / missing severities (and the schema's ``unverified`` downgrade
    marker, ranked below zero) never meet a real threshold, so a malformed
    finding can never sneak past the gate.
    """
    rank = SEVERITY_RANK.get(str(severity).strip().lower(), -99)
    return rank >= threshold_rank


def select_findings(
    findings: list[dict[str, Any]],
    *,
    threshold: str = DEFAULT_SEVERITY_THRESHOLD,
) -> list[dict[str, Any]]:
    """Filter ``findings`` to those eligible to spawn a remediation task.

    A finding is selected only when BOTH:

    * ``status == "confirmed"`` (unverified / suppressed are dropped), and
    * its ``severity`` is at or above ``threshold`` (default ``high``).

    Order is preserved. ``threshold`` must be one of the schema severities;
    an unknown threshold is rejected so a typo cannot silently widen the gate.
    """
    key = threshold.strip().lower()
    if key not in SEVERITY_RANK or key in ("unverified", "info"):
        raise ValueError(
            f"invalid severity threshold {threshold!r}; expected one of "
            "low/medium/high/critical"
        )
    threshold_rank = SEVERITY_RANK[key]
    selected: list[dict[str, Any]] = []
    for finding in findings:
        if str(finding.get("status")).strip().lower() != "confirmed":
            continue
        if not _meets_threshold(finding.get("severity"), threshold_rank=threshold_rank):
            continue
        selected.append(finding)
    return selected


def load_report_findings(path: Path) -> list[dict[str, Any]]:
    """Read a checker report file (or every ``*.json`` in a dir) into findings.

    Accepts the exact shape the bash checkers emit: a top-level object with a
    ``findings`` array. A bare JSON array is also accepted (a caller that has
    already extracted ``.findings``). When ``path`` is a directory, every
    ``*.json`` file directly inside it is read and the findings concatenated
    (a malformed file raises, surfacing the bad report rather than silently
    skipping it). Non-mapping finding entries are ignored defensively.
    """
    if path.is_dir():
        findings: list[dict[str, Any]] = []
        for report in sorted(path.glob("*.json")):
            findings.extend(load_report_findings(report))
        return findings

    raw = path.read_text(encoding="utf-8")
    data = json.loads(raw) if raw.strip() else {}
    if isinstance(data, list):
        items = data
    elif isinstance(data, dict):
        items = data.get("findings") or []
    else:
        items = []
    return [item for item in items if isinstance(item, dict)]


class CheckerFindingIntake:
    """Turn confirmed at/above-threshold checker findings into pipeline tasks.

    Construct with an injected ``coordinator`` (anything exposing
    ``start_task(task_text=..., transport_name=...)``), an optional severity
    ``threshold`` (default ``high``), and the ``transport_name`` the resulting
    remediation tasks should deliver clarifier questions over. Then call
    :meth:`ingest_findings` (in-memory findings) or :meth:`ingest_reports`
    (report files / a directory).

    De-dup is in-memory only (see the module docstring): the set of ingested
    finding identities lives on the instance, so re-reading the same report never
    double-ingests within one process, but a restart loses the set. Durable
    de-dup is the known follow-up.

    Nothing here touches the network or any SDK: it reads local JSON and calls
    the injected coordinator, so the whole intake is unit-testable with a stub
    coordinator and in-memory findings.
    """

    def __init__(
        self,
        *,
        coordinator: Any,
        threshold: str = DEFAULT_SEVERITY_THRESHOLD,
        transport_name: str = CHECKER_TRANSPORT_NAME,
    ) -> None:
        """Bind the intake to one coordinator, severity threshold, and transport.

        Args:
            coordinator: The intake target. Must expose
                ``start_task(task_text=..., transport_name=...)``: the live
                :class:`~agent_team.coordinator.Coordinator` in production.
            threshold: Minimum severity a confirmed finding must reach to spawn a
                task (``low``/``medium``/``high``/``critical``; default ``high``).
                An invalid value is rejected up-front.
            transport_name: Channel the resulting task's clarifier question-sets
                route over (default ``github``).
        """
        # Validate the threshold eagerly (reuses select_findings' guard).
        select_findings([], threshold=threshold)
        self._coordinator = coordinator
        self._threshold = threshold.strip().lower()
        self._transport_name = transport_name
        # In-memory de-dup set (P5 scope: best-effort, NOT durable across a
        # restart; see the module docstring). Tracks finding identities already
        # turned into tasks so re-reading a report does not double-ingest.
        self._ingested: set[str] = set()

    @property
    def threshold(self) -> str:
        """The configured severity threshold (read-only)."""
        return self._threshold

    @property
    def ingested_ids(self) -> frozenset[str]:
        """A snapshot of finding identities ingested this process (read-only)."""
        return frozenset(self._ingested)

    def ingest_findings(self, findings: list[dict[str, Any]]) -> list[str]:
        """Select, de-dup, and start one task per unique eligible finding.

        For each finding that passes :func:`select_findings` and is not already
        ingested this process, calls ``coordinator.start_task(task_text=<rendered
        finding>, transport_name=<configured>)`` and records its identity so a
        subsequent pass does not re-ingest it. The identity is recorded ONLY
        after ``start_task`` returns, so a failing intake leaves the finding
        eligible for retry rather than silently dropping it (mirrors
        github_intake).

        Returns the list of finding identities ingested on THIS pass (empty when
        nothing new), so an operator loop can meter intake volume.
        """
        ingested_now: list[str] = []
        for finding in select_findings(findings, threshold=self._threshold):
            identity = finding_identity(finding)
            if identity in self._ingested:
                _LOG.debug(
                    "checker-intake: finding %s already ingested; skip",
                    _sanitize(identity, max_len=120),
                )
                continue

            checker = str(finding.get("check") or "")
            task_text = finding_task_text(finding, checker=checker)
            # task_text is already sanitized field-by-field; log a sanitized
            # summary (never the raw repo/title) to avoid log injection.
            _LOG.info(
                "checker-intake: starting remediation task for %s "
                "(repo=%s severity=%s checker=%s)",
                _sanitize(identity, max_len=120),
                _sanitize(finding.get("repo"), max_len=120),
                _sanitize(finding.get("severity"), max_len=40),
                _sanitize(checker, max_len=80),
            )
            self._coordinator.start_task(
                task_text=task_text,
                transport_name=self._transport_name,
            )
            self._ingested.add(identity)
            ingested_now.append(identity)

        return ingested_now

    def ingest_reports(self, paths: list[Path]) -> list[str]:
        """Load checker report files / dirs and ingest their eligible findings.

        Each entry in ``paths`` may be a report file or a directory of ``*.json``
        reports (see :func:`load_report_findings`). All loaded findings are
        concatenated, then handed to :meth:`ingest_findings` (so de-dup spans the
        whole batch). Returns the finding identities ingested on this call.
        """
        findings: list[dict[str, Any]] = []
        for path in paths:
            findings.extend(load_report_findings(path))
        return self.ingest_findings(findings)