open-swe/evals/reviewer/config.toml

dataset_name = "openswe-reviewer-v1"
experiment_prefix = "openswe-reviewer-verified-deployed"
max_concurrency = 2

# Leave blank to use LANGGRAPH_URL or local dev.
langgraph_url = ""
assistant_id = "reviewer"
model_id = "anthropic:claude-opus-4-7"
reasoning_effort = "high"

# Use "surfaced_findings" to score only findings that pass the production
# severity threshold and cap.
score_mode = "all_findings"
severity_threshold = "medium"
cap = 4
feat: Adds ability to run evals against deployment (#1311) * feat: tighten reviewer eval workflow Require the reviewer to verify and dedupe findings before recording them, and make benchmark runs safe to execute against deployed reviewer graphs without posting GitHub reviews. * chore: move reviewer eval settings to config Load reviewer benchmark settings from the default eval config file so deployed eval runs do not require a wide CLI surface. * feat: allow reviewer eval model overrides Pass reviewer model and reasoning effort from the eval config into reviewer runs so isolated benchmark deployments can test Opus 4.7 high thinking. * fix: use adaptive thinking for Opus 4.7 Switch Opus 4.7 model overrides to Anthropic adaptive thinking with effort instead of the deprecated budgeted thinking payload rejected by the API. * refactor: use latest Anthropic effort API Remove legacy Anthropic budget-token thinking support and route Anthropic efforts through adaptive thinking plus effort. * revert prompting 2026-05-18 15:47:13 -07:00			`dataset_name = "openswe-reviewer-v1"`
			`experiment_prefix = "openswe-reviewer-verified-deployed"`
			`max_concurrency = 2`

			`# Leave blank to use LANGGRAPH_URL or local dev.`
			`langgraph_url = ""`
			`assistant_id = "reviewer"`
			`model_id = "anthropic:claude-opus-4-7"`
			`reasoning_effort = "high"`

			`# Use "surfaced_findings" to score only findings that pass the production`
			`# severity threshold and cap.`
			`score_mode = "all_findings"`
			`severity_threshold = "medium"`
			`cap = 4`