{
  "module": "SDD-B11 — Tau: The Reference Harness to Attack and Harden",
  "course": "2B — Securing & Attacking Harnesses and LLMs",
  "version": "1.0.0",
  "duration_minutes": 45,
  "total_questions": 15,
  "bloom_distribution": {
    "target": "20% recall / 40% application / 40% analysis-design",
    "actual": { "recall": 3, "application": 6, "analysis": 6 }
  },
  "passing_score_percent": 70,
  "questions": [
    {
      "id": "Q01", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What is tau, and what is its relationship to Pi (Course 1 DD-01)?",
      "options": [
        "A commercial competitor to Pi, built by a different vendor with a proprietary license.",
        "Hugging Face's educational reimplementation of Pi (built by Alejandro AO), ~3,000 LOC across three packages (tau_ai / tau_agent / tau_coding). Same architecture and philosophy as Pi, but with a three-layer package split that makes the extension points cleaner to teach.",
        "A sandboxing framework that wraps Pi to add security controls.",
        "A benchmark suite for evaluating coding agents like Pi."
      ],
      "answer_index": 1,
      "rationale": "Tau is Pi's educational clone under the Hugging Face organization. Same architecture, same philosophy, but the three-layer package split (tau_ai provider layer, tau_agent reusable brain, tau_coding coding app) makes the extension points cleaner to teach than Pi's single-package layout. If DD-01 was 'the simplest harness that works,' SDD-B11 is 'the simplest harness we can break and fix.'"
    },
    {
      "id": "Q02", "bloom": "recall", "type": "multiple_choice",
      "prompt": "Name tau's three layers and the load-bearing boundary between them.",
      "options": [
        "tau_input / tau_process / tau_output, with the boundary at the output formatter.",
        "tau_ai (provider/model streaming) / tau_agent (reusable brain: harness, loop, tools, events) / tau_coding (coding app: CLI/TUI, read/write/edit/bash, sessions). The load-bearing boundary is tau_agent ↔ tau_coding: the core (tau_agent) knows nothing about rendering, and frontends consume events. Enforcement belongs above (tau_agent); attack surface lives below (tau_coding).",
        "tau_core / tau_api / tau_ui, all in a single package with no real boundary.",
        "tau_model / tau_data / tau_network, separated by network protocol."
      ],
      "answer_index": 1,
      "rationale": "The three layers are tau_ai (provider-neutral streaming), tau_agent (the reusable brain that holds the harness, loop, tools, events), and tau_coding (the coding app with filesystem/credential/session/bash surfaces). The load-bearing boundary is between tau_agent and tau_coding: AgentHarness is the reusable brain, CodingSession is the environment, and the TUI is just one frontend. The core does not know about Textual, Rich, or rendering — it emits events. This is what makes the extension points clean: enforcement attaches in tau_agent, attack surface lives in tau_coding."
    },
    {
      "id": "Q03", "bloom": "recall", "type": "multiple_choice",
      "prompt": "Why is tau the ideal teaching target for Course 2B? State the three properties.",
      "options": [
        "It is fast, it is popular, and it has a large community.",
        "It is real (a bash tool that actually runs asyncio.create_subprocess_shell; a credential store that actually holds plaintext API keys on disk), it is small (~3,000 LOC across three packages, readable in an afternoon), and it is undefended (none of the controls B2–B8 build exist today — every control you add is a net-new feature, and its absence is the vulnerability you're closing).",
        "It is free, it is open source, and it has no dependencies.",
        "It is written in Rust, it is memory-safe, and it has no known vulnerabilities."
      ],
      "answer_index": 1,
      "rationale": "The three properties are decisive. (1) Real — not a stub or simulation; attacking tau means attacking a system with actual subprocess execution, actual file reads, actual plaintext credential storage. (2) Small — ~3,000 LOC readable end-to-end, so a student understands every security-relevant line. (3) Undefended — tau has none of the B2–B8 controls, making it the 'before' picture. Every control is a missing feature, and the delta between unmodified tau and the hardened session is the defense scorecard the capstone measures."
    },
    {
      "id": "Q04", "bloom": "application", "type": "multiple_choice",
      "prompt": "You want to add a B2 taint gate to tau so tool output is tagged <untrusted> before it enters the transcript. Where do you attach it, and why not in run_agent_loop?",
      "options": [
        "Patch run_agent_loop (_execute_tool_calls) to inspect each tool call before execution — the loop is the central execution path.",
        "Wrap the executor (Extension Point 1): dataclasses.replace(tool, executor=gated) where gated() calls gate.inspect() pre-execution and gate.sanitize() post-execution, returning the AgentToolResult with tagged content before _tool_result_message (loop.py:260) ever sees it. You do not patch the loop because the loop delegates to tool.execute → tool.executor; enforcement belongs in the tool, not the loop.",
        "Subscribe an event listener to ToolExecutionEndEvent and rewrite the result — events fire before the transcript is written.",
        "Modify the SessionStorage JSONL writer to tag content as it is persisted."
      ],
      "answer_index": 1,
      "rationale": "The architectural lesson: enforcement belongs in the tool, not the loop. The loop delegates to tool.execute, which delegates to tool.executor. Wrapping the executor (EP1) means the gate runs before execution (inspect) and after execution (sanitize), and the returned AgentToolResult already has tagged content before _tool_result_message converts it to a ToolResultMessage. The loop stays generic and untouched; the gate is a property of the tool (auditable in the tool list). Events fire after the decision (read-only, for observability), and the session writer is downstream of the problem."
    },
    {
      "id": "Q05", "bloom": "application", "type": "multiple_choice",
      "prompt": "Your team adds a B7 sandbox by wrapping the bash tool in the harness tool list (Extension Point 1) and reports 'bash is now sandboxed.' What is wrong with this claim?",
      "options": [
        "Nothing — the harness tool list is the only place bash tools exist.",
        "The terminal-command bar (CodingSession.run_terminal_command, session.py:1183) constructs its OWN create_bash_tool outside the harness tool list and calls it directly (session.py:1187), bypassing the tool-list wrap. Commands issued through the bar are unsandboxed. The fix is to wedge at the factory level — replace create_bash_tool itself (Extension Point 2) so both call sites receive the sandbox policy.",
        "The sandbox will not work because tau has no plugin system.",
        "The sandbox needs to be added to the event listener instead."
      ],
      "answer_index": 1,
      "rationale": "This is the two-bash-tool finding. Both bash paths go through create_bash_tool (tools.py:574), but only the harness-list path (create_coding_tools at tools.py:96) enters CodingSessionConfig.tools. The terminal-command path (run_terminal_command at session.py:1183) builds its own tool and calls it directly, never entering the tool list. A tool-list-level wrap (EP1) is bypassed. The fix is the factory-level wedge (EP2): replace create_bash_tool so both paths receive the hardened executor. The factory is the chokepoint, not the instance."
    },
    {
      "id": "Q06", "bloom": "application", "type": "multiple_choice",
      "prompt": "You implement a B5 credential vault that implements the CredentialReader protocol and inject it via create_model_provider(credential_store=my_vault) at provider_runtime.py:46. Are credentials now secure against agent exfiltration?",
      "options": [
        "Yes — the vault replaces the plaintext store, so the agent can no longer read credentials.",
        "No — partially. Two issues remain: (1) a SECOND FileCredentialStore is created in CodingSession.__init__ (session.py:253-255) and must also be replaced for full coverage; (2) even with both replaced, the agent's bash tool can still run 'cat ~/.tau/credentials.json' and 'printenv' — B7 (the bash sandbox denylist) is required to block the exfiltration channel. B5 secures at rest; B7 closes the channel. They are coupled.",
        "No — the vault does not implement the OAuth methods the Codex resolver needs.",
        "Yes — once the vault is injected, no further controls are needed."
      ],
      "answer_index": 1,
      "rationale": "B5 alone is insufficient. First, there are two FileCredentialStore instantiation sites: create_model_provider (provider_runtime.py:46, line 56) for the provider layer, and CodingSession.__init__ (session.py:253-255) for the session. Both must be replaced. Second, even with the vault in place, the agent's bash tool can exfiltrate via 'cat ~/.tau/credentials.json' (if the file still exists) or 'printenv' (leaking OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENAI_CODEX_ACCESS_TOKEN from the environment). B5 secures credentials at rest and at the provider boundary; B7 blocks the exfiltration channel. B5 and B7 are coupled — both are needed."
    },
    {
      "id": "Q07", "bloom": "application", "type": "multiple_choice",
      "prompt": "A security engineer wants to use Extension Point 4 (harness.subscribe) to BLOCK suspicious tool calls by having the listener raise an exception when drift is detected. Why does this fail, and where should enforcement go?",
      "options": [
        "It works — the listener can raise an exception to halt execution.",
        "It fails because events fire AFTER the decision in _execute_tool_calls — the listener is read-only and cannot prevent a tool call already in progress. Enforcement belongs in the executor wrapper (Extension Point 1), where gate.inspect() runs BEFORE awaiting inner.executor. Events (EP4) are for observability (B8 — intent tracking, drift detection, alerting); enforcement is in the tool/executor.",
        "It fails because the listener needs to be registered in a different file.",
        "It works if the listener is async rather than sync."
      ],
      "answer_index": 1,
      "rationale": "Events fire after the decision in _execute_tool_calls. By the time ToolExecutionStartEvent reaches the listener, the tool has already been selected and execution is in progress; the listener cannot prevent it (and an exception would crash the harness, not cleanly block). Enforcement belongs in the executor wrapper (EP1): the wrapped gated() function runs gate.inspect(inner.name, arguments) before awaiting inner.executor, so it can return a denied AgentToolResult before execution. Events are read-only observability (B8); the executor wrapper is enforcement (B2/B7). The architectural rule: enforcement in the tool, observability in the event stream."
    },
    {
      "id": "Q08", "bloom": "application", "type": "multiple_choice",
      "prompt": "You are designing B3's memory-write gate for tau sessions. The poisoned tool result must be caught before it persists to ~/.tau/sessions/ JSONL. Where are the two viable wedge points?",
      "options": [
        "In run_agent_loop and in the TUI renderer.",
        "(1) The executor wrapper (Extension Point 1) — gate.sanitize() runs on the AgentToolResult BEFORE it is converted to a ToolResultMessage, so the poison never enters the message that gets persisted. (2) The persist chokepoint _persist_messages_since (session.py:1378) — the single site where every message becomes durable; a gate here validates what enters the JSONL regardless of source. Either closes the sleeper attack (inject in session 1, JSONL persists, session 2 loads).",
        "In the provider layer and in the OAuth resolver.",
        "In the event listener and in the credential store."
      ],
      "answer_index": 1,
      "rationale": "B3's memory-write gate has two viable wedge points. The executor wrapper (EP1) is upstream: gate.sanitize() transforms the AgentToolResult before it becomes a ToolResultMessage, so the poison never reaches the transcript or the JSONL. The persist chokepoint _persist_messages_since (session.py:1378) is the single site where every message becomes durable — a gate here catches poison regardless of its source (tool result, user message, anything). The sleeper attack is: inject via a tool result in session 1, the JSONL persists it, session 2 loads it back into context. Either wedge point closes it. The executor wrapper is preferred because it also closes Surface 2 (untrusted tagging) at the same seam."
    },
    {
      "id": "Q09", "bloom": "application", "type": "multiple_choice",
      "prompt": "An engineer proposes using tau's shell_command_prefix mechanism (tools.py:586) as an allowlist: set the prefix to a script that rejects disallowed commands. Why is this an anti-pattern?",
      "options": [
        "It is a valid approach — the prefix runs before the command, so it can reject.",
        "shell_command_prefix is a BLIND PREPEND: it does f'{prefix}\\n{command}'. The prefix is evaluated as a shell preamble (setup like env vars, sourcing), and the command runs unconditionally AFTER it. The prefix mechanism CANNOT see or reject the command — it cannot inspect 'command' or return a deny decision. The security control is the executor wrapper inside the hardened create_bash_tool factory (Extension Point 2), where policy.check(command) runs before execution.",
        "It works but is too slow for production.",
        "It works only for local models, not for cloud providers."
      ],
      "answer_index": 1,
      "rationale": "The shell_command_prefix (_prefixed_shell_command, tools.py:586) is a blind prepend that concatenates prefix and command into a single shell script. The prefix is setup (environment variables, sourcing scripts), and the command runs after it unconditionally. There is no mechanism for the prefix to inspect or reject the command — it cannot see the 'command' argument, and even if the prefix script checked an environment variable, the command still runs. Trying to overload it as a security policy is an anti-pattern. The correct control is the executor wrapper in the hardened create_bash_tool factory (EP2), where policy.check(command) evaluates the actual command and can return a denied AgentToolResult before execution."
    },
    {
      "id": "Q10", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why does tau having NO plugin system (no register_tool, no hooks, no middleware) make it a BETTER teaching target than a harness with a rich plugin/middleware framework?",
      "options": [
        "It does not — a plugin system would make tau easier to extend and therefore a better target.",
        "Three reasons: (1) AUDITABILITY — every wedge point is an explicit, named site in the code (tools.py:61, tools.py:574, provider_runtime.py:46, harness.py:124, session.py:286); a student can point at the exact line. A plugin framework hides the attachment behind registration/discovery/middleware chains. (2) TEACHABILITY — no framework magic to learn first; the student reads a frozen dataclass, a Protocol, and a dataclasses.replace. (3) COMPOSITION — four of five extension points are pure wrapping/additive, stacking as function composition without ordering/short-circuit semantics.",
        "It is better only because plugin systems are insecure by definition.",
        "It is better because tau is smaller, and small codebases never need plugins."
      ],
      "answer_index": 1,
      "rationale": "The absence of a plugin system forces clarity. (1) Auditability: the extension points are named code sites, not registrations hidden in a framework's lifecycle. A security reviewer can grep for the exact attachment. (2) Teachability: the concept 'wrap the executor' is one function and a dataclasses.replace — no lifecycle hooks, priority ordering, or registration timing to learn before the security concept lands. (3) Composition: four of five extension points (EP1 executor wrap, EP2 factory replacement, EP4 additive subscription, EP5 tool-list injection) are pure wrapping or additive; they stack as function composition without the ordering and short-circuit semantics that complicate plugin middleware. Only EP3 (credential store) is a straight replacement."
    },
    {
      "id": "Q11", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "How does the two-bash-tool finding generalize as an audit principle for any agent harness, beyond bash specifically?",
      "options": [
        "It only applies to bash; other tools always have a single construction site.",
        "PRINCIPLE: any control that wedges at the TOOL-LIST layer (Extension Point 1 or 5) is BYPASSABLE by any code path that constructs its own tool OUTSIDE the list. The factory is the chokepoint. When auditing a harness for a control's coverage, do NOT ask 'is the tool in the list?' — ask 'is there any code path that constructs this tool outside the list?' The factory-level wedge (replacing the factory so all call sites use it) is the only one that covers every path.",
        "The principle is that all tools should be removed from the harness and called only via the CLI.",
        "The principle is that bash should never be a tool — it should be a built-in function."
      ],
      "answer_index": 1,
      "rationale": "The two-bash-tool finding generalizes: any tool with multiple construction sites can bypass a tool-list-level control. In tau, bash is the concrete instance (run_terminal_command builds its own), but the audit principle applies to any tool. The correct audit question is not 'is the tool in the list?' but 'is there any code path that constructs this tool outside the list?' The factory-level wedge — replacing the factory itself so every call site receives the hardened version — is the only attachment point that covers every construction path. This is why Extension Point 2 is at create_bash_tool (the factory), not at the individual tool instances."
    },
    {
      "id": "Q12", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Trace the indirect-injection attack through tau's surfaces 1, 2, and 3. For each leg, identify where it is undefended and which extension point closes it.",
      "options": [
        "All three surfaces are closed by the event listener.",
        "(1) Surface 2 (tool output, loop.py:260): a read tool returns an AgentToolResult with poison; _tool_result_message appends it to the transcript unfiltered. UNDEFENDED. CLOSED BY EP1 executor wrapper — gate.sanitize() tags output <untrusted> before it reaches the transcript. (2) Surface 1 (agent loop, loop.py:190): the model complies and calls bash; _execute_tool_calls executes on request. UNDEFENDED. CLOSED BY EP1 — gate.inspect() checks before execution. (3) Surface 3 (sessions, session.py:1378): the ToolResultMessage persists to JSONL unvalidated. UNDEFENDED. CLOSED BY EP1 (sanitize before it becomes a message) OR B3's memory-write gate at _persist_messages_since.",
        "All three are closed by patching run_agent_loop.",
        "Surfaces 1 and 2 are closed by the vault; surface 3 by the sandbox."
      ],
      "answer_index": 1,
      "rationale": "The indirect-injection attack chains three surfaces. (1) Tool output (Surface 2): the read tool returns poisoned content; without the executor wrapper, _tool_result_message appends it unfiltered — the InjecAgent ~50% vulnerability. EP1's gate.sanitize() closes it by tagging before the transcript. (2) Agent loop (Surface 1): the model, now seeing the poison, calls bash; _execute_tool_calls runs it. EP1's gate.inspect() closes it with a pre-execution check. (3) Sessions (Surface 3): the ToolResultMessage persists to JSONL; on resume/branch, the poison reloads — the sleeper attack. Closed by EP1 (sanitize upstream) or B3's gate at the _persist_messages_since chokepoint (session.py:1378). The executor wrapper is the highest-leverage single wedge because it closes surfaces 1, 2, and the upstream leg of 3 at one seam."
    },
    {
      "id": "Q13", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why is patching run_agent_loop (_execute_tool_calls) to add a pre-execution gate the wrong abstraction, even though it would technically intercept every tool call?",
      "options": [
        "It is the correct abstraction — the loop is the only place that sees every tool call.",
        "It violates the architectural lesson (enforcement belongs in the tool, not the loop) and has three concrete costs: (1) every future loop change can break the gate; (2) the gate is invisible to anyone reading the tool definitions (not auditable in the tool list); (3) it doesn't compose — stacking two gates requires two loop patches. Wrapping the executor (EP1) makes the gate a property of the TOOL (auditable, composable via function composition), and the loop stays generic and untouched. The frozen dataclass + Protocol + replace is the designed extension point.",
        "It is wrong because the loop is written in C and cannot be patched.",
        "It is wrong because _execute_tool_calls is private and should not be modified for style reasons only."
      ],
      "answer_index": 1,
      "rationale": "Patching the loop fights the framework's designed seam. The loop delegates to tool.execute → tool.executor; the executor field of the frozen AgentTool dataclass is the designed extension point. Wrapping it (EP1) means: (1) the gate survives loop refactors because it is attached to the tool, not the loop; (2) the gate is visible in the tool list — a reviewer sees wrapped executors; (3) multiple gates compose by nesting wrapped executors (function composition). A loop patch has none of these properties: it is fragile, invisible, and non-composable. The architectural lesson — enforcement in the tool, not the loop — is not aesthetic; it is the difference between a maintainable and an unmaintainable control."
    },
    {
      "id": "Q14", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Design the hardened session (create_hardened_session) that assembles all five extension points. For each, state what it attaches and why no single extension point is sufficient.",
      "options": [
        "A single extension point (the executor wrapper) suffices if implemented correctly.",
        "EP5+EP1: inject hardened tools at CodingSession.load (session.py:286); every executor wrapped with scope gate (B0/B1) + taint gate (B2), output tagged <untrusted>. EP2: hardened create_bash_tool replaces the factory; both bash paths covered; default-deny egress + allowlist + caps (B7). EP3: vault replaces FileCredentialStore in BOTH sites (provider_runtime.py:46 + session.py:253); credentials never plaintext (B5). EP4: intent tracker subscribed via harness.subscribe; read-only (B8). NO SINGLE EP SUFFICES because surfaces overlap and controls are coupled: B5 alone doesn't stop cat (needs B7); B7 alone doesn't secure the secret (needs B5); B2 alone doesn't stop JSONL poison (needs B3). The composition is the point.",
        "Only EP2 (bash factory) and EP3 (vault) are needed; the rest are redundant.",
        "Only the event listener (EP4) is needed if it is sophisticated enough."
      ],
      "answer_index": 1,
      "rationale": "The hardened session composes all five because the surfaces overlap and the controls are coupled. EP5+EP1 closes surfaces 1, 2, and the upstream leg of 3 (scope gate, taint gate, untrusted tagging). EP2 closes surface 6 (sandbox) and covers both bash paths including the terminal-command bypass. EP3 closes surface 5 at rest (vault). EP4 closes surface 7 (observability). But no single point suffices: B5 secures credentials at rest but the agent can still cat them without B7's denylist; B7 blocks exfiltration but bash still runs injected commands without B2's taint gate; B2 taints tool output but JSONL still persists poison without B3's memory-write gate. The scorecard measures the delta between unmodified tau and this composition — the artifact Capstone B1 ships."
    },
    {
      "id": "Q15", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "You audit a tau deployment and find the team replaced FileCredentialStore with a vault in create_model_provider (provider_runtime.py:46) but left CodingSession.__init__ unchanged. Analyze the residual exposure and prescribe full remediation.",
      "options": [
        "No residual exposure — the provider factory is the only site that matters.",
        "RESIDUAL EXPOSURE: CodingSession.__init__ (session.py:253-255) still creates a SECOND FileCredentialStore that reads/writes the plaintext file at ~/.tau/credentials.json. Depending on which code path reads credentials, plaintext may still be on disk and reachable by the agent's bash tool. FULL REMEDIATION: (1) replace BOTH FileCredentialStore sites (provider_runtime.py:46 AND session.py:253) with the vault; (2) add B7's bash denylist blocking 'cat ~/.tau/credentials.json' and 'printenv' (closing the exfiltration channel for OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENAI_CODEX_ACCESS_TOKEN); (3) ensure the vault implements the full CredentialReader protocol including get_oauth/set_oauth for the Codex resolver. B5 (at rest) + B7 (channel) are coupled — both required.",
        "Replace the vault with a stronger encryption algorithm.",
        "Move the credentials to an environment variable instead of a file."
      ],
      "answer_index": 1,
      "rationale": "The residual exposure is real: the second FileCredentialStore in CodingSession.__init__ (session.py:253-255) independently reads and writes the plaintext file. The provider-layer replacement (provider_runtime.py:46) only covers the provider's credential reads, not the session's. Full remediation requires three steps: (1) replace both instantiation sites with the vault so no plaintext file exists; (2) add the B7 bash denylist so even if a plaintext file or env vars exist, the agent cannot read them ('cat ~/.tau/credentials.json', 'printenv'); (3) ensure the vault implements the full CredentialReader protocol (get plus get_oauth/set_oauth for the Codex OAuth resolver). B5 secures credentials at rest and at the provider boundary; B7 blocks the exfiltration channel. They are coupled — B5 without B7 leaves the channel open; B7 without B5 leaves the secret in plaintext."
    }
  ]
}