PROPOSAL_TCP_RECORD_REPLAY.md

· erock's pastes · raw

expires: 2026-11-14

  1# Proposal: Transparent Rootless TCP Record & Replay for `pici`
  2
  3**Status**: Draft  
  4**Target Platform**: Linux (x86_64, aarch64)  
  5**Dependencies**: Zero external runtime dependencies (Pure Go binary)  
  6
  7---
  8
  9## 1. Executive Summary & Objective
 10
 11When iterating on local CI pipelines or writing test suites against `pici`, running real tasks often triggers destructive or stateful side effects—such as uploading GitHub releases, pushing Git tags, deploying to SSH servers, or consuming rate-limited third-party APIs.
 12
 13This proposal introduces a **fully transparent, zero-configuration TCP Record & Replay engine** built natively into `pici`. By leveraging rootless Linux kernel namespaces (`CLONE_NEWUSER`, `CLONE_NEWNET`, `CLONE_NEWNS`) and an in-process user-space TCP/IP stack (`gVisor netstack`), `pici` can intercept, record, and mock all outbound network traffic (HTTP, HTTPS, SSH, Git) without requiring `sudo`, environment variable modifications, or tool-specific configuration.
 14
 15---
 16
 17## 2. System Architecture
 18
 19```mermaid
 20flowchart TB
 21    subgraph Host["pici Host Process (Go Runtime)"]
 22        NETSTACK["In-Process TCP/IP Stack\n(gVisor pkg/tcpip)"]
 23        RECORDER["Record / Replay Engine\n(L4 Stream + L7 HTTP/SSH Parsers)"]
 24        CASSETTE[("Cassette Store\n.pici/cassettes/<name>/")]
 25        
 26        NETSTACK <--> RECORDER <--> CASSETTE
 27    end
 28
 29    subgraph Sandbox["Rootless Sandbox (unshare -U -n -m)"]
 30        TUN["Virtual TUN Device (Default Route: 10.0.2.1/24)"]
 31        SSL_MOUNT["Ephemeral /etc/ssl Trust Store\n(Dynamically mounted in-memory)"]
 32
 33        subgraph Tasks["pico.sh & Subprocesses (zmx / Docker / CLI tools)"]
 34            T1["curl / npm / apt / gh"]
 35            T2["git push git@github.com:..."]
 36            T3["ssh pipe.pico.sh 'pub build.status'"]
 37        end
 38    end
 39
 40    Tasks -->|"Raw Outbound TCP (All Ports & IPs)"| TUN
 41    TUN -->|"Packet Stream (UNIX Socketpair FD)"| NETSTACK
 42    RECORDER -.->|"Record Mode: Real WAN\nReplay Mode: Serves Cassettes"| WAN["External Network / APIs"]
 43```
 44
 45---
 46
 47## 3. Kernel Primitives & Sandbox Isolation
 48
 49The sandbox is spawned directly by `pici` without external utilities (`slirp4netns` or `podman` are not required):
 50
 51### 1. User & Network Namespace (`CLONE_NEWUSER | CLONE_NEWNET`)
 52* `pici` spawns the job wrapper with `CLONE_NEWUSER` and sets a single UID/GID mapping (`Host UID -> Container UID 0`). This gives the child process full administrative privileges *inside the namespace only*, with zero privileges on the host.
 53* `CLONE_NEWNET` provides a completely isolated network stack with its own loopback and routing tables.
 54* A virtual `TUN` device is created and set as the default gateway (`0.0.0.0/0 -> 10.0.2.1`).
 55
 56### 2. Mount Namespace (`CLONE_NEWNS`) for Ephemeral TLS Trust
 57* To intercept HTTPS traffic without modifying the user's host system, `pici` generates an in-memory Root CA keypair.
 58* Inside the mount namespace, `pici` bind-mounts a generated bundle containing this CA over standard CA trust paths:
 59  * `/etc/ssl/certs/ca-certificates.crt`
 60  * `/etc/pki/tls/certs/ca-bundle.crt`
 61  * `/etc/ssl/cert.pem`
 62* When the job finishes and the namespace is destroyed, all certificate mounts disappear without leaving any trace on the host.
 63
 64---
 65
 66## 4. Transparent Protocol Handlers
 67
 68### 1. HTTP & HTTPS
 69* **TLS Termination**: When the user-space stack receives a connection to port 443 (or any TLS port), `pici` dynamically mints a TLS certificate for the target SNI hostname signed by the ephemeral Root CA.
 70* **Recording**: Captures `{ method, url, request_headers, request_body }` and paired `{ status_code, response_headers, response_body }`.
 71* **Header Normalization**: Headers like `Date`, `User-Agent`, `X-Request-Id`, and authentication tokens are filtered or normalized to ensure deterministic replay matching.
 72
 73### 2. SSH & Git over SSH (`git push`, `ssh pipe.pico.sh`)
 74* When traffic to port 22 or starting with the `SSH-2.0-` banner is intercepted:
 75  * **Record Mode**: `pici` performs a transparent SSH proxy handshake, records the server host key, captures channel requests (`exec`, `subsystem`, `pty`), and streams data bidirectionally while logging the conversation.
 76  * **Replay Mode**: `pici` completes the SSH handshake using the recorded host key, accepts public key or password auth unconditionally, and replays recorded stdout/stderr streams and exit codes for matching commands.
 77
 78### 3. Raw TCP Fallback
 79* For unrecognized protocols or raw TCP sockets, `pici` records sequential bidirectional packet streams indexed by `Destination IP:Port`.
 80
 81---
 82
 83## 5. Cassette Storage & Secret Sanitization
 84
 85Cassettes are stored in `.pici/cassettes/<name>/` as clean, human-readable JSON files designed to be checked into source control:
 86
 87```
 88.pici/cassettes/release-v1.0/
 89├── manifest.json       # Metadata, timestamps, sanitized environment
 90├── http.jsonl          # HTTP/HTTPS requests & responses
 91└── ssh.jsonl           # SSH exec commands, terminal streams, exit statuses
 92```
 93
 94### Automatic Secret Scrubbing
 95Before writing any cassette to disk:
 961. `pici` inspects all active environment variables (e.g. `GITHUB_TOKEN`, `AWS_SECRET_ACCESS_KEY`, `SSH_AUTH_SOCK`).
 972. Any string matching an active secret value in headers or payloads is replaced with a deterministic placeholder: `{{ REDACTED_SECRET_<KEY> }}`.
 983. During **Replay Mode**, the placeholder is dynamically matched against whatever secret is passed in the local environment.
 99
100---
101
102## 6. Developer Experience & CLI Interface
103
104### CLI Commands
105```bash
106# 1. Execute job live and record all network traffic to default cassette (.pici/cassettes/default)
107pici --record
108
109# 2. Execute job live and record to a named cassette
110pici --record-to .pici/cassettes/release-flow
111
112# 3. Replay job completely offline (0 network calls, 0 side effects)
113pici --replay-from .pici/cassettes/release-flow
114
115# 4. Strict Replay Mode (fails immediately on any unrecorded network call)
116pici --replay-from .pici/cassettes/release-flow --strict-network
117```
118
119### Go Test Harness Integration
120Developers can write end-to-end tests for their CI pipeline directly in Go:
121
122```go
123func TestPicoCI_ReleaseJob(t *testing.T) {
124    runner := pici.NewRunner(&pici.Config{
125        WorkspaceDir:   ".",
126        ReplayCassette: "./testdata/cassettes/release-flow",
127        StrictNetwork:  true,
128    })
129    
130    res, err := runner.Run(context.Background())
131    require.NoError(t, err)
132    require.Equal(t, 0, res.ExitCode)
133}
134```
135
136---
137
138## 7. Implementation Roadmap
139
140| Milestone | Deliverables |
141| :--- | :--- |
142| **Phase 1: Sandbox & TCP Bridge** | Implement `unshare` runner with `CLONE_NEWUSER \| CLONE_NEWNET \| CLONE_NEWNS`, create in-memory TUN socketpair, and wire into `gvisor.dev/gvisor/pkg/tcpip`. |
143| **Phase 2: HTTP/S MITM & TLS Mounting** | In-memory CA generation, ephemeral `/etc/ssl` bind mounting, dynamic SNI cert generation, and JSONL HTTP cassette recorder/replayer. |
144| **Phase 3: SSH & Git MITM Engine** | Transparent SSH proxy handler for `git push` and `ssh ...` commands; recording exec channel payloads and exit codes. |
145| **Phase 4: Secret Sanitization & CLI Flags** | Automatic token redaction, `--record` and `--replay` CLI flags, summary status output in UI reporter. |
146
147---
148
149> [!NOTE]
150> This design maintains `pici`'s zero-dependency single-binary philosophy (`CGO_ENABLED=0 go build`) and works seamlessly on any modern Linux kernel without requiring root or modifying host configurations.