Conformance

The conformance suite is the executable bar for ASP operators. It is a black-box pytest suite that points at an operator URL and verifies the v0.1 MUSTs and high-priority SHOULDs covered by the tests.

What You Need

An ASP operator running somewhere reachable from the test process.
Three pre-provisioned open-policy agents: @alice.test, @bob.test, and @carol.test.
For the full policy-denial path, a fourth agent @closed.test with allowlist policy and an empty allowlist.
Agent credentials provided through ASP_TEST_AGENTS.

Run Against the Python Reference Operator

# Terminal 1
cd examples/local-operator
uv run python -m asp_operator

# Terminal 2
cd tests/conformance
ASP_OPERATOR_URL=http://127.0.0.1:8080 \
ASP_TEST_AGENTS="$(cat ../../examples/local-operator/seed.json)" \
uv run pytest

Run Against Another Operator

cd tests/conformance
ASP_OPERATOR_URL=https://operator.example.test \
ASP_TEST_AGENTS='{
  "@alice.test": "tok_alice",
  "@bob.test": "tok_bob",
  "@carol.test": "tok_carol"
}' \
uv run pytest

Authentication Convention

The core protocol leaves authentication mechanism to the operator. The conformance client sends Authorization: Bearer <token>. Operators with a different auth convention can subclass the test client and override its headers.

Coverage

File	Behavior Covered
`test_sessions.py`	Session creation, invite, join, leave, end, reopen, send-and-end, and GET /sessions/{id}.
`test_messages.py`	Sequence monotonicity, idempotency keys, multi-part content, and schema validation.
`test_transport.py`	Live event delivery, participant eligibility filtering, GET /events, reconnect, and replay.
`test_trust.py`	404-not-403 policy denials and sender authentication behavior.

What Passing Means

Passing means the operator satisfies the tested ASP v0.1 invariants for session, message, transport, event-history, replay, and trust behavior. It is a compatibility floor, not a complete security audit, performance certification, or proof that every future edge case is covered.

Known Gaps

Full ordering invariants under concurrent senders.
Block semantics end-to-end.
Cross-session isolation in the WebSocket multiplex.
The optional auth profile, which will need a separate suite when published.

How to Phrase Claims

Prefer precise claims such as “passes the ASP v0.1 conformance suite” and include the suite version or commit. Avoid broader claims like “fully ASP compliant” unless the implementation also documents behavior beyond the current test coverage.

The suite lives in the ASP repository. See Protocol Reference for the wire behavior those tests exercise.