Private RAG workshop
Design and evaluate a permission-aware knowledge assistant using synthetic documents.
Lab objective
Deploy and test the LAB-230-01 synthetic private retrieval service. The system must preserve document permissions through ingestion, retrieval, context assembly, citation, and deletion while treating all retrieved text as untrusted data.
- Content review
- 2026-08-03
- Automated lab validation
- 2026-08-03
- Independent reproduction
- Not yet completed
Architecture
The executable source is maintained in academy/labs/LAB-230-01. It contains the local service, Compose definition, synthetic corpus, identities, policy, evaluation cases, tests, evidence schema, reset path, and troubleshooting guide.
Compatibility and claim status
| Environment | Status | Last automated validation |
|---|---|---|
| Python 3.11+ standard library | Automated tests passing | 2026-08-03 |
| Docker Compose | Definition supplied; runtime validation pending | Not yet validated |
| Independent operator | Reproduction requested | Not yet completed |
Run the acceptance suite
Open the lab directory
Run from the repository root so the generated evidence remains beside the lab files.
cd academy/labs/LAB-230-01
python scripts/validate.py
Start the local query service
Use the dependency-free Python service or the restricted Compose service.
python -m app.server
# Alternatively: docker compose up --build --wait
Exercise an allowed request
Query as the synthetic Finance user. The response must cite only FIN-BUDGET-2026 version 3.0.
$body = '{"user_id":"fiona.finance","question":"What is the finance budget approval threshold?"}'
Invoke-RestMethod -Uri http://127.0.0.1:8080/query -Method Post -ContentType application/json -Body $body
Failure cases and acceptance gates
| Case | Input | Passing condition |
|---|---|---|
| Denied content | HR user asks for Finance threshold | No Finance document enters retrieval context; approved no-answer is returned |
| Mixed relevance | Permitted and denied sources share terms | Permission filtering happens before scoring |
| Outdated conflict | Versions 2.0 and 3.0 exist | Only active version 3.0 can be cited |
| Deletion | Question targets deleted forecast | Document is absent from index, context, cache-equivalent state, and citations |
| Prompt injection | Retrieved document contains override text | Instruction text is detected and removed; governing behavior is unchanged |
| Unsupported claim | Question has no permitted support | Exact approved no-answer response is produced |
Evidence contract
Every query evidence record identifies the corpus, index, retriever, permission policy, generation component, governing prompt, and evaluation dataset versions. It also records citation resolution and injection detection without storing the original question.
{
"lab_id": "LAB-230-01",
"versions": {
"corpus": "sha256-prefix",
"index": "sha256-prefix",
"permission_policy": "1.0.0",
"retriever": "stdlib-token-overlap-0.1.0",
"evaluation_dataset": "synthetic-questions-0.1.0"
},
"controls": {
"permission_filter_before_scoring": true,
"all_citations_resolve": true,
"injection_detected": false
},
"citations": [
{
"source_id": "FIN-BUDGET-2026",
"version": "3.0"
}
]
}Participant extension
Replace the deterministic retriever or context assembler with an approved local component. Record its exact version, retain permission filtering before retrieval, and rerun every acceptance gate. Add failure cases before adding performance claims.
Deliverables
Submit the validation report, one allowed and one denied query evidence record, any modified policy or corpus versions, test output, known limitations, and a short reproduction log. A second operator must be able to follow the log without relying on undocumented steps before the lab can move from prototype to reproducible.