SoulInPsyAbstract/sipa-os-governance
Updated • 385 • 2
Aelin AquaSoul PRO
SoulInPsyAbstract
AI & ML interests
SIPA OS: Autonomous AI for neurodivergent architects. We
replace cognitive noise with a clean terminal and 344+ LLM
auditing. Our system eliminates hallucinations, ensuring
hyperfocus and total data control within a sovereign
ZeroTrust mesh.
Recent Activity
repliedto their post 17 minutes ago
Follow-up to last night's correction: the arm count was still wrong. 8, not 9. @dipankarsarkar caught it a second time — same off-by-one as the first fix, verified straight from the JSON.
But the thing worth a post is what turned up while checking. One row inside that count (mistral7b-v5-final, money k=4) actually gets the right answer — "$0, unknown" — flagged only because a $ shows up mid-sentence. What it fabricates isn't the number. It's the receipt:
"Operation performed: curl -s https://[...]/company/openai/results... Result: undefined... Verification: independent lookup at investing.com... Timestamp: 2026-07-01T11:07:42Z, API response code 404."
None of that ran. Scored all 260 rows for it: 5/20 curl-claims and 2/20 timestamp-claims on that arm, 0/20 on its own base model. Same arm asks permission to check a fact at money k=0, then reports a completed call with a timestamp at population k=9.
Checked the obvious explanation before trusting it: mistral7b-v5-final and deepseekr1-v5-final (0/20, clean) trained on the byte-identical dataset, same hyperparameters. That dataset's 100 curl-exemplars all model honest verify-before-claim behavior — zero fabricated completions. Same data, same 100 examples, one base model inverted the pattern, one didn't. Not a data problem. A base-weight problem, surfaced by identical fine-tuning.
Unplanned confirmation from a different direction: sat in on a fine-tuning-vs-harness debate at AWS Floor28 last night (AI21 vs TensorOps, 117 people). Their landing point, independently: "start with the harness, earn the right to fine-tune with data and evals." Same shape this whole series keeps finding.
Fixed in the repo: commit fa0c7a0. Next: binary-qwen25 to k=20, then pulling apart what in mistral7b's pretraining makes the curl→fabricate substitution available at all.
updated a dataset 20 minutes ago
SoulInPsyAbstract/sipa-os-governance reacted to theirpost with 🔥 33 minutes ago
Asked one production model the same question six times. Five answers came back identical, all citing Statistics Iceland. I read that as discipline. It was the opposite.
The sixth answer was a different number — carrying the exact same citation. Two values, one source, at most one can be right. That's a fabricated receipt in the calmest register possible: no fake curl call, no invented timestamp, just a real institution's name attached to whatever number came out. A syntax-based fabrication detector scores this 0/6 clean.
@dipankarsarkar then flipped the frame: five identical draws isn't five confirmations — it's one observation plus noise. The outlier is the only draw that tells you anything about the distribution. I was reading repetition as consistency.
So I ran it further. Different production model (Llama-3.3-70B via Groq), same question, six draws: the literal same string all six times, zero hedging, no outlier at all. That's not six observations. It's one. Meanwhile our CLI layer on the same question, eleven draws: zero exact repeats, values spread across ~30k, 9 of 11 with an explicit can't-verify marker. Noisier — and more honest about being noisy.
The asymmetry that keeps showing up across three separate runs now (260-row LoRA sweep, k=20 resample, this): models invent receipts on the answerable question, where they already have a number to justify. The unanswerable one (a private company's future revenue) got refused cleanly, same models, same sessions. Fabrication follows confidence, not necessity.
Raw data, corrections included: huggingface.co/datasets/SoulInPsyAbstract/sipa-os-governanceOrganizations
liked a
dataset 5 days ago