Distributed Cognition: A Sovereign Agentic AI on a 4 GB Consumer GPU

Abstract

We present ORDIS, a sovereign, API-independent AI assistant and agent that runs on a single consumer laptop (NVIDIA RTX 3050 Laptop, 4 GB VRAM; i5-11400H; 30 GB RAM). Rather than scaling parameters in one model, ORDIS treats capability as an emergent property of a system: a single local core (4-bit, GPU-layer-offloaded) handles interactive language and control; a registry of real tools, retrieval-augmented memory, and a difficulty-aware router supply everything the core cannot compute itself; and an optional external expert is consulted only for the heaviest architecture/verification problems the interactive loop never depends on. We report measured inference throughput on the deployment hardware, a reproducible meaning-fidelity evaluation (deterministic rule-flags + a temperature-0 LLM judge), and a live, publicly testable product built on the same stack. We are explicit about structural limits — including a prior tiered ("deep-think tank") design that was measured, deployed, and later removed after live telemetry showed it caused GPU contention that degraded the interactive loop (§8). This is a systems demonstration with real measurements, not a benchmark-beating claim.

1Introduction

Capable agentic assistants are typically delivered by very large (70B–405B+) models hosted in the cloud. This imposes three costs: infrastructure and energy overhead, data egress (user data leaves the device), and recurring per-token billing. For a large class of personal and small-business tasks, these costs are disproportionate.

We ask a different question: how much useful, honest autonomy is achievable on fixed, modest, offline hardware if capability is not forced entirely into one model's weights? ORDIS is our answer — an existence proof that a single locally-served model, anchored by real tools, memory, and disciplined routing, can drive a genuinely useful agentic system whose design decisions can be measured rather than asserted — including the decision to remove a tier that measurement showed was not earning its cost.

Contributions

1. A concrete distributed-cognition architecture for sovereign edge AI, and its measured operating envelope on 4 GB.
2. A self-extension pipeline gated by anti-theater verification — synthesized tools must survive syntax, sandbox, and a placeholder / fake-success rejection filter before registration.
3. Grounded self-perception — a runtime-telemetry path that lets a small model answer questions about its own architecture and load without fabricating specifications.
4. A human-in-the-loop co-development methodology (“professor–apprentice”) in which an LLM architect and a human operator iteratively strip non-functional (“theater”) subsystems — and roll back architectural tiers that live telemetry proves are net-negative (§8).

2Background & Motivation

The deployment target is a single Ubuntu 26.04 workstation with an RTX 3050 Laptop GPU (4 GB VRAM). The design constraint is absolute: the system must remain useful offline, with no external inference API, and with no user data leaving the machine. Naively, a 4 GB budget suggests models beyond ~4B parameters are ruled out at 4-bit quantization. In practice, partial GPU-layer offload (§5) lets a larger model run with only a fraction of its layers GPU-resident and the remainder on CPU/RAM — trading throughput for capacity. The engineering problem is therefore not “which giant model,” but “how to allocate 4 GB, and how much throughput to trade for capability.”

3Architecture: Distributed Cognition

Capability is composed, not monolithic — one core model plus tools, memory, and a router:

Core. A single locally-served ~12B instruct model, 4-bit, with a deliberately tuned fraction of its layers offloaded to the GPU (§5) — the rest run on CPU/RAM. Handles dialogue, intent, and control. It does not compute what it can delegate; arithmetic, retrieval, I/O and code execution are pushed to tools.

External expert (heaviest only, optional). For very large refactors or from-scratch architecture, the router may consult an external frontier model when one is configured — the single deliberate concession to non-sovereignty, gated to the heaviest tasks; the interactive loop never depends on it, and honestly reports when none is available rather than degrading silently.

Tools & memory. A live registry of executable capabilities (filesystem, shell, browser control, web crawl, PDF, OSINT, finance, content generation) returns ground truth; a vector store on CPU/disk holds long-term memory so persistence does not consume scarce VRAM or context.

Difficulty-aware router. Each request is graded: trivial → core answers directly; very-hard architecture/refactor → external expert when available; action-requiring → agent loop. Chat stays responsive because the core never blocks waiting on a network call.

An earlier iteration of this architecture also ran a second, heavier local model (a “deep-think tank”, ~14B, invoked asynchronously) as an intermediate tier between the core and the external expert. It was measured, deployed, and later removed — not because the idea was wrong in principle, but because live production telemetry showed it competing for the same single GPU with the interactive core, degrading real chat latency during ordinary use. See §8 for the measured failure mode. We report this because a negative, measured result is still a result.

4Key Mechanisms

4.1 · Self-extension with an anti-theater verification gate

The system can synthesize new tools from a natural-language capability description. Generated code passes through (i) syntax validation, (ii) sandbox execution, and (iii) an anti-theater gate that rejects code exhibiting simulated success — placeholder markers, or a hard-coded “completed successfully” return with no real I/O, network, or computation. Only code surviving all three is registered. The gate exists because sandbox-callability alone is insufficient: a stub that returns a success string is callable yet useless.

4.2 · Grounded self-perception (anti-hallucination)

Small models asked to describe their own infrastructure tend to confabulate plausible-but-false stacks (e.g., inventing an unrelated framework or dependency). ORDIS instead answers such questions from live telemetry — querying its inference server, GPU, error log, and tool registry — and defaults to “I need to check my logs” under uncertainty. This converts a generative task (prone to hallucination) into a retrieval task over ground truth.

4.3 · Professor–apprentice co-development

Development proceeds as an iterative loop: an LLM architect proposes and audits; the human operator directs and adjudicates; non-functional subsystems (“theater”) are detected and either converted to real implementations or removed on a recurring cadence. This is a methodology claim, not a performance claim.

5Implementation OBSERVED IN REPOSITORY

6Measured Inference Throughput ✓ MEASURED ON THIS MACHINE

Single-stream generation, measured live against the deployed ollama endpoint on the deployment hardware, with 18 of 48 transformer layers offloaded to the 4 GB GPU (the rest run on CPU/RAM — VRAM is the binding constraint).

Finding. Throughput on this box is dominated by contention, not just layer-count: the same model/offload configuration swings roughly ±40% depending on what else the system is doing on the single GPU at that instant. This is the direct, measured consequence of the tiering failure described in §8 — it is also why the earlier 4-tier throughput comparison (4B/8B/14B/32B, superseded by this migration) is not reproduced here as a like-for-like number: those figures described a model family and offload ladder that is no longer deployed. We report what is currently live rather than retain a comparison that would be accurate history but a misleading present-tense claim.

7Evaluation

Feasibility. The full stack (interactive ~12B core + runtime tool registry + RAG + self-diagnostics) runs within 4 GB VRAM / 30 GB RAM, offline for the core loop. Cost / sovereignty. $0 marginal inference cost for the core; no data egress from the local core loop. The essential agent loop remains operational without the optional external-expert escalation path.

Meaning-fidelity harness (reproducible). Persona and loss curves miss semantic drift — a model can be fluent yet misread. We score responses to a held-out set of adversarial prompts (idioms, polysemy, negation-scope, referent tracking) with two layers: a deterministic rule-flag layer (formality leaks, calque, theater, identity confusion) and an impartial LLM judge at temperature 0 scoring meaning-fidelity and fluency 0–5. Because judge and prompts are fixed, iterations are directly comparable. Historical baseline (prior 4B-family model, now superseded): mean meaning-fidelity 1.5/5, fluency 2.68/5, 0 rule-flags — surface-clean but semantically weak, exactly the failure the harness exposes. The harness has not yet been re-run end-to-end against the current ~12B core; we report this gap explicitly rather than imply a re-measurement that has not happened. The harness ships with the system precisely so each future iteration is directly comparable to this baseline.

A falsifiable, live artifact. The same sovereign stack hosts a live public micro-service — an offline secret/PII scanner — at gaia-api.soundbygaia.com. A reader can POST /scan a snippet and receive structured findings, verifying that the described stack produces a real, running product. Part of this paper is testable by anyone, right now.

8Limitations (stated plainly)

• Reasoning depth of the core. A ~12B model at 4-bit cannot match frontier (70B–405B) zero-shot abstract reasoning; hard problems are routed to an external expert when one is configured — otherwise the system says so rather than guessing, and stays local (seconds-to-minutes when it does escalate).
• Non-sovereign escape hatch. The heaviest-task path may consult an external model — a deliberate, bounded concession, not used by the interactive loop.
• Semantic ceiling. The ~12B core still misses some polysemy/idiom traps; the meaning harness treats this as a first-class, tracked target rather than a footnote — though the current core has not yet been re-scored on it (§7).
• Tiering is not automatically a win. A prior second-model “deep-think tank” tier was built, measured, and later removed: it was correct in isolation but competed for the same single GPU as the interactive core in production, degrading real chat latency during ordinary use. We keep this in the paper as a documented negative result, not a footnote to omit.
• Single hardware / single operator. Throughput numbers are specific to this box and not yet reproduced across configurations.
• No external leaderboard yet. Claims concern measured system properties and relative iteration, not comparative accuracy against published models.

9Discussion & Future Work

Under hard resource and sovereignty constraints, architecture and measured model-selection can substitute for raw parameters across a meaningful band of tasks — but only when each added layer of architecture is itself held to the same measurement standard as the base system; tiering that is not verified under real, concurrent production load can quietly cost more than it returns. Future work: (1) a reproducible tool-success and task-completion harness; (2) cross-hardware reproduction of the throughput numbers; (3) re-running the meaning-fidelity harness against the current core and publishing the full score history across iterations; (4) formalizing the anti-theater gate and model-agnostic self-perception as reusable components.

10Conclusion

ORDIS is an engineered demonstration that useful, sovereign, self-extending agentic AI is achievable on a 4 GB consumer laptop — provided intelligence is treated as a property of a system (model, tools, memory, routing), model and architecture choices are measured against the hardware rather than assumed, and the system is held to an honesty standard: every capability maps to something real, self-reports come from measurement, architecture that measurement disproves gets removed rather than kept for appearances, and part of the result is publicly falsifiable.

References

This whitepaper is written to be falsifiable. The system is open to inspection; its resource ledger, hardware limits, and failure modes are stated so a reader can verify or refute them directly.