Gautama-AI

Executive Summary

Gautama-AI proposes to train a complete 3.33-billion-parameter Vision-Language Model from scratch — a GPT-style language model that also sees and understands images — on a single NVIDIA H100 GPU, in 89 days. The result is a fully open model, tokenizer, dataset recipe, and training pipeline that anyone can reproduce, study, and build on.

Frontier VLMs cost millions of dollars and thousands of GPUs to train. This project demonstrates that a genuinely capable multimodal model — image captioning, visual question answering, chart reading, document understanding, and 16K-token long-context reasoning — is achievable at hobbyist-accessible budgets by combining a carefully engineered architecture with cutting-edge training efficiency research, including Lighthouse Attention (Nous Research, 2026) for long-context training.

Why it matters: every design decision, cost figure, and training hour in this proposal is pre-computed and verified against arithmetic derived from the model code — this is not an estimate, it is an engineering plan.

What We Are Building

🗣 Language Backbone — 3.015B

GPT-style decoder-only Transformer: 28 layers, 3072-dim, grouped-query attention, RoPE, SwiGLU, RMSNorm. Same architecture family as Llama — fully compatible with the open-source ecosystem.

👁 Vision Encoder — 303M

CLIP ViT-L/14 encodes any image into 257 visual tokens. The language model attends to these tokens exactly like text — one unified sequence, one unified model.

🌉 Vision Projector — 12.6M

A compact 2-layer MLP that translates vision features into the language model's embedding space — the bridge that turns a GPT into a VLM.

📏 16K Long Context

Lighthouse Attention — a 2026 hierarchical attention method — extends the model from 4K to 16K tokens with full dense-attention quality recovered before release.

One model, two modes

With no image, Gautama-VLM behaves as a pure GPT: completion, Q&A, reasoning, code. With an image, it becomes a visual assistant: describe, answer, read charts, extract text. The same 3.33B weights serve both.

Technical Innovation

Filling every byte of the GPU

The H100's 80 GB is used at 85% during peak training — achieved through fp32-master AdamW, activation checkpointing, and chunked cross-entropy (which cuts the loss computation's memory peak from 9.8 GB to 0.6 GB). A smaller model would waste the hardware; a larger one would not fit. 3B is the engineered sweet spot.

Lighthouse Attention for long-context training

Standard attention cost grows quadratically with context length — the main reason long-context training is expensive. Lighthouse Attention (Peng, Ghosh & Quesnelle, Nous Research, arXiv:2605.06554) pools queries, keys, and values into a multi-resolution pyramid, selects the most important entries with a parameter-free scorer, and runs standard FlashAttention on the compressed sequence — then a brief dense-attention resumption restores full quality. The paper demonstrates 1.4–1.7× faster training at equal or better final loss.

Our integration: we computed the exact break-even point (8,192 tokens) and apply Lighthouse only where it wins — a dedicated 16K long-context extension phase (L=3 levels, p=2 pooling, k=1536 selection budget, dilated scorer), with the first and last two layers kept dense for stability. The delivered model uses standard attention at inference: Lighthouse is a training accelerator, not a dependency.

Verified engineering, not guesswork

Training sub-phase	Tokens	GPU hours
D1 · Text pretraining (seq 4K)	30B	378.3
D2 · Long-context extension @16K (Lighthouse)	5B	54
D3 · Vision alignment (projector)	0.5B	2.1
D4 · VLM instruction tuning	5B	68.8
D5 · DPO preference alignment	0.2B	2.8
Spot interruption buffer	—	60
Total	40.7B	566

Throughput 22,026 tok/s at 45% MFU on H100 SXM5 (989 TFLOPS BF16) — all figures derived from the project's own planning calculator and cross-checked against the model code.

Project Plan - Aug 3 → Oct 31, 2026

Aug 3–7Phase A · Scoping & Design
Architecture lock, data specification, evaluation targets, budget sign-off. No GPU cost.
Aug 8–16Phase B · Data Curation
120 GB text corpus (filtered + deduplicated), 3.4M image-caption pairs, 615k instruction pairs, 110k preference pairs. CPU-only.
Aug 17–19Phase C · Tokenization
Custom 32k SentencePiece tokenizer with vision special tokens; all corpora sharded into memory-mapped binaries.
Aug 20 – Sep 18Phase D · Training & Alignment
Five GPU sub-phases on a single H100 (21.1 GPU-days) with automated checkpoint/resume, per-phase quality gates, and a 7-day contingency buffer.
Sep 19–25Phase E · Evaluation
Language (HellaSwag, MMLU), long-context (needle-in-haystack @16K), vision (VQAv2, GQA, TextVQA, ChartQA), hallucination (POPE), plus human review.
Sep 26 – Oct 17Phase F · Deployment
Quantized release (2.1 GB GGUF runs on an 8 GB consumer GPU), OpenAI-compatible API server, model card, full documentation, and public release.
Oct 18 – Oct 31Phase G · Spot Interruption Buffer
14-day reserved window to absorb spot preemptions, checkpoint restarts, and any phase overruns. Ensures Oct 31 delivery regardless of cloud interruptions.

Success Metrics

Capability	Benchmark	Target
Common-sense reasoning	HellaSwag (0-shot)	≥ 62%
World knowledge	MMLU (5-shot)	≥ 46%
Long-context recall @16K	Needle-in-a-Haystack	≥ 70%
Visual question answering	VQAv2 (test-dev)	≥ 60%
Visual reasoning	GQA	≥ 58%
Text-in-image reading	TextVQA	≥ 45%
Hallucination control	POPE (F1)	≥ 80%
Deployment footprint	GGUF q4_k_m size	≤ 2.5 GB

Honest framing: at 3.33B parameters, Gautama-VLM will not match 7B+ commercial VLMs. It will demonstrate — reproducibly and openly — that first-generation-LLaVA-class multimodal capability is achievable on a single H100 within 90 days, and every artifact of the process will be public.