Gemma 2 · Google DeepMind · the base we fine-tune from

Gemma 2 2B base

Google's pretrained, instruction-tuned Gemma 2 2B, exactly as released. We did not pretrain or fine-tune this model. It is the foundation our Gemma legal fine-tunes start from. Ask it anything and see what the raw base already knows.

HuggingFace weights → SLM Arena 21: compare all 21 models →

2.6B

parameters

256K

vocab

8,192

context

4-bit

NF4 quantized

Google

pretrained by

Validation metrics along the Gemma 2 2B lineage

Each perplexity is measured on that stage's own validation set, so read the trend as 'how well the model fits its own stage's data', not as one curve on one dataset. DPO and RLAIF optimize preferences rather than likelihood, so they log preference margin and reward instead of perplexity. Click a stage to open that model.

Base

n/a

Google's weights, not trained by us

→

QA SFT

ppl 4.26

QA val

→

Instruct

ppl 1.90

instruction val

→

DPO

QLoRA-DPO

preference-trained, no ppl

RLAIF

loss 0.63→0.27

best-of-4 SFT, no ppl

→

RAFT on DPO

ppl 1.22

RAFT val

RAFT on RLAIF

ppl 1.25

RAFT val

ask the base model a question

ready

The base model's answer will appear here.

What this is pretrained base · not ours

This is google/gemma-2-2b-it, Google DeepMind's instruction-tuned Gemma 2 2B. Unlike our 125M and 500M models (which we trained from random weights), this one arrives already pretrained on Google's corpus and instruction-tuned to answer questions. It is the starting point, not an output, of our Gemma work.

With ~2.6B parameters it holds roughly 20x the usable knowledge of our 125M and 5x the 500M, so it answers general questions fluently out of the box. What it does not know is our legal corpus. That gap is exactly what our fine-tunes close: teaching it to recall specific holdings and filings, or to answer grounded in a retrieved passage.

Served 4-bit quantized (bitsandbytes NF4) on a single GPU, scale-to-zero, so the first question may take ~20–40s while the model wakes from idle. The same endpoint also serves our fine-tuned Gemma, which is what the comparison sites below query.

closed-book Q&A: base vs fine-tuned → legal QLoRA fine-tune → our 125M base → our 500M base →

what this model cost to build

$0 our cost to obtain this base model

Google pretrained Gemma 2 2B and released the weights, so acquiring it cost us nothing. There is no pre-training line to split here: our spend on this family only begins when we fine-tune.

That is the entire economic argument for building on an open base model. Our own 125M and 500M models each cost roughly $120 of GPU time to pretrain from scratch, and they are 16× and 4× smaller than this one. Gemma arrives already knowing far more, for free.

What we do pay for is behaviour. Fine-tuning and aligning this base came to a few dollars per model:

Model built from this base	Cost to assemble
Gemma 2 2B · QA fine-tune closed-book SFT on 7,141 QA pairs	$2.11
Gemma 2 2B · DPO QA fine-tune + direct preference optimization	$2.79
Gemma 2 2B · RLAIF QA fine-tune + reward model + best-of-N policy	$5.24

Figures are actual invoiced Modal usage (GPU + CPU + memory), not estimates. Google has not published what pretraining Gemma 2 2B cost them; whatever it was, we did not pay it. Serving is billed separately and scales to zero.