Practice

The compact edge:
27B and the small sizes

3.8 27B dense 0.6 — 8B int4

In 3.8 the compact model is the open 27B that accepts images and video. Below it in size come the earlier generations: they are what covers data extraction on a machine without an accelerator.

16 GB of weights for 27B at int4
0.6 GB per 1B at int4
1M context in 3.8

Two different kinds of "small"

In 3.8 the word "compact" means 27 billion parameters: a dense model where every weight works on every step. It accepts images and video, holds a million tokens of context and — most importantly for private work — is open. It is the one worth putting on your own server when documents cannot leave the building.

Below 27B a different territory begins: models of 0.6–8 billion from earlier generations. They do not compete with the flagship on cleverness, but they run on a CPU, fit in a laptop's memory and cost nothing. For field extraction, classification and routing that is enough — and that is the class we measure.

The choice is not between "smarter" and "dumber" but between "fits my perimeter" and "does not".
Class

The size ladder and memory

A simple rule of thumb: about 2 GB per billion parameters at fp16 and 0.6 GB at int4. The second figure is what decides what runs on your machine.

Model Parameters fp16 int4 Role
qwen3-0.6b0.6B1.2 GB0.4 GBFilters, tags, request routing
qwen3-1.7b1.7B3.4 GB1.1 GBField extraction, classification
qwen3-4b4B8.0 GB2.6 GBThe working point: schema JSON
qwen3-8b8B16.0 GB5.0 GBComplex schemas, long context
qwen3.8-27b27B54.0 GB16.0 GBVision, documents, private perimeter

Plus 1–2 GB for context and cache: a long document eats memory more noticeably than the weights themselves. For 27B that already means server-grade memory.

Measurements

Schema extraction: what comes out in practice

Our test: 400 Russian-language documents of four types, a five-field schema, temperature 0, output constrained by a grammar. The machine is eight cores with no accelerator and 32 GB of memory; for 27B we added runs on a single card.

Model int4 memory JSON, % Fields, % tok/s
qwen3-0.6b 0.4 GB 88 71 42
qwen3-1.7b 1.1 GB 96 84 24
qwen3-4b 2.6 GB 99 91 12
qwen3-8b 5.0 GB 99 94 7
qwen3.8-27b 16.0 GB 100 97 38

"JSON, %" is the share of responses that passed the schema on the first try. "Fields, %" is an exact match across all five fields. The gap between the columns is the price of size: everyone holds the syntax, but the small ones trip on unusual layouts.

27B was measured on a single card — hence the high throughput. The numbers give an order of magnitude on our hardware, not a reproducible benchmark: run them on your own documents.

What a compact model does well

  • flat extraction — identifiers, dates, amounts, numbers, contracting parties: one schema, one pass, a stable result;
  • normalisation — mapping addresses, units and names onto a reference dictionary;
  • classification and routing — identify the document type and send it to the right pipeline before calling a heavy model;
  • scan handling — this is where 27B wins: it takes the page as an image and does not depend on an external OCR;
  • pre-labelling — draft entities and relations for later human review.

The common thread: the answer can be verified formally. Where verification requires understanding meaning, a compact model starts guessing — and that is no longer its job.

Techniques

Five things that raise quality

  1. A schema instead of a request. Not "extract the data" but a concrete JSON with types and required fields. Grammar-constrained generation removes a whole class of errors.
  2. Three examples, not thirty. On small models few-shot works up to a point: beyond it the examples start competing with the instruction for attention.
  3. One document, one call. Gluing several documents into one prompt almost always lowers extraction accuracy.
  4. Validation and one retry. A response that failed the schema goes back with the error text — that is enough for models from 1.7B up.
  5. Division of labour. The cheap model handles everything, the large one only the disputed records. Cost falls faster than quality.

The schema for a supply contract looks roughly like this:

schema.json { "type": "object", "required": ["number", "date", "amount"], "properties": { "number": { "type": "string", "pattern": "^[0-9]{4}-[0-9]{6}$" }, "date": { "type": "string", "format": "date" }, "amount": { "type": "number", "minimum": 0 } }, "additionalProperties": false }
Use cases

Where this fits end to end

Mail and attachment triage

offline

A model of 1.7–4B sorts incoming mail by type, pulls the identifiers and drops them into a table. Nothing leaves the building — this scenario is usually where the choice is settled.

Scans without external OCR

vision

A job for 27B: the page is fed in as an image and the model finds the table and the stamps itself. Smaller sizes only work here with text that has already been recognised.

Preprocessing before search

RAG

A compact model cleans and chunks documents, pulls headings and metadata. That is cheaper than running a large model over every paragraph of a corpus.

A gateway to the large model

routing

Half of all requests need nothing complicated. The small model answers on its own and forwards only what genuinely requires reasoning.

What not to expect from compact models

  • long reasoning chains — models up to 4B lose the thread of a task after about the third step;
  • world knowledge — rare facts, brands and narrow terminology have to be supplied in context; the model does not remember them;
  • 27B speed on a CPU — without an accelerator the model runs, but batch processing thousands of documents becomes a night job;
  • arithmetic over large tables — totals and reconciliations are better computed in code, with the model used for matching.

But anything that can be described by a schema and checked by a rule, a compact model does predictably and cheaply — and that is exactly where it pays for itself fastest.

Questions

Common questions about compact models

If the documents are text and the schema is simple, 8B at int4 in 5 GB of memory gives nearly the same result. 27B is chosen for scans, images and very long documents: that is where the accuracy gap becomes noticeable.
For uniform documents — yes, if the answer is constrained by a schema and there are two or three examples. The drop starts on unusual layouts: there a 4B model, or a retry carrying the error text, will help.
For 0.6–8B it is optional: 4B at int4 gives about a dozen tokens per second on eight cores. For 27B an accelerator is practically required, otherwise batch processing stretches into the night.
In money terms, often yes, and it is worth remembering honestly. You run your own perimeter not for the price per token but because the data must not leave it. If that is not a requirement, a hosted service is almost always better value.