# Run a language model from C

Load a Hugging Face model folder and generate text from a C program with the Kwker Runtime - no Python and no PyTorch at run time.

Web page: https://kwker.io/docs/?p=hf-c - every Kwker documentation page for AI assistants: https://kwker.io/llms.txt

The Kwker Runtime (`libkwker_rt`) runs KwkDecoder's models from any program that can call C. Give it a Hugging Face
model folder and a precision. It reads `config.json`, the safetensors files and `tokenizer.json`, and gives you a
tokenizer and a decoder: text in, text out. The logits are the same as KwkDecoder's in Python, and the token ids the same
as Hugging Face's `tokenizers`.

It loads every model family KwkDecoder runs: Llama, Mistral, Qwen2, Qwen3, SmolLM2, SmolLM3, Phi-3,
Phi-3.5, Phi-4-mini, Granite, Granite MoE, Helium, Gemma 2, Gemma 3, Mixtral, Qwen3-MoE, GPT-2, GPT-NeoX / Pythia, Phi-1.5, Phi-2, OPT,
StableLM, StarCoder2, OLMo and GPTBigCode (SantaCoder, StarCoder). A model KwkDecoder cannot run is refused with the reason.

## What you need

- The library and its header. The Python package ships both: `libkwker_rt.so` (`kwker_rt.dll` on Windows,
  `libkwker_rt.dylib` on macOS) next to the `kwker` package, and `kwker/include/kwker_rt.h`. This prints the
  library's path:

  ```sh
  python -c "import kwker.onnx; print(kwker.onnx.library_path())"
  ```

- A model folder with `config.json`, `tokenizer.json` and `model.safetensors` (or its shards and
  `model.safetensors.index.json`), for example a downloaded Hugging Face snapshot.

## The program

The program loads the tokenizer and the model, turns the prompt into token ids, generates tokens with one call until an
end-of-sequence token, and prints them as text. It decodes greedily; set `KWKER_TEMPERATURE` to sample instead.

```c
// Text generation with the Kwker Runtime from C: load a Hugging Face model folder (weights and tokenizer), then continue a
// prompt until an end-of-sequence token. No Python, no PyTorch.
//
//   rt_generate <model folder> <precision> <new tokens> <prompt>
//
// precision is "int4", "int8", "bf16" or "float32". The program prints the generated text - at most <new tokens> tokens.
// Greedy by default; set KWKER_TEMPERATURE (and optionally KWKER_TOP_P, KWKER_SEED) to sample instead.
//
// Build: cc rt_generate.c -I<kwker include dir> -L<lib dir> -lkwker_rt -o rt_generate
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include "kwker_rt.h"

static double env_or(const char* name, double fallback) {
    const char* v = getenv(name);
    return v ? atof(v) : fallback;
}

int main(int argc, char** argv) {
    if (argc != 5) {
        fprintf(stderr, "usage: %s <model folder> <precision> <new tokens> <prompt>\n", argv[0]);
        return 2;
    }
    const char* dir = argv[1];
    const char* precision = argv[2];
    const int64_t n_new = atoll(argv[3]);
    const char* text = argv[4];

    // The tokenizer: the prompt's text as token ids, with the model's start token if it uses one.
    kwker_rt_tokenizer* tok = NULL;
    if (kwker_rt_tokenizer_load(dir, &tok) != 0) {
        fprintf(stderr, "tokenizer: %s\n", kwker_rt_tokenizer_error());
        return 1;
    }
    int64_t n_prompt = 0;
    kwker_rt_tokenizer_encode(tok, text, -1, 1, NULL, 0, &n_prompt);
    int64_t* prompt = malloc(sizeof(int64_t) * (n_prompt > 0 ? n_prompt : 1));
    kwker_rt_tokenizer_encode(tok, text, -1, 1, prompt, n_prompt, &n_prompt);

    // The model: room for the prompt plus the new tokens, one sequence.
    kwker_rt_decoder* d = NULL;
    if (kwker_rt_decoder_load(dir, precision, n_prompt + n_new, 1, &d) != 0) {
        fprintf(stderr, "load failed: %s\n", kwker_rt_decoder_error());
        return 1;
    }

    // The model's end-of-sequence ids: generation stops at the first one.
    int64_t eos[8];
    int64_t n_eos = kwker_rt_decoder_eos(d, eos, 8);
    if (n_eos > 8) n_eos = 8;

    // Temperature 0 is greedy decoding; top-k 0, top-p 1, min-p 0 and repetition penalty 1 turn those off.
    const double temperature = env_or("KWKER_TEMPERATURE", 0.0);
    const double top_p = env_or("KWKER_TOP_P", 1.0);
    const uint64_t seed = (uint64_t)env_or("KWKER_SEED", 0);

    int64_t* out = malloc(sizeof(int64_t) * (n_new > 0 ? n_new : 1));
    int64_t n_out = 0;
    if (kwker_rt_decoder_generate(d, prompt, n_prompt, n_new, eos, n_eos, 0, temperature, 0, top_p, 0.0, 1.0, seed, out, &n_out) != 0) {
        fprintf(stderr, "generate failed: %s\n", kwker_rt_decoder_error());
        return 1;
    }

    // The new tokens as text, without special tokens such as the end-of-sequence one.
    int64_t len = 0;
    kwker_rt_tokenizer_decode(tok, out, n_out, 1, NULL, 0, &len);
    char* reply = malloc((size_t)len + 1);
    kwker_rt_tokenizer_decode(tok, out, n_out, 1, reply, len + 1, &len);
    printf("%s\n", reply);

    kwker_rt_decoder_free(d);
    kwker_rt_tokenizer_free(tok);
    free(reply);
    free(out);
    free(prompt);
    return 0;
}
```

Build it against the library, then run it on a model folder with a prompt:

```sh
cc rt_generate.c -I<include dir> -L<library dir> -lkwker_rt -Wl,-rpath,<library dir> -o rt_generate
./rt_generate ~/models/SmolLM2-135M int4 16 "The capital of France is"
```

It prints the 16 new tokens as text:

```text
 the city of Paris. It is the largest city in France and the capital of
```

## Sample instead

Set a temperature to sample the tokens. `KWKER_TOP_P` keeps only the most likely tokens that together reach that
probability; `KWKER_SEED` picks the random draws, so the same seed prints the same text every time:

```sh
KWKER_TEMPERATURE=0.8 KWKER_TOP_P=0.9 KWKER_SEED=42 ./rt_generate ~/models/SmolLM2-135M int4 16 "The capital of France is"
```

```text
 the name of the palace. It is also the seat of the president of the
```

The settings work as in Hugging Face's `generate`, and sampling follows the same rules as
`KwkDecoder.generate(do_sample=True)` in Python.

## Chat with an instruct model

An instruct model expects its conversation in the format of its chat template. `kwker_rt_tokenizer_chat` renders the
messages with the model's own template, so the prompt is the same text as `apply_chat_template` in transformers. This
program asks one question, with an optional system prompt, and prints the reply:

```c
// Chat with an instruct model through the Kwker Runtime from C: the question (and an optional system prompt) become the
// model's prompt through its own chat template, the model writes its reply. No Python, no PyTorch.
//
//   rt_chat <model folder> <precision> <new tokens> <question> [<system prompt>]
//
// precision is "int4", "int8", "bf16" or "float32". The program prints the reply - at most <new tokens> tokens, greedy.
//
// Build: cc rt_chat.c -I<kwker include dir> -L<lib dir> -lkwker_rt -o rt_chat
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include "kwker_rt.h"

// Append s to out as a JSON string (quotes included); out has room for 6 bytes per byte of s plus 3.
static char* json_string(char* out, const char* s) {
    *out++ = '"';
    for (; *s; s++) {
        unsigned char c = (unsigned char)*s;
        if (c == '"' || c == '\\') {
            *out++ = '\\';
            *out++ = (char)c;
        } else if (c < 0x20) {
            out += sprintf(out, "\\u%04x", c);
        } else {
            *out++ = (char)c;
        }
    }
    *out++ = '"';
    *out = 0;
    return out;
}

int main(int argc, char** argv) {
    if (argc != 5 && argc != 6) {
        fprintf(stderr, "usage: %s <model folder> <precision> <new tokens> <question> [<system prompt>]\n", argv[0]);
        return 2;
    }
    const char* dir = argv[1];
    const char* precision = argv[2];
    const int64_t n_new = atoll(argv[3]);
    const char* question = argv[4];
    const char* system = argc == 6 ? argv[5] : NULL;

    // The conversation as a JSON array of messages: [{"role": "system", ...}, {"role": "user", "content": "..."}].
    char* messages = malloc(64 + 6 * (strlen(question) + (system ? strlen(system) : 0)));
    char* p = messages;
    p += sprintf(p, "[");
    if (system) {
        p += sprintf(p, "{\"role\": \"system\", \"content\": ");
        p = json_string(p, system);
        p += sprintf(p, "}, ");
    }
    p += sprintf(p, "{\"role\": \"user\", \"content\": ");
    p = json_string(p, question);
    sprintf(p, "}]");

    kwker_rt_tokenizer* tok = NULL;
    if (kwker_rt_tokenizer_load(dir, &tok) != 0) {
        fprintf(stderr, "tokenizer: %s\n", kwker_rt_tokenizer_error());
        return 1;
    }

    // The prompt text from the model's chat template, ending where the assistant's reply starts.
    int64_t len = 0;
    if (kwker_rt_tokenizer_chat(tok, messages, 1, NULL, NULL, 0, &len) != 0) {
        fprintf(stderr, "chat template: %s\n", kwker_rt_tokenizer_error());
        return 1;
    }
    char* text = malloc((size_t)len + 1);
    kwker_rt_tokenizer_chat(tok, messages, 1, NULL, text, len + 1, &len);

    // The prompt's token ids. The template already holds the model's special tokens, so none are added.
    int64_t n_prompt = 0;
    kwker_rt_tokenizer_encode(tok, text, len, 0, NULL, 0, &n_prompt);
    int64_t* prompt = malloc(sizeof(int64_t) * (n_prompt > 0 ? n_prompt : 1));
    kwker_rt_tokenizer_encode(tok, text, len, 0, prompt, n_prompt, &n_prompt);

    kwker_rt_decoder* d = NULL;
    if (kwker_rt_decoder_load(dir, precision, n_prompt + n_new, 1, &d) != 0) {
        fprintf(stderr, "load failed: %s\n", kwker_rt_decoder_error());
        return 1;
    }
    int64_t eos[8];
    int64_t n_eos = kwker_rt_decoder_eos(d, eos, 8);
    if (n_eos > 8) n_eos = 8;

    // Greedy: temperature 0, the sampling settings off.
    int64_t* out = malloc(sizeof(int64_t) * (n_new > 0 ? n_new : 1));
    int64_t n_out = 0;
    if (kwker_rt_decoder_generate(d, prompt, n_prompt, n_new, eos, n_eos, 0, 0.0, 0, 1.0, 0.0, 1.0, 0, out, &n_out) != 0) {
        fprintf(stderr, "generate failed: %s\n", kwker_rt_decoder_error());
        return 1;
    }

    // The reply as text, without the end-of-turn token.
    kwker_rt_tokenizer_decode(tok, out, n_out, 1, NULL, 0, &len);
    char* reply = malloc((size_t)len + 1);
    kwker_rt_tokenizer_decode(tok, out, n_out, 1, reply, len + 1, &len);
    printf("%s\n", reply);

    kwker_rt_decoder_free(d);
    kwker_rt_tokenizer_free(tok);
    free(reply);
    free(out);
    free(prompt);
    free(text);
    free(messages);
    return 0;
}
```

Build it the same way and ask the instruct model a question:

```sh
cc rt_chat.c -I<include dir> -L<library dir> -lkwker_rt -Wl,-rpath,<library dir> -o rt_chat
./rt_chat ~/models/SmolLM2-135M-Instruct int8 24 "What is the capital of France?"
```

It prints the reply:

```text
The capital of France is Paris.
```

## The calls

The tokenizer:

| Call | What it does |
|---|---|
| `kwker_rt_tokenizer_load(path, &t)` | Load `tokenizer.json` from a model folder (or the file itself). |
| `kwker_rt_tokenizer_encode(t, text, len, add_special, ids, cap, &n)` | UTF-8 text to token ids: up to `cap` of them into `ids`, `n` is how many there are. `len` -1 reads up to the terminating 0. `add_special` 1 adds the model's special tokens, such as Llama's start token. Call with `cap` 0 to size the buffer. |
| `kwker_rt_tokenizer_decode(t, ids, n, skip_special, text, cap, &len)` | Token ids to UTF-8 text with a terminating 0; `len` is the full length. `skip_special` 1 leaves special tokens out. Call with `cap` 0 to size the buffer. |
| `kwker_rt_tokenizer_chat(t, messages, add_generation_prompt, options, text, cap, &len)` | Chat messages (a JSON array such as `[{"role": "user", "content": "Hi"}]`) to the prompt text the model expects, with its own chat template: the same text as `apply_chat_template` in transformers. `add_generation_prompt` 1 ends with the start of the assistant's turn. `options` is NULL or a JSON object of template variables, such as `{"enable_thinking": false}`. Encode the result with `add_special` 0. |
| `kwker_rt_tokenizer_token_id(t, token)` | A token's id, such as `"<|im_start|>"`, or -1. |
| `kwker_rt_tokenizer_vocab(t)` | The number of token ids. |
| `kwker_rt_tokenizer_free(t)` | Release the tokenizer. |
| `kwker_rt_tokenizer_error()` | The reason the last tokenizer call on this thread returned -1. |

The decoder:

| Call | What it does |
|---|---|
| `kwker_rt_decoder_load(dir, precision, max_cache_len, max_batch, &d)` | Load a model folder. `precision` is `"int4"`, `"int8"`, `"bf16"` or `"float32"`; `max_cache_len` is the most positions a sequence holds; `max_batch` the number of sequences. |
| `kwker_rt_decoder_load_calibrated(dir, "int4", max_cache_len, max_batch, calib, n_calib, calib_ctx, &d)` | Load in int4 with the weights fitted to sample text: `calib` holds `n_calib` token ids (a few thousand tokens of text like the model's use), run in chunks of `calib_ctx` (512 is a good value). Usually a lower perplexity than plain int4, at the same speed. |
| `kwker_rt_decoder_generate(d, prompt, n_prompt, max_new, eos, neos, min_new, temperature, top_k, top_p, min_p, repetition_penalty, seed, out, &nout)` | Generate up to `max_new` tokens after a prompt, stopping at the first token in `eos` (it is the last one written). `temperature` 0 decodes greedily; above 0 it samples with the other settings (`top_k` 0, `top_p` 1, `min_p` 0 and `repetition_penalty` 1 turn them off). A given `seed` gives the same tokens every time. |
| `kwker_rt_decoder_step(d, ids, n, pos, nlog, seq, logits)` | Run `n` tokens of sequence `seq` at positions `pos` to `pos + n - 1`. Writes the logits of the last `nlog` rows (all rows when `nlog` is negative), each row `vocab` floats. |
| `kwker_rt_decoder_step_batch(d, ids, n, pos, seq, nlog, logits)` | Run tokens of several sequences in one call: row `i` is token `ids[i]` of sequence `seq[i]` at position `pos[i]`. |
| `kwker_rt_decoder_greedy(d, tok, pos, n, eos, neos, min_new, out, &nout)` | Generate up to `n` tokens greedily from token `tok` at position `pos`, stopping at the first token in `eos` (it is the last one written). |
| `kwker_rt_decoder_vocab(d)` | The vocabulary size: the length of one logits row. |
| `kwker_rt_decoder_eos(d, ids, cap)` | The model's end-of-sequence ids (from `generation_config.json`, else `config.json`): up to `cap` of them into `ids`; returns how many there are. Pass them to `kwker_rt_decoder_generate` to stop where Hugging Face's `generate` stops. |
| `kwker_rt_decoder_free(d)` | Release the decoder. |
| `kwker_rt_decoder_error()` | The reason the last decoder call on this thread returned -1. |

Every call returns 0, or -1 with the reason in the matching `_error()` call.

## Notes

- `"int4"` and `"int8"` quantize the weights while loading, with KwkDecoder's defaults. `"bf16"` keeps a bfloat16
  checkpoint exactly; `"float32"` keeps a float32 checkpoint exactly.
- Mixture-of-experts models (Mixtral, Qwen3-MoE, Granite MoE) load in `"int4"` or `"int8"`.
- Rotary scaling: "default", "linear", "llama3" and "yarn" load as they are; "longrope" (Phi-3.5, Phi-4-mini) loads when
  `max_cache_len` is at most the model's original context length. Other kinds are refused with a message.
- The tokenizer reads BPE tokenizers: byte-level ones (GPT-2, Llama 3, Qwen2, SmolLM, Phi, Pythia, ModernBERT) and
  SentencePiece-style ones (Llama 2, Mistral, TinyLlama), and the WordPiece tokenizers of BERT-family encoders (MiniLM,
  bge, MPNet, BERT, nomic-embed), and Unigram tokenizers (XLM-RoBERTa, multilingual-e5, bge-m3, T5). The `truncation`
  length in `tokenizer.json` applies; its `padding` does not. It encodes text 2-5 times as fast as `tokenizers` on one
  thread. `kwker_rt_tokenizer_chat` renders the model's chat template (`tokenizer_config.json` or
  `chat_template.jinja`); a template construct it does not support is an error that names it.
- The weight files are mapped, read once and packed; the decoder keeps only its packed copy.
- The first `"int4"` or `"int8"` load of a model takes seconds: quantizing is the slow part. The packed weights are then
  saved in `~/.cache/kwker/packs` (`KWKER_PACK_CACHE` moves it, `0` turns it off; at most 16 GB, `KWKER_PACK_CACHE_GB`),
  and the next load of the same model reads them: SmolLM2-135M in int4 loads in 0.2 s instead of about 2 s. Another
  `max_cache_len` or `max_batch` uses the same file; changed weights or another Kwker build write a new one.
- `kwker_rt_decoder_load_calibrated` calibrates as `KwkDecoder(model, precision="int4", calib=ids)` does in Python. The
  two runs measure the inputs in their own float32 code, so a few weight groups can round differently: on small test
  models the logits differ by under 1% of their range (uncalibrated int4: about 8%). The first load runs the model once in float32 over the
  calibration tokens; the pack cache keeps the result, so later loads with the same tokens are as fast as plain int4 loads.
  GPT-2, GPT-NeoX, Phi, OPT and mixture-of-experts models load without calibration, as in Python.
- `seed` is the value Python's `KwkDecoder.generate(do_sample=True, generator=g)` draws first from `g` with
  `torch.randint(0, 2**62, (1,), generator=g)`: pass that value and C returns the same tokens.

## Related

- [Generate text](https://kwker.io/docs/md/hf-generate.md): the same decoder from Python.
- [int8 and int4 weights](https://kwker.io/docs/md/hf-quantize.md): what each precision costs in quality.
- [ONNX models](https://kwker.io/docs/md/onnxruntime.md): the encoder side of the Kwker Runtime, inside ONNX Runtime.