Kwker Runtime C API reference

typedef struct kwker_rt_tokenizer kwker_rt_tokenizer;

Load a tokenizer.
  path: a model folder with tokenizer.json, or the tokenizer.json file itself
  out:  receives the tokenizer; release it with kwker_rt_tokenizer_free
Returns 0, or -1 with the reason in kwker_rt_tokenizer_error() (for example a tokenizer kind it does not read).

int kwker_rt_tokenizer_load(const char* path, kwker_rt_tokenizer** out);

Build a chat prompt from messages with the model's chat template: the same text as transformers'
apply_chat_template(messages, tokenize=False). The template comes from tokenizer_config.json (chat_template) or
chat_template.jinja beside tokenizer.json.
  messages:              a JSON array of messages, such as [{"role": "user", "content": "Hi"}]
  add_generation_prompt: 1 ends the text with the start of the assistant's turn, so the model answers next
  options:               NULL, or a JSON object whose members become template variables, such as
                         {"enable_thinking": false} or {"tools": [...]}
  text, cap:             room for cap bytes: the text's first cap - 1 bytes and a terminating 0 are written
  len:                   receives the text's length in bytes. Call with cap 0 to size text.
Returns 0, or -1 with the reason in kwker_rt_tokenizer_error(): no chat template, or a message list the template
refuses (its own error message, as Python would raise it).
Notes: tokenize the result with add_special 0 - the template already writes the special tokens it wants.

int kwker_rt_tokenizer_chat(const kwker_rt_tokenizer* t, const char* messages, int add_generation_prompt, const char* options, char* text,
                            int64_t cap, int64_t* len);

Turn UTF-8 text into token ids.
  text, len:   the text; len its length in bytes, or -1 for a 0-terminated string
  add_special: 1 adds the model's special tokens around the text (such as Llama's start token), 0 leaves them out
  ids, cap:    room for cap token ids
  n:           receives how many ids the text has; only the first cap are written. Call with cap 0 to size ids.
Returns 0, or -1 for text that is not valid UTF-8.
Notes: tokenizer.json's truncation length applies; its padding does not.

int kwker_rt_tokenizer_encode(const kwker_rt_tokenizer* t, const char* text, int64_t len, int add_special, int64_t* ids, int64_t cap,
                              int64_t* n);

Turn token ids back into UTF-8 text.
  ids, n:       the token ids
  skip_special: 1 leaves special tokens (such as an end-of-sequence token) out of the text
  text, cap:    room for cap bytes: the text's first cap - 1 bytes and a terminating 0 are written
  len:          receives the text's full length in bytes. Call with cap 0 to size text.
Returns 0, or -1 for a NULL argument.

int kwker_rt_tokenizer_decode(const kwker_rt_tokenizer* t, const int64_t* ids, int64_t n, int skip_special, char* text, int64_t cap,
                              int64_t* len);

int64_t kwker_rt_tokenizer_token_id(const kwker_rt_tokenizer* t, const char* token);

int64_t kwker_rt_tokenizer_vocab(const kwker_rt_tokenizer* t);

void kwker_rt_tokenizer_free(kwker_rt_tokenizer* t);

const char* kwker_rt_tokenizer_error(void);

typedef struct kwker_rt_decoder kwker_rt_decoder;

Load a language model from a Hugging Face model folder (config.json and model.safetensors or its shards).
  dir:           the folder
  precision:     "int4", "int8", "bf16" or "float32"; mixture-of-experts models load in "int4" or "int8"
  max_cache_len: the most token positions one sequence holds (prompt and new tokens)
  max_batch:     the number of sequences kwker_rt_decoder_step_batch runs at once (1 for one conversation)
  out:           receives the decoder; release it with kwker_rt_decoder_free
Returns 0, or -1 with the reason in kwker_rt_decoder_error().

int kwker_rt_decoder_load(const char* dir, const char* precision, int64_t max_cache_len, int64_t max_batch, kwker_rt_decoder** out);

Load a language model in int4 calibrated on sample text: the int4 weights are fitted to the inputs they see on these tokens,
as KwkDecoder(..., calib=ids) does - usually a lower perplexity than kwker_rt_decoder_load's int4 at the same speed.
  dir, precision, max_cache_len, max_batch, out: as kwker_rt_decoder_load; precision "int4"
  calib:      the calibration token ids (a few thousand tokens of text like the model's use)
  n_calib:    how many
  calib_ctx:  the chunk length they are run in (512 as KwkDecoder's default)
Returns 0, or -1 with the reason in kwker_rt_decoder_error(). The first load runs the model in float32 over the tokens; the
pack cache keeps the result, so a later load with the same tokens skips that. Mixture-of-experts models are refused.

int kwker_rt_decoder_load_calibrated(const char* dir, const char* precision, int64_t max_cache_len, int64_t max_batch, const int64_t* calib,
                                     int64_t n_calib, int64_t calib_ctx, kwker_rt_decoder** out);

Generate tokens after a prompt, greedily or by sampling, until an end-of-sequence token or max_new tokens.
  prompt, n_prompt:   the prompt's token ids
  max_new:            the most tokens to generate
  eos, neos:          the end-of-sequence ids (kwker_rt_decoder_eos gives the model's); the one that ends the run is
                      written last
  min_new:            generate at least this many tokens before an end-of-sequence id may come
  temperature:        0 for greedy decoding; above 0 samples with the settings below
  top_k:              keep the k most likely tokens (0: off)
  top_p:              keep the most likely tokens that together reach probability top_p (1: off)
  min_p:              drop tokens less likely than min_p times the most likely one (0: off)
  repetition_penalty: make tokens of the prompt and the text so far less likely (1: off)
  seed:               the random draws: the same seed gives the same tokens
  out, nout:          room for max_new token ids; nout receives how many were generated
Returns 0, or -1 with the reason in kwker_rt_decoder_error() (for example a prompt longer than max_cache_len allows).
Notes: the settings apply in Hugging Face generate's order. seed is the value Python's KwkDecoder.generate(
do_sample=True, generator=g) draws first from g with torch.randint(0, 2**62, (1,), generator=g): with it, both return
the same tokens.

int kwker_rt_decoder_generate(kwker_rt_decoder* d, const int64_t* prompt, int64_t n_prompt, int64_t max_new, const int64_t* eos,
                              int64_t neos, int64_t min_new, double temperature, int64_t top_k, double top_p, double min_p,
                              double repetition_penalty, uint64_t seed, int64_t* out, int64_t* nout);

The model's end-of-sequence token ids, from generation_config.json or else config.json.
  ids, cap: room for cap ids
Returns how many there are (0 for a decoder made by kwker_rt_decoder_new, -1 for NULL); only the first cap are written.

int64_t kwker_rt_decoder_eos(const kwker_rt_decoder* d, int64_t* ids, int64_t cap);

Run tokens through the model and get the next-token scores (logits): your own decoding loop.
  ids, n: n token ids of sequence seq, at cache positions pos to pos + n - 1
  nlog:   how many of the last rows to return logits for (-1: all n)
  logits: receives nlog rows of kwker_rt_decoder_vocab floats
Returns 0, or -1 with the reason in kwker_rt_decoder_error().

int kwker_rt_decoder_step(kwker_rt_decoder* d, const int64_t* ids, int64_t n, int64_t pos, int64_t nlog, int64_t seq, float* logits);

int kwker_rt_decoder_step_batch(kwker_rt_decoder* d, const int64_t* ids, int64_t n, const int64_t* pos, const int64_t* seq, int64_t nlog,
                                float* logits);

int kwker_rt_decoder_greedy(kwker_rt_decoder* d, int64_t tok, int64_t pos, int64_t n, const int64_t* eos, int64_t neos, int64_t min_new,
                            int64_t* out, int64_t* nout);

int64_t kwker_rt_decoder_vocab(const kwker_rt_decoder* d);

void kwker_rt_decoder_free(kwker_rt_decoder* d);

const char* kwker_rt_decoder_error(void);

typedef struct kwker_rt_encoder kwker_rt_encoder;

Load a text encoder from a Hugging Face model folder (config.json and model.safetensors or its shards).
  dir:       the folder, such as a downloaded sentence-transformers model
  precision: "float32" (the checkpoint's weights exactly), "bf16" (faster) or "int8" (faster still; not ModernBERT, NomicBERT or EuroBERT)
  out:       receives the encoder; release it with kwker_rt_encoder_free
Returns 0, or -1 with the reason in kwker_rt_last_error().

int kwker_rt_encoder_load(const char* dir, const char* precision, kwker_rt_encoder** out);

Sentence embeddings: one vector per text, pooled and normalized as the folder's sentence-transformers settings say.
  B, S:    B texts as rows of S token ids (ids[b * S + i])
  lens:    row b holds lens[b] real tokens, the rest is ignored (NULL: all S)
  out:     receives B rows of kwker_rt_encoder_hidden floats
  threads: the most threads to use (0: every core)
Returns 0, or -1 with the reason in kwker_rt_last_error().
Notes: the pooling (mean, first token, max or last token) and the normalization come from modules.json and
1_Pooling/config.json; a folder without modules.json gets mean pooling. Other steps, such as a Dense layer, are refused.

int kwker_rt_encoder_embed(kwker_rt_encoder* e, int64_t B, int64_t S, const int64_t* ids, const int64_t* lens, float* out, int threads);

The model's last hidden state: one vector per token.
  B, S, lens: as for kwker_rt_encoder_embed
  types:      the token type ids, B rows of S (NULL: all 0)
  out:        receives B x S rows of kwker_rt_encoder_hidden floats; rows past a text's length are 0
  threads:    the most threads to use (0: every core)
Returns 0, or -1 with the reason in kwker_rt_last_error().

int kwker_rt_encoder_forward(kwker_rt_encoder* e, int64_t B, int64_t S, const int64_t* ids, const int64_t* lens, const int64_t* types,
                             float* out, int threads);

int64_t kwker_rt_encoder_hidden(const kwker_rt_encoder* e);

void kwker_rt_encoder_free(kwker_rt_encoder* e);

const char* kwker_rt_last_error(void);

typedef struct kwker_rt_cnn kwker_rt_cnn;

Load an image model file.
  path: the file kwker.cnn.save() wrote
  out:  receives the model
Returns 0, or -1 with the reason in kwker_rt_last_error().

int kwker_rt_cnn_load(const char* path, kwker_rt_cnn** out);

void kwker_rt_cnn_input_shape(const kwker_rt_cnn* m, int64_t shape[3]);

int64_t kwker_rt_cnn_output_size(const kwker_rt_cnn* m);

Run the model on a batch of images.
  B:       the number of images
  x:       B images as float32, [B][channels][height][width] (the layout PyTorch uses), preprocessed as for the model
  out:     receives B x kwker_rt_cnn_output_size() floats
  threads: the most threads to use (0: every core)
Returns 0, or -1 with the reason in kwker_rt_last_error(). One call at a time per model.

int kwker_rt_cnn_run(kwker_rt_cnn* m, int64_t B, const float* x, float* out, int threads);

void kwker_rt_cnn_free(kwker_rt_cnn* m);

#define KWKER_RT_F32 0
#define KWKER_RT_BF16 1
#define KWKER_RT_F16 2
#define KWKER_RT_I8 3
typedef struct kwker_rt_view {
    const void* data;
    int32_t dtype, ndim;
    int64_t shape[4];
} kwker_rt_view;

A decoder from a configuration and weight list.
  c, nc / f, nf: the configuration integers and floats (KwkDecoder's DecodeRunner configuration)
  ts, nts:       the weight list
  flags:         KWKER_RT_BORROW to use the views in place (they must then outlive the decoder); 0 copies them
Returns 0, or -1 with the reason in kwker_rt_decoder_error().

int kwker_rt_decoder_new(const int64_t* c, int nc, const double* f, int nf, const kwker_rt_view* ts, int nts, int flags,
                         kwker_rt_decoder** out);

Flags of kwker_rt_encoder_new (and KWKER_RT_BORROW for kwker_rt_decoder_new):
  KWKER_RT_SQ      SmoothQuant factors follow the weights (int8, mode 18)
  KWKER_RT_BORROW  the embedding tables (decoders: every view) used in place; the caller keeps them alive
  KWKER_RT_RELB    MPNet's relative position bias: one more weight, [buckets][heads]; c[9] buckets, c[10] max distance
  KWKER_RT_MODERN  ModernBERT's layers and weight list; c[9] every how many layers one is global, c[10] half the
                   local window; eps[2] / eps[3] the global / local rotary thetas
  KWKER_RT_POSTLN  with KWKER_RT_MODERN, NomicBERT: post-LayerNorm layers, token type embeddings (c[8] types, their
                   table the last weight), no final LayerNorm
  KWKER_RT_RMS     with KWKER_RT_MODERN, EuroBERT: RMSNorms (weights only) throughout, the embedding norm optional

#define KWKER_RT_SQ 1
#define KWKER_RT_BORROW 2
#define KWKER_RT_RELB 4
#define KWKER_RT_MODERN 8
#define KWKER_RT_POSTLN 16
#define KWKER_RT_RMS 32

An encoder from a configuration and weight list.
  c:     [layers, hidden, heads, intermediate, activation, mode (16 bf16, 18 int8, 32 float32; on CPUs with AMX also 1 bf16
         and 8 int8 on AMX tiles), vocabulary, positions, token types], then the flags' extra entries
  eps:   [layer norm epsilon, embedding norm epsilon], then the flags' extra entries
  w:     the weight list as float32 arrays (KwkEncoder's order; NULL for a tensor the model lacks)
  flags: the KWKER_RT_ flags above, or 0
Returns 0, or -1 with the reason in kwker_rt_last_error().

int kwker_rt_encoder_new(const int64_t* c, const double* eps, const float* const* w, int flags, kwker_rt_encoder** out);

Run an encoder made by kwker_rt_encoder_new on token ids with your own positions and mask.
  ids, pos: B rows of S token ids and their position ids
  types:    the token type ids, or NULL (all 0)
  mask:     B rows of S floats, 0 for padding, or NULL (no padding)
  out:      receives B x S rows of hidden floats; padded rows are 0
  threads:  the most threads to use (0: every core)
Returns 0, or -1 with the reason in kwker_rt_last_error(). One call at a time per encoder.

int kwker_rt_encoder_run(kwker_rt_encoder* e, int64_t B, int64_t S, const int64_t* ids, const int64_t* pos, const int64_t* types, const float* mask,
                         float* out, int threads);