Skip to content

assess.yaml

assess.yaml describes one evaluation: the dataset and the split taken from it, the scorer, the models, and how they are asked. assess validate checks a file and prints every refusal with its path, and exits 64 if there is any. The JSON Schema is published beside this page.

What reads it so far

validate, split and selftest are built. split and selftest use the dataset, split and scorer sections. The rest of the file is checked now, and used by assess run, which is not built yet.

A whole file

version: 1
server:
  url: http://127.0.0.1:8000/v1
  manifest: /run/launch/serve.json
  toolCallParser: hermes
  reasoningParser: qwen3
  insecure: false
dataset:
  kind: xlam-function-calling
  path: /data/xlam_function_calling_60k.json
  sha256: 4ef5c6f0dc552f2231f93f5853a9ef431e9e806d7aa514d0f6b615606ce576c6
  source:
    repo: Salesforce/xlam-function-calling-60k
    revision: 26d14ebfe18b1f7b524bd39b404b50af5dc97866
split:
  seed: assess-v1
  size: 2000
scorer:
  name: xlam-tool-call
  ordering: any
prompt:
  system: ""
models:
  baseline: base
  adapters:
    - name: tuned-a
    - name: tuned-b
      path: /shared/ckpt/b
sampling:
  temperature: 0
  topP: 1
  seed: 0
  maxTokens: 512
thinking:
  mode: off
  templateKey: enable_thinking
serving:
  batchInvariant: required
  probe: 8
concurrency: 16

The shortest file

A key is required unless its table below gives a default. Leaving out everything that has one gives this:

version: 1
dataset:
  kind: xlam-function-calling
  path: /data/xlam_function_calling_60k.json
  sha256: 4ef5c6f0dc552f2231f93f5853a9ef431e9e806d7aa514d0f6b615606ce576c6
  source:
    repo: Salesforce/xlam-function-calling-60k
    revision: 26d14ebfe18b1f7b524bd39b404b50af5dc97866
split:
  seed: assess-v1
  size: 2000
scorer:
  name: xlam-tool-call
models:
  baseline: base
sampling:
  temperature: 0
  topP: 1
  seed: 0
  maxTokens: 512
thinking:
  mode: off
  templateKey: enable_thinking
serving:
  probe: 8
concurrency: 16

Rules for the whole file

  • An unknown key is refused at every level, and so is a key written in another case, such as Version or maxtokens.
  • Nothing is interpolated. ${…} anywhere in the file is refused.
  • The file is one YAML document, of at most 1 MiB. A second document after a --- line is refused, because it would otherwise replace values in the first without saying so.
  • A relative path is read from the directory the command runs in, not from the file's.
  • A whole number is at most 2147483647, except sampling.seed.

version

Key Default Meaning
version required 1.

server

Where the models are served. Every key is optional, because a server started by go/launch says the same things in its environment. Reading that arrives with assess run.

Key Default Meaning
url none The server's OpenAI-compatible base URL.
manifest none The path of go/launch's serve manifest.
toolCallParser none The server's tool-call parser, such as hermes.
reasoningParser none The server's reasoning parser, such as qwen3.
insecure false true allows plain http:// to any host.

url is refused unless it:

  • is an http:// or https:// URL that names a host;
  • ends in /v1, with nothing after it;
  • holds no credentials, query or fragment. The API key never goes in the file: it is read from the VLLM_API_KEY environment variable;
  • uses https://, or names 127.0.0.1, ::1 or localhost, or insecure is true.

A refusal of url never repeats the URL.

dataset

Key Default Meaning
kind required xlam-function-calling, the one kind there is.
path required The dataset file.
sha256 required The file's SHA-256: 64 hex digits, in either case. A file with another hash is refused before a row of it is read.
source.repo required Where the file came from.
source.revision required The revision it came from there.

source is a record of where the file came from. Nothing compares it with the file: sha256 is what pins the content.

split

The held-out split is the items the models are scored on. The same dataset, seed and size always give the same split.

Key Default Meaning
seed required Any text. Put one that is all digits in quotes ("42"), or YAML reads it as a number and it is refused.
size required How many items to hold out: at least 1, and at most the dataset's eligible items.

An item is eligible unless two of its tools share a name. The pinned dataset has 60,000 items, 59,740 of them eligible. size is checked against that count once the dataset is read, so validate, which reads only the file, cannot refuse a size that is too large; split and selftest do.

scorer

Key Default Meaning
name required xlam-tool-call, or a scorer your own program registered. A name nothing registered is refused.
ordering any any or strict: whether a reply's calls must come in the answer's order. The scorers reference has the rule.

prompt

Key Default Meaning
system "" A system prompt, sent as written.

models

Key Default Meaning
baseline required The name the server gives the base model.
adapters none The LoRA adapters compared with it.
adapters[].name required The name the server gives the adapter.
adapters[].path none Where the adapter's weights are, for a server that is not serving it yet.

An adapter named twice is refused, and so is one with the baseline's name.

sampling

Key Default Meaning
temperature required 0 or more.
topP required More than 0, and at most 1.
seed required A whole number that fits in 64 bits.
maxTokens required The most tokens a reply may hold: at least 1, and at least 2048 when thinking.mode is on.

thinking

Some models can reason at length before they answer, and a chat template variable switches that on or off. It changes both the answers and how many tokens a reply needs.

Key Default Meaning
mode required off or on. There is no default because models disagree: some think unless told not to, others only when told to.
templateKey required The template's variable for it. Qwen3 uses enable_thinking.

With mode: on, a sampling.maxTokens below 2048 is refused, since fewer would cut a reply off while the model is still thinking. That floor is provisional until it has been measured.

serving

A server may answer several requests in one batch. A model is batch invariant when it gives the same reply to an item whatever else is in the batch with it.

Key Default Meaning
batchInvariant required required, asserted or not-required.
probe required How many of the split's items are sent twice to check it.

probe: 0 is refused unless batchInvariant is not-required, and probe cannot be more than split.size.

concurrency

Key Default Meaning
concurrency required How many requests are in flight at once: at least 1.

What a refusal looks like

Every refusal is reported in one pass, in path order. A line with no path is about the file as a whole.

$ assess validate
assess.yaml refused:
  additional properties 'retries' not allowed
  dataset.sha256: '4ef5c6f0' does not match pattern '^[0-9a-fA-F]{64}$'
  models.adapters.0.name: base is the baseline, which is scored already
  serving.probe: 0 probes nothing, and serving.batchInvariant is required
  thinking: missing property 'mode'
  version: value must be 1
$ echo $?
64