Skip to content

Scorers

A scorer decides whether a model's reply to an item is correct, and gives one reason for its verdict. go/assess has one built in, xlam-tool-call, and accepts others by registration.

No model is scored yet

assess run is not built. assess selftest holds a scorer to the self-test below. This page describes the scorer the run will use, which is built and checked against the reference implementation.

The interface

A scorer implements assess.Scorer:

Method What it gives
Name() the name a run selects the scorer by
Version() recorded with every result, so a change to the scoring is visible
Reasons() every reason Score can give
Key(item) the item's answer as a reply: what a model that got it right would return
Score(item, response) the verdict: correct or not, one of the declared reasons, and a detail in plain words

An error from Score is not a wrong answer. It says the scorer could not judge the item at all. A malformed reply is a wrong answer, with a reason.

assess.Register makes a scorer available under its name. It panics on a nil scorer, an empty name, a name already registered, and the built-in scorer's name.

The self-test

A scorer has to pass two checks on every item it will score:

  1. The answer key. Score(item, Key(item)) is correct.
  2. The null control. A reply with no text and no calls is wrong, unless the item's key is itself such a reply.

The first cannot catch a scorer that marks everything correct. The second does, on every item whose key is not an empty reply. An item whose key is one is left out of the second check, so a scorer whose keys are all empty replies is held to the first alone. A scorer that gives a reason it did not declare in Reasons() is at fault, whichever check it was on.

xlam-tool-call

xlam-tool-call marks a reply correct when the tool calls it makes are the dataset's answer: the same tools, each with equal arguments. Its version is 1.

Reasons

The first that applies is the reason given:

Reason When
truncated the reply reached the token limit
unparsed-call the reply has no calls, and its text holds the server's tool-call marker: the model tried a call the server's parser could not decode
malformed-arguments a call's arguments are not one JSON object
no-call the reply has no calls, and the answer has some
call-count the reply and the answer have different numbers of calls
name a call names a tool the answer does not call, or under strict does not call at that position
arguments a call's arguments are not the answer's
match the reply is correct

An answer with no calls is matched by a reply with no calls, whatever its text, unless the reply is truncated or unparsed-call.

The tool-call marker depends on the tool-call parser the server runs. hermes is the only parser the scorer knows, and its marker is <tool_call>.

Equality

Arguments are compared as JSON values, not as text:

  • objects are equal when they have the same keys with equal values, in any order;
  • arrays are equal when they have equal elements in the same order;
  • numbers are compared as exact decimals, so 5, 5.0 and 5e0 are equal;
  • strings are compared code point by code point, with no normalisation;
  • true, false and null equal only themselves, and a string never equals a number.

An argument the reply supplies and the answer leaves out makes the two unequal, and so does one the answer has and the reply leaves out.

Arguments are read strictly. A key that appears twice, a number that is NaN or Infinity, a lone surrogate, or text after the object makes the call malformed-arguments.

Ordering

ordering A reply matches when
any its calls are the answer's calls in any order
strict its first call is the answer's first, its second the answer's second, and so on

How a tool is sent

Each of a row's tools is sent with its name and description as written, and its parameters as a JSON Schema object. A parameter's type comes from the row's type words:

Type words JSON Schema
str, int, float, bool string, integer, number, boolean
list, List array
List[T] array with items from T
Tuple[T, …, T] array with items from T, and minItems and maxItems the number of members
List[Union[int, float]] array with items of number
Dict object
set array with uniqueItems: true
Callable[[…], T] string

A parameter is required unless it has a default or its type words carry , optional. A default is sent as the row wrote it, and plays no part in scoring.

Type words outside this grammar refuse the whole dataset, naming the row, rather than send a parameter with no type. So does a Tuple whose members are not all one type, and a default in the type words that is not the parameter's default.