Skip to content

Command line

assess takes one verb and that verb's flags.

assess validate [-config assess.yaml]
assess split    [-config assess.yaml] -out split.json
assess selftest [-config assess.yaml]
assess version

-config names the assess.yaml to read, and is assess.yaml in the directory the command runs in when it is left out. An unknown verb or flag, a positional argument or a flag given twice is refused with exit 64.

Not built yet

assess run, which sends the requests and writes the result, and assess report, which prints one. None of the verbs below needs a server.

The output shown on this page is from the sixteen made-up rows the repository tests with (reference/testdata/rows.json), split with seed: twins and size: 6. The real dataset gives other counts and another digest.

assess validate

Checks the file and nothing else: it opens no dataset and reads no environment. It prints nothing and exits 0 for a file it accepts. Otherwise it prints every refusal with its path and exits 64.

It also refuses a scorer.name that nothing has registered.

assess split

Reads the dataset, takes the held-out split the file asks for, and writes it to the file -out names, so that training data can leave those items out.

$ assess split -out split.json
split.json: 6 items held out, 9 in the complement, 1 left out of it as identical to a held-out item
digest 957cef3171b5acf42a671f9bb66b853414f3c0095728957e5013c3b180902ad2

The file is JSON with these members:

Member What it holds
schema assess.split.v1
algorithm split.v1
seed, size, datasetSha256 what the split was taken with
digest a hash of the held-out items' ids and content, in order. Two splits with the same digest hold the same items.
heldOut the items the models are scored on, each an id and the hash of its content
complement the ids of the items training may use
twins the ids of items left out of both: each is identical to a held-out item, so training on it would leak an answer
twinsExcluded how many ids twins holds
ineligible the items that can never be held out, each an id and a reason. They stay in the complement.

The file is written whole or not at all, and replaces one already there.

The split file is made from the dataset

Its ids come from the dataset's rows. Where the dataset's terms restrict sharing, they cover this file too.

assess selftest

Reads the dataset, takes the split, and holds the scorer to its self-test on every held-out item: the item's own answer must be marked correct, and an empty reply must be marked wrong. No model is asked anything.

$ assess selftest
scorer xlam-tool-call 1 passed its self-test on 6 items
  answer keys marked correct: 6
  empty replies marked wrong: 5
  answer keys that are an empty reply: 1
split digest 957cef3171b5acf42a671f9bb66b853414f3c0095728957e5013c3b180902ad2

An item whose answer is itself an empty reply is counted on the last line, and is left out of the second check.

A scorer that fails exits 65, and each failing item is named with the reason the scorer gave:

$ assess selftest
assess selftest: scorer lenient failed its self-test on 3 of 3 items:
  item 8: an empty reply was marked correct, with the reason same
  item 5: an empty reply was marked correct, with the reason same
  item 4: an empty reply was marked correct, with the reason same
$ echo $?
65

A scorer that returns an error, panics, or gives a reason it did not declare has a bug, and the command exits 70.

selftest ignores server.toolCallParser, since it has no server.

assess version

Prints assess and the module's version, or assess unknown for a build that recorded none.

Exit codes

Code Meaning
0 The verb did what it was asked.
64 A flag or assess.yaml was refused, including a split.size the dataset cannot give, or the -out file could not be written.
65 The dataset could not be opened or was refused, or the scorer failed its self-test.
70 go/assess or a scorer failed.

69 is kept for a server that cannot be used, which arrives with assess run.

A program with its own scorer

assess is also a library. A program registers a scorer and hands its arguments to assess.Run, and is then the same command line with one more name for scorer.name:

package main

import (
    "context"
    "os"

    "gitlab.com/phpboyscout/go/assess"
)

func main() {
    assess.Register(myScorer{})
    os.Exit(assess.Run(context.Background(), os.Args[1:], os.Stdout, os.Stderr))
}

The scorers reference says what myScorer has to implement.