assess¶
Evaluate language models served behind an OpenAI-compatible endpoint.
Pluggable scorers, statistics with confidence intervals, and base-versus-tuned comparison across several adapters on one server. Every scorer must award its own answer key full marks before it is allowed to score anything. Part of the phpboyscout Go toolkit.
Being built
assess is not usable yet. Spec 0001
is approved and sets out the first version: an assess command that scores a baseline model and
its LoRA adapters, served by one vLLM server, on a held-out split of a public tool-calling
dataset. So far the command checks an assess.yaml, writes the
split, and holds a scorer to its self-test: the
command-line reference has each verb. The
scorers reference says what is marked correct and why, and the
statistics reference what a result will report. Nothing sends a
request yet.
Documentation¶
The documentation follows the Diátaxis framework:
- Tutorials: learning-oriented lessons.
- How-to guides: task-oriented recipes for specific goals.
- Explanation: how assess works, and why.
- Reference: the API, configuration and file formats.