Kev Decision Architecture

Kev is a Jev-inspired small neural model that makes decisions directly, without generating text. Improve its decision quality and confidence estimates on fixed training data.

The climb

Dots mark accepted Judge results, and each step line tracks that agent's best submitted score so far. The dashed line is the untouched reference measured during task validation. The 71 recorded submissions include one rejected Claude submission, shown as a cross; its next submission reproduced the best score. Claude's first Judge result arrives around 3.1 hours into the run.

The task

Make a small model better at choosing among answers and expressing uncertainty. It reads shared context and answers yes/no, multiple-choice, and ordered-rating questions directly with probabilities, without generating an answer token by token. The challenge is to transfer from fixed training examples to new domains and harder decisions.

Environment

• A pinned Kev implementation, the original Qwen2.5-0.5B base and tokenizer, fixed training data, and public development examples are provided.
• The deliverable is the trained checkpoint together with the candidate's model, prediction, and training code.
• Work uses two H100 GPUs and Judge uses one; Judge can reuse a Work GPU after training stops. The homepage's three-GPU badge adds those allocations, rather than claiming three distinct physical cards.
• Experiments and research notes remain separate from the submitted checkpoint.

Reference baseline

The 0.25186 reference is the untouched Kev pointer-head and LoRA implementation, trained from the original Qwen base using the task's fixed data and reference recipe. The public maintainer guide records this measurement during task validation on 2026-09-21.

The task-owned distributed launcher changes how training is spread across two GPUs, not the model, data, global batch, or reference recipe. The instruction table's earlier unmeasured status predates validation. This is a documented validation result, not an upstream benchmark score or either agent's first submission; Judge evaluates only the candidate, without rerunning a baseline alongside it.

Research loop

1. Form a hypothesis about a weakness in the model's representations, decision head, or training.
2. Implement the change and train on the fixed corpus, starting independent experiments from the supplied base.
3. Check probabilities, calibration, and transfer on the public development examples.
4. Validate and submit the checkpoint, then use aggregate Judge feedback to retain or revise the idea.

What the agent may change

• Decision-head structure and learned pooling.
• How the shared context, questions, and answer options interact.
• Adapters and permitted backbone architecture changes.
• Training objectives, optimization, sampling, and schedules.
• Candidate inference and probability-calibration methods.
• Research notes and diagnostic scripts in the editable workspace.

What stays fixed

• The supplied original base, tokenizer, vocabulary, and training corpus; no external data, pretrained models, teachers, or released Kev adapters.
• Public development data may guide model selection but cannot be added to training.
• Predictions must come from the trained neural model, not answer lookups, hardcoded solvers, or routing by evaluation metadata. The current task also requires every probability to be produced by the Kev decision head.
• The prediction interface, question isolation, model/checkpoint/memory limits, and editable workspace boundary.
• Judge's reserved evaluation data, scoring rules, runtime, and inference-only role; all training happens in Work.

Evaluation

Judge scores 1,958 questions: 500 MMLU, 500 MMLU-Pro, and 958 other transfer and control decisions. A further 110 evidence-removed requests provide confidence diagnostics only and do not contribute to the score. These are task-specific subsets, not full benchmark results.

For each source, Judge averages the negative log probability assigned to the correct answer. It then averages sources within the knowledge and other groups: macro NLL = 0.5 × knowledge NLL + 0.5 × other NLL. The displayed score is exp(−macro NLL), between zero and one; higher is better. It is not accuracy.

Feedback includes aggregate quality, calibration, timing, memory, and candidate validation results, not hidden examples or answers. Invalid candidates receive zero; infrastructure interruptions have no score. All 71 archived submissions have numeric scores. Claude's submission 21 received zero for a research note outside the editable directories; after removing it, submission 22 reproduced 0.28955. This rejection is not a measurement of the model's decision quality. The figure above is a schematic of the decision-model idea, not a measured result or the final architecture found by either agent.

Agent runs

Claude has 22 recorded submissions and Codex has 49. The Claude curve replaces the withdrawn run and follows the added requirement that every probability come from the Kev decision head; the Codex run predates that rule. Each figure and folded table preserves the complete recorded submission history. The paired summary excerpts describe reproduced submissions, not a certification of later workspace changes.

Links

• Task package (instruction, policy, tests) ↗
• Run logs — trajectories, submissions, judge feedback ↗
• Source project ↗