SemCraftBench
Semantic Craft Bench
In preparation
SemCraftBench is a proposed benchmark for language models built around an Infinite Craft–style semantic crafting game. Models will propose the results of combining concepts, providing a setting for studying how they work with meaning and relationships.
Results
No benchmark runs have been published. This page will hold the benchmark output and model comparisons as evaluations become available.
Methodology
A white paper is in preparation. It will describe the crafting task, evaluation procedure, and interpretation of the results.
The planned evaluation uses Jev, an LLM classifier, to assist automated benchmarking by judging and scoring models’ proposed crafting results. The methodology will document Jev’s role and the limits of using a model to evaluate other models.
The benchmark has not yet been implemented. The evaluation procedure and scoring criteria remain to be established.