Fresh suite
Seeded generators vary physical parameters while preserving the governing equations.
tasks.private.jsonlOpen research software · electrical engineering
Measure a model. Keep the evidence.
OhmJudge builds a fresh, committed suite of EE calculations, collects answers from any web interface without exposing the answer key, and grades values and units later with local Python rules.
Python 3.11+ · zero runtime dependencies · MIT License · no account or API key required
Deterministic task verdictILLUSTRATIVE TRACE
50seeded task templates
10EE domains
0runtime dependencies
5auditable artifacts
An evaluation chain another researcher can inspect.
Each step creates a plain file. The private answer key stays separate from the public prompts; the final audit recomputes every verdict instead of trusting the report.
Freeze intent
SHA-256 records the exact suite before model answers are collected.
prompts.public.jsonlWithout the key
Use the answer-free collector, an optional API adapter, or a replayed response set.
responses.jsonlLocal rules
Normalize units, apply declared tolerances, and classify concrete failure modes.
grades.jsonlVerify evidence
Check protocol metadata, coverage, duplicates, and deterministic recomputation.
audit.jsonA narrow decision, stated precisely.
OhmJudge avoids the false precision of a universal intelligence score. It answers one reproducible question about scalar EE calculations and exposes the evidence behind the answer.
Research question
Can a model solve the same committed EE suite with correct values and units?
Fresh parameters reduce exact-answer memorization. They do not make familiar formulas or task structures secret.
Machine-checkable contract
A number, a dimension, a declared unit, and an explicit tolerance.
- response
- {"answer": 15.59, "unit": "kW"}
- judge
- deterministic Python
- evidence
- portable JSONL + HTML
Where it fits—and where it does not.
The project is not the first EE benchmark. Its practical contribution is the combination of user-generated suites, a no-API capture lab, unit-aware local grading, and an artifact-level audit.
| Approach | Best at | Typical requirement | OhmJudge distinction |
|---|---|---|---|
| EEE-Bench / CIRCUIT | Broad curated EE knowledge and multimodal questions | Static published datasets | Fresh seeded scalar suites and local collection |
| PCB-Bench / EDA tasks | Realistic board, layout, and design artifacts | EDA infrastructure and heavier verification | Runs on a laptop; deliberately narrower construct |
| General evaluation frameworks | Many providers, datasets, and judge types | Configuration, APIs, or external services | Purpose-built browser workflow for an EE researcher |
| OhmJudge | Repeatable scalar EE calculation studies | Python 3.11 and a browser | Fresh suite + capture protocol + deterministic audit |
This positioning is a testable product hypothesis, not a claim of unique academic novelty. Independent studies, task-pack contributions, and reproduced results are the evidence that would validate it.
Ten domains with explicit calculation contracts.
The present release targets introductory-to-intermediate numerical reasoning and unit discipline. It does not infer professional hardware-design competence.
Circuits
Dividers, filters, op-amps, Thevenin models, series networks.
Signals
RMS, sampling, aliases, energy, and convolution.
Control
Poles, overshoot, settling, stability, and gain.
Communications
Capacity, path loss, noise, symbols, and link budgets.
Electronics
Diodes, BJTs, MOSFETs, ADCs, and slew rate.
Power
Three-phase loads, transformers, energy, PF, and batteries.
Electromagnetics
Waves, fields, capacitance, magnetic flux, and inductance.
Digital
Timing, clock limits, counters, encodings, and distance.
Embedded
UART, PWM, timers, buffers, and I²C transfers.
Instrumentation
Shunts, sensors, uncertainty, dB, and bridge outputs.
Limits are part of the result.
A score is useful only inside the construct actually measured.
Not professional certification.Scalar calculations do not establish safe hardware-design competence.
Not contamination-proof.Fresh values reduce exact memorization; formulas and template families remain familiar.
Not an official model ranking.Product surface, visible label, settings, tools, date, and retries must be reported.
Not a replacement for simulation.SPICE, HDL, EDA, laboratory measurement, and safety validation remain outside this release.
Collect responses without handing over the answer key.
The standalone collector contains public prompts and the suite commitment, but no expected values, tolerances, scores, network calls, or grading code. The separate report remains synthetic and makes no model-ranking claim.