Methodology

Overview

MarineBench evaluates large language models on domain-specific marine knowledge. Unlike general knowledge benchmarks, it focuses on practical knowledge gaps that could lead to real-world harm when AI systems provide incorrect marine advice.

Knowledge Pillars

• Naval Architecture — Hull physics, stepped hull dynamics, pontoon mechanics, propeller interactions
• Marine Engineering — Diesel systems, electrical/corrosion, mechanical troubleshooting
• Operational Safety — COLREGs, wake sports safety, emergency procedures
• Electronic Interpretation — Sonar reading, chart symbols, radar analysis

Question Types

• Factual — Direct knowledge recall (e.g., "What is wet stacking?")
• Reasoning — Applying physics or logic (e.g., "Why does this happen?")
• Safety-Critical — Questions where wrong answers could cause harm

Test Configuration

How v1 was actually run (honesty note, added 2026-08-03): the published v1 leaderboards came from manual captures via a chat interface — single run per model, sampling settings unrecorded, graded by a human against the answer key. That's a demo protocol, not a benchmark protocol, which is why every v1 board carries a preliminary-data banner and why v2 exists.

v2 API Parameters (current protocol)

Prompt Format

Evaluation Process

1. Each question is sent to the model API with standardized parameters
2. Response is normalized (lowercase, trademark symbols stripped, whitespace and punctuation unified)
3. Response is checked against the accepted answers using word-boundary matching — "type 4" matches "Type 4." but never "type 40", and a bare "41" never matches "$241,000"
4. A response that states a correct answer passes even if it also mentions a trap term; traps flag clearly-wrong answers for analysis
5. Answers the deterministic matcher can't decide go to an LLM judge with a written per-question rubric, then to blind human adjudication on disagreement (v2)

Scoring Methodology

Answer Acceptance

Each question has a list of accepted answers — variations that convey the correct concept. Scoring uses normalized, word-boundary matching:

Scoring Formula

Simple accuracy scoring is used:

Trap Answer Analysis

Trap answers are common misconceptions. When a model gives a trap answer, it's recorded for analysis but scored as incorrect:

• Helps identify systematic knowledge gaps
• Reveals where models have learned incorrect information
• Informs future question development

Question Sourcing & Expansion

MarineBench questions are grouped into versioned packs (e.g., Advanced Physics, Boat Classes). Every question includes accepted answers, trap answers, an explanation, and a source tag for traceability.

Question Bank

Complete list of all MarineBench questions with acceptance criteria. Click any question to expand details.

Known Limitations

Current Version (v0.2)

• Question count — 68 questions is a modest sample; target is 500+
• No multimodal — Text-only; no sonar screenshots or chart images yet
• English only — No multilingual testing
• US/recreational focus — Questions skew toward US recreational boating; limited commercial/international coverage

Evaluation Limitations

• Substring matching — May miss valid paraphrased answers
• No partial credit — Binary correct/incorrect scoring
• Single-turn only — No follow-up questions or clarification

Planned Improvements

• Expand to 500+ questions across all pillars
• Release domain packs (ski/wake, center console, pontoon/tritoon, dealer ops)
• Add multimodal questions (sonar images, charts, configuration photos)
• Compare base, RAG-augmented, and tuned model variants
• Implement LLM-as-judge for more nuanced answer evaluation
• Add scenario-based multi-turn evaluations
• Link each question pack to its research sources
• Include commercial maritime and international regulations