JFinBench Exposes AI's Japanese Finance Knowledge Gap—and Why Web Tools Change the Results

JFinBench Exposes AI's Japanese Finance Knowledge Gap—and Why Web Tools Change the Results
FREE PERSONALIZED ALERTS

Stay ahead of SEO & AI Search.

Follow the topics that matter to you. Get relevant new articles by email every 8 hours.

Create Free Alerts Free to join · Choose your topics · Unsubscribe anytime

A new Japanese financial benchmark reveals a gap that general-purpose AI leaderboards can hide. On November 17, 2026, Tokyo-based Sakana AI released JFinBench, a benchmark built from 321 questions drawn from six recent Japanese professional examinations in accounting, taxation and finance. The results suggest that some open-weight models approaching proprietary systems on coding tasks still lag substantially on questions requiring current knowledge of Japanese financial law and practice. But the same research contains a second, equally important finding: giving models access to web research and terminal tools substantially narrows that gap.

JFinBench is not a stock-picking contest or an investment-return predictor. It measures how models answer professional examination questions under controlled conditions. That distinction matters for anyone evaluating AI for financial services, regulated workflows, retrieval-augmented generation or local AI infrastructure.

What Sakana AI released

JFinBench uses the latest 2026 editions of six Japanese qualifications: the certified public accountant short-answer examination, tax accountant examinations covering eleven subjects, the first-grade financial planning skills test, the first-grade financial counter services skills test, the first-grade bookkeeping examination of the Japan Chamber of Commerce and Industry, and the first-grade commercial bookkeeping test. The examination dates span May through August 2026.

The 321 questions include 193 multiple-choice items and 128 questions requiring written answers, calculations or explanations. Twelve require image interpretation, including diagrams. Sakana AI averages scores equally across the six examinations rather than weighting them simply by the number of questions. Some non-multiple-choice answers are graded by an LLM judge using reference answers and rubrics; therefore these are benchmark scores under a defined evaluation protocol, not directly comparable to official human exam pass rates.

The main results: proprietary models lead without tools

In the primary setting, each model answered without external tools using one API call per question, with maximum supported reasoning effort. Results are averaged over three runs. The top proprietary model, Claude Fable 5, scored 95.8, followed by GPT-6 Astra at 93.6 and Gemini 3.8 Flash at 88.8. The strongest open-weight model, Kimi K3.1 Pro, reached 83.4—12.4 points behind Claude Fable 5 and 10.2 behind GPT-6 Astra.

ModelNo-tool scoreAPI cost per questionOutput tokens per question
Claude Fable 595.8$0.55510.9K
GPT-6 Astra93.6$0.2484.8K
Gemini 3.8 Flash88.8$0.0256.6K
Kimi K3.1 Pro83.4$0.14514.3K
Kimi K379.9$0.25316.6K
DeepSeek V4.1 Flash 101773.1$0.00723.5K
Qwen3.8 Flash62.5$0.02043.2K
MiMo-V2.6-Pro62.1$0.03033.9K
Kimi K2.659.1$0.05814.2K
Mistral Large 458.2$0.06129.0K
GLM 5.3 Flash57.7$0.01326.4K
MiMo-V2.6-Flash50.9$0.01034.4K

Source: Sakana AI, JFinBench, November 17, 2026. Cost figures are reported API costs in the no-tool maximum-effort setting; they are not equivalent to self-hosting costs or total enterprise deployment costs.

Why a coding leaderboard can be misleading

Sakana AI highlights a particularly revealing comparison. Kimi K3.1 Pro was reported at 65.7% on Terminal-Bench 4.0, compared with 58.2% for GPT-6 Astra, using published figures cited by the researchers. Yet on JFinBench the order reverses: GPT-6 Astra scores 93.6 and Kimi K3.1 Pro 83.4. This cross-benchmark contrast should not be treated as a controlled head-to-head under identical benchmark methodologies. It does demonstrate why coding proficiency alone cannot establish financial-domain competence.

The problem is also geographical. A model trained heavily on English-language or US-oriented finance material may know generic accounting principles while lacking precise details of Japan's tax system, statutory rules, accounting conventions and certification requirements. A global leaderboard cannot substitute for a current, jurisdiction-specific evaluation.

What causes the failures? Mostly missing knowledge, not arithmetic

Sakana AI classified incorrect responses into missing knowledge, misreading or reasoning errors, calculation mistakes, formatting problems, truncated reasoning and grading errors. The most frequent category across many models was insufficient knowledge of Japanese institutions and regulations. Pure calculation mistakes were relatively rare. Several models also used up their output-token allowance while reasoning without delivering a final answer.

This distinction has major practical implications. Increasing reasoning effort does not automatically supply an absent legal provision. In fact, the research found that raising reasoning effort generally increased both performance and API cost. For regulated domains, accurate retrieval of current statutes and source documents may be more valuable than simply asking the model to think longer.

What happens when models can browse and use a terminal?

Sakana AI repeated the evaluation with access to web research and a Linux terminal, allowing models to look up statutes and perform calculations. The researchers tested different agent harnesses and reported each model's best result. This was not a single uniform agent configuration: model-specific harness choices affect comparability. Each attempt had a six-hour time limit, and the study audited logs to detect use of prohibited published answer keys.

ModelNo toolsWith toolsGain
GPT-6 Astra93.697.3+3.7
Claude Fable 595.896.4+0.6
Gemini 3.8 Flash88.894.2+5.4
Kimi K3.1 Pro83.492.9+9.5
Kimi K379.991.0+11.1
DeepSeek V4.1 Flash 101773.191.5+18.4
Qwen3.8 Flash62.585.9+23.4
MiMo-V2.6-Pro62.186.3+24.2
Kimi K2.659.174.7+15.6
Mistral Large 458.275.1+16.9
GLM 5.3 Flash57.787.6+29.9
MiMo-V2.6-Flash50.975.8+24.9

The biggest improvement was GLM 5.3 Flash's 29.9-point gain. DeepSeek V4.1 Flash climbed 18.4 points, from 73.1 to 91.5. The highest open-weight result with tools, Kimi K3.1 Pro at 92.9, remained 4.4 points behind GPT-6 Astra's 97.3. Model rankings were still strongly correlated across the two settings (Spearman rank correlation 0.87), suggesting that access to information narrows—but does not eliminate—differences in underlying competence.

In the shared Terminus-2 agent setting, web pages were retrieved in 67% of attempts, with Japan's official e-Gov legal information service among the most frequently accessed sources. This is particularly relevant to AI Search: retrieval quality matters, but so does a model's ability to decide which statute to look up and how to apply it correctly.

Not a pure RAG benchmark—and not proof that browsing solves finance

The tool-enabled configuration is broader than a conventional RAG setup. It includes browsing, shell commands, multiple turns and different agent harnesses. A simple vector database returning snippets is not equivalent to this evaluation. Nor does a 97.3 score mean a model is safe to provide unsupervised financial, tax or legal advice. The study measures exam performance, not actual client outcomes, professional liability, compliance decisions or live financial risk.

It also exposes agent security risks. Sakana AI reviewed 11,556 attempts and assigned zero scores to eight attempts that consulted prohibited answer sources. The authors discuss the need to keep unnecessary credentials out of the environment, clearly restrict external resources and audit tool-use logs. These safeguards are essential whenever an agent can browse or execute commands.

Why the benchmark can remain relevant after 2026

Financial rules change frequently, and old test questions can leak into training datasets. JFinBench addresses both problems by using recent exams and a continuously refreshable construction pipeline. A vision-language model transcribes exam PDFs into structured text, coding agents write exam-specific conversion scripts, and a human reviewer compares every converted question against the original.

Sakana AI reports approximately two hours of human validation per exam, around twelve hours for all six exams and 321 questions. That does not mean the entire benchmark was created in twelve hours: the figure refers to human verification effort, not total machine execution, preparation or research time. The approach is interesting because it offers a repeatable path to domain benchmarks without assuming automated extraction is error-free.

Cost versus accuracy: there is no universal winner

In the no-tool evaluation, Gemini 3.8 Flash reached 88.8 at a reported $0.025 per question, while DeepSeek V4.1 Flash 1017 reached 73.1 at $0.007. GPT-6 Astra scored 93.6 at $0.248, and Claude Fable 5 scored 95.8 at $0.555. Those are meaningful price-performance trade-offs within this particular experimental setup.

However, a cheaper API call that produces a wrong answer about tax obligations may be much more expensive operationally than a higher-cost call that gets it right. Conversely, paying for maximum reasoning on every easy question may waste money. Teams should evaluate cost per correctly completed, reviewed task—not simply tokens per request or benchmark rank. The published cost table does not include a comparable end-to-end price for all tool-enabled runs.

What JFinBench teaches local AI projects

For teams building local AI assistants with Ollama or other open-weight runtimes, the lesson is not that a small offline model can automatically reach frontier performance by connecting to the internet. JFinBench tested substantially different model classes and specific agent configurations. It does suggest a research method: evaluate an offline baseline, add verified retrieval and controlled tools, then compare results on the same domain-specific questions.

A useful local test would retain a fixed set of questions, their authoritative sources, the applicable legal date, a grading rubric and full model traces. Run each question without tools, then with retrieved documents, then with permitted search and calculation tools. Record accuracy, unsupported claims, citations, time, cost and security failures. That would help distinguish missing knowledge from reasoning problems and retrieval errors.

The implications for AI Search and SEO

JFinBench is not a search-engine ranking announcement. Its relevance to AI Search is about the quality of source-grounded answers. The finding that agents frequently retrieved government legislation underlines the importance of authoritative, current, machine-accessible source material for high-stakes queries. A model cannot safely apply a rule if the underlying rule is outdated, missing or retrieved out of context.

For publishers of financial information, practical priorities include clear publication and revision dates, jurisdiction labels, primary legal references, accurate examples and explicit distinctions between educational explanations and regulated advice. These are sound content-quality principles, not newly announced Google or Naver ranking factors. The study does not demonstrate that a particular page format wins more AI citations or organic traffic.

Frequently asked questions

What is JFinBench?

JFinBench is Sakana AI's Japanese financial-domain LLM benchmark, released November 17, 2026, using 321 questions from six recent professional qualification examinations.

Which model scored highest without tools?

Claude Fable 5, with 95.8, followed by GPT-6 Astra at 93.6 and Gemini 3.8 Flash at 88.8, according to Sakana AI's three-run averages.

Which model scored highest with tools?

GPT-6 Astra scored 97.3 using the Terminus-2 harness in the researchers' reported configuration.

Did open-weight models catch up when given web access?

The gap narrowed considerably. Kimi K3.1 Pro reached 92.9 with tools, compared with GPT-6 Astra's 97.3. The two runs used their best reported harness configurations, so the comparison is not a fully controlled identical-tool test.

Why do models fail Japanese finance questions?

The researchers found that missing knowledge of Japanese financial and legal institutions was a more common source of errors than calculation mistakes.

Does the benchmark predict stock prices?

No. It evaluates answers to professional finance, accounting and taxation examinations, not investment returns or market forecasts.

Can a small local model reproduce these scores?

Not established. The study evaluated twelve specified frontier and open-weight models, not a generic small local model. Any local implementation requires its own tests.

Conclusion: expertise is local, and benchmarks should be too

JFinBench challenges the assumption that a model's performance on coding or general reasoning predicts its competence in a country's financial system. The gap between proprietary and open-weight models remains substantial without tools, but web access and agentic workflows can reduce it. The deeper lesson is that AI quality depends on three distinct things: what a model already knows, what trustworthy information it can retrieve, and whether it can apply that information correctly. Measuring all three is more valuable than treating a single leaderboard score as universal intelligence.

Primary source: Sakana AI, “JFinBench: 最新の金融資格試験を用いた日本語金融ベンチマークの公開,” November 17, 2026. The research page links to the paper, dataset, source code and leaderboard. All reported scores, costs, exam counts and evaluation settings are attributed to Sakana AI; NetContentSEO has not independently rerun the benchmark. Barry Schwartz coverage: not checked.

0%