Social Work Meta-Data

Demonstrations · Report 01 · Text-to-SQL · 48 questions · 6 models · July 2026

Text-to-SQL with Small Local Models

Can free, locally-run language models query these databases? We gave 6 open-weight models, 26 B to 36 B parameters, the same instructions any assistant gets and 48 research questions across six task categories, executing their SQL verbatim against the live databases through the public endpoint. Everything ran locally on a single MacBook Pro (Apple M5), with no cloud AI involved.

Brian E. Perron, PhD · University of Michigan

279 / 288
strict passes across all models
(280/288 correct or defensible)
48 / 48
perfect scores: ornith:35b
1 laptop
MacBook Pro (Apple M5) ·
models pulled, run, and purged in place
01 Method

Blind questions, six task categories, pre-registered answers

Every model ran on a MacBook Pro (Apple M5) through Ollama, at temperature 0 so runs are reproducible. Each model was given exactly two things: the project's two published skill files (word for word, the same documents any user downloads) and a one-line instruction to reply with a single SQL statement.

Each model answered 48 research questions, one at a time, split evenly between the SWRD and SSWR databases. The questions fall into six categories of increasing difficulty:

Each model's SQL was executed exactly as written against the live databases, through the same public read-only endpoint available to everyone. The result was compared with a reference answer computed before the run; every reference query was verified to return a stable, deterministic answer. Each answer receives one of three marks:

How the skill files improve: the feedback loop. The skill files are not written once and frozen. They are maintained by testing, in a simple cycle:

  1. Run the benchmark.
  2. Inspect every error and look for a recurring pattern.
  3. State each pattern as a plain rule in the skill files.
  4. Run the benchmark again with the improved files.

A pilot round produced four rules this way: stay inside one database's tables, always call search functions with SELECT, treat the relevance score as a score rather than a row position, and join the papers table when filtering authorship by year. The scores on this page come from the second pass, run against those improved files. That pass caught one more pattern (models asking a search function's output for a journal_id column it does not have; the functions return journal_name directly), and the published skill files now state each search function's exact return columns. The skill files you download today include every rule.

The models

Model (Ollama tag)Architecture · disk sizeStrictCorrect or defensibleAvg / question
gemma4:26b
Google Gemma 4 26B
26B dense · 16 GB47 / 4847 / 486.6 s
qwen3.6:27b
Qwen3.6-27B
27B dense · 17 GB47 / 4847 / 4860.2 s
nemotron-cascade-2:latest
NVIDIA Nemotron-Cascade-2
30B MoE (3B active) · 24 GB44 / 4844 / 4810.1 s
gemma4:31b-mlx
Google Gemma 4 31B
31B dense (MLX) · 20 GB46 / 4847 / 4827.8 s
ornith:35b
Ornith 35B
35B dense · 21 GB48 / 4848 / 4813.8 s
qwen3.6:latest
Qwen3.6-35B
36B MoE · 23 GB47 / 4847 / 4851.9 s

Models not already on the machine were pulled, evaluated, and deleted afterward. Timings are Ollama generation only.

02 Scores by task category

Where difficulty actually bites

Category (strict passes)gemma4:26bqwen3.6:27bnemotron-c2gemma4:31bornith:35bqwen3.6
A · Simple retrieval8/88/88/88/88/88/8
B · Vocabulary fidelity8/88/88/88/88/88/8
C · Joins & relations8/88/88/88/88/88/8
D · Aggregation & trends8/88/88/87/88/88/8
E · Search functions7/87/84/87/88/87/8
F · Complex analytical8/88/88/88/88/88/8

Accuracy by category: Simple retrieval 48/48 · Vocabulary fidelity 48/48 · Joins & relations 48/48 · Aggregation & trends 47/48 · Search functions 40/48 · Complex analytical 48/48. The gradient is the story: simple retrieval and vocabulary questions are near-ceiling for every size, while joins, nested search-function calls, and multi-step analytical queries separate the models. This is where the models genuinely differ.

03 Explore the full results

Every question, every model, every line of SQL

Expand a category to see its questions with per-model marks (✓ strict · △ defensible · ✗ error, in model order from smallest to largest). Expand a question to see the reference answer and each model's verbatim SQL, timing, and outcome.

A · Simple retrieval8 questions · 48/48 strict passes · click to expand
01SWRDHow many records does the SWRD papers table contain in total, across all years including the historical supplement?

Reference answer: 110618

gemma4:26b7.0 scorrect
select count(*) from swrd.papers
qwen3.6:27b26.2 scorrect
select count(*) from swrd.papers
nemotron-c29.4 scorrect
SELECT COUNT(*) FROM swrd.papers
gemma4:31b88.2 scorrect
select count(*) from swrd.papers
ornith:35b5.9 scorrect
SELECT count(*) FROM swrd.papers
qwen3.613.3 scorrect
select count(*) from swrd.papers
02SWRDHow many SWRD records are from before 1989?

Reference answer: 23288

gemma4:26b2.3 scorrect
SELECT count(*) FROM swrd.papers WHERE publication_year < 1989
qwen3.6:27b14.6 scorrect
SELECT COUNT(*) FROM swrd.papers WHERE publication_year < 1989
nemotron-c24.3 scorrect
SELECT COUNT(*) AS n FROM swrd.papers WHERE publication_year < 1989
gemma4:31b88.3 scorrect
select count(*) from swrd.papers where publication_year < 1989
ornith:35b3.8 scorrect
select count(*) from swrd.papers where publication_year < 1989
qwen3.69.0 scorrect
select count(*) from swrd.papers where publication_year < 1989
03SWRDHow many journals are in the SWRD's journal list?

Reference answer: 91

gemma4:26b1.5 scorrect
select count(*) from swrd.journals
qwen3.6:27b9.6 scorrect
SELECT COUNT(*) FROM swrd.journals
nemotron-c24.2 scorrect
select count(*) as n from swrd.journals
gemma4:31b7.4 scorrect
select count(*) from swrd.journals
ornith:35b2.9 scorrect
select count(*) from swrd.journals
qwen3.69.0 scorrect
select count(*) from swrd.journals
04SWRDHow many SWRD articles, across all years, were published in the journal named 'Child & Family Social Work'?

Reference answer: 1624

gemma4:26b3.4 scorrect
select count(*) from swrd.papers p join swrd.journals j on p.journal_id = j.id where j.name = 'Child & Family Social Work'
qwen3.6:27b23.7 scorrect
SELECT count(*) FROM swrd.papers p JOIN swrd.journals j ON j.id = p.journal_id WHERE j.name = 'Child & Family Social Work'
nemotron-c25.4 scorrect
SELECT COUNT(*) AS n FROM swrd.papers p JOIN swrd.journals j ON j.id = p.journal_id WHERE j.name = 'Child & Family Social Work'
gemma4:31b16.4 scorrect
select count(*) from swrd.papers p join swrd.journals j on p.journal_id = j.id where j.name = 'Child & Family Social Work'
ornith:35b7.3 scorrect
select count(*) as n from swrd.papers p join swrd.journals j on j.id = p.journal_id where j.name = 'Child & Family Social Work'
qwen3.67.4 scorrect
select count(*) from swrd.papers p join swrd.journals j on j.id = p.journal_id where j.name = 'Child & Family Social Work'
05SSWRHow many presentations were given at the 2024 SSWR conference?

Reference answer: 1516

gemma4:26b2.0 scorrect
select count(*) from sswr.papers where year = 2024
qwen3.6:27b14.7 scorrect
select count(*) from sswr.papers where year = 2024
nemotron-c24.5 scorrect
SELECT count(*) FROM sswr.papers WHERE year = 2024
gemma4:31b9.3 scorrect
select count(*) from sswr.papers where year = 2024
ornith:35b2.6 scorrect
select count(*) from sswr.papers where year = 2024
qwen3.64.6 scorrect
select count(*) from sswr.papers where year = 2024
06SSWRHow many presentations does the SSWR database contain in total?

Reference answer: 23793

gemma4:26b1.4 scorrect
select count(*) from sswr.papers
qwen3.6:27b14.6 scorrect
select count(*) from sswr.papers
nemotron-c23.2 scorrect
select count(*) as n from sswr.papers
gemma4:31b7.2 scorrect
select count(*) from sswr.papers
ornith:35b1.7 scorrect
select count(*) from sswr.papers
qwen3.611.3 scorrect
select count(*) from sswr.papers
07SSWRHow many canonical (disambiguated) authors are in the SSWR authors table?

Reference answer: 21209

gemma4:26b1.7 scorrect
SELECT count(*) FROM sswr.authors
qwen3.6:27b32.6 scorrect
select count(*) from sswr.authors
nemotron-c23.7 scorrect
select count(*) from sswr.authors
gemma4:31b6.0 scorrect
select count(*) from sswr.authors
ornith:35b3.7 scorrect
select count(*) from sswr.authors
qwen3.64.5 scorrect
select count(*) from sswr.authors
08SSWRHow many presentations were given at the 2005 SSWR conference?

Reference answer: 423

gemma4:26b2.3 scorrect
select count(*) from sswr.papers where year = 2005
qwen3.6:27b15.3 scorrect
select count(*) from sswr.papers where year = 2005
nemotron-c24.8 scorrect
select count(*) as n from sswr.papers where year = 2005
gemma4:31b10.6 scorrect
select count(*) from sswr.papers where year = 2005
ornith:35b3.2 scorrect
SELECT COUNT(*) FROM sswr.papers WHERE year = 2005
qwen3.620.7 scorrect
select count(*) from sswr.papers where year = 2005
B · Vocabulary fidelity8 questions · 48/48 strict passes · click to expand
09SWRDList the exact distinct values of the research_method field in the SWRD.

Reference answer: Mixed-Methods; Qualitative; Quantitative; Review

gemma4:26b2.2 scorrect
SELECT DISTINCT research_method FROM swrd.papers
qwen3.6:27b18.7 scorrect
SELECT DISTINCT research_method FROM swrd.papers
nemotron-c23.8 scorrect
SELECT DISTINCT research_method FROM swrd.papers WHERE research_method IS NOT NULL
gemma4:31b83.4 scorrect
select distinct research_method from swrd.papers
ornith:35b3.4 scorrect
select distinct research_method from swrd.papers order by 1
qwen3.67.9 scorrect
select distinct research_method from swrd.papers order by 1
10SWRDHow many SWRD articles, across all years, have research_method 'Mixed-Methods'?

Reference answer: 1970

gemma4:26b2.5 scorrect
SELECT count(*) FROM swrd.papers WHERE research_method = 'Mixed-Methods'
qwen3.6:27b26.2 scorrect
SELECT count(*) FROM swrd.papers WHERE research_method = 'Mixed-Methods'
nemotron-c25.5 scorrect
select count(*) as n from swrd.papers where research_method = 'Mixed-Methods'
gemma4:31b14.7 scorrect
select count(*) from swrd.papers where research_method = 'Mixed-Methods'
ornith:35b7.0 scorrect
select count(*) from swrd.papers where research_method = 'Mixed-Methods'
qwen3.64.9 scorrect
select count(*) from swrd.papers where research_method = 'Mixed-Methods'
11SWRDHow many SWRD articles, across all years, have research_method 'Review'?

Reference answer: 706

gemma4:26b2.6 scorrect
select count(*) from swrd.papers where research_method = 'Review'
qwen3.6:27b28.6 scorrect
select count(*) from swrd.papers where research_method = 'Review'
nemotron-c24.6 scorrect
SELECT COUNT(*) AS n FROM swrd.papers WHERE research_method = 'Review'
gemma4:31b10.9 scorrect
select count(*) from swrd.papers where research_method = 'Review'
ornith:35b4.4 scorrect
select count(*) as n from swrd.papers where research_method = 'Review'
qwen3.64.7 scorrect
select count(*) from swrd.papers where research_method = 'Review'
12SWRDHow many SWRD articles, across all years, are flagged scientific but NOT flagged empirical?

Reference answer: 27918

gemma4:26b3.4 scorrect
SELECT count(*) FROM swrd.papers WHERE is_scientific AND NOT is_empirical
qwen3.6:27b37.2 scorrect
select count(*) from swrd.papers where is_scientific and not is_empirical
nemotron-c24.8 scorrect
SELECT COUNT(*) FROM swrd.papers WHERE is_scientific AND NOT is_empirical
gemma4:31b11.8 scorrect
select count(*) from swrd.papers where is_scientific = true and is_empirical = false
ornith:35b5.4 scorrect
select count(*) as n from swrd.papers where is_scientific = true and is_empirical = false
qwen3.66.0 scorrect
SELECT count(*) FROM swrd.papers WHERE is_scientific = true AND is_empirical = false
13SSWRList the exact distinct values of the methodology field in the SSWR database.

Reference answer: other; review; mixed_methods; qualitative; quantitative

gemma4:26b1.9 scorrect
SELECT DISTINCT methodology FROM sswr.papers
qwen3.6:27b25.0 scorrect
SELECT DISTINCT methodology FROM sswr.papers
nemotron-c24.6 scorrect
SELECT DISTINCT methodology FROM sswr.papers
gemma4:31b9.0 scorrect
select distinct methodology from sswr.papers
ornith:35b4.8 scorrect
select distinct methodology from sswr.papers order by 1
qwen3.612.1 scorrect
SELECT DISTINCT methodology FROM sswr.papers ORDER BY 1
14SSWRHow many SSWR presentations have methodology 'other'?

Reference answer: 240

gemma4:26b1.8 scorrect
select count(*) from sswr.papers where methodology = 'other'
qwen3.6:27b22.6 scorrect
SELECT count(*) FROM sswr.papers WHERE methodology = 'other'
nemotron-c24.6 scorrect
SELECT COUNT(*) FROM sswr.papers WHERE methodology = 'other'
gemma4:31b9.4 scorrect
select count(*) from sswr.papers where methodology = 'other'
ornith:35b2.5 scorrect
select count(*) from sswr.papers where methodology = 'other'
qwen3.65.2 scorrect
SELECT count(*) FROM sswr.papers WHERE methodology = 'other'
15SSWRHow many SSWR presentations have methodology 'mixed_methods'?

Reference answer: 2172

gemma4:26b2.5 scorrect
SELECT count(*) FROM sswr.papers WHERE methodology = 'mixed_methods'
qwen3.6:27b43.4 scorrect
SELECT count(*) FROM sswr.papers WHERE methodology = 'mixed_methods'
nemotron-c25.4 scorrect
select count(*) as n from sswr.papers where methodology='mixed_methods'
gemma4:31b10.5 scorrect
select count(*) from sswr.papers where methodology = 'mixed_methods'
ornith:35b2.5 scorrect
select count(*) from sswr.papers where methodology = 'mixed_methods'
qwen3.66.2 scorrect
select count(*) from sswr.papers where methodology = 'mixed_methods'
16SSWRHow many distinct values does the format field take in the SSWR papers table?

Reference answer: 7

gemma4:26b1.7 scorrect
SELECT count(DISTINCT format) FROM sswr.papers
qwen3.6:27b24.9 scorrect
SELECT count(DISTINCT format) FROM sswr.papers
nemotron-c24.8 scorrect
select count(distinct format) as distinct_formats from sswr.papers
gemma4:31b5.7 scorrect
select count(distinct format) from sswr.papers
ornith:35b3.3 scorrect
SELECT COUNT(DISTINCT format) FROM sswr.papers
qwen3.615.9 scorrect
select count(distinct format) from sswr.papers
C · Joins & relations8 questions · 48/48 strict passes · click to expand
17SWRDWhich journal has published the most SWRD articles from 1989 onward, and how many?

Reference answer: 5931 · British Journal of Social Work

gemma4:26b5.2 scorrect
SELECT j.name, COUNT(*) as article_count FROM swrd.papers p JOIN swrd.journals j ON p.journal_id = j.id WHERE p.publication_year >= 1989 GROUP BY j.name ORDER BY article_count DESC LIMIT 1
qwen3.6:27b40.9 scorrect
select j.name, count(*) as n from swrd.papers p join swrd.journals j on j.id = p.journal_id where p.publication_year >= 1989 group by j.id, j.name order by n desc limit 1
nemotron-c28.0 scorrect
SELECT j.name AS journal_name, COUNT(*) AS article_count
FROM swrd.papers p
JOIN swrd.journals j ON p.journal_id = j.id
WHERE p.publication_year >= 1989
GROUP BY j.name
ORDER BY article_count DESC
LIMIT 1
gemma4:31b25.0 scorrect
select j.name, count(*) as n from swrd.papers p join swrd.journals j on p.journal_id = j.id where p.publication_year >= 1989 group by j.name order by n desc limit 1
ornith:35b3.9 scorrect
select j.name, count(*) as n from swrd.papers p join swrd.journals j on j.id = p.journal_id where p.publication_year >= 1989 group by 1 order by 2 desc limit 1
qwen3.610.0 scorrect
select j.name, count(*) as n from swrd.papers p join swrd.journals j on j.id = p.journal_id where p.publication_year >= 1989 group by j.name order by n desc limit 1
18SWRDHow many SWRD articles from 1989 onward appeared in the journal named 'Research on Social Work Practice'?

Reference answer: 2911

gemma4:26b5.8 scorrect
SELECT count(*) FROM swrd.papers p JOIN swrd.journals j ON p.journal_id = j.id WHERE p.publication_year >= 1989 AND j.name = 'Research on Social Work Practice'
qwen3.6:27b50.1 scorrect
select count(*) as n from swrd.papers p join swrd.journals j on j.id = p.journal_id where j.name = 'Research on Social Work Practice' and p.publication_year >= 1989
nemotron-c26.0 scorrect
select count(*) as n from swrd.papers p join swrd.journals j on p.journal_id = j.id where p.publication_year >= 1989 and j.name = 'Research on Social Work Practice'
gemma4:31b24.1 scorrect
select count(*) from swrd.papers p join swrd.journals j on p.journal_id = j.id where j.name = 'Research on Social Work Practice' and p.publication_year >= 1989
ornith:35b13.6 scorrect
select count(*) from swrd.papers p join swrd.journals j on j.id = p.journal_id where p.publication_year >= 1989 and j.name = 'Research on Social Work Practice'
qwen3.613.8 scorrect
select count(*) from swrd.papers join swrd.journals on journals.id = papers.journal_id where journals.name = 'Research on Social Work Practice' and papers.publication_year >= 1989
19SWRDHow many distinct organizations are linked (via author affiliations) to SWRD papers published in 2020?

Reference answer: 2463

gemma4:26b6.9 scorrect
SELECT COUNT(DISTINCT aa.organization_id) FROM swrd.author_affiliations aa JOIN swrd.papers p ON aa.paper_id = p.id WHERE p.publication_year = 2020
qwen3.6:27b63.4 scorrect
select count(distinct aa.organization_id) from swrd.papers p join swrd.author_affiliations aa on aa.paper_id = p.id where p.publication_year = 2020
nemotron-c27.1 scorrect
SELECT COUNT(DISTINCT o.id) AS org_count FROM swrd.papers p JOIN swrd.author_affiliations a ON a.paper_id = p.id JOIN swrd.organizations o ON o.id = a.organization_id WHERE p.publication_year = 2020
gemma4:31b22.2 scorrect
select count(distinct aa.organization_id) from swrd.papers p join swrd.author_affiliations aa on p.id = aa.paper_id where p.publication_year = 2020
ornith:35b11.5 scorrect
select count(distinct aa.organization_id) as n_organizations from swrd.papers p join swrd.author_affiliations aa on aa.paper_id = p.id where p.publication_year = 2020
qwen3.614.2 scorrect
select count(distinct swrd.author_affiliations.organization_id) from swrd.papers join swrd.author_affiliations on swrd.author_affiliations.paper_id = swrd.papers.id where swrd.papers.publication_year = 2020
20SWRDHow many SWRD authorship records (paper-author links) exist for papers published in 2019?

Reference answer: 9953

gemma4:26b4.0 scorrect
SELECT count(*) FROM swrd.paper_authors pa JOIN swrd.papers p ON pa.paper_id = p.id WHERE p.publication_year = 2019
qwen3.6:27b84.8 scorrect
select count(*) from swrd.paper_authors pa join swrd.papers p on p.id = pa.paper_id where p.publication_year = 2019
nemotron-c26.1 scorrect
SELECT COUNT(*) FROM swrd.paper_authors pa JOIN swrd.papers p ON pa.paper_id = p.id WHERE p.publication_year = 2019
gemma4:31b18.9 scorrect
select count(*) from swrd.paper_authors pa join swrd.papers p on pa.paper_id = p.id where p.publication_year = 2019
ornith:35b3.8 scorrect
select count(*) from swrd.paper_authors pa join swrd.papers p on p.id = pa.paper_id where p.publication_year = 2019
qwen3.69.3 scorrect
SELECT count(*) FROM swrd.paper_authors pa JOIN swrd.papers p ON p.id = pa.paper_id WHERE p.publication_year = 2019
21SSWRWhich single institution had the most first-authored SSWR presentations between 2020 and 2026, using the normalized institution field? Give the institution and the count.

Reference answer: 290 · Washington University in St. Louis

gemma4:26b6.1 scorrect
SELECT pa.institution_normalized, COUNT(*) as n FROM sswr.papers p JOIN sswr.paper_authors pa ON p.id = pa.paper_id WHERE p.year BETWEEN 2020 AND 2026 AND pa.author_order = 1 GROUP BY 1 ORDER BY 2 DESC LIMIT 1
qwen3.6:27b51.2 scorrect
SELECT pa.institution_normalized, COUNT(*) AS presentation_count FROM sswr.paper_authors pa JOIN sswr.papers p ON p.id = pa.paper_id WHERE p.year BETWEEN 2020 AND 2026 AND pa.author_order = 1 GROUP BY pa.institution_normalized ORDER BY presentation_count DESC LIMIT 1
nemotron-c29.4 scorrect
SELECT pa.institution_normalized AS institution, COUNT(*) AS cnt FROM sswr.paper_authors pa JOIN sswr.papers p ON p.id = pa.paper_id WHERE pa.author_order = 1 AND p.year BETWEEN 2020 AND 2026 GROUP BY pa.institution_normalized ORDER BY cnt DESC LIMIT 1
gemma4:31b38.0 scorrect
SELECT pa.institution_normalized, count(*) as n FROM sswr.paper_authors pa JOIN sswr.papers p ON p.id = pa.paper_id WHERE pa.author_order = 1 AND p.year BETWEEN 2020 AND 2026 GROUP BY 1 ORDER BY 2 DESC LIMIT 1
ornith:35b6.7 scorrect
select institution_normalized, count(*) as n from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where pa.author_order = 1 and p.year between 2020 and 2026 group by 1 order by 2 desc limit 1
qwen3.618.2 scorrect
select institution_normalized, count(*) as n from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where pa.author_order = 1 and p.year between 2020 and 2026 group by 1 order by 2 desc limit 1
22SSWRHow many presentations is the SSWR canonical author with id 108089 linked to?

Reference answer: 89

gemma4:26b3.0 scorrect
SELECT count(*) FROM sswr.paper_authors WHERE author_id = 108089
qwen3.6:27b43.1 scorrect
select count(*) from sswr.paper_authors where author_id = 108089
nemotron-c27.2 scorrect
SELECT COUNT(DISTINCT paper_id) FROM sswr.paper_authors WHERE author_id = 108089
gemma4:31b13.8 scorrect
select count(*) from sswr.paper_authors where author_id = 108089
ornith:35b22.1 scorrect
select total_papers from sswr.authors where id = 108089
qwen3.65.9 scorrect
select count(*) from sswr.paper_authors where author_id = 108089
23SSWRHow many distinct canonical authors are linked to presentations at the 2015 SSWR conference?

Reference answer: 2089

gemma4:26b7.2 scorrect
SELECT count(DISTINCT pa.author_id) FROM sswr.papers p JOIN sswr.paper_authors pa ON p.id = pa.paper_id WHERE p.year = 2015
qwen3.6:27b34.2 scorrect
select count(distinct pa.author_id) from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where p.year = 2015
nemotron-c26.8 scorrect
SELECT COUNT(DISTINCT pa.author_id) AS distinct_canonical_authors FROM sswr.papers p JOIN sswr.paper_authors pa ON pa.paper_id = p.id WHERE p.year = 2015
gemma4:31b22.5 scorrect
select count(distinct pa.author_id) from sswr.papers p join sswr.paper_authors pa on p.id = pa.paper_id where p.year = 2015
ornith:35b5.9 scorrect
select count(distinct pa.author_id) from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where p.year = 2015
qwen3.611.5 scorrect
select count(distinct pa.author_id) from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where p.year = 2015
24SSWRHow many first-authored SSWR presentations, across all years, list 'University of Michigan' as the normalized institution?

Reference answer: 665

gemma4:26b5.6 scorrect
SELECT count(*) FROM sswr.paper_authors WHERE author_order = 1 AND institution_normalized = 'University of Michigan'
qwen3.6:27b70.8 scorrect
select count(*) from sswr.paper_authors where author_order = 1 and institution_normalized = 'University of Michigan'
nemotron-c25.4 scorrect
SELECT COUNT(*) FROM sswr.paper_authors WHERE author_order = 1 AND institution_normalized = 'University of Michigan'
gemma4:31b31.1 scorrect
select count(*) from sswr.paper_authors where author_order = 1 and institution_normalized = 'University of Michigan'
ornith:35b11.5 scorrect
select count(*) from sswr.paper_authors pa where pa.author_order = 1 and pa.institution_normalized = 'University of Michigan'
qwen3.620.7 scorrect
select count(*) from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where pa.author_order = 1 and pa.institution_normalized = 'University of Michigan'
D · Aggregation & trends8 questions · 47/48 strict passes · click to expand
25SWRDBetween 1989 and 2023, which single publication year had the most SWRD scientific articles with abstracts?

Reference answer: 2023

gemma4:26b4.9 scorrect
SELECT publication_year FROM swrd.papers WHERE publication_year >= 1989 AND publication_year <= 2023 AND is_scientific = true AND abstract IS NOT NULL GROUP BY publication_year ORDER BY count(*) DESC LIMIT 1
qwen3.6:27b62.8 scorrect
SELECT publication_year, count(*) as n FROM swrd.papers WHERE publication_year >= 1989 AND publication_year <= 2023 AND is_scientific = true AND abstract IS NOT NULL GROUP BY publication_year ORDER BY n DESC LIMIT 1
nemotron-c212.5 scorrect
SELECT publication_year, COUNT(*) AS n FROM swrd.papers WHERE is_scientific = true AND abstract IS NOT NULL AND publication_year BETWEEN 1989 AND 2023 GROUP BY publication_year ORDER BY n DESC LIMIT 1
gemma4:31b26.0 scorrect
select publication_year from swrd.papers where publication_year >= 1989 and publication_year <= 2023 and is_scientific = true and abstract is not null group by publication_year order by count(*) desc limit 1
ornith:35b8.2 scorrect
select publication_year, count(*) as n from swrd.papers where publication_year >= 1989 and publication_year <= 2023 and is_scientific = true and abstract is not null group by 1 order by 2 desc limit 1
qwen3.617.0 scorrect
select publication_year, count(*) as n from swrd.papers where publication_year >= 1989 and publication_year <= 2023 and is_scientific and abstract is not null group by 1 order by 2 desc limit 1
26SWRDHow many SWRD scientific articles with abstracts were published in 2020 and flagged empirical?

Reference answer: 1729

gemma4:26b3.1 scorrect
SELECT count(*) FROM swrd.papers WHERE publication_year = 2020 AND is_scientific = true AND is_empirical = true AND abstract IS NOT NULL
qwen3.6:27b91.3 scorrect
select count(*) as n from swrd.papers where publication_year = 2020 and is_scientific = true and abstract is not null and is_empirical = true
nemotron-c210.4 scorrect
SELECT COUNT(*) FROM swrd.papers WHERE publication_year = 2020 AND is_scientific = TRUE AND is_empirical = TRUE AND abstract IS NOT NULL
gemma4:31b16.8 scorrect
select count(*) from swrd.papers where is_scientific = true and abstract is not null and publication_year = 2020 and is_empirical = true
ornith:35b12.6 scorrect
select count(*) as n from swrd.papers where publication_year = 2020 and is_scientific and is_empirical and abstract is not null
qwen3.614.7 scorrect
SELECT count(*) FROM swrd.papers WHERE publication_year = 2020 AND is_empirical = true AND is_scientific = true AND abstract IS NOT NULL
27SWRDHow many SWRD scientific articles with abstracts were published between 2010 and 2019 inclusive?

Reference answer: 21719

gemma4:26b3.8 scorrect
SELECT count(*) FROM swrd.papers WHERE is_scientific = true AND abstract IS NOT NULL AND publication_year >= 2010 AND publication_year <= 2019
qwen3.6:27b42.4 scorrect
select count(*) from swrd.papers where publication_year between 2010 and 2019 and is_scientific = true and abstract is not null
nemotron-c29.3 scorrect
SELECT COUNT(*) FROM swrd.papers WHERE is_scientific AND abstract IS NOT NULL AND publication_year >= 2010 AND publication_year <= 2019
gemma4:31b18.1 scorrect
select count(*) from swrd.papers where is_scientific = true and abstract is not null and publication_year >= 2010 and publication_year <= 2019
ornith:35b5.5 scorrect
select count(*) from swrd.papers where publication_year between 2010 and 2019 and is_scientific and abstract is not null
qwen3.614.6 scorrect
select count(*) from swrd.papers where publication_year between 2010 and 2019 and is_scientific = true and abstract is not null
28SWRDWhat is the mean number of authors per SWRD paper for papers published from 1989 onward, rounded to 2 decimal places?

Reference answer: 2.5

gemma4:26b21.5 scorrect
SELECT ROUND(AVG(author_count)::numeric, 2) FROM (SELECT COUNT(pa.author_id) AS author_count FROM swrd.papers p JOIN swrd.paper_authors pa ON p.id = pa.paper_id WHERE p.publication_year >= 1989 GROUP BY p.id) AS subquery
qwen3.6:27b153.4 scorrect
SELECT ROUND(AVG(cnt), 2) FROM (SELECT p.id, COUNT(pa.author_id) AS cnt FROM swrd.papers p JOIN swrd.paper_authors pa ON p.id = pa.paper_id WHERE p.publication_year >= 1989 GROUP BY p.id) t
nemotron-c220.9 scorrect
SELECT ROUND(AVG(author_cnt),2) AS mean_authors_per_paper FROM (SELECT p.id, COUNT(DISTINCT pa.author_id) AS author_cnt FROM swrd.papers p JOIN swrd.paper_authors pa ON pa.paper_id = p.id WHERE p.publication_year >= 1989 GROUP BY p.id) sub
gemma4:31b107.8 sexpected one of ['2.5'], got ['2.43']
SELECT ROUND(COUNT(pa.author_id)::numeric / COUNT(DISTINCT p.id), 2) FROM swrd.papers p LEFT JOIN swrd.paper_authors pa ON p.id = pa.paper_id WHERE p.publication_year >= 1989
ornith:35b24.2 scorrect
SELECT ROUND(AVG(cnt), 2) AS mean_authors_per_paper FROM (SELECT pa.paper_id, COUNT(*) AS cnt FROM swrd.paper_authors pa JOIN swrd.papers p ON p.id = pa.paper_id WHERE p.publication_year >= 1989 GROUP BY pa.paper_id) sub
qwen3.628.4 scorrect
select round(avg(cnt), 2) from (select paper_id, count(*) as cnt from swrd.paper_authors join swrd.papers on swrd.papers.id = swrd.paper_authors.paper_id where swrd.papers.publication_year >= 1989 group by paper_id) t
29SSWRWhat is the earliest SSWR conference year with more than 1,000 presentations?

Reference answer: 2015

gemma4:26b4.0 scorrect
SELECT year FROM sswr.papers GROUP BY year HAVING COUNT(*) > 1000 ORDER BY year ASC LIMIT 1
qwen3.6:27b25.4 scorrect
select year, count(*) as n from sswr.papers group by 1 having count(*) > 1000 order by 1 asc limit 1
nemotron-c28.5 scorrect
SELECT year FROM sswr.papers GROUP BY year HAVING COUNT(*) > 1000 ORDER BY year ASC LIMIT 1
gemma4:31b22.3 scorrect
SELECT year FROM sswr.papers GROUP BY year HAVING count(*) > 1000 ORDER BY year ASC LIMIT 1
ornith:35b9.7 scorrect
select year, count(*) as n from sswr.papers group by 1 having count(*) > 1000 order by 1 asc limit 1
qwen3.69.6 scorrect
select year, count(*) as n from sswr.papers group by 1 having count(*) > 1000 order by 1 asc limit 1
30SSWRWhich SSWR conference year had the most presentations overall?

Reference answer: 2026

gemma4:26b2.6 scorrect
SELECT year FROM sswr.papers GROUP BY year ORDER BY COUNT(*) DESC LIMIT 1
qwen3.6:27b17.0 scorrect
SELECT year, count(*) as n FROM sswr.papers GROUP BY year ORDER BY n DESC LIMIT 1
nemotron-c26.8 scorrect
SELECT year, COUNT(*) AS n FROM sswr.papers GROUP BY year ORDER BY n DESC LIMIT 1
gemma4:31b12.5 scorrect
select year, count(*) as n from sswr.papers group by 1 order by 2 desc limit 1
ornith:35b4.2 scorrect
select year, count(*) as n from sswr.papers group by 1 order by 2 desc limit 1
qwen3.66.0 scorrect
select year, count(*) as n from sswr.papers group by 1 order by 2 desc limit 1
31SSWRHow many presentations at the 2010 SSWR conference used qualitative methods?

Reference answer: 82

gemma4:26b3.3 scorrect
SELECT count(*) FROM sswr.papers WHERE year = 2010 AND methodology = 'qualitative'
qwen3.6:27b28.9 scorrect
select count(*) as n from sswr.papers where year = 2010 and methodology = 'qualitative'
nemotron-c28.4 scorrect
select count(*) as n from sswr.papers where year = 2010 and methodology = 'qualitative'
gemma4:31b13.0 scorrect
select count(*) from sswr.papers where year = 2010 and methodology = 'qualitative'
ornith:35b6.7 scorrect
select count(*) from sswr.papers where year = 2010 and methodology = 'qualitative'
qwen3.65.7 scorrect
select count(*) from sswr.papers where year = 2010 and methodology = 'qualitative'
32SSWRWhich SSWR conference year had the most mixed_methods presentations?

Reference answer: 2025

gemma4:26b3.5 scorrect
SELECT year FROM sswr.papers WHERE methodology = 'mixed_methods' GROUP BY year ORDER BY COUNT(*) DESC LIMIT 1
qwen3.6:27b31.9 scorrect
select year, count(*) as n from sswr.papers where methodology = 'mixed_methods' group by year order by n desc limit 1
nemotron-c28.5 scorrect
select year, count(*) as n from sswr.papers where methodology = 'mixed_methods' group by year order by n desc limit 1
gemma4:31b14.4 scorrect
select year, count(*) as n from sswr.papers where methodology = 'mixed_methods' group by 1 order by 2 desc limit 1
ornith:35b7.2 scorrect
select year, count(*) as n from sswr.papers where methodology = 'mixed_methods' group by 1 order by 2 desc limit 1
qwen3.68.3 scorrect
select year, count(*) as n from sswr.papers where methodology = 'mixed_methods' group by 1 order by 2 desc limit 1
E · Search functions8 questions · 40/48 strict passes · click to expand
33SWRDUsing the SWRD keyword search function, what is the id of the top-ranked article about 'food insecurity'?

Reference answer: 68376

gemma4:26b5.0 scorrect
SELECT id FROM swrd.search_papers_keyword('food insecurity', 1)
qwen3.6:27b72.5 scorrect
select id from swrd.search_papers_keyword('food insecurity', 1)
nemotron-c220.2 sSQL error: column "paper_id" does not exist
select paper_id from swrd.search_papers_keyword('food insecurity', 1)
gemma4:31b17.5 scorrect
select id from swrd.search_papers_keyword('food insecurity', 1)
ornith:35b19.9 scorrect
select id from swrd.search_papers_keyword('food insecurity', 1)
qwen3.626.4 scorrect
SELECT id FROM swrd.search_papers_keyword('food insecurity', 1)
34SWRDUsing the SWRD keyword search function with match_count 20, how many of the top 20 results for 'homelessness' were published in 2015 or later?

Reference answer: 11

gemma4:26b10.9 scorrect
SELECT count(*) FROM swrd.search_papers_keyword('homelessness', 20) WHERE publication_year >= 2015
qwen3.6:27b101.3 scorrect
select count(*) as n from swrd.search_papers_keyword('homelessness', 20) where publication_year >= 2015
nemotron-c220.5 sSQL error: a column definition list is redundant for a function with OUT parameters
SELECT COUNT(*) FROM (
   SELECT p.publication_year
   FROM swrd.search_papers_keyword('homelessness', 20) AS s(paper_id int, rank int, similarity float)
   JOIN swrd.papers p ON p.id = s.paper_id
   WHERE p.publication_year >= 2015
) t
gemma4:31b25.1 scorrect
SELECT count(*) FROM swrd.search_papers_keyword('homelessness', 20) WHERE publication_year >= 2015
ornith:35b50.0 scorrect
select count(*) from swrd.search_papers_keyword('homelessness', 20) where publication_year >= 2015
qwen3.620.3 scorrect
select count(*) from swrd.search_papers_keyword('homelessness', 20) where publication_year >= 2015
35SWRDUsing the SWRD keyword search function, which journal published the top-ranked article for 'child welfare workforce turnover'?

Reference answer: Social Work

gemma4:26b13.1 sSQL error: column p.journal_id does not exist
SELECT j.name FROM swrd.search_papers_keyword('child welfare workforce turnover', 1) AS p JOIN swrd.journals j ON j.id = p.journal_id
qwen3.6:27b101.8 scorrect
select j.name from swrd.search_papers_keyword('child welfare workforce turnover', 1) sp join swrd.papers p on p.id = sp.id join swrd.journals j on j.id = p.journal_id
nemotron-c222.6 scorrect
SELECT pj.journal_name
FROM swrd.search_papers_keyword('child welfare workforce turnover', 1) AS k
JOIN swrd.papers_with_journals pj ON pj.id = k.id
gemma4:31b37.1 sSQL error: column p.journal_id does not exist
SELECT j.name FROM swrd.search_papers_keyword('child welfare workforce turnover', 1) p JOIN swrd.journals j ON p.journal_id = j.id
ornith:35b31.8 scorrect
select j.name as journal_name from swrd.search_papers_keyword('child welfare workforce turnover', 1) s join swrd.papers p on p.id = s.id join swrd.journals j on j.id = p.journal_id
qwen3.61800.0 snon-terminating generation: at temperature 0 the model reasons in circles and never emits an answer (>94,000 tokens over a 30-minute limit; reproduced three tim
(no SQL produced)
36SWRDUsing the SWRD keyword search function with match_count 10, how many of the top 10 results for 'social work education' appeared in the journal named 'Journal of Social Work Education'?

Reference answer: 0

gemma4:26b29.5 scorrect
SELECT count(*) FROM swrd.search_papers_keyword('social work education', 10) AS s JOIN swrd.papers p ON s.id = p.id JOIN swrd.journals j ON p.journal_id = j.id WHERE j.name = 'Journal of Social Work Education'
qwen3.6:27b163.1 sSQL error: column spk.journal_id does not exist
select count(*) from swrd.search_papers_keyword('social work education', 10) spk join swrd.journals j on j.id = spk.journal_id where j.name = 'Journal of Social Work Education'
nemotron-c218.0 sSQL error: syntax error at or near "SELECT"
SELECT id
   FROM swrd.search_papers_keyword('social work education', 10)
)
SELECT COUNT(*) FROM top10 t JOIN swrd.journals j ON j.id = t.id WHERE j.name = 'Journal of Social Work Education'
gemma4:31b55.1 scorrect
select count(*) from swrd.search_papers_keyword('social work education', 10) s join swrd.papers p on s.id = p.id join swrd.journals j on p.journal_id = j.id where j.name = 'Journal of Social Work Education'
ornith:35b37.3 scorrect
select count(*) from swrd.search_papers_keyword('social work education', 10) s join swrd.papers p on p.id = s.id join swrd.journals j on j.id = p.journal_id where j.name = 'Journal of Social Work Education'
qwen3.658.2 scorrect
SELECT count(*) FROM swrd.search_papers_keyword('social work education', 10) AS res JOIN swrd.papers p ON p.id = res.id JOIN swrd.journals j ON j.id = p.journal_id WHERE j.name = 'Journal of Social Work Education'
37SSWRUsing the SSWR keyword search function, what is the id of the top-ranked presentation about 'opioid overdose'?

Reference answer: 2021-P-0475

gemma4:26b4.6 scorrect
SELECT id FROM sswr.search_papers_keyword('opioid overdose', 1)
qwen3.6:27b29.7 scorrect
select id from sswr.search_papers_keyword('opioid overdose', 1)
nemotron-c214.7 scorrect
select id from sswr.search_papers_keyword('opioid overdose', 1) limit 1
gemma4:31b12.9 scorrect
select id from sswr.search_papers_keyword('opioid overdose', 1)
ornith:35b7.2 scorrect
select id from sswr.search_papers_keyword('opioid overdose', 1)
qwen3.612.0 scorrect
select id from sswr.search_papers_keyword('opioid overdose', 1)
38SSWRUsing the SSWR keyword search function with match_count 20, how many of the top 20 results for 'food insecurity' are from year 2020 or later?

Reference answer: 9

gemma4:26b7.3 scorrect
SELECT count(*) FROM sswr.search_papers_keyword('food insecurity', 20) WHERE year >= 2020
qwen3.6:27b87.7 scorrect
select count(*) from sswr.search_papers_keyword('food insecurity', 20) where year >= 2020
nemotron-c215.1 scorrect
SELECT COUNT(*) FROM (SELECT year FROM sswr.search_papers_keyword('food insecurity', 20)) AS sub WHERE year >= 2020
gemma4:31b25.5 scorrect
select count(*) from (select year from sswr.search_papers_keyword('food insecurity', 20)) as results where year >= 2020
ornith:35b33.4 scorrect
select count(*) from (select year from sswr.search_papers_keyword('food insecurity', 20)) as t where year >= 2020
qwen3.627.1 scorrect
SELECT count(*) FROM sswr.search_papers_keyword('food insecurity', 20) AS t WHERE t.year >= 2020
39SSWRUsing the SSWR keyword search function, in what year was the top-ranked presentation about 'artificial intelligence' given?

Reference answer: 2026

gemma4:26b6.3 scorrect
SELECT year FROM sswr.search_papers_keyword('artificial intelligence', 1) LIMIT 1
qwen3.6:27b23.6 scorrect
select year from sswr.search_papers_keyword('artificial intelligence', 1)
nemotron-c29.4 scorrect
SELECT year FROM sswr.search_papers_keyword('artificial intelligence', 1) LIMIT 1
gemma4:31b18.2 scorrect
select year from sswr.search_papers_keyword('artificial intelligence', 1) limit 1
ornith:35b12.7 scorrect
select year, title from sswr.search_papers_keyword('artificial intelligence', 1) limit 1
qwen3.67.1 scorrect
select year from sswr.search_papers_keyword('artificial intelligence', 1)
40SSWRUsing the SSWR keyword search function with match_count 15, how many of the top 15 results for 'telehealth' use quantitative methodology?

Reference answer: 10

gemma4:26b11.1 scorrect
SELECT count(*) FROM sswr.search_papers_keyword('telehealth', 15) AS results JOIN sswr.papers p ON p.id = results.id WHERE p.methodology = 'quantitative'
qwen3.6:27b79.9 scorrect
select count(*) from sswr.search_papers_keyword('telehealth', 15) where methodology = 'quantitative'
nemotron-c217.2 sSQL error: function sswr.search_articles_keyword(unknown, integer) does not exist
SELECT COUNT(*) FROM (SELECT methodology FROM sswr.search_articles_keyword('telehealth', 15)) sub WHERE methodology = 'quantitative'
gemma4:31b31.4 scorrect
select count(*) from (select methodology from sswr.search_papers_keyword('telehealth', 15)) as results where methodology = 'quantitative'
ornith:35b50.4 scorrect
select count(*) from sswr.search_papers_keyword('telehealth', 15) where methodology = 'quantitative'
qwen3.614.7 scorrect
select count(*) from sswr.search_papers_keyword('telehealth', 15) as res join sswr.papers p on p.id = res.id where p.methodology = 'quantitative'
F · Complex analytical8 questions · 48/48 strict passes · click to expand
41SWRDAmong SWRD journals with at least 1,000 articles from 1989 onward, which journal has the highest proportion of Qualitative articles among its empirical articles? Give the journal name.

Reference answer: Qualitative Social Work

gemma4:26b16.8 scorrect
SELECT j.name FROM swrd.papers p JOIN swrd.journals j ON p.journal_id = j.id WHERE p.publication_year >= 1989 GROUP BY j.name HAVING COUNT(p.id) >= 1000 ORDER BY SUM(CASE WHEN p.is_empirical = true AND p.research_method = 'Qualitative' THEN 1 ELSE 0 END)::float / NULLIF(SUM(CASE WHEN p.is_empirical = true THEN 1 ELSE 0 END), 0) DESC LIMIT 1
qwen3.6:27b174.7 scorrect
select j.name from swrd.papers p join swrd.journals j on j.id = p.journal_id where p.publication_year >= 1989 group by j.id, j.name having count(*) >= 1000 order by sum(case when p.is_empirical and p.research_method = 'Qualitative' then 1 else 0 end)::float / nullif(sum(case when p.is_empirical then 1 else 0 end), 0) desc limit 1
nemotron-c230.1 scorrect
SELECT jc.name AS journal_name
FROM (
    SELECT p.journal_id,
           COUNT(*) AS emp_total,
           SUM(CASE WHEN p.research_method = 'Qualitative' THEN 1 ELSE 0 END) AS qual_count
    FROM swrd.papers p
    WHERE p.publication_year >= 1989 AND p.is_empirical
    GROUP BY p.journal_id
) e
JOIN (
    SELECT j.id, j.name, COUNT(*) AS total_papers_1989plus
    FROM swrd.papers p
    JOIN swrd.journals j ON j.id = p.journal_id
    WHERE p.publication_year >= 1989
    GROUP BY j.id, j.name
) jc ON e.journal_id = jc.id
WHERE jc.total_papers_1989plus >= 1000
ORDER BY (e.qual_count::numeric / NULLIF(e.emp_total,0)) DESC
LIMIT 1
gemma4:31b82.7 scorrect
SELECT j.name FROM swrd.journals j JOIN (SELECT journal_id, COUNT(*) as total_count, SUM(CASE WHEN is_empirical THEN 1 ELSE 0 END) as empirical_count, SUM(CASE WHEN is_empirical AND research_method = 'Qualitative' THEN 1 ELSE 0 END) as qual_count FROM swrd.papers WHERE publication_year >= 1989 GROUP BY journal_id) p ON j.id = p.journal_id WHERE p.total_count >= 1000 AND p.empirical_count > 0 ORDER BY (p.qual_count::float / p.empirical_count) DESC LIMIT 1
ornith:35b70.5 scorrect
select j.name
from swrd.papers p
join swrd.journals j on j.id = p.journal_id
where p.publication_year >= 1989
group by j.name
having count(*) >= 1000
order by (count(*) filter (where is_empirical and research_method = 'Qualitative')::numeric / nullif(count(*) filter (where is_empirical), 0)) desc
limit 1
qwen3.645.2 scorrect
select j.name, count(case when p.is_empirical and p.research_method = 'Qualitative' then 1 end) * 1.0 / count(case when p.is_empirical then 1 end) as prop from swrd.papers p join swrd.journals j on j.id = p.journal_id where p.publication_year >= 1989 group by j.name having count(*) >= 1000 order by prop desc limit 1
42SWRDHow many SWRD papers from 1989 onward have more than 5 authors (based on authorship records)?

Reference answer: 6439

gemma4:26b21.2 scorrect
SELECT COUNT(*) FROM (SELECT p.id FROM swrd.papers p JOIN swrd.paper_authors pa ON p.id = pa.paper_id WHERE p.publication_year >= 1989 GROUP BY p.id HAVING COUNT(pa.author_id) > 5) AS subquery
qwen3.6:27b117.9 scorrect
SELECT COUNT(*) FROM (SELECT pa.paper_id FROM swrd.paper_authors pa JOIN swrd.papers p ON p.id = pa.paper_id WHERE p.publication_year >= 1989 GROUP BY pa.paper_id HAVING COUNT(pa.author_id) > 5) AS t
nemotron-c216.8 scorrect
SELECT COUNT(*) FROM ( SELECT p.id FROM swrd.papers p JOIN swrd.paper_authors pa ON p.id = pa.paper_id WHERE p.publication_year >= 1989 GROUP BY p.id HAVING COUNT(pa.author_id) > 5 ) AS sub
gemma4:31b29.5 scorrect
select count(*) from (select p.id from swrd.papers p join swrd.paper_authors pa on p.id = pa.paper_id where p.publication_year >= 1989 group by p.id having count(pa.author_id) > 5) as subquery
ornith:35b12.7 scorrect
SELECT COUNT(*) FROM (SELECT p.id FROM swrd.papers p JOIN swrd.paper_authors pa ON p.id = pa.paper_id WHERE p.publication_year >= 1989 GROUP BY p.id HAVING COUNT(pa.author_id) > 5) subquery
qwen3.627.0 scorrect
select count(*) from (select p.id from swrd.papers p join swrd.paper_authors pa on pa.paper_id = p.id where p.publication_year >= 1989 group by p.id having count(pa.author_id) > 5) t
43SWRDBetween 1989 and 2023, which single publication year had the most SWRD articles with research_method 'Review'?

Reference answer: 2022

gemma4:26b5.9 scorrect
SELECT publication_year FROM swrd.papers WHERE publication_year >= 1989 AND publication_year <= 2023 AND research_method = 'Review' GROUP BY publication_year ORDER BY COUNT(*) DESC LIMIT 1
qwen3.6:27b72.9 scorrect
select publication_year, count(*) as n from swrd.papers where publication_year >= 1989 and publication_year <= 2023 and research_method = 'Review' group by publication_year order by n desc limit 1
nemotron-c29.6 scorrect
SELECT publication_year, COUNT(*) AS n FROM swrd.papers WHERE publication_year >= 1989 AND publication_year <= 2023 AND research_method = 'Review' GROUP BY publication_year ORDER BY n DESC LIMIT 1
gemma4:31b18.7 scorrect
select publication_year from swrd.papers where publication_year >= 1989 and publication_year <= 2023 and research_method = 'Review' group by 1 order by count(*) desc limit 1
ornith:35b5.7 scorrect
select publication_year, count(*) as n from swrd.papers where publication_year >= 1989 and publication_year <= 2023 and research_method = 'Review' group by 1 order by 2 desc limit 1
qwen3.68.8 scorrect
select publication_year, count(*) as n from swrd.papers where research_method = 'Review' and publication_year between 1989 and 2023 group by 1 order by 2 desc limit 1
44SWRDHow many SWRD journals published at least one article in 1989 AND at least one article in 2025?

Reference answer: 27

gemma4:26b11.2 scorrect
SELECT count(*) FROM (SELECT journal_id FROM swrd.papers WHERE publication_year IN (1989, 2025) GROUP BY journal_id HAVING COUNT(DISTINCT publication_year) = 2) AS common_journals
qwen3.6:27b128.6 scorrect
select count(*) from (select journal_id from swrd.papers where publication_year in (1989, 2025) group by journal_id having count(distinct publication_year) = 2) t
nemotron-c213.1 scorrect
SELECT COUNT(DISTINCT j.id) FROM swrd.papers p1 JOIN swrd.journals j ON j.id = p1.journal_id WHERE p1.publication_year = 1989 AND EXISTS (SELECT 1 FROM swrd.papers p2 WHERE p2.journal_id = j.id AND p2.publication_year = 2025)
gemma4:31b23.9 scorrect
select count(*) from (select journal_id from swrd.papers where publication_year = 1989 intersect select journal_id from swrd.papers where publication_year = 2025) as common_journals
ornith:35b11.5 scorrect
SELECT COUNT(*) FROM (SELECT journal_id FROM swrd.papers WHERE publication_year IN (1989, 2025) GROUP BY journal_id HAVING COUNT(DISTINCT publication_year) = 2) sub
qwen3.615.8 scorrect
SELECT count(*) FROM swrd.journals j WHERE EXISTS (SELECT 1 FROM swrd.papers p WHERE p.journal_id = j.id AND p.publication_year = 1989) AND EXISTS (SELECT 1 FROM swrd.papers p WHERE p.journal_id = j.id AND p.publication_year = 2025)
45SSWRWhich SSWR canonical author (give the name) had the most first-authored presentations between 2010 and 2015?

Reference answer: Michal Grinstein-Weiss

gemma4:26b13.4 scorrect
SELECT a.name FROM sswr.authors a JOIN sswr.paper_authors pa ON a.id = pa.author_id JOIN sswr.papers p ON p.id = pa.paper_id WHERE p.year BETWEEN 2010 AND 2015 AND pa.author_order = 1 GROUP BY a.name ORDER BY COUNT(*) DESC LIMIT 1
qwen3.6:27b67.8 scorrect
select a.name from sswr.papers p join sswr.paper_authors pa on p.id = pa.paper_id join sswr.authors a on pa.author_id = a.id where p.year between 2010 and 2015 and pa.author_order = 1 group by a.id, a.name order by count(*) desc limit 1
nemotron-c214.5 scorrect
SELECT a.name
FROM sswr.paper_authors pa
JOIN sswr.authors a ON a.id = pa.author_id
JOIN sswr.papers p ON p.id = pa.paper_id
WHERE pa.author_order = 1
  AND p.year BETWEEN 2010 AND 2015
GROUP BY a.name, a.id
ORDER BY COUNT(*) DESC
LIMIT 1
gemma4:31b40.5 scorrect
SELECT a.name FROM sswr.paper_authors pa JOIN sswr.papers p ON pa.paper_id = p.id JOIN sswr.authors a ON pa.author_id = a.id WHERE p.year BETWEEN 2010 AND 2015 AND pa.author_order = 1 GROUP BY a.name ORDER BY COUNT(*) DESC LIMIT 1
ornith:35b26.4 scorrect
select a.name, count(*) as n from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id join sswr.authors a on a.id = pa.author_id where pa.author_order = 1 and p.year between 2010 and 2015 group by 1 order by 2 desc limit 1
qwen3.625.3 scorrect
select a.name, count(*) as n from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id join sswr.authors a on a.id = pa.author_id where pa.author_order = 1 and p.year between 2010 and 2015 group by a.name order by n desc limit 1
46SSWRHow many SSWR canonical authors presented at BOTH the 2005 and 2026 conferences?

Reference answer: 79

gemma4:26b19.6 scorrect
SELECT count(*) FROM (SELECT pa.author_id FROM sswr.paper_authors pa JOIN sswr.papers p ON p.id = pa.paper_id WHERE p.year IN (2005, 2026) GROUP BY pa.author_id HAVING COUNT(DISTINCT p.year) = 2) AS common_authors
qwen3.6:27b174.3 scorrect
select count(*) from (select pa.author_id from sswr.paper_authors pa join sswr.papers p on pa.paper_id = p.id where p.year in (2005, 2026) group by pa.author_id having count(distinct p.year) = 2) t
nemotron-c219.0 scorrect
SELECT COUNT(*) AS n FROM (SELECT DISTINCT author_id FROM sswr.paper_authors pa JOIN sswr.papers p1 ON p1.id = pa.paper_id WHERE p1.year = 2005) a JOIN (SELECT DISTINCT author_id FROM sswr.paper_authors pa JOIN sswr.papers p2 ON p2.id = pa.paper_id WHERE p2.year = 2026) b USING (author_id)
gemma4:31b45.0 scorrect
select count(*) from (select pa.author_id from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where p.year = 2005 intersect select pa.author_id from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where p.year = 2026) as common_authors
ornith:35b53.7 scorrect
select count(*) from (select distinct pa.author_id from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where p.year = '2005' intersect select distinct pa.author_id from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where p.year = '2026') as t
qwen3.621.4 scorrect
select count(*) from (select pa.author_id from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where p.year = 2005 intersect select pa.author_id from sswr.paper_authors pa join sswr.papers p on p.id = pa.paper_id where p.year = 2026) as common
47SSWRWhich SSWR conference year had the largest number of distinct normalized institutions among first authors?

Reference answer: 2026

gemma4:26b8.1 scorrect
SELECT p.year FROM sswr.papers p JOIN sswr.paper_authors pa ON p.id = pa.paper_id WHERE pa.author_order = 1 GROUP BY p.year ORDER BY COUNT(DISTINCT pa.institution_normalized) DESC LIMIT 1
qwen3.6:27b90.6 scorrect
select p.year, count(distinct pa.institution_normalized) as distinct_institutions from sswr.papers p join sswr.paper_authors pa on p.id = pa.paper_id where pa.author_order = 1 and pa.institution_normalized is not null group by p.year order by distinct_institutions desc limit 1
nemotron-c210.4 scorrect
SELECT p.year
FROM sswr.paper_authors pa
JOIN sswr.papers p ON p.id = pa.paper_id
WHERE pa.author_order = 1
GROUP BY p.year
ORDER BY COUNT(DISTINCT pa.institution_normalized) DESC
LIMIT 1
gemma4:31b29.4 scorrect
SELECT p.year FROM sswr.papers p JOIN sswr.paper_authors pa ON p.id = pa.paper_id WHERE pa.author_order = 1 GROUP BY p.year ORDER BY count(distinct pa.institution_normalized) DESC LIMIT 1
ornith:35b10.1 scorrect
select p.year, count(distinct pa.institution_normalized) as n_institutions from sswr.papers p join sswr.paper_authors pa on pa.paper_id = p.id where pa.author_order = 1 group by 1 order by 2 desc limit 1
qwen3.69.0 scorrect
select p.year, count(distinct pa.institution_normalized) as n_institutions from sswr.papers p join sswr.paper_authors pa on pa.paper_id = p.id where pa.author_order = 1 group by p.year order by n_institutions desc limit 1
48SSWRHow many SSWR presentations have more than 6 authors (based on authorship records)?

Reference answer: 1083

gemma4:26b4.3 scorrect
SELECT count(*) FROM sswr.papers WHERE author_count > 6
qwen3.6:27b132.3 scorrect
select count(*) from sswr.papers where author_count > 6
nemotron-c214.7 scorrect
SELECT COUNT(*) FROM (SELECT paper_id FROM sswr.paper_authors GROUP BY paper_id HAVING COUNT(*) > 6) AS sub
gemma4:31b26.3 scorrect
select count(*) from (select paper_id from sswr.paper_authors group by paper_id having count(author_id) > 6) as subquery
ornith:35b6.0 scorrect
select count(*) from (select paper_id from sswr.paper_authors group by paper_id having count(*) > 6) t
qwen3.620.9 scorrect
select count(*) from sswr.papers where author_count > 6
04 Honest limits & reproduction

What this shows, and how to rerun it

It shows that free, locally run models can reliably answer routine research questions against these databases, that precise skill files carry compact models a long way, and that the errors that remain concentrate in search-function usage and complex analytical queries.

It doesn't show multi-turn analysis, semantic-search orchestration, or robustness across phrasings: the benchmark is single-turn, one run per question, at temperature 0. It is a structured existence proof, not a leaderboard.

Reproduce with any Ollama model: python migration/08_eval_local_models.py <model> …. The harness, questions, reference queries, and raw outputs are in the repository.