How Far Have Open-Weight LLMs Come? Benchmarking Red- and Green-Flag Extraction on SEC 8-Ks

June 28, 2026 • 8 min read • Arkadij Kummer

#LLM #Benchmarks #AI #Evaluation #SEC Filings #Red Flags #Green Flags #Open Weight Models #Frontier Models #Model Risk #Langfuse #Methodology

We run flag extraction over SEC filings as a product feature. A model reads an 8-K and returns the signals an analyst should look at first.

On the warning side, those signals are red flags: going-concern warnings, debt defaults, toxic convertibles, auditor resignations. On the positive side, they're green flags: earnings beats, buybacks, contract wins, regulatory approvals.

The question was model selection. How much recall do we lose, or gain, if we use open-weight models instead of closed frontier models for this kind of filing work?

We tested 5 current LLMs, 2 older open-weight models, and a deterministic keyword baseline. Each model read the same real 8-Ks 3 times, once for red flags and once for green flags.

The main score is pooled_coverage: the share of consensus material flags a model found. A score of 0.70 means the model recovered 70% of the frozen reference flags for that task.

The result varies by task and by model. On green flags, GLM-5.2 finishes 2nd, ahead of GPT-5.5 and Sonnet. On red flags, Opus and Sonnet stay on top, GLM-5.2 ties GPT-5.5 just behind them, and Kimi lands far back.

The generational story also splits. GLM roughly doubled its green-flag score between GLM-5 and GLM-5.2 and gained 0.27 on red. Kimi moved a little on red and stayed flat on green.

The task and the setup

Each model gets the full filing text. It has to return material findings, each with a category, a severity, and a verbatim evidence quote.

The red and green tasks use the same shape. Red asks for warning signs. Green asks for positive signals. Running both lets us see which conclusions repeat across mirrored tasks.

There's no labeled answer key for this at scale, and hand-labeling thousands of filings is the work we're automating. So the eval is reference-free: outputs get judged against the filing itself and against a frozen pooled consensus.

The current-model contestants are Opus 4.8, Sonnet 4.6, GPT-5.5, GLM-5.2, and Kimi K2.6. We also include GLM-5 and Kimi K2.5 to measure family-level progress, plus a deterministic keyword baseline that emits the matched span as evidence and costs nothing.

Each filing runs through our production screener prompts: financial, governance, and general for red; the matching positive-signal set for green. Findings are merged and deduped by category. Every model runs every filing 3 times, with per-call cost recorded.

The red corpus is 23 real 8-Ks: 16 from companies our production detector had already flagged heavily, 7 from a random recent sample. The green corpus is 20 real 8-Ks picked the same way plus 2 built clean controls. Under the consensus reference, 13 red filings carry 39 material flags and 10 come back clean; on green it's 16 filings with 34 flags and 6 clean.

How we score without gold labels

The eval has no hand-labeled answer key, so the first step is building a reference.

Gemini 3.1 Pro reads the full filing plus the union of every contestant's material findings, then writes one deduplicated list of supported flags. That list is built once and frozen.

The leaderboard metric is pooled_coverage: the share of frozen reference flags a model found. Every run of every model is graded against the same list.

Two checks sit beside that score. relevance asks whether findings are real and material rather than padding. evidence_grounding asks whether cited quotes are exact substrings of the filing.

Relevance is high almost everywhere, 0.88 to 1.0 on red flags. The exceptions are Sonnet, lowest among the frontier models on both tasks (0.80 red, 0.84 green), and the keyword baseline on green (0.63).

Strict grounding rejects about a third of findings, mostly for stitched multi-passage quotes, punctuation drift, and compressed quotations. Spot checks found real filing text behind them.

The figures use pooled_coverage. Alternate graders test whether the ranking depends on who matches model findings to the frozen reference.

The synthesizer can add a flag it reads in the filing that no contestant reported. Those flags penalize every contestant equally, and the pool-bias checks below drop them.

The honest caveat: the model that writes the reference also grades against it. The stress tests below attack that from 3 sides: an open-weight grader from a different family, a no-LLM lexical grader, and a manual audit of all 39 red flags against the filing text.

Grader agreement

Before ranking models, we check whether the ranking depends on the grader. The next chart shows each model's average score under 3 graders: Gemini, an open-weight judge, and a deterministic lexical matcher.

Coverage by grader · per model, mean of 3 runsGemini judgeGLM judgelexical

Each row is a model; the 3 dots are its score under each grader, so a tight row means the grading method barely matters. The hollow lexical dots trail the LLM judges furthest for the highest-scoring models. On red flags GLM-5.2's dots sit inside the closed models' range and it takes the top lexical score; Kimi K2.6 sits far left under all 3 graders. On green flags Opus and GLM-5.2 lead under every grader.

The shape repeats under every grader. On red flags, Opus and Sonnet sit on top. GLM-5.2 ties GPT-5.5, and Kimi trails by roughly 0.3 under all 3.

Grader choice moves the level and barely moves the order. That makes the ranking more useful than any single score.

We report models rather than a closed-versus-open average because the group gap depends on which open models you include. Against GLM-5.2 alone, the closed trio leads by about 0.03, inside run-to-run noise. Against Kimi, it leads by about 0.32.

The lexical grader is plain word matching. A reference flag counts as covered if one of the model's findings repeats at least 40% of the words in the flag's name: a finding titled "Delisting risk after Nasdaq notice" covers the flag "Nasdaq Non-Compliance / Delisting Risk", while a paraphrase like "listing standards problem" misses it.

Word-overlap metrics inherit the wording of their reference, and our flag names are LLM-written, so we treat this grader as the weakest of the 3: a deterministic check that removes judge randomness at match time rather than an LLM-free source of truth.

Its penalty also grows with the score. On red flags, the baseline and Kimi each give up about 0.03 under word matching, while the top cluster gives up about 0.10. Green stretches further.

That shape is expected. The extra flags a stronger model finds are the subtle ones, where its wording and the reference's drift apart, so the lexical grader taxes exactly the recall that separates the leaders.

The leaderboard on both tasks

One run of a stochastic model is a draw, and the draws differ. Kimi K2.6 pulled 0.46, 0.41, and 0.28 on 3 identical red runs.

So the leaderboard reports the mean of 3 runs, with the run-to-run spread shown as error bars.

pooled_coverage by task · mean ±std, N=3closedopenolderbaselinered rankgreen rank

Both task panels share the model order; the tabs switch it between red-flag rank and green-flag rank. Opus (0.73) and Sonnet (0.70) top red flags with GLM-5.2 and GPT-5.5 tied at 0.68; on green flags GLM-5.2 moves to 2nd while Sonnet and both Kimi versions drop sharply. The keyword baseline is the floor on each (0.13 red, 0.09 green).

Opus and Sonnet top red flags. GLM-5.2 ties GPT-5.5 at 0.68.

Green reshuffles the order. GLM-5.2 climbs to 2nd while Sonnet falls to 4th, and the whole field scores lower.

The reshuffle is task sensitivity. From red to green, Opus and GLM-5.2 each give up about 0.11 while GPT-5.5 gives up 0.20 and Sonnet 0.29; Opus's margin over GLM-5.2 stays near 0.05 on both tasks.

The leaderboard says how much each model found. The flag matrix shows what it found.

Each column is a consensus flag. Each cell shows how many of the model's 3 runs covered that flag.

Every consensus flag · runs found of 3by difficultyby severityred flagsgreen flags

runs found:0-3criticalhighmediumlow

Red flags: the left half is common ground (the grep catches 5 of the first 6); the right tail is caught almost only by the closed trio, and the last column escapes every model (a flag the synthesizer took straight from the filing text; no analyst reported it). Per-model totals reproduce the frozen scores except 1 borderline match (GPT-5.5, run 1).

Green flags: 3 consensus flags escape every model (synthesizer-added from the text, or reworded past their source finding), and Opus alone catches the deal fine print (reverse termination fees, voting agreements). Totals reproduce the frozen scores except 3 borderline matches.

The flags that separate the field sit in deal fine print rather than headline events. Every model catches a bankruptcy filing or a delisting notice.

The columns covered almost only by the closed trio come from 2 long transaction filings: a tax receivable agreement that routes 85% of certain tax benefits to the new controlling shareholder, a listing obligation that can leave shareholders unable to sell for up to 5 years, and a super-priority loan at SOFR plus 9% with interest paid in kind.

The rows show stability. Opus covers 28 or 29 of the 39 red flags in every run, 21 of them the same flags each time. Kimi K2.6 swings between 11 and 18.

Recall per dollar

Extraction cost vs recall · N=3red flagsgreen flags

Extraction cost only (screener calls; the fixed judge overhead is excluded, so the baseline is $0). Faded points are the previous open generation. GLM-5.2 ($0.34) ties GPT-5.5 (closed, $1.91) on recall at about a sixth of the cost.

Extraction cost only (screener calls; the fixed judge overhead is excluded, so the baseline is $0). Faded points are the previous open generation. GLM-5.2 ($0.41) beats GPT-5.5 (closed, $2.41) on recall at a sixth the cost.

The keyword baseline prices the floor. It costs $0 a run, catches obvious going-concern and default language, and misses subtle findings like nepotism, related-party structures, and dilution in a financing footnote.

Its evidence is verbatim by construction. The frontier models recover 3 to 6 times as much of the consensus, so the cost question is what that extra recall costs.

GLM-5.2 is the cost-efficient pick. It ties GPT-5.5 on red at $0.34 a run against $1.91, and beats it on green at $0.41 against $2.41. The recall crown costs $2.27 a run at Opus.

How fast the open frontier is moving

Jump from slightly older version · red vs greenGLMKimi

GLM gained +0.27 on red flags (0.40 to 0.68) and doubled on green (+0.30, from below GPT-5.5 to well above it). Kimi gained +0.11 on red and did not move on green (K2.6 equals K2.5 at 0.23).

GLM gained +0.27 on red between GLM-5 and GLM-5.2, roughly 5 times its remaining distance to Opus. On green, it doubled and crossed GPT-5.5 on the way up.

Kimi gained +0.11 on red and stayed flat on green.

The open frontier moves at 2 speeds. Benchmark the specific model you plan to ship; category labels tell you very little.

Stress-testing the result

We tried to break the red result 4 ways.

The grader check is the chart above: the ordering holds under an open-weight judge and under plain word matching.

Pool composition gives the same answer. Scoring only flags at least 2 models found, and again with a leave-one-out rule that drops flags only the graded model contributed, keeps the same top-to-bottom order. Both corrections also remove the synthesizer-added flags no model covered.

Severity calibration points the same way. Closed and open models rate severity within 0.01 of each other, so the material-pool filter favors neither side.

The manual audit checked all 39 consensus red flags against the filing text. Every evidence quote checked out as verbatim. Our read is 34 solid flags, 4 with debatable severity, and 1 mislabeled; dropping that one moves scores by under 0.03.

The audit also found 2 material flags the consensus missed. A pooled reference is a floor, so a coverage of 1.0 would still mean "found everything the pool surfaced" rather than "found everything."

What we'd actually conclude

The model pick barely depends on the task. GLM-5.2 ties GPT-5.5 on red flags and beats it on green, at about a sixth of the cost either way. Opus buys its constant 0.05 edge at roughly 7 times GLM's price ($2.27 a run on red, $3.21 on green), and Sonnet keeps Opus's pace only on red.

The closed-versus-open question has become model-specific. GLM-5.2 sits inside the closed cluster on both tasks, while Kimi trails everywhere.

Opus's lead over GLM-5.2 is the same 0.05 on both tasks, showing up on red in the fine-print tail of the flag matrix; Sonnet matches that pace only on red. A group average would blur that model-level split.

Set against that, GLM gained +0.27 in a jump between 2 versions, roughly 5 times its remaining distance to Opus. We read the picture as a real lead for the top closed models with a short half-life, for one open family at least.

The limits are narrow. The eval covers 39 red flags on 13 filings and 34 green flags on 16 filings. The intervals measure run-to-run reproducibility on fixed corpora of 23 red and 22 green filings.

The reference is LLM-written, frozen, lexically cross-checked, and audited flag by flag. Our read found 1 mislabel in 39 red flags.

Contamination is handled by construction. The eval is reference-free, so there's no gold label a model could have memorized, and the full filing sits in the prompt at inference, so having seen it in training confers no edge.

The lesson for anyone standing up an LLM eval: put a deterministic baseline in the lineup, benchmark on real data rather than synthetic fixtures (planted flags are easy for every model, and an easy task hides gaps), build the reference with a strong model and freeze it, run the stochastic models more than once, and draw the error bars before you draw conclusions. Where you can, run a mirror task; a second benchmark that should replicate is the cheapest way to catch a finding that only holds on one dataset.

Where we sit

We build Bollwerk for the second line of defense: risk, compliance, and financial-crime teams. Flag extraction over filings, on the warning side and the positive side, is one of the model-backed features inside that product, which is why the model choice gets an eval rather than a vibe.

The methodology here is the same one we'd want behind any model-risk decision: a deterministic floor, repeated runs, a mirror task, and error bars that decide what's real before the leaderboard does. If your team is making model-selection calls for compliance workloads and would find it useful to compare notes, write to hello@bollwerk.ai.