Could Your Best AI Model Be The Wrong Fit For A News Pipeline?

🔍 Read the full analysis: Could Your Best AI Model Be The Wrong Fit For A News Pipeline? on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get smart everyday buys delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Writer Thorsten Meyer says he tested 71 AI model configurations on real news sources and selected GPT-6 Luna for production, despite GPT-6 Astra receiving the highest writing scores. In Meyer’s test, Astra scored 4.00 out of 5 in both English and German but cost about $0.12 per article, compared with roughly $0.0012 for Luna. The results also led Meyer to remove two backup models after they generated fabricated quotations or failed to produce usable output.

Meyer’s test, described in a report published October 13 on ThorstenMeyerAI.com, ran from September 8 to October 13, 2026. It used the same prompt as his production pipeline and covered API, subscription, free-tier and locally run open-weight models. Each source was turned into an English and a German article, then evaluated by two AI judges against the source for fidelity, fabrication, clarity about uncertainty, news value, structure and polish; German articles were also graded for language quality.

The initial comparison used two sources: a 6,340-character product filing and a 619-character political news item. Meyer said the short source better reflected much of his production work. Stronger configurations also took part in a robustness run that generated 10 additional articles, mostly from short sources. Meyer reported 95% bootstrap intervals and said differences below about 0.3 points in the two-article test should be treated as noise.

Astra scored 3.93 across 12 articles in the robustness run, while Luna scored 3.67. Meyer reported that Astra’s interval, 3.83–4.02, did not overlap Luna’s, 3.59–3.76. But Luna produced valid output for every article in the run, with source fidelity of 4.71 out of 5. Meyer said its production cost over the previous week was about $0.0013 per article, with a median generation time of 17 seconds; Astra took around 50 seconds.

Cost and Quality in Production

The findings highlight a practical distinction for newsrooms using automated writing: the model with the best score in a test may not be the right one to run at scale. Based on Meyer’s figures, Astra’s reported per-article cost was roughly 100 times Luna’s. At high volume, that difference could outweigh a modest improvement in judged writing quality, provided the lower-cost model reliably meets the publisher’s standards.

The test also draws attention to the risk of judging a pipeline only by its main model. Meyer said the two backup systems in his route generated invented quotations or unusable output. In a news workflow, a fallback can become the active writer when a primary service is unavailable. If it is not tested against the same source-fidelity requirements, it can introduce errors precisely when normal checks or operating conditions are strained.

These results are one publisher’s evaluation, not an independent industry-wide ranking. They suggest that organizations should test models on their own source material, languages, prompts and output requirements before choosing a production configuration. They do not establish how the same models would perform for other publishers or subject areas.

How the Newsroom Test Worked

Meyer’s benchmark was designed around a specific task: convert a supplied news source into a finished article using his existing production prompt. It was not a general test of model intelligence or a coding benchmark. The two judges, GPT-6.1 Sol and Claude Opus 5.5, each saw the original source and assessed the articles on a five-point scale.

The report also compared different reasoning-effort settings for Luna. Meyer said the results showed no meaningful gain: the highest setting took 211 seconds and cost about 1.43 cents per article, versus 17 seconds and 0.12 cents at the “none” setting. He reported that score differences were within the test’s noise range, so production remained at the lower setting. That conclusion applies to this source-to-article task and this evaluation, rather than to every use of reasoning settings.

The test’s model comparisons also ran counter to the assumption that performance transfers across tasks. Meyer reported that Claude Opus 5.5, which he described as a leading coding model, scored 3.46 in English and 3.39 in German as a news writer, with source fidelity of 3.75. He said Claude Sonnet 5.5 scored close to Luna but cost about 30 times as much per article at the API-equivalent price used in his report.

“The cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13 report

Limits of the Model Comparison

The results come from one publisher’s test, using a particular prompt, two source examples in the initial comparison, two languages and two AI judges. Although the robustness run added 10 articles for stronger configurations, the report does not establish how the rankings would change across different news topics, longer investigations, breaking-news updates, other languages or another publisher’s editing standards.

The judges were AI systems, and the source material does not describe a separate human grading panel or a blinded evaluation. The report supplies score intervals for its robustness results, but the available account does not provide enough detail to independently verify the scoring process or reproduce all cost calculations. Prices and model availability may also change. The reported figures should be read as Meyer’s measurements for the stated test period, not as guaranteed current costs or a universal model ranking.

Production Testing After the Switch

Meyer said he removed GPT-4.1 nano and gpt-oss-120b from the news route on October 7, leaving GPT-6 Luna as the production model. His report says the latter configuration produced valid articles in the test, but it does not give a longer-term evaluation of accuracy, correction rates or performance across a broader range of live stories.

The next useful evidence would be continued monitoring of Luna on real production sources, including checks for unsupported details, quote accuracy, output-format failures and performance in both language editions. Meyer’s account does not specify a future publication date for another benchmark or say whether the fallback chain will be changed again. The current results therefore describe a reported test and a production choice, while the durability of that choice remains to be established through ongoing use.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the initial test. It scored 3.93 across 12 articles in the robustness run, according to Meyer.

Why did Meyer choose GPT-6 Luna for production?

Meyer said Luna delivered consistent, valid output at roughly $0.0012 to $0.0013 per article, while Astra cost about $0.12 per article. He judged Luna’s quality sufficient for his production needs at the lower reported cost.

Did more reasoning effort improve the articles?

Meyer reported no reliable improvement across Luna’s six reasoning-effort settings. The maximum setting was reported to cost about 12 times as much and take about 12 times as long as the “none” setting, without a clear score gain in this test.

What happened with the backup models?

Meyer said GPT-4.1 nano received a 1.00 source-fidelity score and generated invented quotations in the evaluation. He reported that gpt-oss-120b produced no valid articles in four attempts and also generated fabricated named quotations. Both were removed from his news route on October 7.

Do these results show which model is best for every newsroom?

No. The report covers Meyer’s prompt, sources, languages and production requirements. Other publishers would need to test models on their own material and standards before drawing conclusions about fit or cost.

Source: ThorstenMeyerAI.com