How My News Pipeline Handles 71 Different AI Model Setups

🔍 Read the full analysis: How My News Pipeline Handles 71 Different AI Model Setups on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he tested 71 AI model configurations on news-writing tasks and kept GPT-6 Luna in production despite GPT-6 Astra receiving the highest scores. In a report published Oct. 13, Meyer said Astra scored 4.00 out of 5 in both English and German, but cost about 100 times more per article; he also said two backup models were removed after tests found fabricated quotations or unusable output.

The test, conducted from Sept. 8 to Oct. 13, used Meyer’s production prompt and real news sources. It covered API and subscription models, free-tier options on OpenRouter, and open-weight models run on his own hardware. Each source was turned into an English and a German article, then evaluated by two AI judges: GPT-6.1 Sol and Claude Opus 5.5.

Meyer tested a longer source of 6,340 characters and a shorter one of 619 characters, which he said better reflects many items in his workflow. The judges assessed source fidelity, fabricated material, clarity about uncertainty, news value, structure and polish; German articles also received a language-quality score. A later robustness run used 10 additional sources for the strongest configurations. Meyer said its results are more reliable than the initial two-source comparison and reported 95% bootstrap intervals.

Astra scored 3.93 across 12 articles in that robustness run, with a reported interval of 3.83 to 4.02. Luna scored 3.67, with an interval of 3.59 to 3.76, and a source-fidelity score of 4.71 out of 5 in the run. Meyer reported Astra at about $0.12 per article and roughly 50 seconds per article, compared with about $0.0012 and 17 seconds for Luna. In recent production, he said Luna cost about $0.0013 per article at a median of 17 seconds, and every Luna article in the test produced valid output.

Why the Cheaper Model Stayed

The results highlight a practical trade-off for publishers using AI at scale: the model with the highest writing score may not be the one that fits a production budget. Meyer says Astra performed better in his test, but Luna’s lower cost and consistent valid output made it the production choice. That is a decision about one workflow, not a general ranking of models or a finding that Luna is best for every newsroom.

The test also points to a risk beyond the main model’s average score. Meyer reported that two backup models generated invented quotations, a serious failure for news copy because readers may mistake fabricated statements for sourced evidence. He says both were removed from the route on Oct. 7. The report does not independently verify the test or the models’ performance; the findings reflect Meyer’s own setup and judging process.

How the News Pipeline Was Tested

Meyer framed the test as a task-specific alternative to relying on public model leaderboards. Those rankings measure performance on selected benchmarks, he wrote, but do not directly establish which model is suitable for a particular publication’s articles. His comparison instead used the same prompt as his production pipeline and sources drawn from real news material.

He set a rule that differences below about 0.3 points in the initial two-article test should be treated as noise. The larger robustness run was intended to give a more dependable comparison. Meyer also counted an article only when it produced valid output after the same repair step used in production, so a high writing score alone was not enough for a model to qualify.

One test varied GPT-6 Luna’s reasoning-effort setting across six levels. Meyer said the scores showed no reliable improvement from higher effort: the maximum setting cost about 12 times as much and took about 12 times as long as the no-effort production setting, without a score gain he considered meaningful. He kept the no-effort setting in use.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13 report

Limits of the Model Comparison

The findings are Meyer’s own test results, not an independent evaluation. The report describes two judges, a limited initial source set and a 10-article robustness run, but it does not establish how the results would generalize to other publishers, prompts, languages, topics or model versions. The judges also rely on AI assessments, and the source material does not provide independent validation of their scoring.

The report says two backup models were removed on Oct. 7, but the supplied account ends with an incomplete description of the replacement route. It says Luna now runs first and then repeats Luna as the next model; whether that reflects an intended duplicate route or a reporting omission is unclear. Meyer also does not provide enough detail here to independently reproduce every configuration, cost calculation or repair-step outcome.

Production Route After the Test

Meyer says GPT-6 Luna remains the production model, while GPT-6 Astra was not adopted because of its reported cost. The two fallback models he identified were removed from the news route on Oct. 7. The report does not specify a complete replacement failover chain, so it is not clear what model would take over if Luna became unavailable or returned invalid output.

The account describes the test through Oct. 13 but does not announce another evaluation date or a planned switch. Any later decision would depend on fresh results, the pipeline’s quality threshold and costs under actual production conditions. For now, the published evidence supports a narrow conclusion: in Meyer’s tested setup, Astra scored higher, while Luna offered a much lower reported cost and remained his chosen production option.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It scored 3.93 across 12 articles in the robustness run, according to Meyer.

Why did Meyer keep GPT-6 Luna in production?

Meyer said Luna cost about $0.0012 per article in the comparison, around one-hundredth of Astra’s reported $0.12 cost, and consistently produced valid output. He reported its production cost over the past week at about $0.0013 per article.

What happened when the backup models were tested?

Meyer said GPT-4.1 nano scored poorly and generated invented quotations. He reported that the local gpt-oss-120b setup produced zero valid articles out of four and also invented named quotes. He said both were removed from the news route on Oct. 7.

Did higher reasoning effort improve Luna’s articles?

Meyer said score differences across six effort settings were within the test’s noise, with no reliable gain. The maximum setting took and cost about 12 times as much as the no-effort setting, so production stayed at no effort.

Can the results be applied to other newsrooms?

Not directly. The results come from Meyer’s prompt, sources, pipeline and AI judges. They may inform how other publishers design evaluations, but the report does not establish that the same rankings or costs would hold in another newsroom.

Source: ThorstenMeyerAI.com