📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva LLM, trained from scratch with extensive Italian data, underperformed on Italian academic benchmarks despite impressive technical results. This raises questions about the scale of native-language investment needed for country-specific knowledge.
Italy’s Minerva-3B, a sovereign language model trained from scratch on 2.5 trillion tokens with roughly 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, highlighting significant challenges in achieving country-specific language understanding despite large-scale investments.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research infrastructure, trained a 7-billion-parameter model from scratch using a dataset of 2.5 trillion tokens, half of which was Italian. The project aimed to demonstrate the feasibility of a European sovereign LLM built entirely with open data and weights, contrasting with other models like Portugal’s AMÁLIA which layered specialization onto multilingual foundations.
Despite the technical success and impressive infrastructure, Minerva-3B’s performance on the INVALSI Italian exam was notably poor, at only 4.9%, a near-chance score. Researchers noted that while dataset size and parameters are crucial, they may not be sufficient for complex language tasks requiring deep country-specific knowledge. This result suggests that simply scaling data and parameters may not guarantee the desired country-specific expertise.
This empirical finding complicates the narrative that larger investments and data volumes automatically lead to better country-specific language understanding in sovereign models. It indicates that a more nuanced approach to native-language investment and model scaling is necessary, with potential implications for European AI policy and infrastructure development.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
large language model training datasets
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Engineering a Small AI Language Model: Training, Evaluation, and Deployment Without Myth
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
open source language model weights
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign LLM Strategies
The poor performance of Minerva on Italian academic tests despite extensive training highlights a crucial challenge for European sovereign-language models: scale alone may not suffice to develop deep country-specific knowledge. This finding questions assumptions about the effectiveness of large-scale training in achieving functional language understanding for complex tasks and suggests that future investments may need to focus on more targeted, high-quality, native-language data and model architectures. The outcome impacts policy debates around national AI sovereignty and strategic resource allocation, emphasizing that achieving true language and knowledge depth requires more than just data volume and model size.
European Sovereign LLM Development and the Scaling Debate
Italy’s Minerva project emerged as a significant case in the European sovereign-LLM movement, which debates whether to train models from scratch or adapt existing multilingual models through continuation pre-training. Minerva trained from scratch on an unprecedented 2.5 trillion tokens, with a focus on Italian content, and published open weights and data, contrasting with Portugal’s AMÁLIA, which layered specialization onto a multilingual foundation. Despite the large-scale effort, Minerva’s performance on Italian benchmarks was disappointing, especially on complex academic tests, revealing limitations in current scaling strategies for country-specific knowledge.
This development underscores ongoing debates about the optimal approach for European AI sovereignty, the importance of native-language data, and the realistic expectations of model scaling. It also reflects broader challenges faced by non-English language models in achieving parity with their English counterparts in complex tasks.
“While the pre-training dataset composition is important, the overall size of the dataset and the number of parameters are more crucial for handling complex language tasks.”
— Research team of Minerva project
Unclear Impact of Native-Language Data Quality and Model Architecture
It remains uncertain whether alternative data curation strategies, model architectures, or training methodologies could improve Minerva’s performance on complex country-specific tasks. The ongoing research aims to explore these avenues, but definitive conclusions are not yet available.
Next Steps for European Sovereign Language Model Development
The Minerva team plans to continue iterating on training methodologies, potentially increasing the scale or refining the data focus, to improve country-specific knowledge. Further benchmarks and evaluations will determine if these adjustments can address the current performance gaps. Additionally, policymakers and AI strategists are expected to reassess investment levels and approaches based on these empirical findings to better align future efforts with realistic outcomes.
Key Questions
Why did Minerva perform so poorly on the Italian exam despite large-scale training?
Research indicates that dataset composition, size, and parameters are important but may not be sufficient alone. Achieving deep country-specific knowledge likely requires targeted, high-quality native-language data and specialized training strategies.
Does this mean training from scratch is ineffective for country-specific models?
Not necessarily. It suggests that scale alone may not be enough. Effective models may need a combination of large-scale data, architecture optimization, and targeted native-language content.
What are the implications for European AI sovereignty efforts?
The findings imply that European projects should reconsider resource allocation and focus on native-language data quality and model design, rather than relying solely on increasing data volume or model size.
Is Minerva an ongoing project, and will its performance improve?
Yes, the team continues to iterate on methodologies, and future training runs may address current limitations. The results so far serve as a critical learning point for future development.
Source: ThorstenMeyerAI.com
