📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva LLM, developed from scratch with extensive Italian data, underperformed on academic benchmarks despite impressive technical results. This highlights the complex challenge of scaling native-language models for country-specific knowledge.
Italy’s Minerva project, a sovereign large language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, despite significant institutional investment. This result questions assumptions about the relationship between training scale and language understanding in national AI initiatives.
Minerva, led by Sapienza University of Rome and supported by Italy’s National Research Council, trained models ranging from 350 million to 7 billion parameters on a dataset of 2.5 trillion tokens, half of which was Italian. The project was publicly open, with weights, data, and code released from inception, and utilized Italy’s CINECA supercomputing infrastructure.
While Minerva’s models outperform comparable multilingual models on Italian benchmarks, the 3B version scored only 4.9% on the INVALSI Italian school-exam tests. Experts note that despite the large dataset and native focus, the low score indicates that scale alone may not suffice for complex language understanding, especially in academic contexts. The evaluation underscores a broader challenge: the need for even larger investments or different methodologies to achieve meaningful country-specific knowledge in AI models.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Rebooting the Machines: A New Human Vision for Artificial Intelligence
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
OpenAI Evals Cookbook: Designing Benchmarks for Product‑Grade LLM Features
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Supercomputing Frontiers: 4th Asian Conference, SCFA 2018, Singapore, March 26-29, 2018, Proceedings (Lecture Notes in Computer Science Book 10776)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-Language Models
This development suggests that large-scale native-language training, even with extensive data and resources, may still fall short in enabling models to perform well on complex, real-world tasks like academic assessments. It raises questions about the necessary scale and approach for European countries to develop effective, knowledge-rich AI systems tailored to their languages and contexts. The findings challenge the assumption that more data and larger models automatically lead to better language understanding in specialized domains.
Background of Italy’s Sovereign-Language AI Effort
Italy’s Minerva project emerged as a response to debates within the European AI community about sovereignty and language-specific models. Unlike Portugal’s AMÁLIA, which layered language specialization onto a multilingual foundation, Minerva was built from scratch, trained on a massive dataset with a focus on Italian. The project was part of Italy’s broader national AI strategy, supported by significant institutional infrastructure—including CINECA’s supercomputers—and aimed to demonstrate the feasibility of sovereign AI development.
Prior to Minerva, European efforts often focused on multilingual models or continuation training of existing models with limited native-language data. Minerva’s approach was to create a large, native-language model from scratch, reflecting a different architectural philosophy. The project publicly released its weights and data, marking a notable step in transparency and open science within European AI initiatives.
“The Minerva results challenge the assumption that native-language models trained from scratch can automatically reach country-specific knowledge depth.”
— Thorsten Meyer, AI researcher
Unresolved Questions About Model Effectiveness
It remains unclear whether the low INVALSI score is primarily due to the model’s scale, training methodology, or intrinsic limitations in current AI architectures. The ongoing research aims to refine training approaches, but definitive conclusions about the necessary investment levels or alternative strategies are not yet available.
Next Steps for European Sovereign-Language AI Projects
The Minerva team plans to continue refining their models, including experiments with larger parameter counts and different training techniques. Further evaluations on diverse academic and real-world benchmarks are expected to clarify whether scale alone can improve performance. Additionally, European policymakers and researchers are likely to reassess strategies based on these empirical findings, potentially shifting toward even larger investments or hybrid approaches combining native and multilingual training.
Key Questions
Why did Minerva perform poorly on the Italian exam?
Despite extensive training data and native focus, the low score suggests that simply increasing data and model size may not be enough for complex language understanding. Factors like data quality, model architecture, and task-specific tuning are also critical.
What does this mean for other European language models?
It indicates that European projects may need to consider larger investments or different training methodologies to achieve country-specific language expertise comparable to native human performance.
Is the Minerva project considered a failure?
No. While the performance on academic benchmarks was disappointing, the project demonstrated important insights into the scaling challenges of sovereign-language models and set a transparent example for open research.
Will this influence future European AI policy?
Yes, empirical results like these are likely to inform policy discussions about the scale and scope of investments needed to develop effective, country-specific AI systems.
Source: ThorstenMeyerAI.com
