How does AI model size affect cost and sustainability?

Yes, I admit it: at first glance, this topic seems a little “nerdy”. But that is precisely why I think it matters to anyone who wants to do more with AI than try a few prompts. Especially with the wave of AI agents now approaching.

Size does matter…

Over the last few months, I have tested many smaller open-source AI models (LLMs), particularly those with fewer parameters. Why does this matter? Because the number of parameters directly determines model size and therefore hardware requirements, energy use and costs. Ultimately, this determines economic viability and sustainability.

Until now, my view was that larger models produced better results. Reduced, quantised variants never quite convinced me in practice.

…but bigger does not always mean more sustainable

Until now! Over the last few weeks, I had something of a revelation with Gemma 3. It is an open-source model from Google that looks small on paper but, in my view, delivers very good results. You could call Gemma Gemini's little sister. According to Google, its Chatbot Arena Elo score falls between DeepSeek R1 671B and Llama 4 400B, both models requiring many times more resources.

In real RAG applications – knowledge management, chatbots and support – Gemma 3 delivers what counts:

• Low energy demand
• Low operating costs
• Reliable performance

A clear look at costs and technology

• Costs: At the time of writing, one million tokens (input and output, model as a service) with Gemma 3 27B cost around €0.10–€0.20. Llama 3.3 is around €0.50–€0.70. GPT-4o is approximately €2 for input and €10 for output. That is a substantial difference.
• Hardware: While other leading models require 8, 16 or more high-performance GPUs, Google says Gemma 3 27B can run on a single NVIDIA H100.
• Quantisation versus everyday performance: Even the Gemma 3 12B QAT model produces very good chat results on my MacBook with an M4 processor.

More details from Google: Gemma 3 QAT Models: Bringing state-of-the-Art AI to consumer GPUs.

Why should anyone outside IT care?

Because AI projects are about more than the technology alone. Key questions include:

• How much effort and how many resources go into ongoing operations?
• Which models can be integrated cleanly and safely into existing processes, including privacy and data security requirements?
• What does the cost-benefit ratio look like when the model is used in everyday work rather than only in a pilot?

There is no need to dramatise it: few projects fail because of hardware or energy costs.

But when an AI-driven application costs €120 per user per month to operate, compared with €2.90 using a more efficient approach, the architecture deserves a closer look. A lean model saves energy and cost and also makes it easier to meet privacy requirements and keep operational effort under control.

What does this mean for companies?

“Good enough” is an approach I currently find very promising. With little additional effort, hybrid architectures can make in-house AI solutions significantly more efficient without losing quality.

For many chatbots, RAG-based assistants or internal sales agents, “good enough” is the more sustainable route. Lightweight models can be used for tasks such as classification or guardrails – quickly checking and filtering AI answers for plausibility and compliance – while larger models are used where maximum precision in text is needed.

It is like a well-run kitchen:

You do not need a Michelin-starred chef for every evening meal, but when it matters, the team is ready.

The strategic difference: process innovation through hybrid model architectures

• Efficiency and sustainability: Less energy, lower costs and high quality.
• Flexibility: The architecture grows with requirements without reinventing the system landscape.
• Economic advantage: Modular systems reduce long-term investment costs and make AI projects more resilient to technological disruption.

A real advantage

Those who want to can run these smaller models on their own hardware. Those who prefer flexibility can use them in a private cloud, with fewer hurdles and full control.

A conclusion with a wink

This may all sound a little “nerdy”, but in practice, technical elegance also helps determine whether a project takes off.

My advice: Assemble the right orchestra of models: cost-efficient, flexible and easy to integrate.

How do you see it? Have you experimented with smaller AI models or hybrid architectures? What have you experienced, particularly around costs, operations or integration with your existing systems?

I welcome feedback, ideas and shared experiences. Get in touch at any time if you have questions or need support for your projects.