In the previous article, we discussed the best strategy to integrate gen AI into your product. Now, we want to share with you how much it will cost. We won’t be covering the cost of developing the actual software that includes UI/UX design, API integrations, front and backend development, and so on. Here, we are going to focus solely on the cost of integrating generative AI.
The key factors affecting the cost of generative AI implementation are the choice of a model and how you decide to implement it. Let’s start with the cost of integrating LLM.
The price for integrating generative AI will depend on the provider you choose. For example, a 500-word response will cost 8.4 cents with GPT-4 and only 0.07 cents with Llama 2.
Now, why do prices differ? The final cost depends on three aspects: the model’s capabilities, the size of input/output, and the context window. We'll discuss the first aspect later. Let’s focus on the input/output and context window first.
In plain language, input is the data provided to the model for processing. Think of it as a story you're telling to your friend. After receiving the input, the LLM uses its existing knowledge to give you an answer (like your friend responding to what you've said). That answer is the output.
A context window is the maximum amount of text a model can consider when delivering a response. For example, in real life, your friend might forget parts of a long conversation, so if you give the LLM too much information at once like telling your friend a super long story, it forgets the earlier information, affecting the relevance of the final output.
Input, output, and the context window are counted and charged in tokens. These are characters, words, or other segments of text or code that an LLM can process. Just for reference, a paragraph of 750 words is roughly equal to 1,000 tokens.
The more advanced the model, the higher the price. For example, here is a comparison between GPT-3.5 Turbo and GPT-4 Turbo:
As you can see, GPT-3.5 is less capable, but it’s less expensive. It's possible that your AI-driven application might not require the advanced features of GPT-4 Turbo. In such cases, the 16,000 token context provided by GPT-3.5 Turbo could suffice perfectly for your requirements and you can confidently pick the cheaper option. However, if your AI-enabled solution requires a larger context window and the latest information, it’s better to use GPT-4 Turbo, as it was trained on the data up to April 2023 and supports the context size of 128,000 tokens.
But the story doesn’t end here. There are many alternatives you can choose from. So let’s overview several LLM providers and compare their pricing and LLM capabilities.
All AI models can be divided into open-source – freely available for use and modification, and commercial (proprietary) – providing APIs for a specific fee.
At first glance, it might seem that open-source models are cheaper as you don’t have to pay for them. But it isn’t entirely true. While you don’t pay for the model itself, you have to spend significant resources on setting up on-premises or cloud infrastructure and manage it at your own expense.
The cost of generative AI in this case will include computational costs like the value of specialized hardware which is rather expensive, and cloud services. With this in mind, integrating a commercial AI model might be a more cost-effective option.
Let’s take a look at the latest and most advanced Open AI model – GPT 4o. It’s a commercial AI model charging $5 per 1 million input tokens and $15 per 1 million output tokens.
Compared to GPT-4 Turbo, GPT-4o is way cheaper. But once again, you don’t have to choose between the two options solely. The particularities of your project inform your final decision. So if GPT-3.5 Turbo is enough, go for it.
One more commercial model to highlight is Gemini 1.5 Flash by Google. It’s a multimodal model (the one that can process either text or media) charging per character. As they explain on the official website, characters are counted by UTF-8 code points excluding white spaces, resulting in around 4 characters per token. So, Gemini 1.5 Flash will cost you $0.000125 per 1K characters of text input and $0.000375 per 1K characters of output. Just for comparison, GPT-4o charges $0.01 per 1K input tokens and $0.02 per 1K output tokens.
Now that we’ve covered the differences between some popular models and how much they charge, it’s time to move on to the implementation stage. At this stage, you’ll have to adapt the LLM to your project needs so that it accurately responds to user inputs. There are two ways to achieve this: fine-tune the model or implement RAG.
To build an AI-enabled application, you need to interface with LLM models. However, this process can become costly when you overload the prompt with excessive information (which, by the way, you're charged for). To enhance the model’s response, developers can fine-tune them or implement Retrieval-Augmented Generation (RAG). Let’s consider both options in detail.
RAG is an advanced technique that enhances the capabilities of LLMs by linking them to external sources, like documents and databases, to retrieve contextual and up-to-date information. It allows developers to extend the initial knowledge of LLM with specific information without the need to retrain the model.
The core idea of RAG is that you only retrieve the chunks of data required for LLM to provide a relevant output. This way, the system is not overloaded with unnecessary data, reducing costs as you don’t pay for the tokens your LLM doesn’t need.
RAG includes three stages – indexing, retrieval, and generation. Here’s a brief overview.
To streamline the process of RAG implementation, you can use LangChain. It’s a framework that provides a set of building blocks to implement RAG easier and faster. These are components like prompt templates, output parsers, text splitters, agents, tools, chains, and more that eliminate the need to build everything from scratch. It speeds up and simplifies the development of LLM-based applications.
While using LangChain components for RAG implementation is free, it requires a vector database implementation, a component that makes LLMs generate outputs faster.
As mentioned, vector databases allow for fast retrieval of relevant information, which makes them an optimal solution for LLM-based apps.
When looking for a vector database to integrate with, you will find both free and paid solutions. For example, Chroma is an open-source vector database that offers tools to embed documents and queries, store and search embeddings, and more. Another example is Pinecone. Although it isn’t free, Pinecone provides a fully managed infrastructure and advanced features to seamlessly handle large datasets.
If you decide to fine-tune an AI model, it means you take a pre-trained LLM and adjust its parameters to create a custom version that is exposed to your specific context, reducing the amount of information required in each prompt. With a fine-tuned model, you don’t need to include fixed data repeatedly since much of it will already be part of the trained model.
However, fine-tuning comes with some challenges you should know about. Firstly, it’s expensive as it requires significant computational resources and you might also be charged for fine-tuning itself. For instance, if you decide to fine-tune the most hyped LLM models of our time – OpenAI's solutions, here is what you have to pay:
Secondly, it can be challenging to fine-tune the model correctly. Therefore, most organizations currently focus on RAG as a cost-effective and straightforward approach to connect the model with fresh data.
Supposing you’ve chosen to implement RAG for your gen AI project, the next and final step is deployment.
You can deploy your LLM-based application in two ways:
Although the LangChain itself is free, its ecosystem includes additional services like LandServe and LangSmith. LangServe is a platform that simplifies the deployment of LLM-based solutions, while LangSmith is a unified solution to develop, deploy, test, and monitor AI-enabled applications. It offers four pricing plans for various teams – Startups, Developer, Plus, and Enterprise. Here is a brief overview of these plans.
Should you opt for LangSmith/LangServe, bear in mind that the expenses will be factored into the total costs of your AI project. But in this scenario, you're investing in specialized features that accelerate the deployment process.
If this option doesn’t fit your budget, consider more cost-effective solutions like deploying your app on the GCP with a custom backend. Unlike LangSmith, this approach doesn’t provide you with specialized monitoring tools, but you get full control over deployment and observability. At Modeso, we use GCP for most of our apps and manage packaging, deployment, and monitoring.
Another alternative to gain specialized observability is LangFuse. It’s an open-source LLM engineering platform with a comprehensive set of observability, analytics, and experimentation features. You can deploy LangFuse on GCP while paying only the infrastructure costs of running the LangFuse app.
By now, we’ve covered all the expenses associated with integrating a gen AI model into your application. Now let’s see how you can optimize them.
When working with AI models, cost management is crucial because you’re using LLMs and other services that aren’t free. Here are some strategies to help you control and optimize the costs.
Don’t opt for the most expensive model like GPT-4 Turbo just because it’s “the best and most advanced model out there”. It might turn out your project doesn’t even need it. Instead, carefully assess the tasks that your generative AI should perform and select a model with the capabilities to fulfill these tasks. For example, if your AI-powered app deals with simple queries, use a more cost-effective LLM that can handle them without compromising on quality.
The retrieval mechanism, a core component of RAG, minimizes redundant data and makes sure only the relevant information goes to the model. It reduces the context size and overall processing costs. What’s more, with RAG on board, your AI-powered solution will get access to the latest data, making its output more contextually appropriate and specific.
As most models operate on a pay-as-you-go basis (and their prices change), you should continuously refine your approach to prompt engineering to maintain quality while reducing unnecessary processing. Keep track of your project as it grows to adapt to its evolving needs and identify potential areas for optimization.
Now you know what costs you should be prepared for when integrating generative AI and how to minimize them. If you are good with what is ahead, let’s talk about your AI project.
Privacy is important to us, so you have the option of disabling certain types of storage that may not be necessary for the basic functioning of the website. Blocking categories may impact your experience on the website. More information
