Fastly improves artificial intelligence performance with an AI Accelerator | Digital strategies

Fastly has announced the launch of Fastly AI Accelerator, an AI solution designed to create a better experience for developers by helping to improve performance and reduce costs through the use of similar prompts for large language model (LLM) applications. .

Built on Fastly’s edge cloud platform, and leveraging its caching technology, AI Accelerator uses a specialized API gateway to dramatically improve the performance of applications using popular LLMs, starting with ChatGPT and expanding support to include additional models. .

Using Fastly AI Accelerator, developers only need to update their application to use a new API endpoint, which typically only requires changing a single line of code. Fastly AI Accelerator will transparently implement semantic caching for OpenAI-compatible APIs. This approach goes beyond traditional caching, as Fastly AI Accelerator is able to understand the context of requests and queries, and will send a similar response if two or more requests are similar.

The most popular AI applications can process hundreds of thousands of API calls or questions per day. Many of your user questions are likely to be very similar and have been asked before. Without semantic caching, each call to the LLM requires returning to the provider to obtain the information, which increases costs and latency. However, Fastly’s semantic caching, AI Accelerator, provides a cached response for repeated queries directly from Fastly’s high-performance edge platform, rather than back to the AI ​​provider, helping to deliver better experience by improving performance while reducing costs.

“At Fastly, we’re always listening to developers, to understand both what excites them and what their biggest pain points are,” said Anil Dash, VP of Developer Experience at Fastly. “Fastly AI Accelerator gives developers exactly what they want, making the experience of their favorite LLMs much faster and more efficient, so they can focus on what makes their app or site unique, and what keeps their customers happy users.”