Blog.Kriativity

How do you reduce AI API costs in production?

Reduce AI costs by choosing the right model for each task, caching repeated prompts, batching requests, compressing context, and setting output token limits.

AILLM

Overview

AI API costs can grow quickly in production. The good news is most applications overspend by 60-80% because they use the largest model for every task, never cache repeated content, and send unnecessarily long prompts. A systematic cost-reduction strategy typically cuts spend in half without any quality loss.

Key Points

  • •Model selection is the biggest lever: GPT-4o-mini costs 20x less than GPT-4o with 80% of the quality for simple tasks.
  • •Prompt caching reduces cost by 75-90% for repeated system prompts (OpenAI, Anthropic both support this).
  • •Output tokens cost 3-5x more than input tokens - set max_tokens to a tight limit and instruct the model to be concise.
  • •Batch API (OpenAI) processes requests at half price with a 24-hour SLA - perfect for offline jobs.
  • •Semantic caching: store previous (question, answer) pairs and return the cached answer for similar future questions.
  • •RAG compression: retrieve only the most relevant chunks rather than stuffing the entire document into context.

Examples

Route tasks to the right model by complexity

typescript
type TaskType = 'classify' | 'extract' | 'summarise' | 'reason' | 'generate';

const MODEL_ROUTING: Record<TaskType, string> = {
  classify:  'gpt-4o-mini',   // simple label, 1 token output        $0.15/M
  extract:   'gpt-4o-mini',   // structured data extraction           $0.15/M
  summarise: 'gpt-4o-mini',   // paragraph generation                 $0.15/M
  reason:    'gpt-4o',        // complex reasoning, multi-step        $2.50/M
  generate:  'gpt-4o',        // high-quality long-form content       $2.50/M
};

async function smartCompletion(task: TaskType, messages: Message[]) {
  return client.chat.completions.create({
    model:      MODEL_ROUTING[task],
    messages,
    max_tokens: task === 'classify' ? 10 : task === 'extract' ? 300 : 1000,
    temperature: task === 'classify' || task === 'extract' ? 0 : 0.7,
  });
}

// Classifying 100k support tickets/month:
// Without routing: 100k x ~500 tokens x $2.50/M = $125
// With routing:    100k x ~500 tokens x $0.15/M = $7.50  (94% saving)

Semantic caching with embeddings

typescript
const cache = new Map<string, { embedding: number[]; answer: string }>();
const SIMILARITY_THRESHOLD = 0.95;

async function cachedCompletion(question: string): Promise<string> {
  // 1. Embed the incoming question
  const queryEmb = await getEmbedding(question);

  // 2. Check cache for a semantically similar previous question
  for (const [, entry] of cache) {
    const similarity = cosineSimilarity(queryEmb, entry.embedding);
    if (similarity >= SIMILARITY_THRESHOLD) {
      console.log('Cache hit! Similarity:', similarity.toFixed(3));
      return entry.answer; // return saved answer, zero LLM cost
    }
  }

  // 3. Cache miss: call the LLM
  const response = await client.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: question }],
  });
  const answer = response.choices[0].message.content ?? '';

  // 4. Store for future similar questions
  cache.set(question, { embedding: queryEmb, answer });
  return answer;
}

OpenAI Batch API for offline jobs

typescript
// Batch API: 50% cheaper, results within 24 hours
// Perfect for: overnight report generation, bulk classification, data enrichment

// 1. Create a .jsonl batch file
const requests = articles.map((article, i) => ({
  custom_id: `article-${i}`,
  method: 'POST',
  url: '/v1/chat/completions',
  body: {
    model: 'gpt-4o-mini',
    messages: [
      { role: 'system', content: 'Classify the sentiment as positive, negative, or neutral.' },
      { role: 'user',   content: article.text },
    ],
    max_tokens: 5,
  },
}));

const jsonl = requests.map(r => JSON.stringify(r)).join('\n');
const file = await client.files.create({
  file: new Blob([jsonl], { type: 'application/jsonl' }),
  purpose: 'batch',
});

// 2. Submit batch
const batch = await client.batches.create({
  input_file_id: file.id,
  endpoint: '/v1/chat/completions',
  completion_window: '24h',
});

// 3. Poll later and retrieve results
const completed = await client.batches.retrieve(batch.id);
if (completed.status === 'completed') {
  const results = await client.files.content(completed.output_file_id!);
  // parse JSONL results...
}

Pro Tip

Profile before optimising. Add logging that captures model, input tokens, output tokens, and cost per request. After a week you will see which routes drive 80% of your spend - those are the ones to optimise first with caching or model downgrade.

← Back to Quick Answers