Blog.Kriativity

What are tokens in AI language models?

Tokens are the chunks that an LLM reads and generates - roughly 3/4 of a word on average. Token count determines cost, speed, and whether your text fits in the context window.

AILLM

Overview

LLMs do not process text character by character or word by word - they use "tokens", sub-word units produced by an algorithm called byte-pair encoding (BPE). A token is roughly 3–4 characters or about 3/4 of a word in English. Understanding tokens matters because API pricing is per token, context window limits are in tokens, and long documents can quickly exceed model limits.

Key Points

  • β€’Tokenisation splits text into sub-word pieces: "running" β†’ ["run", "ning"]; "GPT-4" β†’ ["G", "PT", "-", "4"].
  • β€’Common English words are usually 1 token; rare words, code, and non-English text use more.
  • β€’Numbers are expensive: "1234567" β†’ 4 tokens (["123", "456", "7"]); consider formatting large numbers differently.
  • β€’Code tends to be token-inefficient: whitespace, variable names, and symbols all cost tokens.
  • β€’Rule of thumb: 1000 tokens β‰ˆ 750 words β‰ˆ 1-2 pages of text.
  • β€’Most APIs bill for both input tokens (your prompt) and output tokens (the model's response).

Examples

Counting tokens before sending to an API

typescript
import { encoding_for_model } from 'tiktoken';

// Count tokens before making an API call (avoids surprise costs/errors)
const enc = encoding_for_model('gpt-4o');
const text = 'Hello, how many tokens is this sentence?';
const tokens = enc.encode(text);

console.log(`"${text}"`);
console.log(`Token count: ${tokens.length}`);  // 8
// token IDs: [9906, 11, 1268, 1690, 11460, 374, 420, 11914, 30]

enc.free(); // release memory

How tokenisation splits different text types

python
import tiktoken
enc = tiktoken.get_encoding('cl100k_base')  # GPT-4 / Claude compatible

examples = {
    'English word':    'running',         # β†’ ['running']         = 1 token
    'Rare word':       'serendipitous',   # β†’ ['seren', 'dipit',
                                          #    'ious']             = 3 tokens
    'Number':          '1234567',         # β†’ ['123', '456', '7'] = 3 tokens
    'Python code':     'def hello():',    # β†’ 5 tokens
    'Chinese text':    'δ½ ε₯½δΈ–η•Œ',         # β†’ 4 tokens (each char ~ 1 token)
}

for label, text in examples.items():
    count = len(enc.encode(text))
    print(f"{label:15s}: {count} token(s) β†’ {text!r}")

Estimating cost before making an API call

typescript
const INPUT_PRICE_PER_1M  = 3.00;  // USD - GPT-4o as of 2025
const OUTPUT_PRICE_PER_1M = 12.00;

function estimateCost(inputTokens: number, outputTokens: number): string {
  const cost = (inputTokens / 1_000_000) * INPUT_PRICE_PER_1M
             + (outputTokens / 1_000_000) * OUTPUT_PRICE_PER_1M;
  return `$${cost.toFixed(6)}`;
}

// A prompt with 500 input tokens and 200 output tokens:
console.log(estimateCost(500, 200));  // $0.003900

Pro Tip

To reduce token usage and cost: use structured output formats (JSON with short keys), strip unnecessary whitespace, summarise long documents before passing them to a model, and use cheaper models for simple classification tasks. Saving 30% of tokens on a high-volume app translates directly to 30% cost reduction.

← Back to Quick Answers