Quiz - Fast LLM Inference with Cerebras (Short Course @ DeepLearning.ai)

View Course on DeepLearning.AI    View Other Courses Audited By Us    « Previously

---

View Course on DeepLearning.AI    View Other Courses Audited By Us    « Previously Tags: Agentic AI,Large Language Models,Generative AI,

SQLite-Vector: Vector Search in Your Pocket

See All on GenAI    « Previously

Vector Search in Your Pocket

How sqlite-vector brings AI-powered similarity search to any device — no cloud required

Imagine you have a mobile app that needs to find the most similar image, the best product recommendation, or the right document from a pile of data — all while the user is offline. Traditionally, that would mean sending data to the cloud, running a heavy vector database, and waiting for results. But what if your SQLite database could do all of that, right on the device, with just 30 MB of memory and no indexing wait time? That's exactly what sqlite-vector delivers.

SQLite is already the world's most used database — it's in your phone, your browser, your car, and probably your smart fridge. Sqlite-vector is an extension that adds vector search to SQLite. In plain terms, it lets you store "embeddings" (think of them as mathematical fingerprints of images, text, or audio) and then find the closest matches at lightning speed — all using standard SQL.

Why vector search matters (and why you want it offline)

Modern AI models — from ChatGPT to image recognizers — turn everything into vectors: long lists of numbers that represent the "meaning" of a piece of data. When you want to find something similar, you don't search for exact matches; you search for the nearest neighbors in this high-dimensional space.

Think of it like finding the closest cities on a map — except the map has hundreds of dimensions. That's what vector search does, and it powers:

• Semantic search — finding documents that are conceptually similar to your query
• Image retrieval — showing visually similar photos
• Recommendation systems — matching users with products, videos, or music
• Voice and audio search — identifying sounds or voice queries
• Anomaly detection — spotting outliers in sensor data

Until now, doing this on a phone or a low-power device was tricky. You'd need a separate vector database like FAISS or Weaviate, which often means running a server, setting up complex indexes, and waiting hours for preprocessing. Sqlite-vector flips that script.

What makes sqlite-vector different?

Most vector search tools are heavyweight. They require special virtual tables, pre‑indexing phases that can take hours, and external servers. Sqlite-vector takes a radically simpler approach:

✓ Works with ordinary SQLite tables — no special schemas

✓ No preindexing — start searching immediately

✓ Zero‑cost updates — add or change vectors on the fly

✓ Offline first — works without internet

✓ Cross‑platform — iOS, Android, Windows, Linux, macOS

✓ Memory‑efficient — just 30 MB RAM by default

It's built in pure C with SIMD acceleration, which means it runs blazingly fast even on mobile CPUs. And because it's just a SQLite extension, you can drop it into any existing project with minimal effort.

The secret sauce: TurboQuant

One of the coolest features is TurboQuant — a clever quantization technique inspired by a Google Research paper. Instead of storing full-precision vectors (which take up a lot of space), TurboQuant compresses them into 2‑bit, 3‑bit, or 4‑bit representations.

This dramatically reduces memory and storage while still keeping search results accurate. For example, on a dataset of 1 million vectors with 768 dimensions each, raw 32‑bit floats would take about 3 GB. TurboQuant 4‑bit shrinks that to just 396 MB — about 13% of the original size. And the search is still 15 times faster than brute force.

Here's a quick look at the performance on a Mac with ARM64 (NEON):

Mode	Quantized storage	Full scan / query	TurboQuant / query	Speedup	Recall@10
TurboQuant 4‑bit	396 MB	3248 ms	218 ms	14.9×	0.84
TurboQuant 3‑bit	300 MB	1727 ms	188 ms	9.2×	0.74
TurboQuant 2‑bit	204 MB	3265 ms	85 ms	38.3×	0.48

The 4‑bit mode is a great starting point — it gives a solid balance of speed, memory, and accuracy. For really tight edge budgets, 2‑bit can be a lifesaver, though you'll want to test it with your own data.

Getting started (it's really this simple)

Sqlite-vector is available as a pre‑built binary for all major platforms — Linux, macOS, Windows, Android, and iOS. You can also load it as a WASM module for browsers.

Here's the basic flow in SQL:

-- 1. Load the extension
.load ./vector

-- 2. Create a regular table (no virtual tables needed!)
CREATE TABLE images (
    id INTEGER PRIMARY KEY,
    embedding BLOB,   -- store vectors as binary blobs
    label TEXT
);

-- 3. Insert a vector (as a blob or JSON array)
INSERT INTO images (embedding, label)
VALUES (vector_as_f32('[0.3, 1.0, 0.9, 3.2, ...]'), 'cat');

-- 4. Initialize the vector column
SELECT vector_init('images', 'embedding', 'type=FLOAT32,dimension=384');

-- 5. Quantize for blazing-fast search (TurboQuant 4‑bit)
SELECT vector_quantize('images', 'embedding', 'qtype=TURBO,qbits=4');

-- 6. Search for the top 20 nearest neighbors
SELECT e.id, v.distance
FROM images AS e
JOIN vector_quantize_scan('images', 'embedding', ?, 20) AS v
ON e.id = v.rowid;

That's it. No external servers, no complex indexing, no waiting. Your vector search is ready to go.

💡 Pro tip: You can also use vector_quantize_preload() to load the quantized data into memory for a 4‑5× speedup — perfect for interactive apps.

Where does it shine?

Sqlite-vector is built for Edge AI — scenarios where you need intelligence on the device, not in the cloud.

• Mobile apps that do on‑device image search, face recognition, or voice commands
• Privacy‑first applications where data never leaves the user's device
• Offline‑first tools like note‑taking apps with semantic search
• Embedded systems in robots, drones, or IoT devices

Because it's a SQLite extension, you also get all the benefits of a full relational database — transactions, joins, filters, and ACID guarantees — combined with vector search.

The bigger picture

Sqlite-vector is part of a larger ecosystem from SQLite AI that's turning SQLite into a complete runtime for intelligent, distributed data. There's also sqlite‑sync for offline‑first sync, sqlite‑ai for on‑device LLM inference, and sqlite‑agent for autonomous AI agents — all living inside your SQLite database.

If you don't want to manage it yourself, SQLite Cloud offers a hosted version with sync, auth, edge functions, and a free tier that gives you 512 MB and 20 connections — no credit card required.

Wrapping up

Sqlite-vector is a game‑changer for anyone building AI‑powered applications that need to work offline, on mobile, or at the edge. It's fast, tiny, and dead simple to use. You don't need to learn a new database or wrestle with complex indexing — just SELECT your way to similar items.

Whether you're building a photo app, a recommendation engine, or a privacy‑first search tool, sqlite‑vector gives you superpowers right inside your SQLite database. And with TurboQuant, you get enterprise‑grade performance on devices that fit in your pocket.

Ready to try it? Head over to the GitHub repository, grab the binary for your platform, and start searching in minutes. The era of on‑device AI is here — and it speaks SQL.

Resources: GitHub · Docs · SQLite AI · Releases

All performance numbers and benchmarks are from the project's official documentation and were measured on macOS ARM64 with the NEON backend. Your mileage may vary depending on hardware and data.

See All on GenAI    « Previously
Tags: Generative AI,Database

See All on GenAI    « Previously    Next »

Laptop For Running an LLM Locally (Comparison Specific to India)

Generated: 2026-07-04
Context: User asked about running LLMs on a Dell Precision with NVIDIA GB10, then explored pricing, shifted to laptops in ₹1‑2 lakh range, upgraded to 12–16 GB VRAM, asked for refurbished MSI Stealth 18, and finally compared a large list of laptops.

---

1. Initial Query: Dell Precision with NVIDIA GB10

User: what kind of llm can run on this laptop: Dell Pro Precision portfolio with the NVIDIA GB10 Grace Blackwell Superchip

Assistant Clarification:

• The GB10 is an ARM‑based SoC (NVIDIA Grace CPU + Blackwell GPU)
• If your Dell Precision has the GB10 chip, it features 128 GB unified memory and 1 PetaFLOP of FP4 AI performance.
• It can run:
  • Up to ~200B parameter models on a single node (e.g., Llama 3.1 70B, Qwen 2.5 72B, up to 200B with quantization).
  • Two units linked can run 405B models (e.g., Llama 3.1 405B) using 4‑bit quantization.
• Real‑world speed: ~47 tokens/sec on a 70B coding model.
• Software: runs NVIDIA DGX OS (Ubuntu for ARM) with PyTorch, Python, Jupyter, and NVIDIA AI Enterprise.

Important: The Intel‑based Dell Precisions with discrete GPUs have far lower VRAM (typically 16–48 GB) and can only run up to ~70B models at best.

---

2. Pricing of the GB10 System

User: what's it's cost

Starting prices (base model) across regions:

Region	Starting Price	Currency
USA	~$3,000 – $4,000+	USD
Australia	~10,207.49	AUD
Austria	~6,443.62	EUR
Canada	~8,703.69	CAD
China	~67,270.03	CNY
Hong Kong	~64,365.00	HKD
India	~3,99,000	INR
Ireland	~6,889.75	EUR
New Zealand	~15,328.59	NZD

Note: These are base models. Customisation (storage, memory) increases the price. The GB10 system is specifically the Dell Pro Max line, not all Dell Precisions.

---

3. More Affordable and Higher‑End Dell Pro Max Options

User asked for cheaper and higher‑end options from the Dell Pro Max line.

Affordable (no GB10, Intel/AMD based)

Model	Approx. Price (CNY)	Core Specs
Dell Pro Max 14 (AMD)	¥8,999 – ¥13,633	AMD Ryzen AI 5 PRO 340, 32GB
Dell Pro Max Micro (desktop)	¥8,599 – ¥9,252	Intel Core Ultra
Dell Pro Max 16 (laptop)	¥11,999 – ¥24,999	Core Ultra 7/9, up to 64GB, RTX PRO 2000
Dell Pro Max Tower T2 (mid)	¥26,899 – ¥27,899	Core Ultra 9, 64GB, RTX 4080S
Dell Pro Max 14 Premium	~¥21,000+	Core Ultra

High‑End (no GB10, but powerful discrete GPUs)

Model	Approx. Price	Specs
Dell Pro Max 16 Premium	~$4,583 USD / ¥29,889	Core Ultra 9, RTX PRO 2000
Dell Pro Max 16 Plus	~$6,611 AUD	Intel/AMD + RTX PRO
Dell Pro Max Tower T2 (fully loaded)	~$14,349 USD	Core Ultra 9 285K, 128GB ECC, RTX PRO 6000 (96GB VRAM)
Dell Precision 7865 Tower	~€3,004 (Ireland)	AMD Threadripper PRO, up to 64 cores

Summary: Only the GB10‑based Dell Pro Max (starting ~₹3.99 lakh in India) can run 200B models locally. The tower with RTX PRO 6000 can also handle large models thanks to 96GB VRAM.

---

4. Laptops in ₹1–2 Lakh for AI Workloads

User: give me some recommendations in 1lakh to 2lakh INR in India for running local AI workloads

Recommended models (all with RTX 4060, 8GB VRAM, ~13B max):

Model	Key Specs	Approx. Price (INR)	Best For
HP Omen 17‑cm2002TX	i7‑13700HX / RTX 4060 / 8GB / 16GB	~1,21,999	Best value, large screen
Dell G15 5530	i7‑13650HX / RTX 4060 / 8GB / 16GB	~1,21,990	Reliable Dell build
Acer Nitro V16	Ryzen 7 8845HS / RTX 4060 / 8GB / 16GB	~98,790	Most affordable entry
ASUS ROG Zephyrus G16	Ryzen AI 9 HX 370 / RTX 4060 / 8GB / 32GB	~1,84,990	Premium, portable, OLED
MSI Pulse 16 AI	Ultra 7 155H / RTX 4060 / 8GB / 32GB	~1,84,990	High RAM, 140W GPU
Dell Pro 16 Plus (AMD)	Ryzen PRO / Integrated NPU (no dGPU)	~96,562	Enterprise, not for LLMs

Key advice: 8GB VRAM is minimum; NPU cannot replace a dGPU. Avoid models without discrete NVIDIA GPUs.

---

5. Upgrading to 12–16 GB VRAM Laptops

User asked for options with 12‑16 GB VRAM.

Model	Specs (CPU/GPU/VRAM/RAM)	Approx. Price (INR)	Max LLM Size	Best For
Dell Alienware m16	Ryzen 9 7845HX / RTX 4080 / 12GB / 32GB	1,49,990	~30‑40B	Best value 12GB
HP Omen 17	i7‑13700HX / RTX 4080 / 12GB / 16GB	~2,39,990 – 2,69,990	~30‑40B	Large screen, powerful
ASUS ROG Zephyrus G16 (2024)	Ultra 9 185H / RTX 4080 / 12GB / 32GB	2,76,411	~30‑40B	Premium & portable
ASUS ROG Strix SCAR 16 (2024)	i9‑14900HX / RTX 4080 / 12GB / 32GB	~2,77,994	~30‑40B	High‑refresh MiniLED
MSI Raider GE78 HX (14th)	i9‑14900HX / RTX 4080 / 12GB / 32GB	2,89,990	~30‑40B	17‑inch, excellent cooling
MSI Raider A18 HX	Ryzen 9 7945HX / RTX 4080 / 12GB / 32GB	2,89,990	~30‑40B	18‑inch, Wi‑Fi 7
MSI Stealth 18 AI Studio	Ultra 9 185H / RTX 4090 / 16GB / 32GB	2,59,990	~70B	Best AI value (16GB)
Razer Blade 18	i9‑14900HX / RTX 4090 / 16GB / 32GB	3,49,990	~70B	Premium build
MSI Raider GE78 HX (4090)	i9‑14900HX / RTX 4090 / 16GB / 32GB	4,19,990	~70B	Max 16GB performance

Key: The MSI Stealth 18 AI Studio with RTX 4090 (16GB) is the most affordable way to run 70B‑class models (~₹2.6 lakh). The Alienware m16 (12GB) is the cheapest 12GB option at ₹1.5 lakh.

---

6. Refurbished MSI Stealth 18 AI Studio in India

User asked: can I get a refurbished model of this at a lower price in India: MSI Stealth 18 AI Studio. Give me some references from Amazon (maybe)

• New price: RTX 4090 variant: ~₹2,94,990 to ₹3,99,990 depending on retailer. RTX 4080 variant: ~₹3,49,990.
• Refurbished availability: Not found on Amazon India. International refurbishers (e.g., Reebelo US) list it around $1,658 (~₹1.38 lakh), but shipping + customs + no warranty make it risky.
• Alternative channels: OLX, Quikr, local computer markets (Nehru Place, Ameerpet).
• Precautions: Verify in person, check specs, battery health, warranty, and accessories.

Conclusion: Refurbished units are not yet common in India. You may have to wait or buy new.

---

7. Amazon Options for 12–16 GB VRAM Laptops

User asked to research Amazon and share options.

Model	Specs	Approx. Price (INR)	Max LLM
MSI Stealth 18 AI Studio (RTX 4090)	Ultra 9 185H / 16GB / 32GB	~3,99,990	70B
Dell Alienware x16 (RTX 4080)	Ultra 9 185H / 12GB / 32GB	Price not listed	40B
Dell Alienware m16 R1 (RTX 4080)	i9‑13900HX / 12GB / 32GB	~3,94,760	40B
Dell Alienware M18 R2 (RTX 4080)	i9‑14900HX / 12GB / 32GB	Price not listed	40B
HP OMEN (ck2004TX) RTX 4080	i9‑13900HX / 12GB / 32GB	Price not listed	40B
HP Omen 17 (17‑ck2008AX)	i7‑13700HX / 12GB / 16GB	Price not listed	40B

Observation: Most 4080/4090 laptops are above ₹3 lakh. For a ₹2‑lakh budget, you are limited to 8GB VRAM options.

---

8. Final Comparison of All Laptops Provided

User dumped a long list of laptops and asked to compare with specific columns.

Here is the comprehensive comparison table with all models ranked by VRAM and capability:

Laptop Model	Key Specs (CPU / GPU / VRAM / RAM)	Approx. Price (INR)	MAX LLMs Size Supported	OS Support	Best For
Alienware 16 Area‑51	Ultra 9‑275HX / RTX 5090 / 24GB / 64GB	₹4,84,990	~70B+	Win 11 + MSO	Maximum AI performance
Lenovo Legion Pro 7 2025	Ultra 9‑275HX / RTX 5090 / 24GB / 64GB	₹4,72,490	~70B+	Win 11 + Office	Best value for 24GB VRAM
ASUS ProArt P16 OLED (2025)	Ryzen AI 9 HX 370 / RTX 5090 / 24GB / 64GB	₹4,19,990	~70B+	Win 11 + M365	Creator-focused, 4K OLED
NXTGN XP4 (Desktop)	i9‑14900K / RTX 5060 Ti / 16GB / 64GB	Not listed	~70B	Win 11 Pro	Desktop upgradeability
Lenovo Legion 9	i9‑13980HX / RTX 4090 / 16GB / 32GB	₹4,49,510	~70B	Win 11 + Office	Previous‑gen flagship
ASUS ROG Strix SCAR 16	Ultra 9‑275HX / RTX 5080 / 16GB / 32GB	₹3,79,990	~70B	Win 11 + M365	High‑end gaming & AI
ASUS Zenbook 14 (2026)	Ultra 9‑285H / Integrated iGPU / shared / 32GB	₹1,19,990	Light AI only	Win 11 + M365	Ultra‑portable productivity
ASUS Zenbook S16 (2026)	Ryzen AI 9‑465 / Integrated iGPU / shared / 32GB	₹1,69,990	Light AI only	Win 11 + M365	Premium ultraportable
Lenovo Yoga Slim 7	Ultra 7‑155H / Integrated iGPU / shared / 16GB	~₹90,990	Light AI only	Win 11 + Office	Affordable ultraportable
Lenovo IdeaPad Slim 5	Ultra 7‑355 / Integrated iGPU / shared / 16GB	₹1,36,990	Light AI only	Win 11 + MSO	Next‑gen AI PC (Copilot+)
MSI Stealth 16 AI Studio	Ultra 9‑185H / RTX 4070 / 8GB / 32GB	₹2,43,990	~13B	Win 11 Pro	Slim & portable AI
ASUS ROG Strix G16	Ultra 9‑275HX / RTX 5070 / 8GB / 32GB	₹2,59,990	~13B	Win 11 + M365	Gaming & AI entry
HP Omen 16 Max	Ryzen AI 9 HX 375 / RTX 5070 Ti / 12GB / 32GB	₹2,67,990	~40B	Win 11 + Office	Best mid‑range AI value
HP Omen (an0015TX)	Ultra 7‑255H / RTX 5060 / 8GB / 24GB	₹1,51,490	~13B	Win 11 + M365	Budget gaming & AI
HP Victus (fa2382tx)	i5‑14450HX / RTX 4050 / 6GB / 24GB	₹1,02,990	~7B	Win 11 + M365	Most affordable option
HP Victus (fa2531TX)	i7‑13650HX / RTX 4050 / 6GB / 24GB	₹1,19,990	~7B	Win 11 + M365	Budget AI with better CPU

---

9. Final Recommendation (Based on Budget)

If Budget ~₹4‑5 Lakh (for 70B models)

• Best Value: Lenovo Legion Pro 7 2025 (₹4.72L) – similar to Alienware but cheaper.
• Best for Creators: ASUS ProArt P16 OLED (₹4.19L) – lighter, superior display.
• Best Performance: Alienware 16 Area‑51 (₹4.84L) – top‑tier cooling and build.

If Budget ~₹2.5‑3 Lakh (best balance)

• HP Omen 16 Max (₹2.67L) – 12GB RTX 5070 Ti allows up to 40B models – the sweet spot.

If Budget ₹1‑2 Lakh (entry‑level)

• HP Omen (an0015TX) (₹1.51L) – 8GB RTX 5060, runs up to 13B.
• HP Victus (fa2382tx) (₹1.02L) – 6GB RTX 4050, runs up to 7B.

Avoid these for LLMs

• All laptops with only integrated graphics (Intel Arc, AMD Radeon, or NPU) – they cannot run models larger than a few billion parameters efficiently.

---

10. Critical Takeaways to Avoid Wrong Decisions

1. VRAM is the absolute king – more VRAM = larger models you can run locally.
2. System RAM does NOT substitute VRAM – the model must fit into GPU memory.
3. NPUs (AI accelerators) are for light, energy‑efficient tasks – they cannot run large LLMs (7B+).
4. Price vs. capability:
   • Up to 13B → 8GB VRAM (₹1‑1.8L)
   • Up to 40B → 12GB VRAM (₹2.6‑3L)
   • Up to 70B → 16‑24GB VRAM (₹3.8‑4.8L)
5. Refurbished high‑end laptops are rare in India – be prepared to buy new or wait.
6. Always verify GPU model and VRAM before purchase – don't rely on “AI PC” marketing.
7. Linux compatibility is generally good for NVIDIA GPUs – but integrated NPUs may have limited driver support.

---

See All on GenAI    « Previously    Next »

ROUGE Score (Evaluating Machine Translation and Other GenAI Use Cases)

See All on GenAI    « Previously    Next »
You may recall that BLEU score was precision oriented. Next, we will see a Recall oriented metric to evaluate Machine Translation and other GenAI use cases.

See All on GenAI    « Previously    Next »

BLEU Score (Evaluating Machine Translation and Other GenAI Use Cases)

See All on GenAI    Next »

See All on GenAI    Next »
Tags: Generative AI,

Basic Attention Operation (Additive attention)

See All on GenAI    « Previously    Download the 2014 paper on Additive Attention    View the Lab Work From DeepLearning.ai

🧠 Basic Attention Operation

A Layman's Guide to Understanding & Implementing Attention from Scratch

0. The Big Picture: What Problem Does Attention Solve?

Imagine you are translating a long French sentence into English. The old-school way (before attention was invented) worked like this: you would read the entire French sentence, compress its meaning into a single, fixed-size summary in your head, and then spit out the English translation from that summary alone. This is hard—especially for long sentences—because that little summary can't possibly hold all the details. Important nuances get lost.

Attention changes the game. Instead of forcing the model to cram everything into one final thought, attention lets the decoder (the part that writes the output) look back at every single input word whenever it needs to produce the next output word. Think of it as the model having a highlighter: at each step of generating the translation, it highlights which input words are most relevant right now.

🔎 Everyday Analogy: You're at a crowded party. Someone across the room is talking to you, but there's a lot of background noise. Your brain does "attention" naturally—it focuses on the speaker's voice while tuning out the clinking glasses and other conversations. In a machine translation model, the "speaker's voice" is the relevant input word, and the "background noise" is the less-important input words. Attention is just a mathematical way of doing that focusing.

In this ungraded lab, you implement a basic version of attention from the influential paper by Bahdanau et al. (2014). You'll use only NumPy—no deep learning frameworks like PyTorch or TensorFlow—so you can see every moving part up close. The entire attention mechanism can be broken into three simple steps:

1. Step 1 – Alignment Scores: Compare the decoder's current "mood" (hidden state) with each encoder word's "meaning" (hidden state) to get raw similarity numbers.
2. Step 2 – Weights (Softmax): Turn those raw scores into percentages that add up to 100%, telling us exactly how much attention each input word deserves.
3. Step 3 – Context Vector: Use those percentages to blend the encoder's hidden states into one summary vector that captures what's important right now.

Let's walk through every line of code in this notebook, with plenty of examples and analogies.

---

1. The Setup Code: Imports and the Softmax Function

The very first code cell in the notebook sets up the tools we'll need:

import numpy as np

def softmax(x, axis=0):
    return np.exp(x) / np.expand_dims(np.sum(np.exp(x), axis=axis), axis)

1.1 What is NumPy?

numpy (imported as np) is Python's numerical powerhouse. It gives us arrays—think of them as supercharged lists that can hold numbers in rows and columns, like a spreadsheet. But unlike regular Python lists, NumPy lets us do math on entire arrays at once. If you have a list of 1,000 numbers and want to multiply each by 2, with NumPy you write arr * 2 and it's done in one go—no loops needed. This is called vectorized computation, and it's the secret sauce behind all modern machine learning.

1.2 The Softmax Function — Turning Numbers into "Attention Percentages"

The softmax function is one of the most important functions in all of deep learning. Let's build an intuition for it step by step.

🍕 Pizza Analogy: Imagine three friends ordering pizza. Friend A is starving and rates their hunger as 10. Friend B is moderately hungry and rates it as 5. Friend C just ate and rates their hunger as 1. How do we split one pizza fairly? We could just say A gets 10 slices, B gets 5, C gets 1—but that's 16 slices, not one pizza! Softmax solves this: it converts those raw "hunger scores" (10, 5, 1) into percentages that sum to 100%. A might get ~62%, B gets ~31%, and C gets ~7%. Everyone gets a fair share of the one pizza.

Mathematically, softmax does this:

1. Exponentiate each number (np.exp(x)): This makes all numbers positive and amplifies differences. A small gap becomes a bigger gap.
2. Sum up all the exponentiated numbers (np.sum(np.exp(x), axis=axis)).
3. Divide each exponentiated number by that sum. Now every number is between 0 and 1, and they all add up to exactly 1.

🔢 Concrete Example: Let's say our raw scores are [2.0, 1.0, 0.1].

• Step 1 – Exponentiate: e² ≈ 7.39, e¹ ≈ 2.72, e⁰·¹ ≈ 1.11 → [7.39, 2.72, 1.11]
• Step 2 – Sum: 7.39 + 2.72 + 1.11 = 11.22
• Step 3 – Divide: [7.39/11.22, 2.72/11.22, 1.11/11.22] → [0.66, 0.24, 0.10] (roughly)

Result: 66% attention to item 1, 24% to item 2, 10% to item 3. Notice they sum to 100%.

The axis parameter controls which direction the softmax is applied. axis=0 means "each column sums to 1" (operate down the rows), and axis=1 means "each row sums to 1" (operate across the columns). In our attention code, we'll use the default axis=0 because our scores come as a column vector (shape [5, 1]).

The function uses np.expand_dims to make sure the division broadcasts correctly—this is a NumPy technicality that ensures the shapes align. Don't worry too much about it; just know that it makes the math work.

---

2. The Synthetic Data: Our "Toy" Example

Before we dive into the attention code, the notebook sets up some synthetic (fake) data so we can test our functions:

hidden_size = 16
attention_size = 10
input_length = 5

np.random.seed(42)

encoder_states = np.random.randn(input_length, hidden_size)
decoder_state = np.random.randn(1, hidden_size)

layer_1 = np.random.randn(2*hidden_size, attention_size)
layer_2 = np.random.randn(attention_size, 1)

2.1 What Do These Numbers Mean?

Variable	Value	What It Represents
hidden_size = 16	16	The size of each "thought vector." Every word gets encoded as 16 numbers that capture its meaning. In a real translation model, this might be 256, 512, or even 1024.
attention_size = 10	10	The size of the "attention network's" middle layer. Think of it as 10 little "judges" that help decide how relevant each word is.
input_length = 5	5	Our pretend input sentence has 5 words. For example: "The cat sat on mat" → 5 word-vectors.

2.2 The Arrays, Explained

📦 encoder_states – shape (5, 16)
This is a 5×16 matrix. Each of the 5 rows is one input word's "meaning vector"—16 numbers that encode what that word means in context. Row 0 = word 1 ("The"), row 1 = word 2 ("cat"), etc. This is what the encoder produced after reading the entire input sentence.

🧭 decoder_state – shape (1, 16)
This is the decoder's current "state of mind"—a single 16-number vector representing what it has produced so far and what it's thinking about next. In a real model, this updates after every output word. For this lab, we just have one decoder state to work with.

⚖️ layer_1 – shape (32, 10)
The first layer of our tiny neural network. It takes the concatenated encoder+decoder information (32 numbers = 16+16) and transforms it into 10 "relevance features." These are weights—numbers that get multiplied with the input. In a real model, they'd be learned through training.

⚖️ layer_2 – shape (10, 1)
The second layer. It takes those 10 relevance features and condenses them into a single score per word. Again, these are weights that would be learned during training.

np.random.seed(42) ensures that every time you run this notebook, you get the same random numbers. This is important for reproducibility—so the expected outputs in the notebook always match.

---

3. Step 1 — Calculating Alignment Scores

This is the heart of the attention mechanism. We need to answer: "For each input word, how relevant is it to what the decoder is trying to produce right now?"

🎯 Target Analogy: Imagine you're playing darts. You (the decoder) stand at the throwing line with a dart in hand. On the wall, there are 5 dartboards (the 5 encoder words). Before you throw, you need to aim at each board and decide which one you want to hit. The alignment score is essentially your "aim quality" for each board—a high score means you're well-aligned with that board; a low score means you're not really aiming at it.

3.1 The Mathematical Formula

e_ij = v_a^⊤ · tanh( W_a · s_i−1 + U_a · h_j )

This looks intimidating, so let's unpack it piece by piece:

• h_j – The hidden state of the j-th encoder word (the "meaning" of input word j). In our code, this is encoder_states[j], a vector of 16 numbers.
• s_i−1 – The decoder's hidden state from the previous step. In our code, decoder_state, also 16 numbers.
• W_a and U_a – Weight matrices that transform the decoder state and encoder state respectively. In practice, we concatenate the two states and use one big weight matrix (our layer_1).
• tanh – The hyperbolic tangent function. It squashes numbers into the range [-1, 1]. It adds non-linearity, which allows the network to learn complex relationships.
• v_a – A final weight vector (our layer_2) that collapses everything into a single score.

3.2 The Implementation, Line by Line

Here's the alignment function we need to fill in:

def alignment(encoder_states, decoder_state):
    # Step A: Concatenate
    inputs = np.concatenate(
        (encoder_states, decoder_state.repeat(input_length, axis=0)),
        axis=1
    )
    
    # Step B: First layer + tanh
    activations = np.tanh(np.matmul(inputs, layer_1))
    
    # Step C: Second layer (no tanh)
    scores = np.matmul(activations, layer_2)
    
    return scores

Step A — Concatenation: Gluing Two Things Together

We have encoder_states which is shape (5, 16) and decoder_state which is shape (1, 16). We need to combine them so that each encoder word gets paired with the same decoder state. This is like putting two spreadsheets side by side.

📊 Visual Example: Imagine the encoder has 3 words (not 5) and hidden size is 4 (not 16):

Encoder states (3×4):

Word 0: [a₁, a₂, a₃, a₄]
Word 1: [b₁, b₂, b₃, b₄]
Word 2: [c₁, c₂, c₃, c₄]

Decoder state (1×4):

        [d₁, d₂, d₃, d₄]

After repeating the decoder state (3×4):

        [d₁, d₂, d₃, d₄]
        [d₁, d₂, d₃, d₄]
        [d₁, d₂, d₃, d₄]

After concatenation (3×8):

Word 0: [a₁, a₂, a₃, a₄, d₁, d₂, d₃, d₄]
Word 1: [b₁, b₂, b₃, b₄, d₁, d₂, d₃, d₄]
Word 2: [c₁, c₂, c₃, c₄, d₁, d₂, d₃, d₄]

decoder_state.repeat(input_length, axis=0) copies the decoder state 5 times (once for each input word) so it becomes shape (5, 16). Then np.concatenate(..., axis=1) glues the encoder states (5×16) and the repeated decoder states (5×16) side by side along axis 1 (the column axis), producing a (5, 32) matrix. Each of the 5 rows now has 32 numbers: 16 from an encoder word + 16 from the decoder state.

🧩 Puzzle Analogy: Think of each encoder word as a puzzle piece with a unique shape (16 numbers). The decoder state is like a "target shape" you're trying to match. Concatenation puts each puzzle piece next to the target shape so you can compare them. It's like laying a key next to each lock to see which one fits best.

Step B — Matrix Multiplication + tanh: The "Comparison Engine"

np.matmul(inputs, layer_1) performs matrix multiplication between our concatenated data (5, 32) and the first layer weights (32, 10). The result is a (5, 10) matrix.

What does matrix multiplication actually do here? Each of the 32 input numbers gets multiplied by a corresponding weight in layer_1, and then products are summed in specific combinations to produce 10 output numbers per word. You can think of the 10 outputs as 10 different "aspects of relevance" that the network checks:

• Aspect 1: "Do the subject nouns match?"
• Aspect 2: "Is the tense consistent?"
• Aspect 3: "Are the word positions close?"
• ...and 7 more learned aspects.

After the multiplication, we apply np.tanh(). The tanh function looks like an S-curve:

• Very negative numbers → close to -1
• Zero → 0
• Very positive numbers → close to +1

Tanh serves two purposes: (1) it bounds the values so nothing explodes to infinity, and (2) it adds non-linearity. Without non-linearity, stacking layers would be pointless—two linear transformations in a row are equivalent to just one. Tanh breaks that linearity, letting the network learn richer patterns.

Step C — Second Layer: Producing a Single Score

np.matmul(activations, layer_2) multiplies our (5, 10) activations with the (10, 1) weights of layer_2. The result is a (5, 1) column vector—one number per input word.

No tanh here! The notebook explicitly reminds us: "Remember that you don't need tanh here." Why? Because we want raw scores that can be any real number (positive or negative). The softmax in the next step will handle normalizing them. If we applied tanh, all scores would be squashed between -1 and 1, which would make the softmax produce very uniform (uninteresting) weights.

🎯 Expected Output: If you implemented the function correctly, running the test cell should print:

[[4.35790943]
 [5.92373433]
 [4.18673175]
 [2.11437202]
 [0.95767155]]

Interpretation: Word 2 (index 1, score 5.92) is the most relevant to the decoder's current state. Word 5 (index 4, score 0.96) is the least relevant. But these are raw scores—they're not percentages yet. That's the next step!

---

4. Step 2 — Turning Alignment Scores into Attention Weights

Now we have 5 raw scores, one per input word. But they're on an arbitrary scale. Word 2's score of 5.92—is that a lot? A little? We need to convert these into percentages that sum to 100%, so we know exactly how to split our attention.

α_ij = exp(e_ij) / Σ_k=1^K exp(e_ik)

This is exactly what our softmax function does. Let's trace through the math with our actual expected scores:

🔢 Step-by-step with expected scores:

Scores:        [4.36,  5.92,  4.19,  2.11,  0.96]

exp(scores):   [78.1,  373.0,  65.8,   8.3,   2.6]
               (approximate values)

Sum of exps:   78.1 + 373.0 + 65.8 + 8.3 + 2.6 = 527.8

Weights:       [78.1/527.8, 373.0/527.8, 65.8/527.8, 8.3/527.8, 2.6/527.8]
             = [0.148,      0.707,       0.125,      0.016,     0.005]

Interpretation: ~14.8% attention on word 1
                ~70.7% attention on word 2  ←  THE MOST IMPORTANT WORD!
                ~12.5% attention on word 3
                 ~1.6% attention on word 4
                 ~0.5% attention on word 5

📢 Megaphone Analogy: Imagine you're at a concert. Five singers are on stage, each singing at a different volume. The sound engineer has a mixing board with 5 sliders. The alignment scores are like the raw volume readings from each microphone. The softmax is like the engineer adjusting the sliders so the total output volume is constant, but each singer contributes a different proportion. Singer 2 (70.7%) is the lead vocalist—you hear them the most. Singer 5 (0.5%) is barely audible.

Key property: The exponential function (eˣ) makes softmax peaky. Notice how a score of 5.92 becomes weight 0.707 while 0.96 becomes 0.005. The exponential amplifies differences, so the model can be very decisive about where to focus.

---

5. Step 3 — Weighting & Summing: The Context Vector

This is the final step. We have our attention weights (percentages), and we have our encoder hidden states (meaning vectors for each word). Now we combine them:

c_i = Σ_j=1^K α_ij · h_j

In plain English: "Multiply each word's meaning vector by its attention weight, then add them all up."

5.1 The Implementation

def attention(encoder_states, decoder_state):
    # Step 1: Get alignment scores
    scores = alignment(encoder_states, decoder_state)
    
    # Step 2: Convert scores to weights via softmax
    weights = softmax(scores)
    
    # Step 3: Multiply each encoder vector by its weight
    weighted_scores = encoder_states * weights
    
    # Step 4: Sum up to get one context vector
    context = np.sum(weighted_scores, axis=0)
    
    return context

Step 3 — Element-wise Multiplication

encoder_states * weights uses NumPy broadcasting. encoder_states is (5, 16) and weights is (5, 1). NumPy automatically stretches the weights across all 16 columns, so each row of the encoder states gets multiplied by its corresponding weight.

📊 Visual Example (simplified to 3 words, hidden size 4):

Encoder states (3×4):         Weights (3×1):
[a₁, a₂, a₃, a₄]              [0.15]
[b₁, b₂, b₃, b₄]              [0.70]
[c₁, c₂, c₃, c₄]              [0.15]

Weighted states (3×4):
[0.15×a₁, 0.15×a₂, 0.15×a₃, 0.15×a₄]    ← word 1, scaled down to 15%
[0.70×b₁, 0.70×b₂, 0.70×b₃, 0.70×b₄]    ← word 2, LOUD AND CLEAR at 70%
[0.15×c₁, 0.15×c₂, 0.15×c₃, 0.15×c₄]    ← word 3, scaled down to 15%

Notice how the middle row (word 2, weight 0.70) keeps most of its original magnitude, while words 1 and 3 are significantly dampened. This is the model saying: "Word 2 matters most right now, so I'll preserve its information almost fully. Words 1 and 3? Meh, I'll take just a little from each."

Step 4 — Summation

np.sum(weighted_scores, axis=0) adds up all 5 rows column-wise. The result is a single vector of 16 numbers—the context vector.

🍲 Soup Analogy: Imagine making a soup. You have 5 ingredients (words), each with its own flavor profile (hidden state). The attention weights tell you how much of each ingredient to add. Word 2 (weight 0.70) is like adding a full cup of carrots. Word 5 (weight 0.005) is like a tiny pinch of salt. When you stir everything together (np.sum), you get one pot of soup (the context vector) that tastes mostly of carrots but has subtle hints of the other ingredients. This soup is what the decoder "tastes" to decide on the next output word.

5.2 Expected Output

✅ If your attention function is correct, the context vector should be:

[-0.63514569  0.04917298 -0.43930867 -0.9268003   1.01903919 -0.43181409
  0.13365099 -0.84746874 -0.37572203  0.18279832 -0.90452701  0.17872958
 -0.58015282 -0.58294027 -0.75457577  1.32985756]

This 16-number vector is the attention-weighted summary of all 5 input words, biased toward whichever word the model deemed most relevant.

---

6. The Complete Picture: Tying Everything Together

Let's zoom out and see how all three steps connect:

1 Alignment Scores

Input: Encoder states (5×16) + Decoder state (1×16)

Operation: Concatenate → Multiply by layer_1 → tanh → Multiply by layer_2

Output: 5 raw scores (5×1)

Purpose: "How relevant is each input word to the decoder's current state?"

2 Attention Weights

Input: 5 raw alignment scores

Operation: Softmax (exponentiate → sum → divide)

Output: 5 percentages that sum to 1.0

Purpose: "What fraction of my attention should each word get?"

3 Context Vector

Input: Encoder states (5×16) + Attention weights (5×1)

Operation: Multiply each row by its weight → Sum all rows

Output: One context vector (16 numbers)

Purpose: "A weighted blend of all input meanings, focused on what matters now."

🎯 End Result

The context vector is fed into the decoder alongside its own hidden state to predict the next output word. In the next time step, the decoder updates its state, we recalculate attention (now with a new decoder state), and the cycle repeats.

---

7. Why Is Attention Such a Big Deal?

Before attention, sequence-to-sequence models (like early Google Translate) had a critical bottleneck: the entire input sentence had to be compressed into a single fixed-size vector. For a 50-word sentence, that's like trying to summarize "War and Peace" on a Post-it note. Attention removes this bottleneck by giving the decoder direct access to every input word.

7.1 Three Revolutionary Benefits of Attention

1. Handles Long Sequences: No matter how long the input is (5 words or 500), attention can "look back" at any position. The context vector is always a fresh blend tailored to the current decoding step.
2. Interpretability: We can visualize the attention weights as a heatmap and actually see which input words the model focused on when producing each output word. This is incredibly valuable for debugging and understanding model behavior.
3. Better Translation Quality: Especially for languages with different word orders (like English → Japanese), attention lets the model reorder concepts naturally because it can attend to input words in any order.

7.2 From Bahdanau to Transformers

The attention mechanism you just implemented is additive attention (also called "Bahdanau attention"), published in 2014. It uses a small neural network (our layer_1 and layer_2) to compute alignment scores. A year later, a simpler variant called multiplicative attention (or "Luong attention") was proposed, which just uses a dot product: score = decoder_state · encoder_state.

Then in 2017, the famous paper "Attention Is All You Need" introduced the Transformer architecture, which uses a more sophisticated version called scaled dot-product attention. The Transformer discards recurrence entirely and relies solely on attention mechanisms. This is the architecture behind GPT, BERT, and virtually all modern large language models.

🌟 The lineage:

1. Bahdanau Attention (2014) — Additive attention, the one you just coded. The original breakthrough.
2. Luong Attention (2015) — Dot-product attention, simpler and faster.
3. Scaled Dot-Product Attention / Transformers (2017) — Multi-head self-attention. Powers ChatGPT, Claude, Gemini, and all modern LLMs.

Every single one of these builds on the same core idea you just implemented: compare a query (decoder state) with keys (encoder states) to get scores, softmax into weights, and blend values (encoder states) into a context vector.

---

8. Complete Code Walkthrough Recap

Here's every meaningful line of code in this notebook, explained one more time at a glance:

Line	What It Does	Layman's Translation
import numpy as np	Imports the NumPy library for fast array math.	"Bring in my Swiss Army knife for number crunching."
def softmax(x, axis=0):	Defines the softmax function.	"Here's my recipe for turning any list of numbers into percentages."
np.exp(x)	Applies eˣ to every element.	"Make all numbers positive and amplify the differences."
np.sum(np.exp(x), axis=axis)	Sums the exponentiated values along an axis.	"Add up all the amplified numbers to get a total."
np.expand_dims(..., axis)	Adds a dimension for correct broadcasting.	"Reshape the total so the division lines up properly."
hidden_size = 16	Sets the size of hidden state vectors.	"Each word's meaning is captured in 16 numbers."
attention_size = 10	Sets the middle layer size.	"We'll use 10 'relevance detectors' in the attention network."
input_length = 5	Sets how many encoder words we have.	"Our pretend sentence has 5 words."
np.random.seed(42)	Fixes the random number generator.	"Make sure we all get the same random numbers."
encoder_states = np.random.randn(...)	Creates fake encoder hidden states.	"Pretend the encoder already processed 5 words into 5 meaning-vectors."
decoder_state = np.random.randn(...)	Creates a fake decoder hidden state.	"Pretend the decoder has a current 'state of mind.'"
layer_1 = np.random.randn(...)	Creates fake first-layer weights (32→10).	"Pretend we've already learned these comparison weights."
layer_2 = np.random.randn(...)	Creates fake second-layer weights (10→1).	"Pretend we've learned how to combine the 10 features into one score."
decoder_state.repeat(input_length, axis=0)	Copies the decoder state 5 times.	"Clone the decoder state so each word can be compared to it."
np.concatenate((enc, dec), axis=1)	Glues encoder and decoder states side by side.	"Put each word's meaning next to the decoder state for comparison."
np.matmul(inputs, layer_1)	Matrix multiplication: inputs × weights.	"Run the combined info through the first layer of the attention network."
np.tanh(...)	Applies the tanh activation function.	"Squash the results between -1 and 1 to keep things stable."
np.matmul(activations, layer_2)	Matrix multiplication: activations × weights.	"Run through the second layer to get one score per word."
softmax(scores)	Converts raw scores to percentages.	"Turn the scores into a proper attention distribution (sums to 1)."
encoder_states * weights	Element-wise multiplication with broadcasting.	"Scale each word's meaning by how much attention it gets."
np.sum(weighted_scores, axis=0)	Sums all weighted rows into one vector.	"Blend everything into one context vector—the final attention summary."

---

9. Common Questions (and Their Answers)

Q: Why do we use tanh in the first layer but not the second?

The first layer's tanh adds non-linearity, which is essential for the network to learn complex patterns. The second layer produces raw scores that get fed into softmax. If we applied tanh to the second layer, all scores would be between -1 and 1, making the softmax weights nearly uniform—every word would get roughly equal attention, defeating the purpose!

Q: Why is the hidden_size 16 and attention_size 10? Are these magic numbers?

Not at all! They're arbitrary choices for this educational example. In a real model, hidden_size might be 256, 512, or 1024, and attention_size is typically the same as hidden_size. The authors chose small numbers so the code runs instantly and the shapes are easy to reason about.

Q: What's the difference between encoder_states and decoder_state?

Encoder states are the "meanings" of the input words after the encoder has read the entire input sentence. There's one vector per input word. Decoder state is the "state of mind" of the output-producing side, representing what it has generated so far and what it's thinking about producing next. The attention mechanism bridges these two: "Given what I want to say next (decoder state), which input words should I pay attention to (encoder states)?"

Q: Are layer_1 and layer_2 learned during training?

Yes! In this lab, we use random numbers for simplicity. In a real model, these weights are initialized randomly and then gradually improved through backpropagation during training. The model learns which patterns in the encoder+decoder combination indicate relevance. After training on millions of sentence pairs, the weights encode sophisticated linguistic knowledge.

Q: How does this relate to the attention in ChatGPT?

ChatGPT uses self-attention within the Transformer architecture. The core math is very similar—query, key, value projections; dot-product scores; softmax; weighted sum—but instead of comparing a decoder state to encoder states, self-attention compares every word to every other word in the same sentence. This lets the model understand relationships like "the pronoun 'it' refers to the noun 'cat' mentioned 5 words ago." The basic building block you implemented here is the conceptual ancestor of all modern attention mechanisms.

---

10. Final Thoughts: You Just Built the Foundation of Modern AI

Take a moment to appreciate what you've done. With fewer than 30 lines of NumPy code, you've implemented the core mechanism that:

• Revolutionized machine translation in 2014
• Earned a citation count of over 30,000 for the Bahdanau paper
• Paved the way for the Transformer architecture in 2017
• Underlies every modern large language model, including GPT-4, Claude, and Gemini

The three-step recipe you coded—score, weight, blend—is the same pattern used by models with hundreds of billions of parameters. They just do it many times in parallel (multi-head attention) and stack many layers. But the fundamental operation is exactly what you wrote:

1. Compare a query with keys to get scores
2. Softmax the scores into weights
3. Use the weights to blend values into an output

This notebook has given you the conceptual and practical foundation to understand all of it. Well done! 🎉

Explanation generated for the Basic Attention Operation ungraded lab — Course 4, Week 1 of the NLP with Attention Models Specialization.

See All on GenAI    « Previously    Download the 2014 paper on Additive Attention    View the Lab Work From DeepLearning.ai Tags: Generative AI,Large Language Models,