Distance, and why high dimensions are weird
Clustering, nearest-neighbor search, and chapter 22's retrieval all stand on one primitive: a distance (or similarity) between two feature vectors. That primitive behaves less and less like your geometric intuition as dimensions grow — and your embeddings have hundreds of dimensions.
Run the measurement
The editor drops random points into d-dimensional space and compares the nearest neighbor's distance to the farthest point's. In 2-D, nearest is a small fraction of farthest — being "close" means something. By 1000-D the ratio pushes toward 1: everything is almost the same distance from everything. That's the curse of dimensionality, measured live in twelve lines.
Practical consequences, not doom:
- Contrast shrinks; it doesn't vanish. Real data (unlike this uniform noise) lives near lower-dimensional structure, which is why embeddings still work. But margins get thin — which is why chapter 22 told you to compare relative similarity rankings, not absolute scores.
- Scale dominates. In any metric space, an unscaled feature with a big range owns the distance (last lesson's warning, now with a mechanism).
- Cosine vs Euclidean. Cosine compares direction, ignoring length; Euclidean compares absolute position. For text embeddings the convention is cosine (chapter 22's function). For physical features, Euclidean-after-standardizing is the default.
Dimensionality reduction, in one paragraph
Sometimes you shrink dimensions on purpose: PCA finds the directions of largest variance and keeps the top few — good for compressing correlated features and for 2-D "maps" of your data. Visualization-first tools (t-SNE, UMAP) make prettier maps but distort global distances — treat their plots as suggestive neighborhoods, never as measurements. Reduce → then cluster is a common, sane pipeline for wide data.
Where AI specifically gets this wrong
- Euclidean on raw mixed-unit features. The generated k-means "worked," and every cluster is secretly the income column.
- Reading absolute cosine scores as confidence. 0.83 vs 0.79 across different models/dims is noise (chapter 22 said it; today's demo is the why).
- Presenting t-SNE plots as truth. "The clusters are clearly separated" — in a projection built to separate them. Verify in the original space.