Context Connect: A Word Context Game

Introduction

In the realm of Natural Language Processing (NLP), understanding the meaning and relationships between words is fundamental. Word embeddings, like GloVe, provide a powerful way to represent words as numerical vectors, capturing their semantic nuances. By measuring the similarity between these vectors, we can determine how closely related words are in meaning. This concept of semantic similarity can be harnessed to create engaging and educational word games.

The code we’re about to explore implements a “Word Context Game” that leverages this principle. The game challenges users to guess a target word by providing words with varying degrees of semantic similarity. The game uses cosine similarity to quantify the relationship between the guessed word and the target word. This game is a great example of how word embeddings can be used to create interactive and insightful NLP applications.

Core Concepts

Word Embeddings

The heart of the game lies in word embeddings, specifically GloVe (Global Vectors for Word Representation) in this case. These embeddings represent words as dense vectors in a high-dimensional space, capturing semantic relationships between words. The load_word_embeddings function reads a GloVe file, mapping words to their corresponding vectors. This allows the game to quantify the similarity between words.

Cosine Similarity

The calculate_similarity function utilizes cosine similarity to measure the semantic relatedness between the user’s guess and the target word.

$$ \text{Cosine Similarity}(A, B) = \frac{A \cdot B}{\|A\| \|B\|} $$

Where A and B are the vectors of the two words. A higher cosine similarity indicates a closer semantic relationship.

Game Logic

The WordContextGame class encapsulates the game’s logic. It manages the target word, difficulty level, and similarity calculations. The set_difficulty method allows users to adjust the game’s challenge. The set_target_word method selects a random word from the GloVe vocabulary, or permits the developer to set a specific word. The get_feedback function provides textual feedback based on the similarity score and the selected difficulty level.

Flask Web Application

The code utilizes Flask to create a web-based interface. The / route renders the index.html template, providing the game’s interface. The /guess route handles user input, calculates similarity, and returns feedback as JSON.

Text Preprocessing

The preprocess_text function cleans the input text by lowercasing, removing punctuation, tokenizing, and removing stop words. This ensures that the similarity calculations are based on meaningful words.

What GloVe actually stores, and why cosine is the right comparison

GloVe is not trained on running text the way word2vec is. It factorises a global co-occurrence matrix X, where X_ij counts how often word j appeared inside the context window of word i across the entire corpus. The training objective is a weighted least-squares fit of dot products to log counts:

J = ∑_{i,j}  f(X_ij) · ( w_i · w~_j + b_i + b~_j − log X_ij )²

f(x) = (x / x_max)^α   if x < x_max        (x_max = 100, α = 0.75)
     = 1                 otherwise

So the dot product between two learned vectors is fitted to approximate the logarithm of a co-occurrence count. Two consequences fall straight out of that, and both matter for the game.

First, the similarity is distributional, not semantic. The model never saw a definition; it only ever saw which words share neighbours. Section five below is entirely about the places where those two things come apart.

Second, the vector norm carries frequency information. The weighting function f deliberately gives frequent pairs more influence on the loss, and the fitted biases absorb frequency effects, so magnitudes end up systematically different between common and rare words. If calculate_similarity used Euclidean distance, common words would look "close" to everything purely for being common. Cosine divides the norm out and keeps only direction, which is the part that encodes context. That is why cosine is the correct primitive here rather than an arbitrary stylistic choice.

A third consequence is less obvious and bites later: the space is anisotropic. The mean of all 400,000 vectors is nowhere near the origin, so every pair of words shares a common component and random pairs do not score near zero — they cluster in the low positives. Subtracting the corpus mean before normalising, and optionally projecting out the top two or three principal components, recentres the distribution and makes the numbers you eventually show a player far more informative. The same trick applies to any embedding built by factorising a count matrix, including graph embeddings such as DeepWalk.

Loading 400,000 vectors without a 90-second cold start

load_word_embeddings as written parses a text file. glove.6B.300d.txt is roughly a gigabyte of ASCII: 400,000 lines, each a token followed by 300 space-separated decimal strings. Parsing it means about 120 million calls to float(), which costs tens of seconds to well over a minute on ordinary hardware — and it happens on every Flask autoreload while you are developing.

Convert once, then memory-map:

# ---- one-off conversion ----
words, rows = [], []
with open("glove.6B.300d.txt", encoding="utf-8") as fh:
    for line in fh:
        tok, *vals = line.rstrip().split(" ")
        words.append(tok)
        rows.append(np.asarray(vals, dtype=np.float32))

V = np.vstack(rows)                              # (400000, 300) f32 = 480 MB
V -= V.mean(axis=0)                              # kill the anisotropy offset
V /= np.linalg.norm(V, axis=1, keepdims=True)    # pre-normalise ONCE
np.save("glove300.npy", V)
json.dump(words, open("vocab.json", "w"))

# ---- at serve time ----
V     = np.load("glove300.npy", mmap_mode="r")
words = json.load(open("vocab.json"))
index = {w: i for i, w in enumerate(words)}

Two things changed. Start-up is now a memory map: pages arrive from the OS page cache on demand instead of being reconstructed from text, so the process is ready immediately. And because every row is already unit length, cosine similarity collapses to a plain dot product — float(V[i] @ V[j]), no division, no norm computation per request.

That second point is what makes the rest of the design affordable. Scoring one guess against the entire vocabulary is a single matrix-vector product V @ q: 400,000 × 300 = 120 million multiply-accumulates, which BLAS dispatches in a few milliseconds. A full-vocabulary scan being cheap is exactly what lets you replace the raw similarity score with something much better.

Raw cosine makes a bad game - rank makes a good one

The instinct is to show the player the cosine directly. Do that and the game feels broken, because the number has almost no dynamic range in the region where the player is actually working.

Suppose the target is guitar. A near-miss like bass might score 0.55; a genuinely-wrong-but-adjacent piano scores 0.62; an unrelated refrigerator scores 0.12. The player sees 0.55 against 0.62 and cannot tell which is meaningfully closer, and meanwhile every one of their first ten exploratory guesses lands in a narrow band around 0.15 and looks identical. The information the player needs — where does this sit relative to everything else? — is not in the raw value.

The fix is to score by rank, not by value. When the target is chosen, score the whole vocabulary once and keep the head of the list:

q    = V[index[target]]
sims = V @ q                                    # (400000,) one sgemv, ~ms
k    = 1000
top  = np.argpartition(-sims, k)[:k]            # O(n) selection, not O(n log n)
top  = top[np.argsort(-sims[top])]              # sort only the k survivors
rank = {words[j]: r for r, j in enumerate(top)} # rank 0 == the target itself

Now a guess reports "the 41st closest word out of 400,000" instead of "0.55". Rank stretches the interesting region out: the gap between rank 41 and rank 380 is immediately legible to a human, while the cosines behind them differ by 0.04. Words outside the precomputed head get a "cold" verdict plus their raw similarity, which is all the resolution a cold guess needs.

argpartition earns its place here: a full argsort of 400,000 floats is O(n log n) work to produce an ordering you will never display past position 1,000. And because the whole ranking is built once when the target is set, every subsequent guess is a dictionary lookup — the request path never touches the matrix.

Difficulty tuning: frequency rank is the dial that actually works

set_difficulty is the piece most implementations get wrong, because they tune the feedback thresholds when the real lever is target selection.

The released GloVe files list tokens in descending corpus frequency, so a word's line number is a frequency rank you get for free, with no extra resource to ship. (Verify it on the file you actually downloaded rather than assuming it — a five-line check on the first and last hundred tokens settles it.) That single property gives you a clean difficulty axis: sample from the first few thousand rows and you get words every player knows; sample from the tens of thousands and you get words that are recognisable but hard to reach by free association.

Do not sample raw rows. GloVe's 6B vocabulary is lowercased Wikipedia and web text: it contains punctuation tokens, bare numerals, URL fragments, misspellings, proper nouns and non-English strings. Intersect the candidate band with a curated English word list, drop anything failing str.isalpha() or shorter than four characters, and decide explicitly whether proper nouns are in scope. A round whose target turns out to be http, de or 1999 is dead on arrival.

The second lever is k, the depth of the neighbour list. Too shallow — say 200 — and almost every guess reads "cold", so the player gets no gradient to climb. Too deep — 5,000 or more — and almost everything reads "warm", which is the same as no signal. Around 1,000 out of a 400,000-word vocabulary keeps roughly 99.75% of guesses cold while preserving a usable slope in the neighbourhood of the answer.

Where distributional similarity betrays the game

The distributional hypothesis — words appearing in similar contexts have similar meanings — is a good approximation and a bad guarantee. Four failure modes show up in essentially every deployment of this design, and each one needs a deliberate decision rather than a bug fix.

Antonyms are near neighbours

hot and cold, up and down, always and never occur in nearly identical contexts, so GloVe places them close together. A player guessing cold against the target hot gets a strong signal that reads as "almost there" and is in fact the exact opposite. There is no fix inside the embedding — the geometry is behaving correctly and the semantics are not what you assumed. You either accept it as part of the game's texture or carry a small antonym blocklist (WordNet's antonym relation covers the common pairs) and label those hits explicitly.

Morphology leaks the answer

run, runs, running and ran sit very close to one another. A player who stumbles on one inflection and sees a hot score can walk the whole paradigm mechanically and land on the target without any insight into the clue. Stem or lemmatise both the guess and the target and reject any guess sharing the target's stem, the same way a crossword refuses to accept the answer's own root as a clue word.

Hubness

In high-dimensional spaces a small number of points appear in a disproportionate share of everyone's nearest-neighbour lists. In GloVe these hubs tend to be generic, very high-frequency words. They score warm against almost any target, which both misleads players and makes difficulty inconsistent from round to round — one target's top-1,000 is full of real associations, another's is padded with hubs. Excluding the top few thousand frequency ranks from the hint pool, and optionally from scoring altogether, flattens most of the variance.

One vector per word, all senses averaged

bank gets a single vector blending the financial and the riverside sense; bass blends the fish and the instrument. Guessing river against target bank returns a mid-range score that is right for one sense and meaningless for the other, and the player has no way to know which. Static embeddings cannot disambiguate. That is precisely the problem contextual models were built to solve, and precisely why they still do not fit here.

Preprocessing: the function that quietly breaks single-word input

preprocess_text lowercases, strips punctuation, tokenizes and removes stop words. That is the correct pipeline for a document. Applied to a one-word guess it is a landmine.

Feed it the, not, over, own or just and stop-word removal returns an empty token list. Whatever comes next — indexing tokens[0], averaging a zero-length array — raises, and the player gets a 500 for typing a legal English word. NLTK's English stop list runs to roughly 180 entries and several of them (not, no, own, same, very) carry real meaning and are entirely reasonable guesses.

Split the pipeline in two. The guess path should lowercase, strip surrounding punctuation, reject anything with whitespace or non-letters, and then do exactly one thing: look the token up in the vocabulary.

guess = raw.strip().lower()

if not guess.isalpha():          return reject("letters only")
if guess not in index:           return reject("not in this dictionary")
if stem(guess) == stem(target):  return reject("that is a form of the answer")
if guess in history:             return reject("already tried")

r = rank.get(guess)
return {"guess": guess,
        "rank":  r,                       # None => outside the top k
        "sim":   float(V[index[guess]] @ q),
        "band":  band_for(r, difficulty)}

The distinction that matters to players is between wrong and not accepted. A guess that is out of vocabulary must not consume a turn or enter the history — otherwise the vocabulary's gaps look like the game lying about the score. Keep the document-oriented preprocess_text for anywhere you genuinely process free text, and keep it away from the guess route entirely.

Serving it: one big matrix behind Flask

The /guess route is cheap — a dictionary lookup and at most one dot product. The expensive resource is the 480 MB matrix, and how much you pay for it across workers depends entirely on how you loaded it.

If you np.load the array into ordinary anonymous memory, gunicorn's default lazy loading gives each worker its own copy: four workers means 1.9 GB resident for a game that needs 480 MB. The fix is to load before the fork:

# gunicorn.conf.py
preload_app = True     # load BEFORE fork; workers share pages copy-on-write
workers     = 4
threads     = 2

preload_app builds the matrix in the parent and forks afterwards, so the pages start out shared. This works better with NumPy than it would with ordinary Python objects: CPython keeps a reference count in every object header, so merely reading an object dirties its page and un-shares it — but a NumPy array's data buffer is one flat allocation with no per-element headers, so the 480 MB genuinely stays shared for the lifetime of the workers.

With mmap_mode="r", as recommended above, the sharing is free and preload_app is no longer load-bearing for memory: every worker maps the same file and the OS page cache holds exactly one copy whether you forked or not. Pre-loading still saves the repeated open and the per-worker rebuild of the 400,000-entry index dict, which is real but measured in tens of megabytes, not gigabytes. The mmap buys something else too — those pages are file-backed, so under memory pressure the kernel evicts them instead of pushing the box into swap.

The per-round precomputation — the V @ q plus the argpartition — is the only real CPU cost, and it happens once per target, not once per guess. If targets rotate daily, compute the ranking in a startup or cron task and cache the resulting dict; the request path then never touches the matrix and the game will serve thousands of guesses per second on a single core.

Finally, session state. The target id and guess history are small, but they must not live in a module-level global — that global is per-worker, and players will watch their history vanish as the load balancer moves them between processes. A signed cookie handles it for a single-round game; Redis handles it once you want resumable sessions or a shared scoreboard.

Evaluating a word game, which is not an NLP benchmark

There is no accuracy number here. What you are tuning is a difficulty distribution, and the only honest instrument is play data. Three signals cover most of it.

Guesses-to-solve distribution. Report the median and the p90, never the mean — the mean is dominated by the players who brute-force a category. A round with a median of 8 and a p90 of 90 is not "medium difficulty", it is two different games sharing a target.

Abandonment rate and abandonment point. Where players quit tells you where the gradient died. Mass abandonment between guesses 15 and 25 usually means the target sits in a sparse region of the space and nothing in a normal player's vocabulary lands inside the top-k.

First-warm latency. The number of guesses before a player's first non-cold result. If that routinely exceeds ten, the round has no on-ramp no matter how reasonable the target looks on paper.

You can also screen targets offline before they ever ship, which is much cheaper than screening them with players. For each candidate, inspect its top-100 neighbours: if they are dominated by inflections of the word itself, by proper nouns, or by tokens an average player would never produce, the round will play badly. Requiring that at least, say, 40 of the top 100 neighbours appear in a common-words list removes most bad rounds automatically and leaves a much smaller pile for a human to read.

When a different tool is the right one

Static embeddings suit this game precisely because they are static. Every word has exactly one vector, so a target's neighbourhood can be precomputed and a guess costs a lookup. Three alternatives are worth understanding, and mostly worth declining.

Contextual embeddings. In a transformer encoder a word has no fixed vector; you would have to invent a sentence to embed the guess in, and that invented context would drive the score more than the word does. Contextual spaces are also strongly anisotropic, so isolated-token cosines compress into a narrow high band — the opposite of the spread you need. Sentence encoders become the right tool only if you move from single words to phrase guesses.

WordNet path similarity. A hand-built sense hierarchy. It knows that hot and cold are antonyms and that bank has distinct senses — exactly the two things GloVe gets wrong. It is also brittle and sparse: coverage stops at the taxonomy's edges, cross-part-of-speech comparisons are undefined, and the scores are coarse and lumpy. It makes an excellent filter layered over the embedding and a poor primary score.

Approximate nearest-neighbour indexes. HNSW or DiskANN would matter if you needed neighbour queries for arbitrary, unseen query vectors at scale. Here the query set is one known vector per round, so an exact brute-force scan computed once is simpler, exact, and faster than maintaining a graph index. See HNSW and vector databases compared for where that tradeoff flips — broadly, when queries are unpredictable and the corpus is large enough that a full scan blows the latency budget.

Conclusion

“Context Connect” showcases the practical use of word embeddings and cosine similarity for interactive language games. It utilizes a Flask web interface to engage users in guessing target words based on semantic similarity. The game’s adjustable difficulty and clear feedback enhance the user experience. This project effectively translates NLP concepts into an entertaining and educational application. Further development could incorporate visual elements and user-generated content for a more immersive experience.