Natural Language Processing

Why language is hard for machines

Language is compact, ambiguous and drenched in context. "I saw her duck" — a bird or a dodge? "The trophy didn't fit in the suitcase because it was too big" — what was too big? (Swap "big"→"small" and the referent flips — the Winograd test.) Resolving these requires world knowledge, not grammar alone. That's why NLP humbled rule-writers for fifty years.

The task landscape

Natural language processing is a family of tasks, most of which you can now map to ML framings you already know:

Task	Example	ML framing
Text classification	spam, sentiment, topic	classification
Named-entity recognition	find people/places/dates	per-token classification
Machine translation	English → Sinhala	sequence-to-sequence
Summarization	article → abstract	sequence-to-sequence
Question answering	answer from documents	retrieval + generation
Speech recognition	audio → text	sequence-to-sequence
Dialogue	chatbots, assistants	next-token generation

Three eras in fast-forward

Rules (1950s–90s) — hand-written grammars and dictionaries; ELIZA (1966) faked therapy with pattern-matching tricks. Brittle beyond toy domains.
Statistics (1990s–2010s) — learn from corpora: n-gram probabilities, tf-idf + classical classifiers, statistical translation. Robust but shallow — "bags of words" with no grasp of meaning or order.
Neural (2013–) — the DL-path storyline in application: embeddings give words meaningful geometry; RNN/LSTM seq2seq made translation fluent (Google Translate's 2016 jump); transformers + pretraining unified everything: models like BERT (read text, power search/classification) and GPT (generate text) are pretrained once on internet-scale corpora, then fine-tuned or simply prompted per task. Today, most classic NLP pipelines have collapsed into "ask a large language model" — one generative model performing every row of the table above.

The pipeline's atoms: tokens

Models don't read letters or words but tokens — learned subword chunks ("unbelievable" → un + believ + able). Tokenization explains familiar LLM quirks: struggles with character-level puzzles or arithmetic (digits split oddly), and why languages underrepresented in training data — Sinhala among them — consume more tokens per sentence and often get weaker performance. Language equity in AI is an open, important problem.

Grounding and evaluation

Text-only learning captures how words co-occur, not what they refer to — critics call it the "stochastic parrot" concern. Multimodal training (text+images+audio) and tool use (calculators, search engines, code) are the field's current answers, along with retrieval-augmentation to pin generations to sources. Evaluation remains genuinely hard: fluent text can be confidently wrong, so benchmarks mix task suites with human preference judgments.

Key takeaways

Ambiguity + world knowledge make language AI's hardest perception problem.
Rules → statistics → neural: each era traded hand-coding for data; transformers unified the task zoo.
Tokens, grounding and language equity explain most practical LLM quirks you'll encounter.