Jeff Dean: Exciting Trends in ML (Rice 2024 watch companion)

If you want one lecture that explains how we got from “computers can’t see” to multimodal chatbots, this is a great pick. Jeff Dean, Google’s chief scientist, gave Exciting Trends in Machine Learning as the Ken Kennedy Institute Distinguished Lecture at Rice University on February 13, 2024. It’s about 72 minutes, classroom style. Here’s a companion to watch alongside it.

(Heads up: clips of this talk circulate on social media with chapter labels like “LLM from scratch” or “100 agents.” Those are marketing overlays. Ignore them.)

The one-line thesis

Scale (compute plus data plus model size) times better algorithms times specialized hardware gives you general multimodal models like Gemini. Those models reverse old machine-learning arrows (they generate, not just classify) and open up science and health uses, with real responsibility for bias, privacy and social benefit.

A rough map

Times are approximate; there are no official chapters.

0:00 to ~8:00: setup, and a decade of progress in perception
~8:00 to ~18:00: image and speech error rates; why ML wants low-precision math; Google’s TPU chips
~18:00 to ~35:00: the language arc, from n-grams to word2vec to the transformer to Gemini
~35:00 to ~50:00: training infrastructure, data quality, chain-of-thought, benchmarks
~50:00 to ~60:00: product demos, Med-PaLM, image and video generation, phone features
~60:00 to ~68:00: science and health; AI principles
~68:00 to end: Q&A

What to listen for

Scale works. More compute, richer data, bigger models give better results, and new capabilities appear once accuracy crosses a usability line. Example: speech word error rates fell from about 13% to about 2.5% in about five years, and dictation became reliable. On ImageNet, accuracy went from about 50.9% in 2011, through the AlexNet jump, to about 91%.

Reversing the arrows. Old ML went pixels to label, audio to transcript. New ML also goes text to image, video and audio.

Specialized chips. Neural networks tolerate reduced precision, and almost everything is linear algebra, so Google built chips (TPUs) for low-precision matrix math. TPU V1 for inference was about 30 to 80x better on energy and performance than CPUs of the time. Later generations scaled into pods of thousands of chips.

The transformer moment. Attending to all words in parallel, instead of updating state one word at a time, gave 10 to 100x less compute for higher accuracy. That multiplies with the hardware gains.

Data quality. Dean argues data quality is as important as architecture, sometimes more. Low-quality data can actually hurt (math is his example).

Show your work. Chain-of-thought prompting (asking the model to reason step by step) gives a big accuracy jump past a certain scale. His demo: a photo of a student’s handwritten physics answer; the model finds the mistake and solves it.

Small teams still matter. In Q&A he notes the original transformer work was done on about eight GPUs.

Pull-quotes

Every time we scale things up, things get better… new capabilities emerge or accuracy crosses a usability threshold.

Simple techniques over large amounts of data are very effective.

Asking models to show their work improves accuracy and interpretability.

An analogy for the whole talk

Think of the talk as the history of the airplane told by someone who helped build the engines. Bigger engines (compute), better fuel (data), a new wing design (the transformer), and a custom factory (TPUs). Each one alone helps. Together they turned short hops into flights across oceans.

Try this

Watch the first 20 minutes, then pause and try his “show your work” trick yourself. Ask a chatbot a tricky word problem, like: “A bat and a ball cost $1.10 together. The bat costs $1.00 more than the ball. How much is the ball?” Then ask the same question in a new chat with “Think step by step and show your work” added. Compare. Then finish the talk and see if you agree with his take that “multimodal” is a better word than “LLM.”