Generalisation

Key question

How do neural networks learn representations?

Check your understanding

Find an example, an application and a limit. Explain the concept without using its name, then compare your explanation with a primary source.

Claims to examine

• Learning representations from data can improve a task while leaving generalisation and distribution shift unresolved.

  Editorial claim · contestedExamine the source →

Follow a learning route

• How do neural networks learn representations?

  Build from differentiation and optimisation to architectures, generalisation and scaling, using small reproducible experiments.

Reading connections

• Deep Learning: Deep feedforward networks

  Ian Goodfellow, Yoshua Bengio · Aaron Courville

• Learning Representations by Back-Propagating Errors

  Rumelhart, Hinton · Williams

• Deep Learning: Optimisation

  Ian Goodfellow, Yoshua Bengio · Aaron Courville

• Deep Learning: Regularisation

  Ian Goodfellow, Yoshua Bengio · Aaron Courville

• Scaling Laws for Neural Language Models

  Kaplan et al.

• Training Compute-Optimal Large Language Models

  Hoffmann et al.