piatra . institutepathways
Explore/AI & computation/Introduction to AI & computation

What does it mean to evaluate an AI system?

Evaluation compares a defined task, dataset and measure; those choices affect what success means.

AI & computationIntroduction · Selected readingBeginner friendly
Background and goal

No prior study required. Read the assigned passage; the route is an introduction, not a survey of the whole subject.

Design an evaluation for a hypothetical image classifier and identify two kinds of mistakes a single accuracy score could hide.

Your pathway

4 steps · 3 hours
  1. 01

    Begin in fifteen minutes

    0.25h · week 1
    Reading notes and task
    First fifteen minutesWrite your initial answer to “What does it mean to evaluate an AI system?” in three sentences. Keep it to compare with your final answer.
    Your outputThree-sentence starting answer
  2. 02

    Read the argument

    1h · week 1
    Datasheets for Datasets

    Timnit Gebru · Jamie Morgenstern · Briana Vecchione · Jennifer Wortman Vaughan · Hanna Wallach · Hal Daumé III · Kate Crawford · paper

    Abstract, Introduction and the overview of documentation questions

    Free full text

    Open the reading
    Reading notes and task

    Datasheets for Datasets. Evaluation compares a defined task, dataset and measure; those choices affect what success means.

    Try thisRead Abstract, Introduction and the overview of documentation questions. Mark one claim, its reason and a passage you do not yet understand.
    Your outputAnnotated argument
    Check your understanding

    Answer this question in your own words: What does it mean to evaluate an AI system?

    Hint

    Name the claim, then explain why a reader might accept it.

    Compare with an example answer

    Evaluation compares a defined task, dataset and measure; those choices affect what success means.

    What to look for

    • States a claim that addresses the question
    • Connects the claim to a passage or observation
    • Separates the author’s argument from personal agreement

    Common mistake: Summarising a biography instead of explaining the argument.

    Start this pathway to keep your answers in My learning.

  3. 03

    Try the idea

    1h · week 1
    Reading notes and task
    Try thisDesign an evaluation for a hypothetical image classifier and identify two kinds of mistakes a single accuracy score could hide.
    Your outputWorked example
    Check your understanding

    What would make your activity a convincing example?

    Hint

    Design an evaluation for a hypothetical image classifier and identify two kinds of mistakes a single accuracy score could hide.

    Compare with an example answer

    A useful response carries out the task: Design an evaluation for a hypothetical image classifier and identify two kinds of mistakes a single accuracy score could hide. It records the observation or passage used and explains how it bears on the question.

    What to look for

    • Completes the specific task
    • Shows evidence or reasoning rather than only a conclusion
    • Identifies an alternative explanation or reading

    Common mistake: Presenting an opinion without showing the example it rests on.

    Start this pathway to keep your answers in My learning.

  4. 04

    Explain what changed

    0.75h · week 1
    Reading notes and task
    Try thisRevise your initial answer. Include one passage, one reason and one limit: A benchmark score does not establish suitability for every use.
    Your outputRevised answer with a reading reference
    Check your understanding

    What is one conclusion this reading does not establish?

    Hint

    Test the boundary: A benchmark score does not establish suitability for every use.

    Compare with an example answer

    A benchmark score does not establish suitability for every use.

    What to look for

    • States a specific limit
    • Explains why the reading does not warrant the stronger conclusion
    • Distinguishes a limitation from dismissing the work

    Common mistake: Treating a useful interpretation as an unrestricted fact.

    Start this pathway to keep your answers in My learning.

All 1 resources

Background, different viewpoints, and further reading.

  • Datasheets for Datasets

    Timnit Gebru · Jamie Morgenstern · Briana Vecchione · Jennifer Wortman Vaughan · Hanna Wallach · Hal Daumé III · Kate Crawfordpapercore

    Evaluation compares a defined task, dataset and measure; those choices affect what success means.

    Abstract, Introduction and the overview of documentation questions

Continue learning

  • Datasets and decisions

    Introduces the vocabulary and a small example before the longer study.

    AI & computationSome background

  • AI as a material system

    Introduces the vocabulary and a small example before the longer study.

    AI & computationSome background