Skip to learning content

DATA SCIENCE PYTHON PLAYGROUND

Data Foundations · A little practice goes a long way.

← Wrangle / Preprocess lessonsTINY TABLES · REAL PYTHON · YOUR PACE
Combining & preparing · W29 · 8 MIN

Handle outliers responsibly

Flag unusual values without deleting evidence.

Exercises within this concept

  1. FollowFollow the techniqueCurrent exercise
  2. ChangeAdapt a requirement
  3. TransferCombine earlier skills

Understand the idea

The IQR rule flags unusually low or high values for review. It does not establish that a value is wrong.

Concept sketch: flag outside fences; do not delete40: flag7.517.510, 11, 12, 13, 14, 40flag outside fences; do not delete
Illustration · not the exercise output

A small example

If Q1=10 and Q3=14, IQR=4. The fences are 4 and 20: 3 and 21 are flagged; 4 and 20 are not.

Follow the code

Apply the idea to the supplied table. Read from top to bottom; the final line displays the result.

q1 = df["price"].quantile(0.25)
q3 = df["price"].quantile(0.75)
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
df["needs_review"] = (df["price"] < lower) | (df["price"] > upper)
df

What each part does

q1 / q3
25th / 75th percentiles
iqr = q3 - q1
spread of the middle half
1.5 * iqr
distance beyond each quartile for screening
(value < lower) | (value > upper)
flag either tail; retain the original value

Your inputs

The editable setup on the right creates df. Run executes the setup and your work from top to bottom.

Candy shop · review measurements · 6 synthetic rows
candyflavourpriceratingshelf
Gummy Bearfruity104.1A
Choco Popchocolate114.6B
Mint Bitemint123.8A
Berry Loopfruity134.4B
Cocoa Cubechocolate144.9A
Lemon Dropfruity404B

Your task · Follow

  1. In df, add needs_review: True when price is below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR, where IQR = Q3 − Q1.
  2. Keep all rows and original values; return df.
  3. Use: quantile().
Hint

The distance extends beyond both quartiles; do not mistake Q3 itself for the upper fence.

Reveal solution

One way to do it. Keep any supplied setup in the editor and use this in the Your work section.

q1 = df["price"].quantile(0.25)
q3 = df["price"].quantile(0.75)
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
df["needs_review"] = (df["price"] < lower) | (df["price"] > upper)
df
Optional stretch

If a sensor is known to saturate at 100, a separate capped column may be defensible. Why should the raw column still be retained?

Your task · Follow
  1. In df, add needs_review: True when price is below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR, where IQR = Q3 − Q1.
  2. Keep all rows and original values; return df.
  3. Use: quantile().

Tab: indent · Shift+Tab: outdent · Esc, then Tab: leave editor

Edit Python. Control or Command plus Enter runs it. Tab indents by four spaces. Shift plus Tab outdents. Press Escape, then Tab or Shift plus Tab to leave the editor.

Each run executes all editor code in a fresh Python session. Display a value by leaving it on the final line.

Python starts when you open a lesson.

Output

Run your code to see what Python returns.