Describe a sampled hierarchy · Data Playground

The question

Describe a Ward hierarchy of a reproducible sample of at most 500 Breast measurement rows, with diagnosis excluded from fitting.

data/ml-learning/ML-X18.csv Complete workflow and evidence

Your inputs

Challenge input · ML-X18

569 observations. One tumour sample represented by cell-nucleus image measurements. Load the CSV file yourself.

Download challenge input CSV · Original source dataset · Source and original dictionary

Ready to load: comma-delimited UTF-8, header row, no saved index. No filtering or target construction is required. This is the 569-row measurement population. Creating the reproducible sample of at most 500 rows remains part of the hierarchy task. Fitting input contains measurements only; species/diagnosis reference labels are not supplied or required.

ML-X18 · first 8 prepared rows
`radius_mean`	`texture_mean`	`smoothness_mean`	`concavity_mean`	`symmetry_mean`
17.99	10.38	0.1184	0.3001	0.2419
20.57	17.77	0.08474	0.0869	0.1812
19.69	21.25	0.1096	0.1974	0.2069
11.42	20.38	0.1425	0.2414	0.2597
20.29	14.34	0.1003	0.198	0.1809
12.45	15.7	0.1278	0.1578	0.2087
18.25	19.98	0.09463	0.1127	0.1794
13.71	20.83	0.1189	0.09366	0.2196

Column meanings and units

Computed image features, with mean, standard-error and worst summaries as defined by the source. Diagnosis is a category.

This is a historical teaching classification dataset, not a clinical decision system. Measurements are assumed available before prediction.

Input schema
Column	Stored type
`radius_mean`	float64
`texture_mean`	float64
`smoothness_mean`	float64
`concavity_mean`	float64
`symmetry_mean`	float64

The file is available in the Python working folder. Load it yourself.

Your task

Describe a Ward hierarchy of a reproducible sample of at most 500 Breast measurement rows, with diagnosis excluded from fitting.
Draw the scaled sampled Ward hierarchy as a visible, labelled dendrogram truncated to the last 20 groups.

samplelinkage_matrixlabelsprofilescut_evidence

Outputs you create

sample: Original-unit dataframe with the reproducible sampled row indices.
linkage_matrix: Ward linkage matrix of scaled_sample.
labels: One cluster label per fitted row; equivalent consistent label names are accepted.
profiles: Original-unit means grouped by aligned cluster labels.
cut_evidence: Silhouette evidence for hierarchy cuts k=2–8.

Deliverables

These conditions describe a complete workflow. Use the declared Python variables beside the editor; Check inspects evidence from your Run.

Python name key

Teal outlines name table columns. Violet left rules name Python objects. Dotted rules mark syntax, functions and parameters. Use each name exactly as shown.

Population & sample
1. Scale the supplied five-measurement population
2. Keep the identities of the seed-42 sample of at most 500 rows.
Hierarchy & cuts
1. Build Ward linkage on the scaled sample
2. Draw the labelled hierarchy truncated to 20 last groups and compare cuts from k=2–8.
Aligned profiles
1. Attach the chosen cut labels to exactly the sampled rows and report original-unit group means.
Interpretation
1. Explain a defensible cut, sample scope and distance assumptions; group identifiers are names, not classes.

The playground workflow

Use the ML → Workflow discovery route: define the observations and measurements, explore and prepare them, fit the grouping or representation, inspect its evidence and explain what the result can support. There is no prediction target or reserved final test in this workflow.

Choose what to discoverHow do Breast Cancer Wisconsin (Diagnostic) · All features continuous rows join into a hierarchy when compared on radius_mean, texture_mean, and 3 more selected inputs?
Explore the dataWhat ranges and overlaps among radius_mean, texture_mean, and 3 more selected inputs might shape the Breast Cancer Wisconsin (Diagnostic) · All features continuous hierarchy?
Prepare the dataHow should radius_mean, texture_mean, and 3 more selected inputs be scaled, and why does Breast Cancer Wisconsin (Diagnostic) · All features continuous use a reproducible sample?
Build the dendrogramWhere do Breast Cancer Wisconsin (Diagnostic) · All features continuous rows join at noticeably different heights before a cut is chosen?
Compare possible cutsWhich dendrogram cut gives a useful description of Breast Cancer Wisconsin (Diagnostic) · All features continuous?
Fit the hierarchyWhat does the selected starting cut produce for Breast Cancer Wisconsin (Diagnostic) · All features continuous?
Explain the groupsWhat do the sampled groups look like in the original units of radius_mean, texture_mean, and 3 more selected inputs for Breast Cancer Wisconsin (Diagnostic) · All features continuous?
Map the hierarchyDescribe two sampled groups in the original units of radius_mean, texture_mean, and 3 more selected inputs for Breast Cancer Wisconsin (Diagnostic) · All features continuous; why is this hierarchy useful for the selected scenario?

Data policy Use only the supplied measurements for fitting. Keep group labels, scores and profiles aligned with the rows they describe.

Help Hints & explained solution

Choose the amount of help you need. Opening one hint does not reveal the others.

Hint 1 — Think

Sample labels belong to sampled rows.

Hint 2 — Tools

linkage, dendrogram, cut_tree and index-aligned profiles.

Revisit: Interpret discovery without inventing truth

Hint 3 — Approach

Scale the population, preserve sample identity through the hierarchy, compare cuts and qualify the scope.

Explained solution

This complete script replaces the editor contents. It includes the input setup and the workflow.

import numpy as np
import pandas as pd
df=pd.read_csv('data/ml-learning/ML-X18.csv')
from sklearn.preprocessing import StandardScaler
from scipy.cluster.hierarchy import linkage, dendrogram, cut_tree
from sklearn.metrics import silhouette_score
import matplotlib.pyplot as plt
X = df[['radius_mean','texture_mean','smoothness_mean','concavity_mean','symmetry_mean']]
scaler = StandardScaler().fit(X)
sample = X.sample(min(500,len(X)), random_state=42)
scaled_sample = scaler.transform(sample)
linkage_matrix = linkage(scaled_sample, method='ward')
cut_evidence = []
for k in range(2,9):
    assignments = cut_tree(linkage_matrix, n_clusters=k).ravel()
    cut_evidence.append({'k':k, 'silhouette':silhouette_score(scaled_sample, assignments)})
labels = cut_tree(linkage_matrix, n_clusters=3).ravel()
profiles = sample.groupby(labels).mean()
fig, ax = plt.subplots(figsize=(7,4))
dendrogram(linkage_matrix, truncate_mode='lastp', p=20, ax=ax)
ax.set(title='Ward hierarchy of sampled measurements', xlabel='Sample groups', ylabel='Ward merge distance')
print(pd.DataFrame(cut_evidence))

View input setup in this script

These opening lines are included in the complete script above. They load the challenge input.

import numpy as np
import pandas as pd
df=pd.read_csv('data/ml-learning/ML-X18.csv')

Why these steps exist

The measurement-only population is scaled before the declared reproducible sample is selected. Original row identity is retained through Ward linkage, cuts and profiles. The 500-row cap bounds runtime, and conclusions describe the sampled rows rather than claiming assignments for all 569 observations.

Another valid approach

Equivalent ordinary Python is welcome. A defensible candidate or grouping need not match the solution’s choice; objective evidence must remain consistent.