Diagnose an overfitting classification tree

The question

Build and diagnose a Car acceptability tree. Compare depth choices and explain class errors and the limits of feature importance.

data/ml-learning/ML-X06.csv Complete workflow and evidence

Your inputs

Challenge input · ML-X06

1728 observations. One categorical car configuration from a decision-model evaluation dataset. Load the CSV file yourself.

Download challenge input CSV · Original source dataset · Source and original dictionary

Ready to load: comma-delimited UTF-8, header row, no saved index. No filtering or target construction is required.

ML-X06 · first 8 prepared rows
`buying`	`maintenance`	`doors`	`persons`	`luggage_boot`	`safety`	`acceptability`
vhigh	vhigh	2	2	small	low	unacc
vhigh	vhigh	2	2	small	med	unacc
vhigh	vhigh	2	2	small	high	unacc
vhigh	vhigh	2	2	med	low	unacc
vhigh	vhigh	2	2	med	med	unacc
vhigh	vhigh	2	2	med	high	unacc
vhigh	vhigh	2	2	big	low	unacc
vhigh	vhigh	2	2	big	med	unacc

Column meanings and units

Buying/maintenance cost, doors, capacity, luggage and safety are categorical ratings, not observed currency prices.

Acceptability is derived from the source decision model. Performance reproduces that rating system and does not establish real-world safety or consumer preference.

Ready to load: comma-delimited UTF-8, header row, no saved index. No filtering or target construction is required.

Input schema
Column	Stored type
`buying`	object
`maintenance`	object
`doors`	object
`persons`	object
`luggage_boot`	object
`safety`	object
`acceptability`	object

How we validate this model

Reserve 20% of the declared population for the final test using split seed 42. Keep class proportions with a stratified split.

Use the same five stratified training folds, shuffled with seed 42, for reference comparison, candidates and selection. Use out-of-fold training predictions for diagnosis. Select with macro F1 and compare with a most-frequent-class reference. Accuracy is supplementary. Open final-test evidence after selection and diagnosis.

The file is available in the Python working folder. Load it yourself.

Your task

Build and diagnose a Car acceptability tree.
Compare depth choices and explain class errors and the limits of feature importance.

cv_resultsreference_resultschosen_namematrixfinal_f1final_accuracy

Outputs you create

cv_results: Dataframe indexed by the requested model IDs, with initial_score, selected_score and settings columns.
Scores are observed mean training-fold scores; settings describes the chosen recipe.
reference_results: cross_validate result for the dummy reference; test_score contains five scores.
chosen_name: Model ID nominated from training evidence.
matrix: Confusion matrix of training-only diagnostic predictions.
final_f1: Macro F1 on final-test class predictions.
final_accuracy: Final-test accuracy, supplementary to macro F1.

Deliverables

These conditions describe a complete workflow. Use the declared Python variables beside the editor; Check inspects evidence from your Run.

Python name key

Teal outlines name table columns. Violet left rules name Python objects. Dotted rules mark syntax, functions and parameters. Use each name exactly as shown.

Words in this task

DataFrame: a table with rows and columns
numeric: a number, rather than text
dummy: a simple reference model used for comparison
folds: training-data groups used to compare models on the same held-out rows
out-of-fold: predictions made for training rows by models that did not fit those rows
aligned: matched by row or category identity, rather than guessed position
residuals: observed values minus predicted values

Data boundary
1. Load the supplied challenge CSV, define the declared X/y and keep aligned row identities through the split.
2. Keep learned preparation inside each training fit.
Validation & selection
1. Compare the requested initial model families with a dummy reference on matching training folds.
2. Evaluate the declared settings where applicable and choose from training evidence.
Diagnosis
1. Use out-of-fold training predictions to inspect residuals or class errors before opening the final test.
Final evidence
1. Fit the nominated model on training rows and evaluate the reserved rows once.
2. Report saved predictions, the requested metrics and labelled evidence
3. Explain limitations without using the final test for further selection.

The playground workflow

Use the same data boundary as ML → Workflow: frame, split, explore, prepare, validate against a reference, diagnose, then make the final evaluation. Each step answers one question.

Choose what to predictWhat changes when every predictor is categorical? Use 6 selected inputs to estimate car acceptability.
Split data and save the test setWhich rows are available for learning, and which are being saved for the final check?
Explore training dataDoes buying show a training pattern related to car acceptability?
Define the preparation recipeWhat preparation does this particular model need?
Build the model pipelineWhich if/then splits form useful class decisions?
Validate the initial modelDo validation results stay reasonably similar across folds?
Compare with a simple referenceDo the selected features beat a simple guess on the same training folds?
Tune the modelHow deep should the tree grow?
Inspect validation errorsWhich classes are most often confused in training-only validation?
Final testHow does the selected model's saved final car acceptability result compare with the initial-model CV and any selected-setting CV for Car Evaluation · All features categorical? Use the training-only diagnostic or optional final confusion view to identify a class pattern, then name one limitation.

Data policy Keep target-derived inputs out of X. Use training-only selection for prediction.

Help Hints & explained solution

Choose the amount of help you need. Opening one hint does not reveal the others.

Hint 1 — Think

Inspect validation error as complexity changes.

Hint 2 — Tools

DecisionTreeClassifier, OneHotEncoder, GridSearchCV and confusion_matrix.

Revisit: Macro F1 and imbalance · Classification trees · Finish once, then report

Hint 3 — Approach

Evaluate depth choices within training folds before selecting a final tree.

Explained solution

This complete script replaces the editor contents. It includes the input setup and the workflow.

import numpy as np
import pandas as pd
df=pd.read_csv('data/ml-learning/ML-X06.csv')
from sklearn.base import clone
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_validate, cross_val_predict, GridSearchCV
from sklearn.dummy import DummyClassifier
from sklearn.metrics import f1_score, accuracy_score, confusion_matrix

import matplotlib.pyplot as plt
import seaborn as sns
X=df[['buying', 'maintenance', 'doors', 'persons', 'luggage_boot', 'safety']]
y=df['acceptability']
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.2,random_state=42,stratify=y)
folds=StratifiedKFold(n_splits=5,shuffle=True,random_state=42)
candidates={}
grids={}
# Prepare inputs and construct the candidate pipeline.
from sklearn.preprocessing import OneHotEncoder
category_encoder = OneHotEncoder(handle_unknown='ignore', sparse_output=False, drop=None)
preprocessor = category_encoder
from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(random_state=42)
pipeline = Pipeline([('prepare', preprocessor), ('model', model)])
model = pipeline
candidates['classification_tree']=model
grids['classification_tree']={'model__max_depth': [3, 5, None]}
# Validate initial candidates on matching training folds.
initial_results={name:cross_validate(candidate,X_train,y_train,cv=folds,scoring='f1_macro') for name,candidate in candidates.items()}
# Compare with a simple reference before tuning.
reference_results=cross_validate(DummyClassifier(strategy='most_frequent'),X_train,y_train,cv=folds,scoring='f1_macro')
rows=[]
selected={}
for name,candidate in candidates.items():
    initial=initial_results[name]
    grid=grids[name]
    if grid:
        search=GridSearchCV(candidate,grid,cv=folds,scoring='f1_macro',n_jobs=1)
        search.fit(X_train,y_train)
        selected[name]=search.best_estimator_
        validation_score=float(search.best_score_)
        settings=search.best_params_
    else:
        selected[name]=candidate
        validation_score=float(np.mean(initial['test_score']))
        settings='Keep defaults: no production search'
    rows.append({'model':name,'initial_score':float(np.mean(initial['test_score'])),'selected_score':validation_score,'settings':str(settings)})
cv_results=pd.DataFrame(rows).set_index('model')
# This solution nominates by mean training-fold score. Other evidence-based choices can be defensible.
chosen_name=cv_results.selected_score.idxmax()
chosen=selected[chosen_name]
oof_predictions=cross_val_predict(chosen,X_train,y_train,cv=folds)
diagnostic_y=y_train
class_labels=sorted(y_train.unique())
matrix=confusion_matrix(diagnostic_y,oof_predictions,labels=class_labels)
fig,ax=plt.subplots(figsize=(6,4))
sns.heatmap(matrix,annot=True,fmt='d',cmap='Blues',ax=ax,xticklabels=class_labels,yticklabels=class_labels)
ax.set(xlabel='Predicted class',ylabel='Actual class',title='Training-only confusion matrix')
fig.tight_layout()
fig.savefig('diagnostic.png',dpi=150,bbox_inches='tight')
final_model=clone(chosen).fit(X_train,y_train)
final_predictions=final_model.predict(X_test)
final_f1=f1_score(y_test,final_predictions,average='macro')
final_accuracy=accuracy_score(y_test,final_predictions)
print('Final macro F1:',final_f1,'; accuracy:',final_accuracy)

View input setup in this script

These opening lines are included in the complete script above. They load the challenge input.

import numpy as np
import pandas as pd
df=pd.read_csv('data/ml-learning/ML-X06.csv')

Why these steps exist

Encoding preserves the Car category schema while training-fold depth comparison controls tree complexity. OOF confusion reveals class errors before final evaluation. Importance describes this fitted tree’s split reductions and must not be presented as a causal ranking.

Another valid approach

Equivalent ordinary Python is welcome. A defensible candidate or grouping need not match the solution’s choice; objective evidence must remain consistent.