Does a neural classifier justify its complexity?

The question

Compare a small neural classifier with logistic regression on the five-feature Breast scenario. Consider predictive evidence, convergence and cost.

data/ml-learning/ML-X16.csv Complete workflow and evidence

Your inputs

Challenge input · ML-X16

569 observations. One tumour sample represented by cell-nucleus image measurements. Load the CSV file yourself.

Download challenge input CSV · Original source dataset · Source and original dictionary

Ready to load: comma-delimited UTF-8, header row, no saved index. No filtering or target construction is required.

ML-X16 · first 8 prepared rows
`radius_mean`	`texture_mean`	`smoothness_mean`	`concavity_mean`	`symmetry_mean`	`diagnosis`
17.99	10.38	0.1184	0.3001	0.2419	M
20.57	17.77	0.08474	0.0869	0.1812	M
19.69	21.25	0.1096	0.1974	0.2069	M
11.42	20.38	0.1425	0.2414	0.2597	M
20.29	14.34	0.1003	0.198	0.1809	M
12.45	15.7	0.1278	0.1578	0.2087	M
18.25	19.98	0.09463	0.1127	0.1794	M
13.71	20.83	0.1189	0.09366	0.2196	M

Column meanings and units

Computed image features, with mean, standard-error and worst summaries as defined by the source. Diagnosis is a category.

This is a historical teaching classification dataset, not a clinical decision system. Measurements are assumed available before prediction.

Ready to load: comma-delimited UTF-8, header row, no saved index. No filtering or target construction is required.

Input schema
Column	Stored type
`radius_mean`	float64
`texture_mean`	float64
`smoothness_mean`	float64
`concavity_mean`	float64
`symmetry_mean`	float64
`diagnosis`	object

How we validate this model

Reserve 20% of the declared population for the final test using split seed 42. Keep class proportions with a stratified split.

Use the same five stratified training folds, shuffled with seed 42, for reference comparison, candidates and selection. Use out-of-fold training predictions for diagnosis. Select with macro F1 and compare with a most-frequent-class reference. Accuracy is supplementary. Open final-test evidence after selection and diagnosis.

The file is available in the Python working folder. Load it yourself.

Your task

Compare a small neural classifier with logistic regression on the five-feature Breast scenario.
Consider predictive evidence, convergence and cost.

cv_resultsreference_resultschosen_nameloss_curvesmatrixfinal_f1final_accuracy

Outputs you create

cv_results: Dataframe indexed by the requested model IDs, with initial_score, selected_score and settings columns.
Scores are observed mean training-fold scores; settings describes the chosen recipe.
reference_results: cross_validate result for the dummy reference; test_score contains five scores.
chosen_name: Model ID nominated from training evidence.
loss_curves: Dictionary of available neural training-loss histories by model ID.
matrix: Confusion matrix of training-only diagnostic predictions.
final_f1: Macro F1 on final-test class predictions.
final_accuracy: Final-test accuracy, supplementary to macro F1.

Deliverables

These conditions describe a complete workflow. Use the declared Python variables beside the editor; Check inspects evidence from your Run.

Python name key

Teal outlines name table columns. Violet left rules name Python objects. Dotted rules mark syntax, functions and parameters. Use each name exactly as shown.

Words in this task

DataFrame: a table with rows and columns
numeric: a number, rather than text
dummy: a simple reference model used for comparison
folds: training-data groups used to compare models on the same held-out rows
out-of-fold: predictions made for training rows by models that did not fit those rows
aligned: matched by row or category identity, rather than guessed position
residuals: observed values minus predicted values

Data boundary
1. Load the supplied challenge CSV, define the declared X/y and keep aligned row identities through the split.
2. Keep learned preparation inside each training fit.
Validation & selection
1. Compare the requested initial model families with a dummy reference on matching training folds.
2. Evaluate the declared settings where applicable and choose from training evidence.
Diagnosis
1. Use out-of-fold training predictions to inspect residuals or class errors before opening the final test.
2. Inspect neural convergence and loss history as well as prediction error.
Final evidence
1. Fit the nominated model on training rows and evaluate the reserved rows once.
2. Report saved predictions, the requested metrics and labelled evidence
3. Explain limitations without using the final test for further selection.

The playground workflow

Use the same data boundary as ML → Workflow: frame, split, explore, prepare, validate against a reference, diagnose, then make the final evaluation. Each step answers one question.

Choose what to predictPredict diagnosis class from 5 selected inputs; each row is one tumour sample.
Split data and save the test setWhich rows are available for learning, and which are being saved for the final check?
Explore training dataDoes radius_mean show a training pattern related to diagnosis class?
Define the preparation recipeWhat preparation does this particular model need?
Build the model pipelineCan one small hidden layer learn nonlinear class patterns?
Validate the initial modelDo validation results stay reasonably similar across folds?
Compare with a simple referenceDo the selected features beat a simple guess on the same training folds?
Tune the modelDoes a different small hidden-layer size help?
Inspect validation errorsWhich classes are most often confused in training-only validation?
Final testHow does the selected model's saved final diagnosis class result compare with the initial-model CV and any selected-setting CV for Breast Cancer Wisconsin (Diagnostic) · All features continuous? Use the training-only diagnostic or optional final confusion view to identify a class pattern, then name one limitation.

Data policy Keep target-derived inputs out of X. Use training-only selection for prediction.

Help Hints & explained solution

Choose the amount of help you need. Opening one hint does not reveal the others.

Hint 1 — Think

Complexity needs evidence.

Hint 2 — Tools

MLPClassifier, LogisticRegression and matching stratified folds.

Revisit: Neural classification checkpoint · Tune logistic regularisation · Read cross-validation evidence

Hint 3 — Approach

Compare both searches and training diagnostics before choosing; either model may be defensible.

Explained solution

This complete script replaces the editor contents. It includes the input setup and the workflow.

import numpy as np
import pandas as pd
df=pd.read_csv('data/ml-learning/ML-X16.csv')
from sklearn.base import clone
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_validate, cross_val_predict, GridSearchCV
from sklearn.dummy import DummyClassifier
from sklearn.metrics import f1_score, accuracy_score, confusion_matrix

import matplotlib.pyplot as plt
import seaborn as sns
X=df[['radius_mean', 'texture_mean', 'smoothness_mean', 'concavity_mean', 'symmetry_mean']]
y=df['diagnosis']
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.2,random_state=42,stratify=y)
folds=StratifiedKFold(n_splits=5,shuffle=True,random_state=42)
candidates={}
grids={}
# Prepare inputs and construct the candidate pipeline.
from sklearn.preprocessing import StandardScaler
preprocessor = StandardScaler()
from sklearn.pipeline import Pipeline
from sklearn.neural_network import MLPClassifier
model = MLPClassifier(hidden_layer_sizes=(24,), max_iter=500, early_stopping=True, random_state=42)
pipeline = Pipeline([('prepare', preprocessor), ('model', model)])
model = pipeline
candidates['mlp_cls']=model
grids['mlp_cls']={'model__hidden_layer_sizes': [(16,), (24,)]}
# Prepare inputs and construct the candidate pipeline.
from sklearn.preprocessing import StandardScaler
preprocessor = StandardScaler()
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(max_iter=2000, random_state=42)
pipeline = Pipeline([('prepare', preprocessor), ('model', model)])
model = pipeline
candidates['logistic']=model
grids['logistic']={'model__C': [0.1, 1.0, 10.0]}
# Validate initial candidates on matching training folds.
initial_results={name:cross_validate(candidate,X_train,y_train,cv=folds,scoring='f1_macro') for name,candidate in candidates.items()}
# Compare with a simple reference before tuning.
reference_results=cross_validate(DummyClassifier(strategy='most_frequent'),X_train,y_train,cv=folds,scoring='f1_macro')
rows=[]
selected={}
for name,candidate in candidates.items():
    initial=initial_results[name]
    grid=grids[name]
    if grid:
        search=GridSearchCV(candidate,grid,cv=folds,scoring='f1_macro',n_jobs=1)
        search.fit(X_train,y_train)
        selected[name]=search.best_estimator_
        validation_score=float(search.best_score_)
        settings=search.best_params_
    else:
        selected[name]=candidate
        validation_score=float(np.mean(initial['test_score']))
        settings='Keep defaults: no production search'
    rows.append({'model':name,'initial_score':float(np.mean(initial['test_score'])),'selected_score':validation_score,'settings':str(settings)})
cv_results=pd.DataFrame(rows).set_index('model')
# This solution nominates by mean training-fold score. Other evidence-based choices can be defensible.
chosen_name=cv_results.selected_score.idxmax()
chosen=selected[chosen_name]
oof_predictions=cross_val_predict(chosen,X_train,y_train,cv=folds)
diagnostic_y=y_train
class_labels=sorted(y_train.unique())
matrix=confusion_matrix(diagnostic_y,oof_predictions,labels=class_labels)
fig,ax=plt.subplots(figsize=(6,4))
sns.heatmap(matrix,annot=True,fmt='d',cmap='Blues',ax=ax,xticklabels=class_labels,yticklabels=class_labels)
ax.set(xlabel='Predicted class',ylabel='Actual class',title='Training-only confusion matrix')
fig.tight_layout()
fig.savefig('diagnostic.png',dpi=150,bbox_inches='tight')
final_model=clone(chosen).fit(X_train,y_train)
final_predictions=final_model.predict(X_test)
final_f1=f1_score(y_test,final_predictions,average='macro')
final_accuracy=accuracy_score(y_test,final_predictions)
print('Final macro F1:',final_f1,'; accuracy:',final_accuracy)
loss_curves={}
for name,candidate in selected.items():
    fitted=candidate.named_steps['model']
    if hasattr(fitted,'regressor_'):
        fitted=fitted.regressor_
    if hasattr(fitted,'loss_curve_'):
        loss_curves[name]=list(fitted.loss_curve_)

View input setup in this script

These opening lines are included in the complete script above. They load the challenge input.

import numpy as np
import pandas as pd
df=pd.read_csv('data/ml-learning/ML-X16.csv')

Why these steps exist

Both neural and logistic candidates use the same five measurements and stratified folds. Candidate searches, OOF class errors and neural loss histories support a nomination that can consider runtime and stability. Final evidence evaluates that decision without reopening it.

Another valid approach

Equivalent ordinary Python is welcome. A defensible candidate or grouping need not match the solution’s choice; objective evidence must remain consistent.