My BSc thesis at COMSATS asked a plain question: from a public health questionnaire, how well can you predict diabetes risk? The dataset was BRFSS 2015 - 253,680 CDC health records, 22 features - and the first thing it taught me was that the obvious way to answer is the wrong one. On an 86/14 imbalance, accuracy is not the question. Sensitivity and ROC-AUC are, because the expensive mistake here is the miss.
The imbalance decides before the model does
BRFSS 2015 is 86/14 imbalanced: non-diabetic records outnumber diabetic ones six to one. That means a model that predicts "not diabetic" for everyone scores 86% accuracy while being clinically useless. Any accuracy figure on this dataset is meaningless until the balance is handled.
We ran three balancing strategies head-to-head - Random Over-Sampling, SMOTE, and ADASYN, with the resampling confined to the training folds so the held-out split stayed untouched:
# Illustrative: the head-to-head that decided the balancing strategy
for sampler in (RandomOverSampler(), SMOTE(), ADASYN()):
X_bal, y_bal = sampler.fit_resample(X_train, y_train)
score = evaluate(model, X_bal, y_bal, X_test, y_test)Plain Random Over-Sampling won, and the reason is worth remembering: SMOTE and ADASYN interpolate synthetic samples, which blurs categorical questionnaire features. There is no meaningful point halfway between "has high blood pressure" and "does not". Duplication preserved the feature distributions; interpolation smeared them.
Eleven models, one honest winner
The benchmark spanned linear, instance-based, tree, boosting, and neural families - eleven classifiers under the same 80/20 split. The tree ensembles led, then the instance-based models, with the linear baseline last.
That ordering was a measured finding, not an assumption. Running the full zoo is tedious, and it is also the only way the sentence "Random Forest was best" means anything.
Accuracy is not the metric that matters
For a screening tool, the expensive mistake is the miss. Telling a healthy person to get a check-up costs an appointment; telling a diabetic person they are fine costs years of untreated progression. So the model was tuned to catch positives rather than to look tidy on a single headline metric - accepting more false alarms in exchange for missing fewer real cases. For screening, that is the right side of the trade to be on.
The constraint that shaped everything: no lab tests
Every BRFSS feature is self-reportable - blood pressure history, BMI, activity, general health. That constraint became the product: a 19-question screen anyone can complete without a lab visit, served from a Flask API with a React front end, returning a risk classification and a future-risk probability. A model needing an HbA1c value would have been more precise and useless to the people the tool is for.
A prediction without a why is a black box
Clinical adoption research is blunt about this: when clinicians cannot trace a prediction to features they recognise, they reject the tool - even when it outperforms their judgement. The thesis handled this with correlation-driven risk-factor analysis: general health was the strongest correlate at -0.41, then high blood pressure at +0.38, high cholesterol and BMI at +0.29 each, with prevalence climbing sharply from age 50 onward. Those are drivers a clinician already reasons with, which is the point: the explanation has to be in their vocabulary, not the model's.
What I would do differently
Ship per-prediction attribution, not just population-level correlations - a risk score without "why this score for this person" is exactly the black box the thesis argued against. And before anything touches a real clinic: calibration and population-shift detection. 93% on one curated dataset does not transfer to a different hospital's intake unchecked, and Platt-scaled probabilities plus a shift detector are the difference between a paper result and a safe tool.