My BSc thesis at COMSATS asked a plain question: from a public health questionnaire, how well can you predict diabetes risk? The dataset was BRFSS 2015 - 253,680 CDC health records, 22 features - and the answer ended up being 93.15% accuracy. But the number I quote is the last thing that happened. The decisions that produced it came earlier, and none of them were about picking a fancier model.
The imbalance decides before the model does
BRFSS 2015 is 86/14 imbalanced: non-diabetic records outnumber diabetic ones six to one. That means a model that predicts "not diabetic" for everyone scores 86% accuracy while being clinically useless. Any accuracy figure on this dataset is meaningless until the balance is handled.
We ran three balancing strategies head-to-head - Random Over-Sampling, SMOTE, and ADASYN:
# Illustrative: the head-to-head that decided the balancing strategy
for sampler in (RandomOverSampler(), SMOTE(), ADASYN()):
X_bal, y_bal = sampler.fit_resample(X_train, y_train)
score = evaluate(model, X_bal, y_bal, X_test, y_test)Plain Random Over-Sampling won, and the reason is worth remembering: SMOTE and ADASYN interpolate synthetic samples, which blurs categorical questionnaire features. There is no meaningful point halfway between "has high blood pressure" and "does not". Duplication preserved the feature distributions; interpolation smeared them.
Eleven models, one honest winner
The benchmark spanned linear, instance-based, tree, boosting, and neural families - eleven classifiers under the same 80/20 split. Random Forest won at 93.15% accuracy, with Decision Tree close behind at 91.22% and KNN at 82.54%. Logistic regression managed 74.5%.
That 19-point gap between the tree ensembles and the linear baseline was a measured finding, not an assumption. Running the full zoo is tedious, and it is also the only way the sentence "Random Forest was best" means anything.
Accuracy is not the metric that matters
For a screening tool, the expensive mistake is the miss. Telling a healthy person to get a check-up costs an appointment; telling a diabetic person they are fine costs years of untreated progression. So the model was tuned to catch positives rather than to look tidy on a single headline metric - accepting more false alarms in exchange for missing fewer real cases. For screening, that is the right side of the trade to be on.
The constraint that shaped everything: no lab tests
Every BRFSS feature is self-reportable - blood pressure history, BMI, activity, general health. That constraint became the product: a 19-question screen anyone can complete without a lab visit, served from a Flask API with a React front end, returning a risk classification and a future-risk probability. A model needing an HbA1c value would have been more precise and useless to the people the tool is for.
A prediction without a why is a black box
Clinical adoption research is blunt about this: when clinicians cannot trace a prediction to features they recognise, they reject the tool - even when it outperforms their judgement. The thesis handled this with correlation-driven risk-factor analysis: general health was the strongest correlate at -0.41, then high blood pressure at +0.38, high cholesterol and BMI at +0.29 each, with prevalence peaking at 63.2% in the 70-74 age band. During the research assistantship that followed, we added SHAP attribution to the deployed model - and chose SHAP over LIME specifically because LIME's local approximations are unstable across runs on the same input. An explanation that changes between refreshes is worse than no explanation.
What I would do differently
Ship the attribution in v1 instead of retrofitting it - a risk score without "why this score" is exactly the black box the thesis argued against. And before anything touches a real clinic: calibration and population-shift detection. 93% on one curated dataset does not transfer to a different hospital's intake unchecked, and Platt-scaled probabilities plus a shift detector are the difference between a paper result and a safe tool.