Skip to main content
All case studies

Machine Learning · Health Technology · Cloud Deployment

Medical Condition Classification Platform

A Random Forest classifier over institutional healthcare data, served through a Flask API on AWS, evaluated honestly as a top-N ranking problem rather than a single-answer one.

Academic · Deployed on AWS
Role
Software Engineer — modelling, API and cloud deployment
Period
2025

Stages covered

  • Architecture
  • Build
  • Deploy

At a glance

2025

Top-1 accuracy
29.26%
Top-3 accuracy
53.79%
Top-10 accuracy
88.76%

Modelling

  • Python
  • scikit-learn
  • Random Forest

Service

  • Flask
  • MySQL

Infrastructure

  • Amazon EC2
  • Amazon RDS
  • Amazon S3
  • Application Load Balancer
  • Route 53
  • AWS Certificate Manager

Overview

A machine-learning platform that suggests likely ICD-10-related condition categories from institutional healthcare records. The model was trained on real institutional data with substantial class imbalance and deployed on AWS behind a load balancer with HTTPS. It is presented here as a decision-support ranking tool, which is what its measured accuracy actually supports.

Top-1 accuracy
29.26%
Top-3 accuracy
53.79%
Top-10 accuracy
88.76%

Problem

Institutional healthcare records carry a large number of condition categories, and their distribution is severely uneven: a handful of categories dominate while a long tail appears rarely. A classifier trained on that data will look competent on aggregate metrics while being close to useless on the categories that most need help.

The framing question was therefore not 'can a model predict the condition' but 'is there a form of prediction that is actually usable by someone doing this work'.

Constraints

  • Real institutional data, with the privacy handling and access limits that implies
  • A large number of target classes relative to the number of examples per class
  • Severe class imbalance that no amount of model tuning removes
  • A clinical context, where a confidently wrong single answer is worse than no answer

Role and contribution

Software Engineer — modelling, API and cloud deployment

  • Prepared and analysed the institutional dataset, including the class distribution
  • Trained and evaluated the Random Forest classifier
  • Designed the top-N evaluation used to judge whether the model was useful
  • Built the Flask inference API and its MySQL persistence
  • Provisioned the AWS deployment: compute, managed database, object storage, load balancing, DNS and TLS

System design

Architecture

Deployment topology

Client requests arrive over HTTPS at an Application Load Balancer, with DNS handled by Route 53 and the certificate issued by AWS Certificate Manager. The load balancer forwards to a Flask application running on Amazon EC2, which loads the trained model artifact from Amazon S3 and reads and writes structured data in Amazon RDS running MySQL.

Modelling
  • Python
  • scikit-learn
  • Random Forest
Service
  • Flask
  • MySQL
Infrastructure
  • Amazon EC2
  • Amazon RDS
  • Amazon S3
  • Application Load Balancer
  • Route 53
  • AWS Certificate Manager

Reading the results honestly

Top-1 accuracy is 29.26%. On its own that is a weak classifier, and presenting it as anything else would be dishonest.

The useful figures are the ranked ones: the correct category is in the model's top three 53.79% of the time and in its top ten 88.76% of the time. The gap between 29% and 89% is the whole finding. It says the model has learned a great deal about which categories are plausible for a given record, and comparatively little about how to choose between the few that remain.

Against a large, heavily imbalanced label space, that is the expected shape. A single prediction forces the model to spend its confidence on a decision the data does not support; a short ranked list matches what it actually knows.

Why top-N was the right operational target

A tool that proposes one category and is wrong seven times out of ten will be ignored after a week. A tool that narrows hundreds of categories to ten, and is right about nine times in ten that the answer is somewhere in that list, removes most of the search while leaving the judgement where it belongs.

That reframing changed what the system had to be. It is not an automated classifier; it is a shortlist generator whose job is to be reliably inclusive rather than occasionally exactly right.

Limitations and clinical risk

This is an academic project. It has not been clinically validated, it is not a medical device, and it must not be used to make or support a diagnosis.

Two limitations matter most. Class imbalance means the rare categories — often the ones where assistance is most valuable — are the ones the model has seen least and predicts worst; aggregate accuracy hides this entirely. And a ranked shortlist creates its own risk: a plausible-looking list can anchor a reader towards its contents and away from a correct answer that is not on it.

  • Not clinically validated; not a medical device
  • Rare categories are the weakest and are hidden by aggregate metrics
  • A shortlist can anchor judgement toward the options it contains
  • Trained on one institution's data, with no evidence of transfer to another

Key decisions and trade-offs

Decision

Evaluate as a top-N ranking task rather than single-label classification

Context
Top-1 accuracy of 29.26% against a large, imbalanced label space reads as a failed model under the default framing.
Alternatives considered
  • Report top-1 accuracy only and treat the project as unsuccessful
  • Reduce the problem by merging categories until top-1 accuracy looks acceptable
Chosen approach
Report top-1, top-3 and top-10 together and design the product around the ranked output.
Reason
The 29% → 89% spread shows the model is informative even when it cannot commit to one answer. Merging categories would have improved the metric by discarding exactly the distinctions the tool exists to help with.
Trade-off
The system cannot be automated — it always requires a human to choose from the shortlist. That is the correct trade-off in a clinical context.
Outcome
The correct category appears in the top ten 88.76% of the time.

Decision

Analyse class imbalance as a first-class result

Context
The dataset's category distribution is severely uneven.
Alternative considered
  • Resample or reweight until aggregate metrics improve, and report only the improved number
Chosen approach
Report the distribution alongside the accuracy figures.
Reason
Aggregate accuracy on an imbalanced dataset mostly measures the majority classes. Without the distribution, the headline number invites a reader to overestimate the model everywhere it matters most.
Trade-off
A less flattering headline result, and more explanation required of every reader.

Result

  • Measured top-1 29.26%, top-3 53.79%, top-10 88.76%
  • Deployed and reachable over HTTPS on AWS behind an Application Load Balancer
  • Reframed from a single-answer classifier into a shortlist tool that matches what the data supports

Lessons learned

  • The evaluation metric is a product decision. Choosing top-N over top-1 changed what the system was for, not just how it scored.
  • Aggregate accuracy on imbalanced data is close to meaningless without the distribution beside it.
  • Deployment surfaced problems training never did — model artifact loading, cold start and certificate handling are not modelling concerns but they decide whether anyone can use the model.
  • In a clinical context, a system that declines to be certain is more trustworthy than one that is confidently wrong.

Next improvements

  • Per-class metrics, especially recall on the rare categories the aggregate figures obscure
  • Calibrated probabilities so the shortlist can express how confident it is, not only what it ranks
  • Comparison against gradient-boosted and linear baselines to establish whether Random Forest is the right family here
  • Validation on data from a second institution before any claim of generality