Status
Published
Role
Co-author
Posted
Revised

De-identifying clinical notes across institutions with deep learning

How well deep learning de-identification models trained on one institution's notes transfer to another's, and what it takes to adapt them, using the 2014 i2b2/UTHealth corpus and a new corpus of University of Florida Health notes.

Model architecture diagram. Character and word embeddings feed a bidirectional LSTM, linguistic and knowledge-based features are added, and a conditional random field layer outputs protected health information tags.

Published figure

Model architecture

Figure 1 of Yang et al., BMC Medical Informatics and Decision Making 2019, reused under CC BY 4.0.

The question

Clinical notes hold information that structured fields do not, but they also hold names, dates, phone numbers and other protected health information. Before notes can be used for research, that information has to be removed. Most de-identification systems had been trained and tested on notes from the same institution. This study asked what happens when a model trained elsewhere meets local notes, and how best to adapt it.

Data and models

The models were trained on the 2014 i2b2/UTHealth de-identification corpus and evaluated on a new corpus of 500 annotated clinical notes from University of Florida Health. The architecture, shown in the figure above, combines character and word embeddings with a bidirectional LSTM and a conditional random field output layer, and adds linguistic features (such as part of speech and word shape) and knowledge-based features drawn from local resources.

De-identification differs from most clinical NLP tasks. It cares about names, contact details and identifiers rather than diagnoses and drugs, and finding every instance matters more than labeling its exact type, because anything missed stays in the text.

What we found

  • Models trained only on the i2b2 corpus lost substantial accuracy on UF notes: strict and relaxed F1 scores fell from 0.9547 and 0.9646 to 0.8568 and 0.8958.
  • Word embeddings pre-trained on general English text worked better than embeddings from de-identified clinical text or biomedical literature, likely because of their broader vocabulary.
  • Linguistic and knowledge-based features improved cross-institution performance.
  • Adapting the models with UF notes and local resources produced the best results, with strict and relaxed F1 scores of 0.9288 and 0.9584. Fine-tuning a pre-trained model was more efficient than retraining on merged data.

Why it matters

A de-identification model is only as good as its fit to the notes it is applied to. Local annotation and adaptation are part of the cost of using clinical text responsibly.

First published on my earlier site, andeeli.github.io/posts/DeID/, and revised for this one.