Some of the most exciting applications of machine learning to medicine involve the kinds of data that cannot be analysed with traditional statistical models: medical imaging, waveforms, and videos. Researchers are training algorithms to take in these complex signals, and output a doctor's interpretation—eg, given a particular retinal fundus photograph, would an ophthalmologist identify diabetic retinopathy? Algorithms based on datasets that pair images or waveforms with “labels” assigned by a doctor have the potential to drive improvements in efficiency and diagnostic accuracy. However, the strength of this approach can also be its weakness: by matching the performance of doctors, algorithms will also incorporate their inherent limitations.