Medical Metrics Considered Overrated

One of the presenters at Metricon 5.0 was comparing IT security to other fields in various aspects of metrics and monitoring. I mentioned I thought she was giving far too much green for good to the field of medicine. This provoked repeated back and forth later.

My point was that 150 years after the invention of epidemiology and 100 years after the discovery of bacterial transmission of disease, in medicine application of known preventive measures is so low that Atul Gawande of Harvard has gotten large (on the order of 30%) reductions in deaths from complications of surgery in many hospitals simply by getting them to use checklists for things like washing hands before surgery.

I have an elderly relative in a nursing home who can't take pills whole due to some damage to nerves in her neck. Again and again visitors sent by the family discover nursing home staff trying to give her pills whole without grinding them up. Why? They don't read instructions about her, and previous shifts don't remind later shifts. This kind of communication problem is epidemic not only in nursing homes but in hospitals. I found my father in a diabetic coma because nurses hadn't paid any attention to him being a diabetic and needing to eat frequently. Fortunately, a bit of honey brought him out of it. Even nurses readily acknowledge this problem, but it persists. I can rattle off many other examples.

To which someone responded, yes, but medicine has epidemiology, and Edward Tufte demonstrated in one of his books that that goes well beyond checklists in to actual analysis, as in a physician's discovery of a well in London being he source of cholera. I responded, yes, John Snow, in 1854: that was the first thing I said when I stood up to address this. But who now applies what he learned? One-shot longitudinal studies are not the same as ongoing monitoring with comparable metrics to show how well one group is doing compared to both the known science and to other groups.

Many people still didn't get it, and kept referring to checklists as rudimentary.

So I tried again. If John Snow were alive today, he wouldn't be prescribing statins for life to people with high blood pressure. He would be compiling data on who has high blood pressure and what they have been doing and eating before they got it. He would follow this evidence back to discover that one of the main contributors to high blood pressure, heart disease, and diabetes in the U.S. is high fructose corn syrup (HFCS). Then he would mount a political campaign to ban high fructose corn syrup, which would be the modern equivalent of his removal of the handle from the pump of the well that stopped the cholera.

To which someone replied, but there are political forces who would oppose that. And I said, yes, of course. Permit me to elaborate.

There were political forces in John Snow's time, too, and he dealt with them:

Dr Snow took a sample of water from the pump, and, on examining it under a microscope, found that it contained "white, flocculent particles." By 7 September, he was convinced that these were the source of infection, and he took his findings to the Board of Guardians of St James's Parish, in whose parish the pump fell.

Though they were reluctant to believe him, they agreed to remove the pump handle as an experiment. When they did so, the spread of cholera dramatically stopped. [actually the outbreak had already lessened for several days]

Snow also investigated several outliers, all of which turned out to involve people actually travelling to the Soho well to get water.

Still no one believed Snow. A report by the Board of Health a few months later dismissed his "suggestions" that "the real cause of whatever was peculiar in the case lay in the general use of one particular well, situate [sic] at Broad Street in the middle of the district, and having (it was imagined) its waters contaminated by the rice-water evacuations of cholera patients. After careful inquiry," the report concluded, "we see no reason to adopt this belief."

So what had caused the cholera outbreak? The Reverend Henry Whitehead, vicar of St Luke's church, Berwick Street, believed that it had been caused by divine intervention, and he undertook his own report on the epidemic in order to prove his point. However, his findings merely confirmed what Snow had claimed, a fact that he was honest enough to own up to. Furthermore, Whitehead helped Snow to isolate a single probable cause of the whole infection: just before the Soho epidemic had occurred, a child living at number 40 Broad Street had been taken ill with cholera symptoms, and its nappies had been steeped in water which was subsequently tipped into a leaking cesspool situated only three feet from the Broad Street well.

Whitehead's findings were published in The Builder a year later, along with a report on living conditions in Soho, undertaken by the magazine itself. They found that no improvements at all had been made during the intervening year. "Even in Broad-street it would appear that little has since been done... In St Anne's-Place, and St Anne's-Court, the open cesspools are still to be seen; in the court, so far as we could learn, no change has been made; so that here, in spite of the late numerous deaths, we have all the materials for a fresh epidemic... In some [houses] the water-butts were in deep cellars, close to the undrained cesspool... The overcrowding appears to increase..." The Builder went on to recommend "the immediate abandonment and clearing away of all cesspools -- not the disguise of them, but their complete removal."

Nothing much was done about it. Soho was to remain a dangerous place for some time to come.

John Snow didn't shy away from politics. He was successful in getting the local politicians to agree to his first experiment, which was successful in helping end that outbreak of cholera. He even drew his biggest opponent into doing research, which ended up confirming Snow's epidemiological diagnosis and extending it further to find the original probable source of infection of the well. But even that didn't suffice for motivating enough political will to fix the problem.

From which I draw two conclusions:

1. Even John Snow is over-rated. Sure, he found the problem, but he didn't get it fixed longterm.

2. Why not? Because that would require ongoing monitoring of likely sources of infection (which sort of happened) compared to actual incidents of disease (which does not appear to have happened), together with eliminating the known likely sources.

Eliminating likely known sources is what Dr. Gawande's checklist is about, 150 years later, which was my original point. And the ongoing monitoring and comparisons appear not to be happening, even yet.

As someone at Metricon said, who will watch the watchers? I responded, yes, that's it!

One-shot longitudinal studies can create great information. That's what John Snow did. That's what much of scientific experiment is about. But even when you repeat the experiment to confirm it, that's not the same as ongoing monitoring. And it's not the same as checklists to ensure application of what was learned in the experiment.

What is really needed is longitudinal experiments combined checklists, plus ongoing monitoring, plus new analysis derived from the monitoring data. That's at least four levels. All of them are needed. Modern medicine often only manages the first. And in the case of high fructose corn syrup (HFCS), until recently even the first was lacking, and most of the experiments that have happened until very recently have not come from the country with the biggest HFCS health problem, namely the U.S. A third of the entire U.S. population is obese, and another third is overweight, with concomittant epidemics of heart disease, diabetes, and high blood pressure. And the medical profession prescribes statins for life instead of getting to the root of the problem and fixing it.

Yes, I think the field of medicine gets rated too much green for good.

And if IT security wants to improve its own act, it also needs all four levels, not just the first or the second.

-jsq