Here’s some good news for a change about cancer: Cancer mortality — the rate of death from cancer — has fallen substantially over the last four decades.

There is also, however, some not-so-good news: Cancer incidence — the rate of cancer diagnoses — has been rising. This doesn’t reflect increasing dangers in our environment, but a danger in our medical system.

In the New England Journal of Medicine, two colleagues and I examined the last four decades of cancer statistics in the United States. The decline in cancer mortality is a good sign. Everyone, including the National Cancer Institute, agrees that a declining cancer death rate is the best measure of progress against cancer.

The biggest single component of this decline? Lung cancer and the recognition of a uniquely powerful causal factor: cigarette smoking.

While many studies helped produce the knowledge that smoking causes lung cancer, arguably the most persuasive one was published in 1956. It followed nearly 35,000 male physicians, the majority of whom smoked. Those who smoked more than a pack a day were 20 times more likely to have died from lung cancer than their non-smoking colleagues.

We aren’t going to find another cancer-causing factor that strong, and that common, ever again.

In 1964, the U.S. surgeon general announced that there was no doubt that cigarette smoking caused lung cancer. It took time for rates of smoking to decline, and even more time to see the effect on lung cancer death. But we are seeing it now. Big time.

The good news doesn’t stop there. There have also been real improvements in cancer treatment. For a few rare, blood-borne cancers, the improvement has been phenomenal. There has also been substantial improvement in the treatment of two more common cancers — breast and prostate cancer — reflecting the recognition that both are typically hormonally responsive diseases.

Add to that the fact that some cancers seem to be slowly disappearing, although we are not entirely sure why. Stomach, cervical, and colorectal cancers are all less commonly diagnosed now than they were in 1975 and are a less common cause of death. Fewer people are treated, fewer die: That’s really good news.

The decline in mortality in Hodgkin’s lymphoma and chronic myeloid leukemia since 1975 reflects improvements in treatment with stable occurrence of cancer not influenced by screening.

Back to the not-so-good news. Even though overall mortality from cancer is falling, the overall incidence is rising. The declines in lung, stomach, cervical, and colorectal cancers have been more than offset by a rise in breast, prostate, thyroid, kidney, and melanoma skin cancers.

Why are more people being told they have these cancers? Blame that on overdiagnosis — the diagnosis of cancers not destined to cause symptoms or death. Overdiagnosis is not a purposeful act; it is an unfortunate side effect of our irrational exuberance for early detection.

This exuberance began with the observation that patients in whom cancer was detected early lived years longer than those in whom cancer was detected later. The simple inference was that they had benefited from early detection. Many doctors recognized the logical fallacy here, one that has nothing to do with overdiagnosis: If we start the clock earlier in the course of disease, patients will always appear to live longer — even if their time of death is unchanged.

But the horse was out of the barn. Screening efforts were initiated with the express purpose of finding small cancers that weren’t causing any symptoms. Overdiagnosis is easy to see in populations — and can be substantial — although doctors don’t know which patients are overdiagnosed.

The initiation of widespread mammography screening during the 1980s led to a 50% increase in the incidence of breast cancer. It never came back down. The advent of PSA testing a few years later doubled the incidence of prostate cancer. And changing PSA testing practices ver the years have produced a roller coaster incidence curve — the likes of which never seen before in cancer epidemiology.

The increase in diagnoses of breast cancer and the fluctuations in prostate cancer, with minimal changes in mortality, reflect the influence of screening.

“The earlier, the better” dictum permeated medicine. Physicians felt compelled to evaluate small spots on the kidney and thyroid that they had stumbled upon while imaging for some other purpose simply because these abnormalities might be cancer. Skin moles became a source of concern and an opportunity for biopsy. The incidence of kidney cancer doubled, thyroid cancer tripled, and melanoma went up sixfold — while their death rates remained stable.

These are not epidemics of disease. They are epidemics of diagnosis.

Paradoxically, overdiagnosis helps fuel the exuberance for early detection. Survival rates skyrocket either because the clock started ticking earlier or the disease was not destined to cause death. More people appear to be cured. And more survivors — as well as more politicians — advocate for more early detection.

Overdiagnosis isn’t the only danger caused by early detection. False alarms are another: test results that initially indicate the possibility of cancer but that are ultimately shown to be wrong. The earlier doctors try to detect things, the more likely we will set off false alarms. It can take us a while to sort out these early red flags, during which more people are subjected to cascades of tests and procedures to prove they don’t have cancer.

And make no mistake: having to prove you are well can take a lot out of you physically, emotionally, and financially.

Not all the exuberance has been either irrational or unintended. Early detection is great for the business of medicine. Regular testing of broad swaths of healthy people has been a boon for the medical testing industry. It’s been equally great for hospitals and health systems, bringing them new “patients” for further investigation and treatment. Plus, it serves as evidence of their paying attention to what’s known as population health — despite the lack of evidence that early cancer detection substantially improves a population’s health or longevity.

This is nothing like getting people to stop smoking. Or helping them eat real food, move regularly, and find purpose. Nor is it addressing the root causes of the rising death rates in younger white adults (hint: it’s not cancer).

If society tries to procure improved population health from the medical-industrial complex, we won’t buy more health, but simply more testing.

Early detection is always a trade-off between benefits and harms. It is influenced by a variety of factors: the biology of the disease, who is screened, how are they tested, and what happens following abnormal tests. The trade-off is most favorable in highly selected settings: individuals at genuinely high risk for cancer (think cigarette smokers) served by organized screening programs that are very attentive to minimizing overdiagnosis and false alarms.

The problem is in the leap of faith: Early detection is increasingly seen as the default solution for all cancer — and, more broadly, disease in general.

Ask your doctor what she thinks. She might point out the topic is too difficult to explain in the time allotted, and it’s much easier just to order the tests. That if she could be candid, she’d say cancer screening is, at best, a reasonable choice — but never a public health imperative. And that she’d like to talk more, but she must meet her system’s performance metric for ordering mammograms.

H. Gilbert Welch is an internal medicine physician and author of Less Medicine, More Health — 7 Assumptions that Drive Too Much Medical Care. This article originally appeared in STAT News.

Image credit: Shutterstock.com