Several years ago, during an annual mammogram, my wife, who is in her 40s, was told a mass had been found in one of her breasts. Anxious and uncertain, she had a biopsy, and we braced for the worst.

My father-in-law, when in his 50s, went through a similarly harrowing experience when a prostate specific antigen (PSA) test given during a routine physical exam came out positive, and he underwent a prostate biopsy.

Fortunately, both my wife and father-in-law were found to be victims not of cancer but of false positives. It’s a common outcome — about three of every 10 mammograms and about seven of every 10 positive PSA testsresult in false alarms. Like my wife and father-in-law, many people may feel that the short-term pain of these tests is worth the reassurance that they are cancer-free. But lately the idea that screening tests are a surefire weapon against cancer has been undermined.

New guidelines, new fears

We have been programmed to believe that early detection through screening saves lives. Seventy-five percent of men over the age of 50 have had a prostate specific antigen (PSA) test; 67 percent of women over the age of 40 have had a mammogram in the past two years; and 55 percent of women get a Pap smear every year.

New guidelines and studies, however, have raised new questions and re-ignited debate over whether these tests may in fact do more harm than good. Earlier this month, the U.S. Preventive Services Task Force (USPSTF), a government-appointed scientific advisory board, recommended against routine PSA tests to screen for prostate cancer. The same panel also suggested that a Pap test every three years is as good as an annual test. Two newly published studies suggest reducing the number of mammograms in women under 50; one went so far as to conclude that “most women with screen-detected breast cancer have not had their life saved by screening. They are instead either diagnosed early (with no effect on their mortality) or overdiagnosed.”

How can this be? We have relied on these tests for decades to screen for cancer in otherwise healthy individuals. The truth is that PSA, mammogram and the Pap smear are poor screeners — and always have been. The levels of prostate specific antigen — the marker for cancer — can be elevated for a variety of reasons, like a benign prostate enlargement or a prostate infection. A mammogram can be read as suspicious due to a cyst, a benign fluid collection; a Pap smear is often reported as abnormal due to abnormal cells caused by a local infection.

The fallacies of these screening tests extend beyond the false positives or excessive mistaken alerts. These tests frequently miss the diagnosis of cancer or result in false negatives. The  PSA will be normal in two of every 10 patients with prostate cancer and the mammogram will be normal in two of every 10 patients with breast cancer. The Pap smear will miss cervical cancer in three of 10 patients.

The tests are also potentially harmful. They lead to subsequent testing such as a biopsy, which can cause complications such as infection or lead to treatment of a cancer that would not have otherwise caused any harm.

‘We need better tests’

It’s hard to argue with my wife’s plea: “We need better tests.” But until new screening methods are developed, we need to adjust our expectations of the ones we have. As Virginia Moyer, a professor at the Baylor College of Medicine and head of the USPSTS, puts it, we need to get away from the false notion that “if some prevention is good, then more is better.”

So what would “a better test,” as my wife puts it, look like? It should be accurate, inexpensive, easy to administer and noninvasive, like the HIV blood test, which is 99 percent accurate and costs only about $15. Researchers are pursuing a variety of improvements for cancer screening tests, including one for prostate cancer that tracks blood calcium levels following a PSA test and another that tests urine for a genetic marker. But such tests are years away from being available to the general public.

In the meantime, should we throw away the PSA test, mammography and the Pap smear? Absolutely not. But we should certainly rethink their administration — limiting these tests to those who may be at higher risk of cancers because of their age, family history or presenting symptoms.

The PSA test has more value in the urologist’s office than in the primary care doctor’s office. It is helpful in patients who have an abnormal rectal exam or a family history of prostate cancer. A routine mammogram saves lives both in women over 50 and younger women with a family history of breast cancer. And a Pap smear done regularly reduces the risk of cervical cancer incidence and mortality by at least 80 percent.

A case-by-case decision

We should also remember that guidelines are just that — recommendations. Doctors need to tailor them to each patient’s situation. When a patient is anxious and requests an inexpensive, noninvasive test, I am willing to agree to it, as long as the patient is informed about the limitations. The question is whether insurers and Medicare are willing to pay for such tests.

Ultimately, the new guidelines and studies add a wrinkle to the already complex medical decision-making process: Are we willing to undergo an imperfect screening test? Or should we forgo it?

My wife has elected to continue with mammograms in her 40s despite the new research. “But if I miss a year or two, I am less worried,” she said.

As for me, when I turn 50 in a few years, I plan to skip the PSA test.

Manoj Jain is an infectious disease physician and contributor to the Washington Post, where this article originally appeared.  He can be reached at his self-titled site, Dr. Manoj Jain.

Submit a guest post and be heard on social media’s leading physician voice.