Since the early years of the AIDS epidemic, gay and bisexual men seeking to lower their risk of contracting HIV during intercourse have relied on condoms for protection. So it may come as a surprise that scientists only recently conducted a truly rigorous analysis to estimate how well condoms work among men who have sex with men (MSM) in the United States. And it may come as a further surprise, given how the ultimate estimate of condoms’ success rate has since been accepted as a solid or near-solid fact by numerous press reports and among the chattering classes, that this figure is still clouded by statistical uncertainty—and probably will always remain something of an enigma.

At the 2013 Conference on Retroviruses and Opportunistic Infections (CROI), Dawn K. Smith, MD, MPH, an epidemiologist at the Centers for Disease Control and Prevention, presented results of research she and CDC colleagues conducted about the effectiveness of condoms among MSM. (“Effectiveness” is synonymous with condoms’ success rate, or how well they reduce HIV risk.) Those MSM who always use condoms, she reported, have a 70 percent lower risk of HIV than those who always bareback.

To condom devotees, this figure may seem alarmingly low, even farfetched. If nothing else, 70 percent is quite a drop in effectiveness when compared with studies showing that latex is a nearly impermeable barrier to HIV, and that condoms’ HIV protection rate is in the high 90 percent range—in an ideal laboratory setting, that is. Real-life safer sex is considerably more complex.

Smith’s risk reduction figure is also lower than condoms’ estimated 80 percent effectiveness among heterosexuals, a figure derived from a meta-analysis of studies of mixed-HIV status couples. There isn’t a statistically significant difference between the respective effectiveness estimates for MSM and heterosexuals. However, Smith and her colleagues believe it’s better to assume the lower figure for gay and bi men, since it actually derives from data about MSM rather than the traditional extrapolation of a figure based on heterosexual sex. (Smith doesn’t cite other reasons for preferring the lower estimate for MSM, but research has shown that condoms are more likely to slip or break during anal sex than vaginal sex. Also, HIV transmits much more easily through anal intercourse.)

Smith’s research also projects that MSM who sometimes use condoms only reduce their risk of contracting HIV by 8 percent. The slight amount of protection condoms apparently afford such individuals wasn’t statistically significant in her study. In other words, people who don’t wear a condom every time appear to benefit from minimal or no protection from HIV in the long run, when compared with those who avoid them all together. This worrisome finding does require a certain amount of context to fully comprehend—more on that later.

Condoms and PrEP: powerhouse protection?
The CDC researchers’ condom study served as a springboard for a new paper, with Smith also as lead author, that has estimated the effectiveness of varying overlapping adherence rates of latex and Truvada (tenofovir/emtricitabine) as pre-exposure prophylaxis (PrEP) at preventing HIV among MSM. Published in the February 2015 edition of Sexually Transmitted Diseases, the study states that MSM who always use condoms and who adhere to the daily regimen of Truvada at a rate of at least 90 percent have an estimated 92 percent lower HIV risk than those who never use condoms or PrEP.

Ninety-two percent effectiveness may seem like a particularly low estimate when the word on the street is that daily PrEP is about 99 percent effective, especially when condoms are providing another layer of armor. The reason for this discrepancy is that Smith and her colleagues based their calculations on PrEP effectiveness estimates that were derived from the self-reported adherence rates of participants of the iPrEx trial. (Published in 2010, iPrEx first proved PrEP’s ability to prevent HIV among MSM.) Of the men assigned to take PrEP in that study, the half who claimed to adhere well to PrEP had a 73 percent lower rate of HIV than those in the placebo arm of the trial.

A 2012 follow-up study to iPrEx took the results of tests of drug blood levels that estimated actual, rather than self-reported, adherence to Truvada, and used a mathematical model called a regression analysis to come up with the 99 percent effectiveness estimate. (The figure derives from comparing the HIV rate of those who adhered well to Truvada with the infection rate of those who were assigned to take the drug, but didn’t.) Researchers in iPrEx’s open-label extension phase also examined blood levels, and using a different kind of modeling (a stratified analysis), estimated, in a paper published in July 2014, that PrEP is 100 percent effective when used four or more days a week.

These modeling studies arguably suggest that 73 percent is too low of an estimate for PrEP’s effectivness—after all, doesn’t a blood test trump self-reports about condom use? But such modeling estimates are by no means perfect. IPrEx researcher Susan Buchbinder, MD, a clinical professor of medicine, epidemiology and biostatistics at the University of California, San Francisco (UCSF), suggests that the men whose blood tests indicated they used PrEP all of the time during the study may have been the sort of people who also had less risky sex, when compared with those who didn’t adhere well to Truvada. This would mean that a lower rate of risky sex (for which the researchers didn’t control when making their PrEP effectiveness estimates) accounted for some of PrEP’s apparent success, and that Truvada’s actual capacity to reduce the risk of HIV is lower than the perfect or near-perect rate that the modeling sugggests. However, the iPrEx open-label extension study found that those who chose to take PrEP had riskier sex than those who chose not to, hinting that perhaps men who favored taking Truvada were aware of their higher risk and were motivated to protect themselves with the drug.

Smith says she and her colleagues drew PrEP effectiveness figures from self-report data for the sake of consistency, since the condom effectiveness data they had to work with was also based on self-reporting. (There really is no other reliable way to measure condom use.) She also wanted the study’s findings to better apply to face-to-face conversations about HIV prevention in a clinical setting.

“When a clinician or an HIV prevention counselor is sitting down with a client to talk about their HIV prevention needs, you don’t have drug level tests in front of you,” Smith explains. “Those are research-level tests. So we wanted to give a very practical sense of how much added protection you can get from PrEP and condom use at different levels of adherence. And we used the same measure of adherence that’s available for both, which is self report.”

So how solid is “70 percent”?
Smith says she is “very confident” about the 70 percent figure, saying it is “the best estimate we have” of the actual real-world effectiveness of condoms in preventing HIV among MSM who use them consistently and correctly.

Others are less confident. “It is important to highlight that the 70 percent estimate of condom effectiveness is precisely that, an estimate, and is not set in stone,” says Alfonso C. Hernández-Romieu, MD, MPH, a research associate in the department of epidemiology at Emory University in Atlanta, who recently published a study in the journal Sexually Transmitted Infections about the considerable rates of condom failure and misuse among MSM. “As our capacity to measure condom use grows, we will likely revise [the estimate].”

“We know with some specificity how good PrEP is,” says Mitchell Warren, executive director of the global HIV advocacy group AVAC. “We don’t actually know exactly how good condoms are.”

Warren says the CDC researchers’ condom effectiveness figure is still subject to statistical scrutiny because “the only thing worse than modeling is self-reporting.”

“People lie about condom use,” says Hernández-Romieu. “Self-reported condom use is not a reliable measure.” Study participants asked to recall their behavior over long periods of time, he says, may do so inaccurately. Also, some may be inclined to give reports that make them look better in the eyes of the researchers. Such “social desirability bias” may have depressed the effectiveness estimate in the recent condom paper.

Finally slated for publication in the Journal of Acquired Immune Deficiency Syndromes (JAIDS), Smith’s condom study derives its effectiveness estimate from calculations based on data concerning 7,725 initially HIV-negative American MSM who participated in either the EXPLORE trial, a study of an HIV-prevention behavioral intervention that ran from 1999 to 2001, or a 1998 to 1999 HIV vaccine trial called VAX 004. (Both the behavioral intervention and the vaccine failed to reduce HIV incidence in these trials.) The men from these studies who were included in the CDC researchers’ analysis were followed for an average of two years and three months, during which time 510 became HIV positive.

The participants in both EXPLORE and VAX 004 were tested for the virus every six months and asked about their sex lives during the previous half year: the serostatus of their partners, how often they used condoms with them, and how many times they had sex. Smith and her coauthors based their analysis solely on men’s condom use rates for intercourse with HIV-positive male partners—“always,” “sometimes,” or “never.” A total of 3,490 men in EXPLORE and VAX 004 said they’d had sex with at least one HIV-positive partner, reporting this at an average of 2.6 of their semi-annual study visits. Out of that group, 225 contracted HIV.

“Men reporting having had sex with someone who was HIV-positive might be more inclined to say they always used a condom [because of] social desirability bias, lowering the measure of effectiveness if some men who actually only sometimes used condoms were classified as always using condoms,” argues Hernández-Romieu.

Granted, the one other significant study that made an estimate about condoms’ effectiveness at reducing HIV risk among MSM, published in JAIDS in 1989, arrived at the same figure as the CDC researchers: 70 percent. This estimate was much less statistically sound, however: The older paper estimated that the actual effectiveness could be between 35 and 88 percent, while the estimate range for the recent Smith paper was a more resolute 58 to 79 percent.

Also, the authors of the 1989 study point out a flaw in the way they questioned the 2,915 initially HIV-negative men during the two-year study, which could have biased the estimate downward. Instead of asking for the men’s condom use rate, they only asked, “Since your last visit, with how many of your partners did you use a condom?” This leaves the possibility that men may have used condoms with all of their partners, but not all the times they had sex with any one of them. When the researchers conducted their analysis, they lumped together all the men who said they used condoms “with all partners” and compared them with men who said they used condoms “with no partners.” If the men in the “with all partners” group actually used condoms at a rate of less than 100 percent, the 70 percent effectiveness figure for always using condoms likely would have been artificially depressed. (When the Smith condom effectiveness paper addresses the similarity between the two studies’ findings, it erroneously refers to the 1989 paper’s calculations as having been based upon “always use” and “never use” condom categories.)

Another factor that may have lowered the apparent effectiveness of condoms in the CDC researchers’ recent paper is incorrect use of condoms, which can lead them to break or slip off. Of course, when considering how well condoms will protect a typical group of MSM who simply use them “consistently,” it is absolutely valuable to factor in incorrect or “incomplete” use. Hernández-Romieu’s study found that white MSM in Atlanta use condoms correctly just 35 percent of the time and take them off before finishing sex 23 percent of the time. (The combined study group of EXPLORE and VAX 004 was 82 percent white.) These findings suggest that even if MSM are always using condoms, the likelihood that many are using them incorrectly—by using oil instead of water- or silicone-based lubricant, for example—and therefore reducing condoms’ ability to protect them, serves as a solid argument for adding PrEP to the mix in order to be as safe as possible.

On the other hand, a condom effectiveness estimate depressed by incorrect use is a less accurate measure for predicting how well condoms tend to work for individuals who do use condoms correctly. And notwithstanding Dawn Smith’s professed confidence that 70 percent applies to those who use condoms consistently and correctly, she and her colleagues did not actually have any data available for their analysis about men’s correct use of condoms.

Considering the fact that the data Smith and her colleagues analyzed from the EXPLORE trial showed a much higher effectiveness rate than the data from the VAX 004 study—87 percent versus 64 percent—it’s possible that the men in the behavioral intervention arm of EXPLORE were taught how to use condoms correctly and that this better protected them against HIV.

Regardless, the gap between the two figures underscores the notion that, as Hernández-Romieu puts it, “The 70 percent effectiveness of condoms is essentially an average how well condom works.” Assuming for a moment that 70 percent is indeed the true average effectiveness, this means that, for some subgroups of the MSM analyzed, condoms probably protected at a better rate than 70 percent, and some were protected at a lower rate.

“I would say that condoms are likely well over 90 percent effective if they’re used properly all of the time,” says UCSF’s Susan Buchbinder, who was a site investigator on VAX 004 and EXPLORE. “We do know you can improve effectiveness by practice-makes-perfect. People who use them more frequently have more success with them. Not being high or drunk or otherwise altered, and using lube—the right kind of lube—is really important.”

No perfect way to do the math
The CDC researchers’ condom effectiveness analysis is structured to assume that the rate of condom use—“always,” “sometimes,” or “never”—with HIV-positive partners is what mediates the risk of acquiring the virus. Smith says this was the best way to determine that there was indeed exposure to HIV, and then to examine how well condoms protected against that known exposure.

No study has estimated what percentage of HIV-diagnosed MSM had undetectable viral loads, making them relatively uninfectious, at the time the EXPLORE and VAX 004 trials were conducted. One researcher contacted ventured a guess that the undetectable rate in this population would likely have been somewhat less than 10 percent—so about that proportion of the HIV-positive partners may not have actually been a significant source of exposure to the virus.

Far more importantly, as Smith’s paper acknowledges in its list of limitations, the men included in the analysis could have contracted the virus from someone they believed to be HIV negative. Consider that the men who said they had not had sex with any HIV-positive men during EXPLORE or VAX 004 contributed to 56 percent of the HIV infections that were tallied in the Smith study’s broader analysis (that is, before the analysis was limited only to those who reported sex with positive partners). And as the CDC researchers’ paper points out, the men in the analysis were less likely to use condoms with partners they believed to be HIV-negative than they were with HIV-positive partners or those whose serostatus they did not know.

“We found in EXPLORE that having a larger number of negative partners was an independent risk factor for becoming infected,” Buchbinder reports. “And that’s because if you have a lot of partners who you believe are negative, some are probably positive—and not only positive, but perhaps highly infectious, because they’re acutely infected.”

So it is very likely the case that the group of men the Smith study presumes had a perfect condom record did not actually use latex every time they were exposed to HIV, and that this lack of latex explains at least some of the HIV transmissions in this group. Also, if in reality the men in the “never use” group did use condoms with partners they incorrectly presumed were HIV negative, condoms may have kept this group’s transmission rate lower than it would have been if it were truly latex free.

Imagine if it were possible to carve out accurately which members of the “always use” group actually remained 100 percent faithful to condoms every time they had sex with someone who was truly HIV positive. Their rate of HIV acquisition would likely be lower than the rate for Smith’s version of the “always group.” And if we could find those members of the “never use” group who really did avoid latex every time they were truly exposed to the virus, their rate of new infections would likely be higher than the rate for Smith’s version of the group. Because the difference between the two infection rates that reflect true condom use would then be larger than difference between the HIV rates of Smith’s groups, the estimate of how successful condoms actually were at mediating HIV infection would rise. Translation: the CDC researchers’ estimate of 70 percent is artificially low. In fact, because there is no way for the men in Smith’s intepretation of the “never” group to have used condoms less, or for those in her “always” group to have used them more, when you are considering true condom use rates with all HIV exposures, the only direction for the true estimate to travel is higher.

Kenneth Mayer, MD, medical research director at Fenway Health in Boston, who was the site principal investigator on EXPLORE and an investigator on VAX 004, is critical of the CDC researchers’ decision to limit their analysis to condom use with partners identified as HIV positive.

“Basing estimates of condom effectiveness should use all relevant information from study participants,” Mayer says. Speaking hypothetically about a scenario in which “HIV-uninfected” is a perception, he continues, “So calculating the effectiveness of condom use or non-use from someone who has one episode of sex with a known HIV-infected partner and 100 episodes with 100 different HIV-uninfected partners needs to include information from all partners with whom there is a potential transmission risk.”

Another factor unaccounted for in the CDC researchers’ analysis was men’s frequency of intercourse and their number of partners. The 1989 JAIDS paper that also estimated condoms’ effectiveness among MSM provides an example of how variations in such behavior patterns can affect the outcome of such an analysis. That paper found that the men who reported using condoms with some of their partners were the most likely to contract HIV, more so than even those who reported not using condoms with any partners. This phenomenon is possibly explained by the fact that the men in the “with some partners” group reported the highest number of different sexual partners; also, that group had the lowest rate of men reporting having only one partner in any given six-month period. In other words, the “with some partners” men appeared to have riskier sex lives irrespective of condoms than those who didn’t use latex.

Smith defends how she and her colleagues structured their analysis. “We did not feel that the data [reporting] men’s partners of unknown or negative status would help answer the effectiveness question, given that we had data from reported HIV-discordant partnerships,” she says. (“HIV-discordant” means a partnership between people of different HIV statuses.)

Conducting a robust analysis that considers rates of condom use with all partners adds many layers of complexity that can be highly challenging for researchers to negotiate. If Smith’s analysis had factored in condom use with partners of an unknown HIV status or who were perceived to be HIV negative, she and her colleagues would have had to try to determine the HIV prevalence in the various communities of all the men studied.

“Some studies try to model this,” Mayer says. “For example, assuming that X percentage of the unknown status partners may be infected. But it is never an exact science, which is why I don’t love to have the discussions focus on specific numbers, which can at best be an approximation of reality.”

Smith also says she wanted to focus her analysis only on condom use with partners identified as HIV positive so that her results could be compared to those in the similarly structured 2002 meta-analysis of mixed-HIV-status heterosexual couples, the study that estimated condoms’ effectiveness at 80 percent.

One trouble with the comparison between these two trials is the statistical shakiness of that 80 percent estimate. The authors of the heterosexual meta-analysis worked with a relatively small data set pooled from various other studies: 11 HIV transmissions among 587 people who always used condoms, who were followed for an average of 1.6 years; and 14 transmissions among 87 those who never used latex, followed for an average of 2.8 years. Consequently, the authors projected a very wide range in which the true effectiveness rate might be.

“This lack of precision is not the fault of the authors and their review,” David Wilkinson, then pro vice chancellor and vice president of the division of health sciences at the University of South Australia, in Adelaide, Australia, wrote in The World Health Organization Reproductive Health Library in 2002. “It is an inherent feature of the available studies [included in the analysis]. As such, it is not really appropriate to estimate condom effectiveness at 80 percent. While 80 percent is the best single estimate of effectiveness, it is also fair to say that the true measure of the effect could be as low as 35 percent or as high as 94 percent, as the authors state in the review.”

Hernández-Romieu further argues that the results of the Smith paper are tricky to apply to contemporary American MSM because, referring to the general period when the data in the analysis was collected, “The state of the epidemic among MSM in the late 1990s is not the same as it is today, and a study like this with current data might show very different estimates of condom effectiveness.”

Buchbinder sums it up in an email: “There is no ‘perfect’ way to measure condom effectiveness.”

PrEP to the rescue?
The finding that those who “sometimes” use condoms with HIV-positive partners have the same essential risk of contracting HIV as those who “never” use them may seem illogical. Don’t condoms count for something if they’re used at all?

This is where a phenomenon known as “cumulative risk” likely comes into play. If members of a group of MSM keep rolling the dice by only sometimes using condoms when they’re exposed to HIV, over time their risk accumulates. Eventually the number of infections in that group will resemble that of a similar group of MSM in which no one used condoms.

A considerable challenge with interpreting the “sometimes” vs. “never” finding is that “sometimes” had a broad definition in the Smith paper’s analysis—it could have reflected anywhere between 1 and 99 percent condom use. So it’s possible that there wasn’t a very large actual gap in condom use rates between the two groups, which would at least partially explain the apparent lack of a relative benefit of less-than-perfect condom use when compared with always barebacking. (Smith says that an analysis based on specific percentages of condom use would ask for trouble by relying on men’s ability to recall their use with such specificity; doing so would only introduce more potential error and bias.)

Enter PrEP, which, as Smith’s paper on Truvada’s combined effectiveness with condoms shows, can be instrumental in lowering men’s risk, even without perfect adherence. The only scenario in which someone starting PrEP might actually raise his risk, according to the analysis, is if he went from always using condoms to sometimes or never using them and then adhered to PrEP at a rate of less than 50 percent. His risk of HIV, relative to someone using neither condoms nor PrEP, would drop from a 70 percent reduced risk to just a 37 percent reduced likelihood of contracting the virus.

But recall how the self-report data fed into the CDC researchers’ analysis may very well have underestimated PrEP’s actual effectiveness rates. According their analysis, adhering to PrEP at a rate of 50 percent or less offers about 32 percent protection. But the iPrEx open-label extension study’s blood test-based modeling estimated that those taking Truvada two or three times a week had an 81 percent reduced risk of HIV. (Although those researchers guessed PrEP taking this often could actually be between 12 percent and 99 percent effective.)

Pre-empting protests from those who predict that PrEP’s roll-out may lead to a rise in HIV risk among some groups of men, AVAC’s Mitchell Warren also argues that the sort of person who uses condoms consistently isn’t likely to be the type to go on PrEP and then adhere poorly to both Truvada and condoms.

“If we imagine someone who is a really good, careful risk mitigator,” Warren says, “it’s hard for me to believe that introducing PrEP is going to make them less focused on protecting themselves.”

When discussing HIV prevention options with everyday MSM, Ken Mayer and many in his field prefer to draw the conversation away from statistics and into a more qualitative way of thinking about safer sex. The two Smith papers, he says, “help give the case that condoms offer substantial protection if used consistently. PrEP offers substantial protection if used consistently. And when the two are used together, even by someone who says they’re always using condoms, there may be some incremental benefit from using PrEP. But particularly for people who are not using condoms regularly, there’s even more of a benefit from using PrEP.”

“Often it seems that condoms and PrEP are pitted against each other,” Buchbinder says, “and they really can be complementary and supplemental.”

To read a POZ feature article that helps explain some of the math behind HIV risk reduction, click here.