How to lie with statistics using a very bad ‘decision rule’

13 Feb

I teach statistics at the undergraduate and graduate level.  When a researcher selects cases (subjects to be included in the analysis, be they individuals, nations, districts, years, etc.) they must adopt a selection criterion, or a ‘decision rule’, by which some cases will be included and some will be excluded from the analysis.  If you are trying to classify subjects/cases as either religious or irreligious, for instance, you have to consistently apply a rule (will a religious person be someone who attends church weekly or monthly? The answer you settle upon is your ‘decision rule’).  This is sometimes called selection bia by selecting upon the dependent or main independent variable.  Relaxing or tightening the decision rule can paint very different portraits of a sample or population.  Here is a classic example of a bad decision rule perpetrated by Planned Parenthood and White House spokespersons in recent days (most relevant part highlighted).

How to lie with statistics, example umpteen [Updated] (What’s Wrong with the World).

How to lie with statistics, example umpteen [Updated]

Recently I received a phone call from my non-Christian (as far as I know) philosopher friend Neil Manson who, because he has an active and fair mind, had been exercised over what seemed to him the high FQ (Fishiness Quotient) of the statistical meme presently going around to the effect that “98% of Catholic women use birth control.” Or something. Maybe “98% of Catholic women have used birth control.” The former is obviously ludicrous, as it would seem to include elderly Catholic women, of whom it seems plausible that there are more than 2% among Catholics. Anyway, Neil wanted to know if I had read anything debunking the statistic.

Well, I had to admit that I hadn’t. This is mostly because the relevance of the claim to the HHS’s mandate is, to put it mildly, obscure. If a large percentage of Jainists are chowing down on hamburgers on the side, it hardly follows that an expressly Jainist charitable organization should be forced by the federal government to fund a plan that buys free hamburgers for its employees. If a bunch of Quakers turn out to have gun licenses, employees of an expressly Quaker organization are not therefore entitled to have their fees paid to a shooting range or their ammo. provided at no cost through an employer plan. There is this commonsense notion that organizations that are explicitly identified as religious are allowed to uphold the actual doctrinal and behavioral standards of their respective religious bodies. Whether the rank and file membership of that religious body follow those standards in daily life should be irrelevant.

Still, it has proven rather interesting to look into the statistical claim.

Here’s how it works. The study is here. The relevant tables are Figure 3 on p. 6 and the second Supplementary Table on p. 8. The survey was limited to women between 15-44. Ah, well, that explains how we weren’t including the elderly, but it also means that the silly “percent of all Catholic women” thing should be chucked out right from the beginning. More strikingly, as Neil pointed out to me after looking up the study, it excluded any women who were a) not sexually active, where that is defined as having had sexual intercourse in the past three months (there go all the nuns), b) postpartum, c) pregnant, or d) trying to get pregnant! In other words, the study was specifically designed (as the prose discussion on p. 8 makes explicit, in bold print) to include only women for whom a pregnancy would be unintended and who are “at risk” of becoming pregnant. Whether or not it included women who considered themselves neither trying nor not trying to get pregnant (there are some such women in the world) is unclear. It’s also unclear whether it included women who have had their reproductive organs removed because of some medical problem. Presumably the study was intended to exclude women in both of these categories, as neither would count as a woman “at risk of an unintended pregnancy.”

Now, consider what all of this means as far as the representativeness of the sample for Catholic women. Surely there are a fair number of Catholic women between 15-44 who are not “at risk of an unintended pregnancy” for various reasons. It is plausible that this number is higher among Catholics than among non-Catholics. For one thing, a faithful Catholic woman in this age category who is not married is supposed to be remaining celibate. Hence she won’t fall into the “at-risk” category, and by the same token she won’t have any use for the “services” that the Obama administration is mandating be provided. Similarly, married Catholic women are probably more likely not to be attempting to avoid pregnancy, even using Natural Family Planning, than non-Catholic women. One would think they are also more likely to be pregnant or postpartum. And so on and so forth. In short, the deliberate design of the study to cover only women who, at the time of the study, were having sexual intercourse while regarding a pregnancy as unintended would be likely to make it unrepresentative of Catholics and particularly unrepresentative of devout Catholics. Yet the study is now being cited to show the percentage of Catholic women generally who are not following the teaching of the Catholic Church in this area! What is wrong with this picture?

To make matters even weirder, this Politifact evaluation of the meme gets it wrong again and again, and in both directions.

First, the Politifact discussion insists that the claim is only about women in this category who have ever used contraception. When I first heard that and hadn’t looked at the study, I immediately thought of the fact that such a statistic would presumably include women who were not at the time of the study using contraception and had used it only once in the past. It was even pointed out to me that it would include adult converts whose use might easily have been prior to their becoming Catholic. However, that isn’t correct, anyway. The study expressly was of current contraceptive use. That’s, in a sense, “better” for the side that wants the numbers to be high.

However, second, that is swamped by the point made above about all the groups (likely to be more highly represented among Catholics, especially faithful Catholics, and not in need of contraceptive “services”) excluded from the study–celibate women, postpartum women, women not trying to avoid pregnancy. And on this point, too, the Politifact evaluation is completely wrong. Politifact implies that only the supplementary table on p. 8 excluded these groups and that Figure 3 on p. 6 included them! But this is wrong. The table on p. 8 is simply supplementary to Figure 3, and both are taken from the same survey using the same restrictions! This is made explicit again and again in the study.

Politifact seems to think that including these excluded groups, which it wrongly thinks are included in Figure 3, increases the total number of “contracepting” Catholic women found, apparently by including women who have used contraception at some earlier time in their lives. This completely ignores the ways in which such exclusions are likely to bias the study away from devout Catholics.

Of course, we don’t actually know what the overall effect of including the excluded groups would be, because they weren’t included. No doubt Guttmacher would say that such groups should be excluded from their survey because they wanted their survey to be about “current contraceptive use.” It is obvious that, for example, pregnant women aren’t going to be using contraception. Well, okay, then. But a statistic based on a study that explicitly excluded those who have no use for contraception is obviously irrelevant to a question about the percentage of Catholic women who have a use for contraception!

If a researcher wanted to design a study that included the excluded groups and then examine probable later contraceptive use among women currently post-partum, pregnant, trying to get pregnant, etc., he could include those groups in the original study and interview those sub-groups further about their later intentions: If they get pregnant, after that baby is born, do they intend to try to avoid pregnancy while continuing to be sexually active? If so, what method do they intend to use? If they get married, do they intend to use contraception? And so forth. It would, of course, be possible to question the objectivity of such future projections on the part of the women, but it would allow a sample in such a study to be more representative of Catholic women, including devout Catholic women, while gathering information relevant to the claim, sure to be made, “Yeah, but they’ll want contraception later on.”

The statistics in the Guttmacher study appear to be okay for the purpose for which the study was originally intended. The intention of the study was to answer something like the following question: “Among women of various religious groups who are now sexually active but do not wish to become pregnant, what percentage use different methods of avoiding pregnancy?” But the purpose for which the statistic for Catholic women from the study is now being used is to argue, “A very high percentage of Catholic women (or, perhaps, Catholic women of child-bearing age) are currently not following the Catholic Church’s teachings on sex and contraception and have a use for contraception forbidden by the Catholic Church.”

For that purpose, these statistics are bogus.

Update 1:
Upon reflection, I have realized clearly an additional major problem with the 98% statistic. It is including all the Catholic women who expressly told researchers that they used “no method” to avoid pregnancy. In the table, that is 11%. The 98% statistic is apparently derived by subtracting only the 2% who said that they used NFP from 100%. So women who said they used no method of contraception are apparently being included in a statistic about how many Catholic women use contraception. How’s that for crazy? And that’s in addition to the problems discussed already in the original post.

Update 2: P. 5 of the prose discussion asserts that, among married women, percentages of pregnant, postpartum, and trying-to-get-pregnant women do not vary by religious affiliation. No data is provided in this document to support that assertion. None of the graphs appear to bear on it.

%d bloggers like this: