Box Turtle Bulletin

The NGLTF Study On Race and Prop 8: The Problem of Margins of Error

Well, I’ve said before I had some homework to do this weekend. It turns out that this weekend was jam-packed with unexpected activities, but I did manage to give the NGLTF report (PDF: 420KB/17 pages) a careful read this morning while sipping tea from my family’s heirloom Fiestaware handed down from my great-great grandmother. Yes, I’m a dish queen.

Margin of Error: The Key
Timothy’s Kincaid’s analysis garnered a lot of controversy last week. Many people privately called and emailed to ask if I agreed with it. My only response at the time is that I hadn’t had a chance to look over the NGLTF report or Timothy’s analysis, but I generally trust his judgment. Well, now I have studied the report, and I do think it falls short, but in very different ways than what Timothy found.

My concerns about this report begin with one important paragraph on page 2:

Table 1 displays findings from a poll of California voters conducted by David Binder Research (DBR) between November 6th and 16th, 2008. The survey included 1,066 respondents selected at random from state voter registration lists, including an oversample of 266 African American, Latino, and Asian-American voters. Participants were asked a series of questions about Proposition 8, as well as basic questions about their demographic background, religion, political views, and other characteristics. The sample in the DBR survey was limited to those who reported voting in the November 4 general election, and its margin of error was 3 percentage points (although the margin is greater for analyses of subgroups within the sample).

The DBR survey is the backbone of this study. That three-percent margin of error applies only to the 1,066 respondents overall, not to the smaller sample of 266 African-American, Latino, and Asian-American voters. The authors acknowledge that “the margin is greater for analyses of subgroups within the sample,” but they don’t tell you what those margins are. This is important, because as sample sizes get smaller, the margin of error gets larger.

A simple calculation for the 266 African-American, Latino, and Asian-American voters reveals that this margin of error is actually plus or minus 6 percentage points. That is margin of error for the three groups combined. Nowhere in this report is a breakdown of the three groups revealed. Of the 266 participants in the subgroup, how many were African-American?

Since they don’t tell us, we’re left to guess. If Blacks made up half of that pool, then responses from African-Americans alone are subject to an 8.5% margin of error. Cut that in about half again to separate the church-going from the non-church-going, then you’re up to about a plus or minus 12 percentage point margin of error for the two groups of African-Americans separately. If Blacks only made up a third of that pool, then the margins of error are greater still — about 10.4% and 14.7% respectively. This is huge. How do these large margins of error affect the rest of the report?

Religiosity As An Explanation
To see, let’s move on to this graphic, which illustrates the religiosity of the four ethnic groups using the DBR survey data with the margins of error we just talked about. You’ll have to click on the image to see it clearly:

According to the DBR survey, 57% of African-American voters attend church service weekly, compared to 40% for Asians, 47% for Latinos, and 42% for White. The authors assert that the differences between African-Americans and the rest of the population is statistically significant, which checks out according to the standard measures for statistical significance. Even with this small sample size and large margin of error, the DBR data does successfully demonstrate that African-Americans are more likely to attend weekly religious services than the other groups.

That then leads us to this graphic, based again on the same DBR survey. Again, you’ll have to click on it to see clearly:

The authors say that the differences shown in this graph between ethnic groups are not statistically significant, and they conclude that this shows that religiosity explains the differences in how African-Americans voted relative to everyone else.

Well, at least one part of their statement is absolutely correct. The differences between ethnic groups in the figures referenced in this table are not statistically significant according to all the standard measures of significance — but that’s because the sample sizes are so small.

There is a logical fallacy in saying that just because this data shows no statistically significant difference, that there is no actual difference. That’s not true. All we can say is that this data is incapable of showing a statistically significant difference based on these results and these small sample sizes. It cannot demonstrate that there is no difference in actuality. Remember, we’re dealing with a probable margin of error for the African-American churchgoing sample of somewhere in the neighborhood of plus or minus 12% to 14.7%. With an uncertainty that large, these numbers could be all over the place and still be a statistical tie. Any assessment of actual differences is completely swamped by the margins of error.

If the study consisted of a larger pool of African-American respondents to get a lower the margin of error, we might have been able to converge on a statistically significant difference. Or maybe then we can prove that there really is no difference in how religious African-Americans voted compared to the other groups. But with this data, we cannot tell either way. The Achilles Heel in this study remains the very small sample size for African-Americans and the resulting large margins of error for that sample. I don’t think they are able to make the case that religiosity explains the African-American vote with this data.

The African-American Vote on Prop 8
So how did African-Americans vote? Let’s go to this graphic from the NGLTF report:

The NGLTF study is being used to throw cold water on CNN’s NEP exit poll, which said that 70% of African-Americans supported Prop 8. The middle set of bars are the NEP exit poll, which shows African-Americans voting 70% for Prop 8 (in gray) versus 52% overall voting for Prop 8 (in black). The graphic also shows two surveys taken before the election (The Field Poll of 10/23 and SurveyUSA on 10/30) and two surveys taken after the election (the DBR poll we’ve already mentioned showing 58% of African-Americans supporting Prop 8 versus 51% overall on 11/11, and the SurveyUSA on 11/19). The study authors note:

As shown in Figure 2, two surveys conducted just before Election Day (by Field and SurveyUSA) found insignificant differences in support for Proposition 8 between African Americans and Californians as a whole. Two surveys conducted in the weeks following Election Day found similar results. On average, the difference in support between African Americans and all voters in these four surveys was just two percentage points. The NEP exit poll finding—that black support for Proposition 8 was 18 points higher than Californians as a whole—is most likely an “outlier,” a result that is very different than what concurrent data trends suggest to be the case. [Emphasis mine]

The authors dismiss the NEP exit poll as an outlier, an assessment that I can agree with. Exit polls, by their nature, don’t include margins of error. But since it is likely that the sample size of African-Americans was very small in this exit poll, I can accept that it is probably not an accurate snapshot of how African-Americans voted.

However, the study authors claim that the four remaining surveys show a difference of just two percentage points on average. True enough, in a strictly mathematical sense. But since the last SurveyUSA was the only survey showing African-Americans actually opposing Prop 8 to a remarkable degree compared to everyone else — that difference is a whopping eight percentage points in the other direction — I don’t see how we can regard that as anything but an outlier as well. So, with the three remaining polls, the difference is now back up to five percentage points.

Is this significant? I can’t tell, since again, we don’t know the sample sizes of African-Americans in these polls to judge whether they are robust enough to draw a reasonable conclusion.

The problem of sample sizes and margins of error, in my mind, does lay to rest one of Timothy’s concerns, and that is this:

In their Table 1, they lay out their breakdown of ethnic voting:

Well sorry, but those numbers don’t get us to 52.3% support. One of those ethnic demographics is understated.

Given the likely margins of error involved, I don’t think that this chart is off base entirely. No poll is likely to mimic the 52.3% of the actual vote at the means, but shoving all of these figures around their margins of error will get there quite easily. (I also wonder if maybe there ought to be an “other” category not included in the table.)

Fifty-eight percent as a very rough ballpark figure could be about right for the African-American vote. But given some of the margins of error we tossed around earlier, that figure could be as high as about 67% to 70%, or as low as 49% to 46%. Which means that if we used the DBR survey as the reference survey as the NGLTF study authors did, then none of those surveys which I (or the NGLTF authors) suggested were outliers may be outliers after all. The DBR survey may well validate all of them.

The study authors then replicate a 58% estimate by using data depicted in this figure, which is based on precinct-level voting data from five California counties:

The line drawn through the figure represents a “running-mean smoother” to show the overall trend as the racial mix of precincts moves from 0% to 100% African-American. Unlike Timothy, I’m satisfied with this representation which the authors use to arrive at a 58% figure for African-Americans, although I am keen to learn the algorithm for the smoother. But generally this verifies what many of us suspect: Those who live in diverse settings are more comfortable with diversity. Those who don’t, aren’t.

The reason I’m okay with this is that the authors also ran this same data set through two other independent analyses which led them to report a degree of comfort with an estimate of 58% of African-Americans voting for Prop 8. They do caution however, that “rather than being treated as definitive, these estimates should be considered as helping to corroborate the individual-level findings discussed earlier in this section of the study” — namely, the discussion of the five surveys we discussed earlier.

But in the end, I do believe the authors were successful in demonstrating that the Black vote may be closer to 58% than 70%. The higher figure, technically speaking, still barely remains in the theoretical realm of possibility, but I think we can safely dismiss it. But I would also caution that 58% might not be accurate either.

Can The Scapegoating End?
But if 58% is plausible, does this mean that the scapegoating of African-Americans can come to an end? Of course it does.

But what if the authors instead determined that the figure was closer to 70%? Would that have meant that blaming African-Americans for Prop 8’s passage was legitimate? Ask yourself this and take a hard look at how you answer, because this is critical to where our movement goes next. The answer to this question speaks loudly to our own character as a community.

If all it takes is a survey to give one oppressed minority the justification it needs to blame another oppressed minority for its woes, then we have a lot more work to do before we can credibly address society’s attitudes about fairness and equality. We will have to change our own attitudes first.

We cannot assume that one oppressed minority ought to automatically empathize with another oppressed minority’s oppression. If that were true, Jews and Palestinians would see themselves in each other and peace would break out all over the Middle East. Well that certainly hasn’t happened, has it?

Just to touch the tip of a few icebergs, gays were never enslaved or lynched in mass numbers. Non-Black gays really have no idea what it’s like to have that in their history. On the other hand, heterosexual Blacks were never obliged to undergo cruel “cures,” nor were they ostracized from their own families because of their Blackness. We really don’t know — internally know — the other’s experiences with history, and we can no longer be so naive in assuming that others will naturally see and recognize our experiences with discrimination just because they were discriminated against in a different way for different reasons.

So we must begin the task of reaching out to the African-American community, and more importantly, we need to work to raise the visibility of African-Americans within our own raucous LGBT family. If we want to confront homophobia in the Black community, we must also deal with examples of both overt and underlying racism within our own.

And we need to talk honestly and listen patiently to each other. We need to do this not to “educate” the other, as though we had some sort of special prize that we wish to arrogantly bestow on some poor, unenlightened folks. Instead, we need to do this with the sincere intent of understanding each other and ourselves better.

We need to do this not because a survey says we ought to. We need to do this because it is the right thing to do.

And we need to do this not just because elections are at stake, but because lives are at stake as well.