Box Turtle Bulletin

Box Turtle BulletinNews, analysis and fact-checking of anti-gay rhetoric
“Now you must raise your children up in a world where that union of man and box turtle is on the same legal footing as man and wife…”
This article can be found at:
Latest Posts

The NGLTF Study On Race and Prop 8: The Problem of Margins of Error

This commentary is the opinion of the author and does not necessarily reflect that of other authors at Box Turtle Bulletin.

Jim Burroway

January 12th, 2009

Well, I’ve said before I had some homework to do this weekend. It turns out that this weekend was jam-packed with unexpected activities, but I did manage to give the NGLTF report (PDF: 420KB/17 pages) a careful read this morning while sipping tea from my family’s heirloom Fiestaware handed down from my great-great grandmother. Yes, I’m a dish queen.

Margin of Error: The Key
Timothy’s Kincaid’s analysis garnered a lot of controversy last week. Many people privately called and emailed to ask if I agreed with it. My only response at the time is that I hadn’t had a chance to look over the NGLTF report or Timothy’s analysis, but I generally trust his judgment. Well, now I have studied the report, and I do think it falls short, but in very different ways than what Timothy found.

My concerns about this report begin with one important paragraph on page 2:

Table 1 displays findings from a poll of California voters conducted by David Binder Research (DBR) between November 6th and 16th, 2008. The survey included 1,066 respondents selected at random from state voter registration lists, including an oversample of 266 African American, Latino, and Asian‐American voters. Participants were asked a series of questions about Proposition 8, as well as basic questions about their demographic background, religion, political views, and other characteristics. The sample in the DBR survey was limited to those who reported voting in the November 4 general election, and its margin of error was 3 percentage points (although the margin is greater for analyses of subgroups within the sample).

The DBR survey is the backbone of this study. That three-percent margin of error applies only to the 1,066 respondents overall, not to the smaller sample of 266 African-American, Latino, and Asian-American voters. The authors acknowledge that “the margin is greater for analyses of subgroups within the sample,” but they don’t tell you what those margins are. This is important, because as sample sizes get smaller, the margin of error gets larger.

A simple calculation for the 266 African-American, Latino, and Asian-American voters reveals that this margin of error is actually plus or minus 6 percentage points. That is margin of error for the three groups combined. Nowhere in this report is a breakdown of the three groups revealed. Of the 266 participants in the subgroup, how many were African-American?

Since they don’t tell us, we’re left to guess. If Blacks made up half of that pool, then responses from African-Americans alone are subject to an 8.5% margin of error. Cut that in about half again to separate the church-going from the non-church-going, then you’re up to about a plus or minus 12 percentage point margin of error for the two groups of African-Americans separately. If Blacks only made up a third of that pool, then the margins of error are greater still — about 10.4% and 14.7% respectively. This is huge. How do these large margins of error affect the rest of the report?

Religiosity As An Explanation
To see, let’s move on to this graphic, which illustrates the religiosity of the four ethnic groups using the DBR survey data with the margins of error we just talked about. You’ll have to click on the image to see it clearly:

According to the DBR survey, 57% of African-American voters attend church service weekly, compared to 40% for Asians, 47% for Latinos, and 42% for White. The authors assert that the differences between African-Americans and the rest of the population is statistically significant, which checks out according to the standard measures for statistical significance. Even with this small sample size and large margin of error, the DBR data does successfully demonstrate that African-Americans are more likely to attend weekly religious services than the other groups.

That then leads us to this graphic, based again on the same DBR survey. Again, you’ll have to click on it to see clearly:

The authors say that the differences shown in this graph between ethnic groups are not statistically significant, and they conclude that this shows that religiosity explains the differences in how African-Americans voted relative to everyone else.

Well, at least one part of their statement is absolutely correct. The differences between ethnic groups in the figures referenced in this table are not statistically significant according to all the standard measures of significance — but that’s because the sample sizes are so small.

There is a logical fallacy in saying that just because this data shows no statistically significant difference, that there is no actual difference. That’s not true. All we can say is that this data is incapable of showing a statistically significant difference based on these results and these small sample sizes. It cannot demonstrate that there is no difference in actuality. Remember, we’re dealing with a probable margin of error for the African-American churchgoing sample of somewhere in the neighborhood of plus or minus 12% to 14.7%. With an uncertainty that large, these numbers could be all over the place and still be a statistical tie. Any assessment of actual differences is completely swamped by the margins of error.

If the study consisted of a larger pool of African-American respondents to get a lower the margin of error, we might have been able to converge on a statistically significant difference. Or maybe then we can prove that there really is no difference in how religious African-Americans voted compared to the other groups. But with this data, we cannot tell either way. The Achilles Heel in this study remains the very small sample size for African-Americans and the resulting large margins of error for that sample. I don’t think they are able to make the case that religiosity explains the African-American vote with this data.

The African-American Vote on Prop 8
So how did African-Americans vote? Let’s go to this graphic from the NGLTF report:

The NGLTF study is being used to throw cold water on CNN’s NEP exit poll, which said that 70% of African-Americans supported Prop 8. The middle set of bars are the NEP exit poll, which shows African-Americans voting 70% for Prop 8 (in gray) versus 52% overall voting for Prop 8 (in black). The graphic also shows two surveys taken before the election (The Field Poll of 10/23 and SurveyUSA on 10/30) and two surveys taken after the election (the DBR poll we’ve already mentioned showing 58% of African-Americans supporting Prop 8 versus 51% overall on 11/11, and the SurveyUSA on 11/19). The study authors note:

As shown in Figure 2, two surveys conducted just before Election Day (by Field and SurveyUSA) found insignificant differences in support for Proposition 8 between African Americans and Californians as a whole. Two surveys conducted in the weeks following Election Day found similar results. On average, the difference in support between African Americans and all voters in these four surveys was just two percentage points. The NEP exit poll finding—that black support for Proposition 8 was 18 points higher than Californians as a whole—is most likely an “outlier,” a result that is very different than what concurrent data trends suggest to be the case. [Emphasis mine]

The authors dismiss the NEP exit poll as an outlier, an assessment that I can agree with. Exit polls, by their nature, don’t include margins of error. But since it is likely that the sample size of African-Americans was very small in this exit poll, I can accept that it is probably not an accurate snapshot of how African-Americans voted.

However, the study authors claim that the four remaining surveys show a difference of just two percentage points on average. True enough, in a strictly mathematical sense. But since the last SurveyUSA was the only survey showing African-Americans actually opposing Prop 8 to a remarkable degree compared to everyone else — that difference is a whopping eight percentage points in the other direction — I don’t see how we can regard that as anything but an outlier as well. So, with the three remaining polls, the difference is now back up to five percentage points.

Is this significant? I can’t tell, since again, we don’t know the sample sizes of African-Americans in these polls to judge whether they are robust enough to draw a reasonable conclusion.

The problem of sample sizes and margins of error, in my mind, does lay to rest one of Timothy’s concerns, and that is this:

In their Table 1, they lay out their breakdown of ethnic voting:

Well sorry, but those numbers don’t get us to 52.3% support. One of those ethnic demographics is understated.

Given the likely margins of error involved, I don’t think that this chart is off base entirely. No poll is likely to mimic the 52.3% of the actual vote at the means, but shoving all of these figures around their margins of error will get there quite easily. (I also wonder if maybe there ought to be an “other” category not included in the table.)

Fifty-eight percent as a very rough ballpark figure could be about right for the African-American vote. But given some of the margins of error we tossed around earlier, that figure could be as high as about 67% to 70%, or as low as 49% to 46%. Which means that if we used the DBR survey as the reference survey as the NGLTF study authors did, then none of those surveys which I (or the NGLTF authors) suggested were outliers may be outliers after all. The DBR survey may well validate all of them.

The study authors then replicate a 58% estimate by using data depicted in this figure, which is based on precinct-level voting data from five California counties:

The line drawn through the figure represents a “running-mean smoother” to show the overall trend as the racial mix of precincts moves from 0% to 100% African-American. Unlike Timothy, I’m satisfied with this representation which the authors use to arrive at a 58% figure for African-Americans, although I am keen to learn the algorithm for the smoother. But generally this verifies what many of us suspect: Those who live in diverse settings are more comfortable with diversity. Those who don’t, aren’t.

The reason I’m okay with this is that the authors also ran this same data set through two other independent analyses which led them to report a degree of comfort with an estimate of 58% of African-Americans voting for Prop 8. They do caution however, that “rather than being treated as definitive, these estimates should be considered as helping to corroborate the individual-level findings discussed earlier in this section of the study” — namely, the discussion of the five surveys we discussed earlier.

But in the end, I do believe the authors were successful in demonstrating that the Black vote may be closer to 58% than 70%. The higher figure, technically speaking, still barely remains in the theoretical realm of possibility, but I think we can safely dismiss it. But I would also caution that 58% might not be accurate either.

Can The Scapegoating End?
But if 58% is plausible, does this mean that the scapegoating of African-Americans can come to an end? Of course it does.

But what if the authors instead determined that the figure was closer to 70%? Would that have meant that blaming African-Americans for Prop 8’s passage was legitimate? Ask yourself this and take a hard look at how you answer, because this is critical to where our movement goes next. The answer to this question speaks loudly to our own character as a community.

If all it takes is a survey to give one oppressed minority the justification it needs to blame another oppressed minority for its woes, then we have a lot more work to do before we can credibly address society’s attitudes about fairness and equality. We will have to change our own attitudes first.

We cannot assume that one oppressed minority ought to automatically empathize with another oppressed minority’s oppression. If that were true, Jews and Palestinians would see themselves in each other and peace would break out all over the Middle East. Well that certainly hasn’t happened, has it?

Just to touch the tip of a few icebergs, gays were never enslaved or lynched in mass numbers. Non-Black gays really have no idea what it’s like to have that in their history. On the other hand, heterosexual Blacks were never obliged to undergo cruel “cures,” nor were they ostracized from their own families because of their Blackness. We really don’t know — internally know — the other’s experiences with history, and we can no longer be so naive in assuming that others will naturally see and recognize our experiences with discrimination just because they were discriminated against in a different way for different reasons.

So we must begin the task of reaching out to the African-American community, and more importantly, we need to work to raise the visibility of African-Americans within our own raucous LGBT family. If we want to confront homophobia in the Black community, we must also deal with examples of both overt and underlying racism within our own.

And we need to talk honestly and listen patiently to each other. We need to do this not to “educate” the other, as though we had some sort of special prize that we wish to arrogantly bestow on some poor, unenlightened folks. Instead, we need to do this with the sincere intent of understanding each other and ourselves better.

We need to do this not because a survey says we ought to. We need to do this because it is the right thing to do.

And we need to do this not just because elections are at stake, but because lives are at stake as well.

Comments

POST COMMENT | COMMENT RSS 2.0 | TRACKBACK URL

Elise
January 12th, 2009 | LINK

“True enough, but since the last SurveyUSA was the only survey showing African-Americans actually *supporting* Prop 8 in numbers higher than all voters overall…”

I think you meant “opposing” here.

Mike
January 12th, 2009 | LINK

I’m tired both of (white) gays comparing our experience to life under Jim Crow, and of (straight) African Americans deriding the very real abuses that LGBT folk have suffered and continue to suffer.

Oppression comes in many different flavors. Then again, prejudice is color-blind and orientation-blind. I’ve heard more than one gay acquaintance make unabashed racist remarks, and I’ve definitely heard some homophobic slurs from African Americans — and Americans of all races for that matter.

So yeah, we’ve got to get past the notion that different minorities automatically understand each other’s experiences, and stop playing “more oppressed than thou.” At the end of the day, we’ve just got to reach out intelligently without preconceptions or assumptions.

Jim Burroway
January 12th, 2009 | LINK

Elise,

You’re right. I’ve corrected the post. Thanks for the catch

homer
January 12th, 2009 | LINK

I am also a dish queen, as you well know.

Mad Professah
January 12th, 2009 | LINK

This is a much more reasoned analysis than Timothy Kincaid’s.

Thank you for that.

Priya Lynn
January 12th, 2009 | LINK

Jim said “gays were never enslaved or lynched in mass numbers”.

Blacks were enslaved in mass numbers, but lynched – I’m not so sure. Correct me if I’m wrong, but I don’t think blacks were lynched in mass numbers.

Mad Professah
January 12th, 2009 | LINK

Priya, what do you consider “mass numbers”?

There were definitely THOUSANDS of Black people (almost always Black men) lynched in the South.

The estimates of documented lynchings is usually in the under 5000 range (between 1860-1950) but those are just the lynchings that we know about!

dairyqueen
January 12th, 2009 | LINK

This article was way too long. You lost me half way in.

John
January 12th, 2009 | LINK

I would agree that gays and lesbians need to find a more effective way of reaching out to all voting constituency, but I find this whole preoccupation with how blacks voted on Prop 8 to be an interesting diversion, but a diversion nonetheless.

Blacks did not write this proposition. Unlike the virtually all white Mormon Church members, I would venture to guess that very few dollars for this campaign came from blacks. This campaign wasn’t manaaged or run by black folks, and the No on 8 campaign ignored the ethnic vote.

The blame for Prop 8 lies sqarely with the authors, promoters and financers of this propostion, and African Americans are very unrepresented in that group.

Priya Lynn
January 13th, 2009 | LINK

Mad Professor, that’s more blacks lynched than I would have thought.

Jaime Grant
January 13th, 2009 | LINK

As director of the NGLTF Policy Institute, I want to thank Jim Burroway for ultimately concluding that the authors of our recent election analysis report on Prop 8 – Ken Sherrill and Pat Egan “were successful in demonstrating that the Black vote may be closer to 58% than 70%.” At no point in the Task Force report do we make a claim that 58 percent is the precise answer. Rather, we stress throughout the study
that the range of data available to us leads us to the conclusion that 58% is much more accurate than 70%.

Burroway is justifiably concerned about sample size. The DBR survey includes 149 African Americans, making the margin of error for that population (as is typically calculated by pollsters) 8 percentage
points. Our analysis of this minority population is of course limited by its sample size in this survey. But unlike other polls, the DBR survey makes a deliberate attempt to rectify this problem by over-sampling African Americans, resulting in an African American sample that is at least
double the size of those found in typical surveys of Californians. This greatly augments the statistical power of our survey to detect differences among racial and ethnic groups.

Burroway says that we conclude that “religiosity explains the
differences in how African-Americans voted relative to everyone else.” This falsely characterizes our conclusions. We say rather that “controlling for
frequency of religious attendance helps explain why African Americans
supported Proposition 8 at higher levels than the population as a
whole.” In other words, if you’re trying to figure out why African Americans voted at higher rates for Proposition 8 than the general population, part of the answer is that they as a group are more religious than the general population–and religious people voted at high rates for Prop. 8. We show this quite clearly.

Thanks for your attention to this study, which we believe points to the value of LGBT-friendly faith based organizing in ballot measure campaigns. As all of us consider how to move the dial just a few more critical points toward marriage equality, taking a close look at the vote, while taking stock of our strategies to date, is an important next step.

Jaime Grant
Director of the Policy Institute

pat
March 12th, 2009 | LINK

Jim said “gays were never enslaved or lynched in mass numbers”.

Gays being exterminated in mass numbers during Nazi Germany doesn’t count for anything?

Same-Sex Marriage in DC and Polling « The NLGJA Blog
March 2nd, 2010 | LINK

[…] the NGLTF study isn’t exactly rock-solid either, as Jim Burroway and Timothy Kincaid at BoxTurtleBulletin have pointed out, raising questions about NGLTF’s […]

Leave A Comment

All comments reflect the opinions of commenters only. They are not necessarily those of anyone associated with Box Turtle Bulletin. Comments are subject to our Comments Policy.

(Required)
(Required, never shared)

PLEASE NOTE: All comments are subject to our Comments Policy.