Pissin’ in the wind….

 Click for cute figure with 2008 GSS Data!   


In March of 2009, the most important survey of American religion was released, but nobody noticed. While media pundits and religious activists tout the findings from surveys with very low response rates and minimal comparability to other studies, the 2008 edition of the National Opinion Research Center’s General Social Survey became public access sometime in March. I’m a regular user, and I couldn’t tell you precisely when the new GSS became available. There was no press conference, and this will be the first you’ve heard of what’s really going on in American religion.

     The cover of USA Today and almost ever other publication presented findings from the American Religious Identification Survey (ARIS) data, which is a nice little study, for some purposes. Last year, findings from the Pew data generated a media circus, and before that, it was the Baylor survey.  What the ARIS, Baylor, and Pew studies have in common are exceptionally low response rates. Few of the targeted respondents actually complete the interview–about 20%, perhaps a little higher in ARIS, Baylor, and Pew. In contrast, the lowest response rate ever achieved in a GSS survey has been 70%, and the 2006 and 2008 surveys garnered 71% of targeted respondents.

     Studies cannot be compared when they use different questions, so the pissing match between pundits and investigators of these studies has no scientific merit. The question asked in ARIS is unique, and interesting. But, you cannot compare the distributions of the ARIS question on religious identification to distributions of religious ties based on other ways of asking the question. Given the exceptionally low response rates of the Baylor, ARIS, and Pew data, legitimate scholars should be embarrassed for having made claims about population parameters which cannot be discerned with reasonable certainty.

     What is really going on? GSS data provide a much more accurate summary of the trends based on comparable data collected over time. Below is a table cumulating evidence from the GSS surveys from each decade. As you can see, the percentage with no religion has increased dramatically; from under 7% in the 1970s, to over 15% in the 2000s. Non-Christians move from under 11% of respondents in the 1970s, to over 20% of respondents since 2000. Notably, in the just-released 2008 GSS data presented above in Figure 1, 16.5% of GSS respondents reported no religious identification, and 21% are non-Christian. This is a higher proportion of non-identifiers than is reported in either the ARIS or Baylor studies, probably because non-identifiers are more likely to be non-respondents in studies with low response rates.

      Further, contrary to ARIS and Baylor, high quality data from the GSS show that the proportion of sectarian Christians (the proper term for what journalists and activists call “evangelicals”) is in decline. Most of the decline comes for Baptists, who are shrinking. But, other sectarian groups peaked in the 1980s and now hold the commitments of fewer Americans.

     Religious identification is declining in the United States. Anyone who claims that irreligion is not growing is simply wrong. And, claims that conservatives continue to grow are based on exceptionally weak evidence. Should we believe ARIS or Baylor? The answer is neither. The GSS shows no growth among sectarian Protestants since the 1990s, and Baptists (the largest sectarian group) are in decline.

     One in five Americans rejects religious identification. The 2008 GSS shows that 20% of Americans believe that the Bible is a book of Fables (up from under 14% in 1984, the first year the GSS asked that particular question), and that 18% do not believe in a “god” (up from 13% in 1988, the first year that particular question was asked).

     Religion isn’t going away, but rejection of religion is growing. Americans have long been forced to accept the hegemony of Christianity, but that hegemony is breaking down, and secular Americans cannot be ignored by anyone serious about contemplating the vagaries and trajectory of American culture.



 Table 2.1  


Trends in Religious Identification: GSS 1972-2006







Liberal Protestant












Moderate Protestant






























Other Protestant
























Other Religion



















8 Responses to “Pissin’ in the wind….”

  1. Conrad Hackett Says:

    I just discovered your blog and have enjoyed reading through it!

    Yes, the GSS has higher quality data than the other big religion surveys that have attracted media attention lately. What surprises me is that with each new survey we get the same headlines about the growth of the non-affiliated as if this story was only unearthed in the latest survey on the scene rather than something that we saw in the GSS years ago.

    I agree we have to be cautious about how the low response rates of ARIS and the Pew Landscape Study taint the reliability of their findings. However, it seems to me that without much more generous funding, low response may be a necessary tradeoff for the large samples that allow, for example, ARIS to make claims about state level religious composition and for the Pew survey to provide demographic profiles of relatively small religious groups. (I can hear the objection–“low response rates mean none of these sub-population estimates are reliable.” But drop ARIS and Pew and none of our standard surveys – GSS, NSFG, NES, Add Health, NLSY, etc. provide much detail on state level religious composition or the current demographics of relatively small religions. I’ll take suboptimal data over no data but I acknowledge that it is difficult to quantify the tradeoff in precision caused by low response rates and that this must be addressed directly in scholarly work that draws on such sources.)

    Non-response patterns are a problem. Though the GSS is relatively good in this regard, it may not be trivial that 3 out of 10 potential GSS respondents are missing. I haven’t looked carefully at differences between the various surveys you mention in terms of their estimates of the evangelical population or the size of the non-religious population but at least with regard to the growth of non-affiliates, it seems they are describing the same trend and producing reasonably similar estimates.

    In this post you suggest that the non-GSS surveys suffer because they are less likely to include the presumably well-educated, progressive non-affiliates. You say, “non-identifiers are more likely to be non-respondents in studies with low response rates.” A few months ago, when discussing surveys about race and politics, you had a very different take on non-response patterns. At that time you claimed that politically conservative respondents (who I presume are also religiously conservative) were more likely to be omitted from survey samples. You wrote, “Everything we know about survey non-response bias (which isn’t much) tells us that the people who aren’t responding are more conservative.” (https://iranianredneck.wordpress.com/2008/10/13/be-afraid-be-very-afraid/).

    How do you reconcile the above positions? Do you believe that a large portion of non-responders are simultaneously secular, racist, and conservative?

  2. Conrad Hackett Says:

    Incidentally, how available is the latest round of the GSS? Release of the latest round isn’t mentioned on NORC GSS pages, nor can I find the data at ICPSR or Roper. The only source I can find is the Berkeley SDA site, which describes the data as preliminary, allows for some analysis, but does not allow for the latest wave to be downloaded.

    The last issue of GSS News mentioned that the 2008 data was supposed to be released in Dec. 08 or Jan. 09. It would be nice if there was some kind of list-serv to keep updated on GSS developments and discuss methodological issues, such as those that arise with the introduction of panels within the new waves.

  3. sherkat Says:

    I don’t think low data quality is a solution to the funding problem. In fact, what we’re seeing is multiple funding agencies essentially wasting money collecting bad data, rather than pooling resources and coordinating high quality research.

    When response rates get down to 20% the biases are so many that it’s hard to determine. What you get, I think, is a generic mash of mostly moderate and lilberal Protestants. You miss normal people who are hard to get (an exceptionally diverse group), you miss marginal populations, and you miss the most conservative Christians who hold misanthropic views of outsiders.

    GSS 2008 are out! I don’t know when. I got them for free from NORC, not even logged into my ICPSR account. Drop me an e-mail if you have trouble finding the link. They used to have a GSSNEWS listserv, but I don’t think that operates anymore.

  4. Conrad Hackett Says:

    Interesting points. I would like to attend a panel at some sociology/ soc of religion conference featuring a debate about the merits of low-response rate surveys like ARIS, all Pew surveys, and the Baylor Religion Survey. Clearly many scholars are invested in such surveys and the surveys seem to have considerable credibility with the media even if they are held in suspicion by many scholars. Darren, didn’t you post on this blog that you have been working with the ARIS team on their data? Do you think some of this data is useful, or not? Any chance you, Barry Kosmin, Rodney Stark, John Green, or other critics/advocates would tackle this at the next SSSR in Denver? It would be good for all of us to be clear about the strengths and limitations of these data sets. I know such conversations occur in private and during the journal article submission process but I think it would be profitable to have a more public discussion.

    I dug around further on the NORC site and found their new GSS portal (http://www.norc.org/GSS+Website/). It is a huge improvement over the old version! It is nice to be able to download the questionnaires from each wave to see what questions were fielded in given years (though this is complicated by the multiple ballot issue).

  5. sherkat Says:

    There is no debate. This isn’t about competing opinions. Samples with response rates of 20% cannot be used to answer questions about broad population parameters, like the % non-religious. They shouldn’t even be used to speculate about such things. To say otherwise is disengenous at best, and moronic at worst. People are making arguments based on their opinions, religious desires, and (more often) their material interests in procuring more money from religiously motivated foundations. I have no interest in “debating” such issues. If people want to find out what is really going on, they can attend one of my talks, where I will present research based on the highest quality data and using the most appropriate statistical models to draw reasoned conclusions from those data.

    Plus, SSSR conflicts with cyclocross season.

    totally awesome what NORC has done!

  6. Conrad Hackett Says:

    As much as I would like to take advantage of unique features of the data/sample size in ARIS, Pew data, and the Baylor Religion Survey, the criticisms you mention are a significant problem. Despite your natural suspicion of those behind these surveys, I would like to hear them defend their data against critics in a public forum. We have author meets critics sessions–why not major national religion study meets critics sessions?

    I know there are many other critics of these studies out there, including scholars who use money from religiously motivated foundations to field surveys with much higher response rates (and I doubt they will be competing in cyclocross races in the fall!). Strong opinions, religious desires, and material interests can coexist with rigorous scholarship, though this is not always the case. I wonder if any parties would be willing to participate in the type of discussion I propose.

    • sherkat Says:

      You are right, Conrad. I’m just being a dick. And, I don’t deal well with those kind of events. I’m fine if I can stay on script on a particular specific research topic, but free form discussions tend to bring out the asshole in me. Actually, Ryan Cragun is planning a session like that for SSSR, and he’s worked with ARIS and GSS data. I’m not saying that Baylor or ARIS data are worthless, they can give good estimates, particularly for relationships among variables. Aris and Pew have the advantage of numbers and geographic scope. Baylor and Pew have rich sets of questions, but are one point studies. Baylor is not a large N study, that is often misreported (as it was in USA Today). I don’t think the studies are compromised by the views of the principals, but there is a deliberate bias in public presentation, and in this case it has been pretty direct (and generally over something neither study can really speak toward). Frankly, I wasn’t going to say shit about any of this, but then I read some essay in the WAPO by an atheist activist calling Stark an ostrich and Kosmin a meerkat, and the author concluded that you can just pick whichever numbers you want and construct your own reality. That drove me over the edge. I think Ryan is going to ask Michael Hout, but you should suggest some people! I’m not really skipping the meetings for ‘cross. I’ve been to way too many meetings and I’m trying to scale back. It would probably cost me close to $1k out of my pocket to go to Denver, and I’m coming off being on half salary for my sabbatical. Plus, it’s cross season…

  7. Conrad Hackett Says:

    I’ll have to touch base with Ryan.

    Agreed on advantages and disadvantages of various studies.

    Had to Google the article you mentioned: http://newsweek.washingtonpost.com/onfaith/guestvoices/2009/03/america_land_of_the_free_think.html
    — very interesting. Not the first or the last leader of a group with preferences for the way one study suggests his group is large and distaste for the study which suggests his group is small. Too bad that rather than get into the significant methodological issues, he simply concludes by saying we can pick whichever data we like.

    The tension between the Baylor study and the others concerning the nonreligious was foreshadowed by the response to Hout and colleague’s ASR piece highlight the big jump in no religion in the GSS. The response basically pointed out that most of those who claim no religion have some vestiges of religious belief. And without reading the study carefully, this seemed to be the point of the Baylor Survey, which asked specific questions to more accurately measure weak religious commitment (including weak congregational ties, I think). Europeans have a similar debate about secularism versus believing without belonging.

    It seems worth noting that the methodological issues at stake in debate about the size of the nonreligious population may have little to do with response rates. This discussion should be about operationalizing concepts rather than sampling methodology, even though we know that sampling biases are likely to affect estimates of religious populations (I like Bob Woodberry’s short ASR piece on this topic).

    ARIS and GSS agree that 15-16% of Americans claim no religion, and this may be the one religious category in which ARIS and GSS estimates are comparable. One can use a plethora of surveys, including the GSS and BRS, to demonstrate that people who claim no religion tend to have some degree of religious belief. And of course, claiming identity as an atheist in America is very distinct from being religiously indifferent.

    I enjoy your SSSR presentations–sorry that you will miss the next meeting. I use Priceline to make my personal and professional travel budget go much further. I will be a good professional organization citizen and pay the full conference hotel fees one day, but not quite yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: