In 1975, Civic Education in Ten Countries: An Empirical Study was published. The study was conducted by the International Association for the Evaluation of Educational Achievement (IEA), which had published results of international tests since the 1960s. The IEA is today most known for its studies TIMSS and PIRLS, measuring performance in math, science, and reading literacy, but its history and present work includes other surveys as well.
The study of civic education rested on the fact that all students, regardless of their country, learned about society and its political institutions. At the same time, civic education was taking place in societies that were very different from one another. With developing and developed societies, Western capitalist and Eastern socialist countries side by side, the study of civic education implied that widely different societies and ideas on education had to be captured by a single, standardized test. The ambition to do so proved challenging to live up to.
One of the challenges will be discussed in this article: how to make countries willing to participate. This willingness is one of the prerequisites of international surveys, one which has easily been taken for granted in a world where numerous countries take part in tests like PISA, but one which on closer inspection has a rather interesting history. The ability to recruit countries to a study includes an initial process of creating interest to participate, and a sustained effort to keep countries in the study. The early IEA studies were special in the sense that they were a long time in the making. The civics study took about nine years to complete, and during the period several countries left the study. The sources for tracing this process—available at the IEA archive located at Stanford University—are rich and have rarely been used in previous research. Correspondence, manuals, memorandums, minutes, drafts of tests, and other kinds of documents provide detailed insights into the deliberations of the organization (Landahl, in press), making it possible to investigate the processes by which social knowledge is produced (Camic et al., 2011).
More exactly, we can gain concrete insights into a broader feature of any international large-scale assessments: the navigation between international standardization and local adaptions. In other words, the article sheds light on the complex process of constructing international comparability. Previous research has referred to this phenomenon as commensuration: the comparison of different entities according to a common metric. Espeland and Stevens have observed that commensuration is the product of a labor-intensive process: “Commensuration as a practical task requires enormous organization and discipline that has become largely invisible to us. Commensuration is often so taken for granted that we forget the work it requires and the assumptions that surround its use” (Espeland & Stevens, 1998, p. 315). The effort to recruit countries demonstrates that the attempt to achieve commensuration was not without complications. Efforts to make phenomena comparable sometimes clashed with claims that certain aspects were incommensurable—that is, impossible to compare. Determining what is incommensurable is seldom self-evident, it depends on negotiations between actors that might have different views on the limits of comparability. As Espeland and Stevens note: “Just as commensuration is a considerable social accomplishment, so too the creation of incommensurables requires work. Some party must draw boundaries around the thing whose value is to be kept, or made, distinctive and then defend the boundaries from encroachment” (Espeland & Stevens, 1998, p. 328).
In other words, the question of what can be compared is a complex process, and both the claim that something can or cannot be compared must be legitimized through credible arguments. The ability to launch an international study can be viewed as the result of the negotiation between those two perspectives. It also involves power dimensions, where different actors might have different ideas about what kind of scientific knowledge is desirable or feasible to accomplish. Thus, the production of knowledge is always the result of a process where some ideas prosper while others are sidelined. In short, the production of scientific knowledge coexists with the non-production of knowledge. As Frickel notes, we need to ask ourselves: “Why are certain kinds of scientific knowledge created, certified and circulated while other kinds are not?” (Frickel, 2014, p. 263). Or put in other terms, why are certain forms of comparison accepted while others are not?
The Civics Survey was part of a larger project managed by the IEA, the so-called Six Subject Survey, covering civics, literature, science, English as a foreign language, French as a foreign language, and reading comprehension. The Six Subject Survey represented a period of expansion for the IEA, with new countries joining the organization and new subjects being studied. In total, 22 education systems participated in the survey, but only one country participated in all six studies: Sweden. Science attracted the most countries (19), followed by Reading comprehension (15 countries). Other studies attracted between 8 and 10 countries. The countries that took part in the civics study were relatively homogeneous in terms of political systems and levels of societal development. The final report identifies ten education systems that participated in the study: Germany, Finland, Iran, Ireland, Israel, Italy, the Netherlands, New Zealand, Sweden, and the United States. However, not all were fully included in the final report. Iran ultimately decided to drop out. Importantly, this withdrawal was preceded by other countries who initially were part of the study but decided to leave at different phases of the process. The reasons varied and were multifaceted.
Iran’s decision to leave the study was made late in the process, after field testing. As a result, only a portion of its data was included in the final report (Torney et al., 1975, pp. 17, 33). The report mentions the withdrawal of Iran, but gives no explanation as to why or when the country dropped out. Correspondence gives a clearer picture. Iran’s decision to withdraw was actually made very late, in September 1973. By that time, a draft of a chapter for the final report had already been circulated. The reasons for withdrawal were “the local climate of opinion,” and the numerous changes that had occurred in Iran since the studies were first organized (Hashemi, 1973, September 18, p. 1). This decision was problematic for the IEA since many statistical calculations already had been done, and since it was a developing country that could provide unique perspectives. Therefore, IEA requests were made that at least some data from Iran could be included, but this was rejected by the representatives of Iran: “We repeat, emphatically, that the Iranian National Center does not grant permission for the publication of the results of the Civics study” (Hashemi, 1973, October 6, p. 1).
Iran was not alone in leaving the study. Other countries that first joined but later withdrew from the test were Australia, Thailand, Hungary, Chile, and England. Australia withdrew early, stating in 1967 that it could not continue with the project, partly due to a lack of enthusiasm for the project (Radford, 1967). This motive is a reminder that international assessments depend not only on shared technical expertise, but also on a shared ethos, a belief in the value of certain studies being conducted.
Three months later, it was time for the next country to drop out. Thailand based its decision on the “apparent irrelevance of the test items and our civic education program,” indicating a gulf between what the test measured and what the schools taught (Saradatta, 1968, p. 1). In 1969, as the cognitive test-items and pre-test of attitude items had been presented, Hungary expressed a similar point as Thailand, and decided to drop out. In particular, it was the attempts to measure civic attitudes that caused the most problems:
Although most items of the cognitive tests are meaningful in our circumstances too, the construction of the Attitude Scales might cause a confusion in the cross-national analysis, because our aims and objectives in Civic Education deviate from yours in many respects. This fact raises another problem too, i.e. not all the aims and objectives of our civic education are represented in the tests. (Kiss, 1969, p. 1)
Yet another year passed, and yet another country—Chile—left the study. A 1970 letter describes how the National Committee had tried out the Civic Education test, but met resistance from teachers and students who “strongly resented the items.” It was felt that “even a thorough revision of the items would not remove the basic weaknesses of the test.” Some questions were deemed too easy and naïve, and others too subjective and based on a provincial conception of democracy (Husén, 1970, p. 1).
One by one, year by year, they left the study: Australia, Thailand, Hungary, Chile, and Iran. They participated to varying degrees, sometimes for a few months, sometimes for several years. The fact that countries could withdraw from a study even after participating for years is itself an illustration of the slow pace at which IEA studies were designed and administered.
In addition to these five dropouts, there was a sixth system that left the study: England. Its departure from the study is interesting given the strong dominance of Anglo-American actors in the study. The main authors of the final book consisted of two persons from the US and one from England. The International Civic Education Committee, responsible for planning the study, consisted of eight people, with two representatives from England and three from the United States, as well as one person each from Sweden, Italy, and Iran. It is a striking paradox that England ultimately decided not to participate in the civics study. The reasons, as stated in a 1971 IEA quarterly progress report, were financial (IEA, 1971).
That might be the true and only reason, but it is also evident that the English team, which was actively involved in the project for several years, had expressed quite hard criticism against the study. In 1968, Charity James, a British contributor to the Civic Education Study, made several complaints when the proposal for a pre-test was made. She wrote that the English group working on the civics study found the pre-test disappointing and less interesting than they had hoped. Some items were “unsuitable for use in the United Kingdom,” while others were “too naïve to make them acceptable for use in the United Kingdom.” There were also items where the level of difficulty was inappropriate for a particular school population, and items that suffered from technical defects. These flaws were serious enough to jeopardize English participation in the project, and poor test items would also risk damaging the reputation of the IEA (James, 1968, p. 1). When Russell Farnen, chairman of the International Civics Committee, received these objections, he wrote a long and detailed letter that almost looks like a reprimand, stressing the international nature of the survey and the inherent impossibility of creating test items that were suitable for each country. In other words, the need for international standardization made it impossible for any country to be satisfied with the design of the test. Furthermore, the need for some kind of international synchronization in the form of adherence to deadlines was emphasized. The English objections to the items threatened to delay the whole project (Farnen, 1968). The issue was infected, and Postlethwaite commented that the IEA Bureau—a small steering group—would have to spend some time trying to find a diplomatic solution (Postlethwaite, 1968).
Time passed, tests were modified, and joint discussions were held. In December 1969, more objections came from England, this time from Leslie Smith, who had succeeded Charity James on the International Committee for Civic Education. By this time the International Civics Committee had been active for more than two years, but Smith made suggestions that more or less signaled the need to start the whole endeavor over again. To begin with, he offered some basic suggestions for the development of the existing test. He wanted new testing techniques to be developed (not only multiple choice, but also open-ended questions), new hypotheses to be developed, new conceptual frameworks, and he opposed dividing the cognitive and affective domains into separate test items. He also proposed a far-reaching reorganization of the central group that ran the study. Apparently, he felt that the International Civics Committee was inadequate, for he proposed forming an entirely new “panel of experts” to make the necessary developments in the Civic Education Project. The panel was to include A. N. Oppenheim and possibly Judith Torney from the existing International Committee, but it was also to include expertise in several other fields, namely, political science, economics, law, civic education and social studies, principles of education, test construction, and, finally, someone familiar with the research processes of the IEA programmes (Smith, 1969, p. 1). Suggesting a new way of organizing collaboration indicates that the English participants had serious doubts about the nature of the project and saw a clear need for development. However, the timing of their criticism was questionable. The issue of the Civics Education Study had recently been discussed in the IEA Council, resulting in new proposals that had to be dealt with. Adding new proposals—including the appointment of a new group of experts—was controversial. One person who reacted negatively was Sixten Marklund, who was part of the International Civic Education Committee. He did not deny that there were some problems with the existing items and frameworks, but described them partly as the result of international compromises, and stressed that many of the proposals had already been considered in previous debates. His message was that the time for fundamental changes had passed. “In a project like this one finally comes to a point, where the leading group has to say: take it or leave it. We are now very close to this point” (Marklund, 1970, p. 2).
In addition to these six countries, correspondence also took place with France and Nigeria, both of which did not participate in the study (Farnen & Marklund, 1973). The fact that several countries expressed initial interest but remained outside had many reasons; closer examination of national sources would be needed to get a fuller picture. Yet, it is reasonable to assume that an important factor was the difficulties of coming up with an international standard that was considered valuable on a national level.
A country that remained in the study, but expressed much criticism, was Germany. Already when test items—many of which were used in all populations—were being constructed and tested, German opposition was heard. An undated document (likely from 1967) on draft tests, argued that there were too few questions dealing with democratic values, and too little emphasis on economic and social matters. The test methods were also questioned: “the shortcomings of the multiple choice method should not be overlooked” (IEA, n.d., p. 1).
The representative at the IEA Council, Walter Schultze, wondered at a 1970 meeting how it was possible to test all populations with the same test instrument. He wrote that several students in the oldest, pre-university population, considered many of the items to be stupid (“Blödsinn”) (IEA, 1970, p. 1). This was likely to some degree a reflection of the fact that the Civics test used similar questions for three different populations, the youngest aged 10. When the tests were administered, it became clear that this skepticism was widespread and had a significant impact on the results. A national report from Germany claimed that some students only completed the cognitive portion and refused to answer the affective scales, or treated the materials as a joke impossible to take seriously. In other cases, schools returned the tests blank, “refusing cooperation after they had seen the tests.” In the two oldest populations, almost 20% of the schools refused the civics tests (IEA, 1972, p. 6). Germany as a country did not withdraw from the study, but many of the individual schools did.
Other countries were not as critical. Ireland, for example, joined the IEA relatively late and only participated in the Civics study. They were eager to participate, although they had to overcome some domestic obstacles for participating. There was some skepticism against the IEA in the Irish Department of Education, so instead industrial firms had been approached for funding (Raven, 1970).
The Civic Education Study was a standardized test aimed for multiple countries. As such, its creation required collaboration across borders. As the case is with most standards, its creation was fundamentally a social act. While a single individual in principle can create a standard, “most standards are built collectively” (Timmermans & Epstein, 2010, p. 75). However, this collective process was also a highly complex one, plagued by tensions and withdrawals. A reason was the distinctly national character of civics education. While subjects like mathematics, science and foreign languages could be quite similar across space (or at least, so it could seem), it was quite obvious that civics was a subject aimed at introducing pupils to a national culture, whether in the form of national literary work or national ways of governing a society, making it difficult to construct an international, standardized test. The civics test does therefore clearly illustrate the inevitable clash between commensuration and incommensurables that all international tests face.
The civics study was directly linked to the social systems and political ideologies of the respective countries. Given this, it is not surprising that the process of creating an international, standardized, test was fraught with conflict, causing some countries to opt out. The ability to hold the scientific community of the IEA together in this particular study was challenged, at least in part, by a world that was fragmented—culturally, politically and economically. A non-standardized world proved inhospitable to a standardized test. Not in the sense that the whole test imploded, as some pessimists had feared early on, but in the sense that several countries chose to withdraw, while others expressed rather strong concerns.
Given those tensions, repeating the study in the future was perhaps an unlikely idea. It would not have been surprising if the survey would have remained a one-time event: a bold or perhaps naïve experiment in the early phases of international assessments that highlighted the limits of comparability as much as it produced the actual comparisons. It would be understandable if future projects would focus on more tangible and straightforward aspects of schooling than civic education. However, after several years, the idea of studying civic education was revived. In 1993, during the IEA General Assembly in Spain, Judith Torney-Purta was invited to prepare a proposal for a new civic education study. Much had happened in the world since the last study, not least the end of the Cold War, and many new countries had joined the IEA. CIVED, the IEA Civic Education Study, as the study was eventually called, was chaired by Torney-Purta, the lead author of the first Civic Education survey (Torney-Purta et al., 1999). Strikingly, 11 post-socialist countries participated in this survey (Torney-Purta & Schwille, 2011, p. 103). It was soon followed by a series of surveys called the International Civic and Citizenship Education Study (ICCS), conducted in 2009, 2016, 2022, and one planned for 2027. The existence of these surveys shows that the idea of studying civic education, after all, proved durable. While the number of countries participating in these studies is lower than in studies such as PISA and TIMSS, it is evident that the quest for international studies of civic education has endured.
References
Camic, C., Gross, N., & Lamont, M. (Eds.). (2011). Social knowledge in the making. University of Chicago Press.
https://doi.org/10.7208/chicago/9780226092102.001.0001
Espeland, W. N., & Stevens, M. L. (1998). Commensuration as a social process. Annual Review of Sociology, 24(1), 313–343.
https://doi.org/10.1146/annurev.soc.24.1.313
Farnen, R. (1968, May 22). [Letter to Charity James]. Hoover Institution Library & Archives Stanford, United States (hereafter Hoover, IEA), box 62.
Farnen, R., & Marklund, S. (1973, June). Civic education and curricular processes in twelve countries: A comparative analysis. [First draft of unpublished book manuscript]. Hoover, IEA, box 71.
Frickel, S. (2014). Not here and everywhere: The non-production of scientific knowledge. In D. L. Kleinman & K. Moore (Eds.), Routledge handbook of science, technology, and society (pp. 263–276). Routledge.
https://doi.org/10.4324/9780203101827.ch15
Hashemi, E. (1973, September 18). [Letter to Russell Farnen]. Hoover, IEA, box 242.
Hashemi, E. (1973, October 6). [Letter to Roy Phillipps]. Hoover, IEA, box 242.
Husén, T. (1970, August 10). [Letter to Richard Noonan]. Hoover, IEA, box 54.
IEA. (n.d.). Comments on tests for pop. II and III. Hoover, IEA, box 59.
IEA. (1970). Minutes of I.E.A. Council Meeting R.I.T.P., Herengracht 520, Amsterdam C, 26 – 30th October, 1970. Ford Foundation records, Rockefeller Archive Center, r-2033.
IEA. (1971). IEA quarterly progress report, March 1971–May 1971. Hoover, IEA, box 60.
IEA. (1972). National report on stage 3 testing, Federal Republic of Germany. Hoover, IEA, box 192.
James, C. (1968, May 14). [Letter to Douglas Pidgeon]. Hoover, IEA, box 62.
Kiss, Á. (1969, ca. September). [Letter to Neville Postlethwaite]. Hoover, IEA, box 59.
Landahl, J. (in press). Global knowledge in the making: The rise of international testing in education. Palgrave Macmillan.
Marklund, S. (1970, January 19). [Letter to Torsten Husén]. Hoover, IEA, box 62.
Postlethwaite, N. (1968, June 4). [Letter to Torsten Husén]. Hoover, IEA, box 357.
Radford, W. C. (1967, October 11). [Letter to Russell Farnen]. Hoover, IEA, box 59.
Raven, J. (1970, August 7). [Letter to Neville Postlethwaite]. Hoover, IEA, box 441.
Saradatta, L. (1968, January 18). [Letter to Neville Postlethwaite]. Hoover, IEA, box 59.
Smith, L. (1969, December 22). [Letter to Douglas Pidgeon]. Hoover, IEA, box 62.
Timmermans, S., & Epstein, S. (2010). A world of standards but not a standard world: Toward a sociology of standards and standardization. Annual Review of Sociology, 36(1), 69–89.
https://doi.org/10.1146/annurev.soc.012809.102629
Torney, J. V., Oppenheim, A. N., & Farnen, R. F. (1975). Civic education in ten countries: An empirical study. Almqvist & Wiksell International; Wiley & Sons.
Torney-Purta, J., & Schwille, J. (2011). The IEA studies in civic education: Their history from the late 1960s to 2005. In C. Papanastasiou, T. Plomp, & E. C. Papanastasiou (Eds.), IEA 1958–2008 (pp. 98–110). Cultural Center of the Kykkos Monastery.
Torney-Purta, J., Schwille, J., & Amadeo, J.-A. (Eds.). (1999). Civic education across countries: Twenty-four national case studies from the IEA Civic Education Project. IEA.
Recommended Citation
Landahl, J. (2026). Recruiting countries to standardized surveys: The case of the IEA civics study. On Education. Journal for Research and Debate, 9(24).
https://doi.org/10.17899/on_ed.2026.24.4
Do you want to comment on this article? Please send your reply to editors@oneducation.net. Replies will be processed like invited contributions. This means they will be assessed according to standard criteria of quality, relevance, and civility. Please make sure to follow editorial policies and formatting guidelines.
