Trouble is brewing in the arcane world of humanities bibliometrics, and it looks as if a major debate on the measurement of what constitutes "quality" in scholarship in fields such as archaeology and classical studies is about to begin.
The catalyst for this is a European Science Foundation initiative to compile a European Reference Index for the Humanities (ERIH), defined (in its first iteration) as "a reference index of top journals in 15 areas of the Humanities, across the continent and beyond" but due to expand "to include book-form publications and non-traditional formats" with the aim of eventually operating "as a backbone to a fully-fledged research information system for the Humanities." Wow!
In 2007, with little fanfare, the first ERIH lists were published, presenting what aimed to be comprehensive catalogues of journals (ca. 1,300 in total and almost 300 in Classical Studies alone), each with an A, B, or C classification. Although the Committees of top scholars in each discipline who compiled them emphasized that "the lists are not a bibliometric tool" and that they therefore advised "against using the lists as the only basis for assessment of individual candidates for positions or promotions or of applicants for research grants," the presumption that universities and other organizations would not start to use them in this way was naive.
The reality is that funding agencies, university administrators, library acquisitions staff, and hiring committees alike have been desperate to find some objective way of measuring the quality of humanities research for years. Although subject to increasing criticism (and attempts to find a web-based metric using Google-like algorithms), the citation-based "impact factor" has been an acceptable measure of article quality in the sciences for decades, since Eugene Garfield invented the measure and later institutionalized it by selling his Institute for Scientific Information (ISI) to the mighty Thomson Corporation. Journals in the humanities, in the meantime, tend to be ranked on the basis of the extremely qualitative and fuzzy scale of "peer perception" which understandably drives busy bureaucrats within the higher establishment wild. There is an Arts and Humanities citation index, and sometimes I will get a panicked call from a junior scholar whose Dean has asked what Hesperia's impact factor is, but the AHCI index has never been widely accepted and ISI does not provide a means of extracting an impact factor from it. Among other problems with AHCI, its coverage of journals is limited and it doesn't acknowledge the important role books play in the humanities.
In an aside, related to previous discussions on this blog and elsewhere about the segmented and "tribal" nature of disciplines like archaeology, I once heard a rumor that ISI didn't produce an impact factor for the Arts and Humanities partly because their statistical analyses tended to find "odd clumping" when analyzing humanities journals, perhaps explained by the tendency of some sub-disciplines to almost exclusively cite themselves. However, I asked Eugene Garfield a question about this at a meeting of the Society for Scholarly Publishing a couple of years ago, and he claimed that his algorithm would work just as well for humanities journals as for sciences. Statistics is far from being my strong suit, so I didn't pursue this further.
In this context, ERIH sounds like an attempt to turn a qualitative measure of "peer perception" into something quantitative, and it undoubtedly has some good motivations.
Firstly, while the method by which the expert panels selected were chosen is obscure, the people on them are distinguished. For Archaeology, the panel consists of Lin Foxhall (Chair), University of Leicester (UK); Csanád Bálint, Hungarian Academy of Sciences, Budapest (HU); Serge Cleuziou, CNRS / Nanterre (FR); Kristian Kristiansen, Göteborgs Universitet (SE), and Jacek Lech, Polish Academy of Sciences, Warzaw (PL). For Classical Studies, Claudia Antonetti (Chair), Università Ca' Foscari, Venice (IT); Angelos Chaniotis, Universität Heidelberg (DE); Antonio Gonzales, Université de Franche-Comté, Besançon (FR); Richard Hunter, University of Cambridge (UK), and Paul Schubert, Université de Genève (CH).
Secondly, the compilation of lists that include important journals from developing as well as developed countries and new periodicals is extremely praiseworthy since almost any library catalogue is incomplete in this regard, and the high quality work of publishing colleagues in some of the new European countries is often unfairly ignored.
Thirdly, humanities is probably losing out on funding by not catering to bureaucrats. ERIH proponents, thinking in European terms, argue that a quantitative measure of humanities research quality would enable the humanities to compete alongside the sciences to access the €7bn funding provided by the European Research Council each year. At a more provincial level, it is probably true that one reason why high-quality but independently-published journals like Hesperia that are seeing steadily declining numbers of institutional subscribers is that librarians don't have a quantitative measure of quality to rely on when making their choices, and therefore tend to make scattershot decisions to subscribe to large commercial packages in the hope that they will hit some of the "core" periodicals. For independent journals as well, quantitative measures in the humanities would probably level the playing field.
On the other hand, a growing body of academics, especially in the traditionally Euro-sceptic UK, are spotting problems with ERIH. They seem to be led, one is somewhat proud to note, by the traditional "awkward squad" disciplines of archaeology and classics.
A good summary of the arguments against ERIH can be found in the PDF minutes of a meeting of the Arts and Humanities Research Council in the UK on February 27, 2008, subsequently reported in the Times Higher Education Supplement of March 19, 2008. Also worth watching may soon be the website of the Council of University Classical Departments (CUCD) which seems to be leading the opposition to ERIH under the control of their distinguished Chairman, Robin Osborne.
The main problems identified were:
1. By categorizing journals into disciplines, the importance of interdisciplinary journals is understated. A single list of journals would be more useful than 15 separate ones.
2. The methodology on categorization (involving quantitative data on % of authors from different countries, acceptance rate, level of peer-review but also qualitative data about who is on the advisory board etc.) is obscure and unscientific. However good the expert panels are, surely their own research preferences and integration into networks would show.
3. The lists were not complete, especially in regard to non-European including North American periodicals, and some of the journals listed were defunct.
The biggest concern was that a system like ERIH that even its proponents agreed was till in a beta-phase was already being to make hiring and funding decisions. Although the evidence for this was anecdotal, the probability that, after such a long period of frustration over the lack of quantitative measures of humanities research quality, ERIH would not be used by data-starved administrators seemed low.
In recent weeks, the generators of ERIH have clearly been acting to head off its critics. For the first time last week, Hesperia received formal notification of the project and a feedback form with which to comment on our rankings. How did we do? Not too bad with category "A" (for "high ranking, international level publication") in Classical Studies and Archaeology and "B" (for "standard, international level publication") in History. The initial list in Art and Art History, another important field in which the journal publishes, is yet to be announced.
It's nice to get grade "A"s, so perhaps my decision to suspend judgement on ERIH for the moment is biased. Peer review works for the contents of journals, so why shouldn't it work for compiling lists of journals themselves? How else would an obvious gap in the market for information on humanities publications be filled than by a major international initiative? Will ERIH's promise to index publications in "non-traditional formats" in the future provide objective measures for the quality of electronic publications that have so far been poorly recognized by employers? However, I also see a lot of validity in the criticism of the project which seems to have been unduly secretive in its development and perhaps naive in its implementation. The important debate about how to measure the quality of publications in the humanities that ERIH has reopened is definitely one to watch.
Saturday, April 5, 2008
Archaeology as Ecology?
I was extremely flattered and amused last week to have stimulated a debate about ecology through my rather glib identification of some of the giants in our field of broader Mediterranean archaeology (Hodder, Davis, Joukowsky, Rose) as "grey panthers," who are confident enough of their positions to take the risk of sharing data.
I think that the distinguished colleagues who have recently identified themselves as "archaeological data critters" (Eric Kansa's great term) are downplaying their role in the food chain; Bill Caraher is more capibara (uber-rodent and one of my favorite animals) than squirrel, Eric Kansa is more an eagle than a bluebird, and Tom Elliott is more like the nicest kind of giant pan-galactic gorilla than cranky space monkey. As for me, Tom's reference to "the Watkinson Archaeological Cyberverse" momentarily transformed me into one of those tiny tree-frogs that temporarily conceal their true minuteness by puffing up.
Before I deflate and go back to my hole in the tree, however, I do want to make an additional comment about the idea of there being different kinds of "archaeological critters" and its relevance to larger debates about how to support emerging digital scholarship.
The use of a biological metaphor in discussions of how scholarship works is, of course, not new. The idea of an "ecology of scholarly communication" seems to be cropping up more and more in discussions of the respective roles of libraries, press, and IT departments in supporting the "information life-cycle" from production, through management and preservation, to dissemination. The nature of academia as an interdependent and complex system of groups with their own behavior patterns is often discussed in terms of either "species" or "tribes" (e.g., the bestselling book, Academic Tribes and Territories: Intellectual Enquiry and the Cultures of Discipline).
Any of us involved in archaeology recognize that our discipline must be one of the most segmented fields of study out there. Look at conferences, for example. As the veteran of many conference displays in a past job at The David Brown Book Company, I know that hardly anybody who goes to the Society of American Archaeology (SAA) meetings attends the Archaeological Institute of America (AIA) or the Society for Historical Archaeology (SHA). Funding constraints mean that most scholars can only attend one big conference a year, and this economic "accident" perpetuates disciplinary divide. Even within an AIA meeting, a multitude of different worlds with their own logics can be observed. Individual conference identities are shaped by scholarly specialty, institutional affiliation, age cohort, qualitative or quantitative orientation etc., and groupings form and reform in a way that is probably better described in terms of theories of "ethnicity" than biology.
The recognition that, in academic terms, fields like archaeology consist of many sub-disciplines with their own logics and modes of presentation and discussion is very clear when working with authors. There are notable exceptions, but my experience in publishing classical archaeology, for example, is that epigraphers and amphora specialists tend to be extremely territorial; reviewers tend to be very opinionated and outspoken, citations seem to always be to a limited range of sources, data sharing seems unusual, and there is usually a long time-delay between discovery and publication.
In a commentary on the "archaeological critters" discussion, my friend and sadly-missed ex-colleague Chuck Jones (some kind of good bear, I think) asked the following question: "it occurs to me that some rather ambitious projects seeking to offer generalized platforms for the archiving and distribution of archaeological data have not yet come up in the conversations, and I wonder why not."
Understanding that there are multiple species, even in a small ecological zone like archaeology, is important for the designers and funders of the evolving cyberinfrastructure. The recognition that "archaeological informatics" is a specific field of information science is becoming more widespread. Archaeology is different from computer science, economics, medicine or physics in the way researchers use, produce, manage, and disseminate information. However, we all need to also recognize that is that the discipline is internally segmented to a degree that (a) any overly ambitious "global" repository- or digital toolkit-building exercise will only gain very limited acceptance, unless it tailors itself to support a range of different communities, and (b) perhaps the most logical level at which to sustain repositories and digital tools is at a "species" rather than "ecological zone" level, meaning that it is likely that each discipline will remain fairly wary of solutions created by "outsiders" (e.g., Americanist archaeologists in the Mediterranean).
Of course, the ecological metaphor loses its usefulness after a certain point. After all, in our quest for interoperability between digital repositories, is what we are really trying to do is to get different species to interbreed? (If so, no wonder it's so difficult!) And, anyway, what kind of sick, demented kind of research project is that? None of the pleasant and visionary archaeological informaticians I have met look like Dr. Moreau.
I think that the distinguished colleagues who have recently identified themselves as "archaeological data critters" (Eric Kansa's great term) are downplaying their role in the food chain; Bill Caraher is more capibara (uber-rodent and one of my favorite animals) than squirrel, Eric Kansa is more an eagle than a bluebird, and Tom Elliott is more like the nicest kind of giant pan-galactic gorilla than cranky space monkey. As for me, Tom's reference to "the Watkinson Archaeological Cyberverse" momentarily transformed me into one of those tiny tree-frogs that temporarily conceal their true minuteness by puffing up.
Before I deflate and go back to my hole in the tree, however, I do want to make an additional comment about the idea of there being different kinds of "archaeological critters" and its relevance to larger debates about how to support emerging digital scholarship.
The use of a biological metaphor in discussions of how scholarship works is, of course, not new. The idea of an "ecology of scholarly communication" seems to be cropping up more and more in discussions of the respective roles of libraries, press, and IT departments in supporting the "information life-cycle" from production, through management and preservation, to dissemination. The nature of academia as an interdependent and complex system of groups with their own behavior patterns is often discussed in terms of either "species" or "tribes" (e.g., the bestselling book, Academic Tribes and Territories: Intellectual Enquiry and the Cultures of Discipline).
Any of us involved in archaeology recognize that our discipline must be one of the most segmented fields of study out there. Look at conferences, for example. As the veteran of many conference displays in a past job at The David Brown Book Company, I know that hardly anybody who goes to the Society of American Archaeology (SAA) meetings attends the Archaeological Institute of America (AIA) or the Society for Historical Archaeology (SHA). Funding constraints mean that most scholars can only attend one big conference a year, and this economic "accident" perpetuates disciplinary divide. Even within an AIA meeting, a multitude of different worlds with their own logics can be observed. Individual conference identities are shaped by scholarly specialty, institutional affiliation, age cohort, qualitative or quantitative orientation etc., and groupings form and reform in a way that is probably better described in terms of theories of "ethnicity" than biology.
The recognition that, in academic terms, fields like archaeology consist of many sub-disciplines with their own logics and modes of presentation and discussion is very clear when working with authors. There are notable exceptions, but my experience in publishing classical archaeology, for example, is that epigraphers and amphora specialists tend to be extremely territorial; reviewers tend to be very opinionated and outspoken, citations seem to always be to a limited range of sources, data sharing seems unusual, and there is usually a long time-delay between discovery and publication.
In a commentary on the "archaeological critters" discussion, my friend and sadly-missed ex-colleague Chuck Jones (some kind of good bear, I think) asked the following question: "it occurs to me that some rather ambitious projects seeking to offer generalized platforms for the archiving and distribution of archaeological data have not yet come up in the conversations, and I wonder why not."
Understanding that there are multiple species, even in a small ecological zone like archaeology, is important for the designers and funders of the evolving cyberinfrastructure. The recognition that "archaeological informatics" is a specific field of information science is becoming more widespread. Archaeology is different from computer science, economics, medicine or physics in the way researchers use, produce, manage, and disseminate information. However, we all need to also recognize that is that the discipline is internally segmented to a degree that (a) any overly ambitious "global" repository- or digital toolkit-building exercise will only gain very limited acceptance, unless it tailors itself to support a range of different communities, and (b) perhaps the most logical level at which to sustain repositories and digital tools is at a "species" rather than "ecological zone" level, meaning that it is likely that each discipline will remain fairly wary of solutions created by "outsiders" (e.g., Americanist archaeologists in the Mediterranean).
Of course, the ecological metaphor loses its usefulness after a certain point. After all, in our quest for interoperability between digital repositories, is what we are really trying to do is to get different species to interbreed? (If so, no wonder it's so difficult!) And, anyway, what kind of sick, demented kind of research project is that? None of the pleasant and visionary archaeological informaticians I have met look like Dr. Moreau.
Tuesday, April 1, 2008
Only Panthers Share Archaeological Data
Sebastian Heath's comments on my last post, where I made some rather self-important statements about the problems of linking data and publication, are (as one would expect) thoughtful and stimulating.
Sebs is right to pick me up on my glib statement that "nobody wants to share their data," and he presents some good examples of admirable projects that are taking a lead in being transparent and open with their findings. As I'll suggest in a moment, however, I think that the projects he highlights are the exceptions rather than the norm, and there are some distinctive features that make them exceptional.
Firstly, however, I would like to point out that I am not alone in my pessimism about the readiness of scholars to share. Since my last post also referred to it vigorously, it may seem that Christine Borgman's synthetic Scholarship in the Digital Age (2007) is the only book I have read. It's not quite (I have just re-read the last Harry Potter!), but I am finding some good stuff in it, especially in Chapter 8 where the author explores four areas of concern that discourage information sharing among scientists, social scientists, and humanists alike:
(1) Reward: Scholars are rewarded for publication through promotion and tenure but rarely acknowledged for managing their information. This means that they feel little obligation to self-archive data for others to use.
(2) Effort: The preparation of data so that it is useful to others is a huge effort. The logical fieldwork database structure is less than half the story. The documentation so that the information is useful to others is the real sweat.
(3) Priority: Although archaeology isn't medicine, being first with information is still the key to prestige and future funding.
(4) Ownership: Confusion about intellectual property rights and related concerns about the ownership and control of data make the prospect of sharing information risky.
These worries ring true to me for archaeology, and I find two of the points particularly persuasive. Having just spent a day watching my editorial colleagues almost literally tearing their hair out over a poorly organized pottery catalogue, I think that it is hard to underestimate the amount of thought that needs to go into preparing raw information so that it is useful (point 2 above). Archaeological data is socially constructed, so it needs context to be properly interpreted. There are norms of presentation to obey, concerns about interoperability to address, and a multitude of structuring decisions that need to be made to differentiate a catalog from a data dump. The question of "when" the records of a project become data is a big one in our field.
Although intellectual property concerns (point 4 above) are often linked to "copyright" and the naughtiness of publishers, other concerns that archaeologists have to wrestle with over the ownership and control of data seem to me to be much more urgent. In multi-institutional projects, which director's institution owns the copyright in the work its staff member is doing (remember that it is usually employers rather than employees who own the rights to work done on "company time")? When working overseas, what rights does the host nation have over the information being extracted? (Mexico, Greece, and France have recently attempted to tax photographs of their "patrimony"). These are all scary considerations, and issues that are made more frightening by being poorly understood.
But let's move back to the examples of good sharers Sebs brings up; Jack Davis with PRAP and MRAP, Ian Hodder at Çatalhöyük, Martha Joukowsky at Petra, and Brian Rose at Troy. Let's face it, Sebastian, these are legendary names, the "gray panthers" who have nothing to prove. Tenured, funded, at the top of their profession, they have little need for further reward, have access to some of the best minds around to help shape their data for other users, have less need than others to retain the right to priority, and are savvy in their abilities to navigate the intellectual property minefield. If you are a powerful feline, the obstacles to data sharing drop away.
When we talk about sharing, we need to look more at scholarly behavior at the starting-out level. Think graduate students and untenured faculty, the baby armadillos and raccoons rather than the panthers of the scholarly ecosystem. With their institutional repositories standing largely empty, libraries are currently puzzling about why these first "net gen" grad students and junior faculty aren't loading up university servers with data sets, conference presentations, articles in progress, course materials, and all the other good digital stuff a lively intellect produces. A glance at Borgman's book may suggest some simple truths about motivations, disincentives, and the law of the grasslands.
Sebs is right to pick me up on my glib statement that "nobody wants to share their data," and he presents some good examples of admirable projects that are taking a lead in being transparent and open with their findings. As I'll suggest in a moment, however, I think that the projects he highlights are the exceptions rather than the norm, and there are some distinctive features that make them exceptional.
Firstly, however, I would like to point out that I am not alone in my pessimism about the readiness of scholars to share. Since my last post also referred to it vigorously, it may seem that Christine Borgman's synthetic Scholarship in the Digital Age (2007) is the only book I have read. It's not quite (I have just re-read the last Harry Potter!), but I am finding some good stuff in it, especially in Chapter 8 where the author explores four areas of concern that discourage information sharing among scientists, social scientists, and humanists alike:
(1) Reward: Scholars are rewarded for publication through promotion and tenure but rarely acknowledged for managing their information. This means that they feel little obligation to self-archive data for others to use.
(2) Effort: The preparation of data so that it is useful to others is a huge effort. The logical fieldwork database structure is less than half the story. The documentation so that the information is useful to others is the real sweat.
(3) Priority: Although archaeology isn't medicine, being first with information is still the key to prestige and future funding.
(4) Ownership: Confusion about intellectual property rights and related concerns about the ownership and control of data make the prospect of sharing information risky.
These worries ring true to me for archaeology, and I find two of the points particularly persuasive. Having just spent a day watching my editorial colleagues almost literally tearing their hair out over a poorly organized pottery catalogue, I think that it is hard to underestimate the amount of thought that needs to go into preparing raw information so that it is useful (point 2 above). Archaeological data is socially constructed, so it needs context to be properly interpreted. There are norms of presentation to obey, concerns about interoperability to address, and a multitude of structuring decisions that need to be made to differentiate a catalog from a data dump. The question of "when" the records of a project become data is a big one in our field.
Although intellectual property concerns (point 4 above) are often linked to "copyright" and the naughtiness of publishers, other concerns that archaeologists have to wrestle with over the ownership and control of data seem to me to be much more urgent. In multi-institutional projects, which director's institution owns the copyright in the work its staff member is doing (remember that it is usually employers rather than employees who own the rights to work done on "company time")? When working overseas, what rights does the host nation have over the information being extracted? (Mexico, Greece, and France have recently attempted to tax photographs of their "patrimony"). These are all scary considerations, and issues that are made more frightening by being poorly understood.
But let's move back to the examples of good sharers Sebs brings up; Jack Davis with PRAP and MRAP, Ian Hodder at Çatalhöyük, Martha Joukowsky at Petra, and Brian Rose at Troy. Let's face it, Sebastian, these are legendary names, the "gray panthers" who have nothing to prove. Tenured, funded, at the top of their profession, they have little need for further reward, have access to some of the best minds around to help shape their data for other users, have less need than others to retain the right to priority, and are savvy in their abilities to navigate the intellectual property minefield. If you are a powerful feline, the obstacles to data sharing drop away.
When we talk about sharing, we need to look more at scholarly behavior at the starting-out level. Think graduate students and untenured faculty, the baby armadillos and raccoons rather than the panthers of the scholarly ecosystem. With their institutional repositories standing largely empty, libraries are currently puzzling about why these first "net gen" grad students and junior faculty aren't loading up university servers with data sets, conference presentations, articles in progress, course materials, and all the other good digital stuff a lively intellect produces. A glance at Borgman's book may suggest some simple truths about motivations, disincentives, and the law of the grasslands.
Subscribe to:
Posts (Atom)
