|
|
 |
|
|
|
|
|
|
|
|
|
|
|
ACM-SIGMOD Digital Review:
Restaurant Ratings for Technical Papers
THE BASIC IDEA:
Create a guide to works likely to be of interest to the SIGMOD membership, to help visitors locate "important" work, through an electronic review and editorial process designed with the on-line world in mind. ACM-SIGMOD Digital Review itself stores no documents, only reviews and pointers. The primary repositories are the ACM archive on computing related research (CoRR) and the ACM SIGMOD Anthology, supplemented as needed by private web pages at authors’ institutions.
THE MOTIVATION :
Our system of refereed journals is based upon a paradigm where information dissemination is expensive. There is a great deal of effort devoted to a review process that is meant to ensure a level of quality in what is published. As publication costs have dropped, the numbers of journals have proliferated, to the point that most of us today find it hard to keep up with all published literature even in our own narrow areas of specialization, let alone in the field at large. Also, the review and physical paper publication process introduces long delays, to the extent that many papers are no longer topical by the time they appear in print, and many researchers choose to focus on conference, rather than journal, publication.
As conference publication has gained in prestige, we have also seen an explosion in the number of conferences. There are conferences today that have few attendees who are not authors, and finance their entire budget out of registration fees paid by authors of accepted papers. Even at prestigious conferences, the quality of reviews often leaves authors dissatisfied. Furthermore, conference publication, though substantially faster than journal publication, introduces a significant delay itself.
The key point is that the costs of disseminating information have fallen dramatically, as has the time required for such dissemination. Review processes designed for a world with high dissemination costs are not necessarily the best gatekeepers to dissemination in the new world. In fact, technical reports are often circulated widely by many authors prior to, and occasionally even in lieu of, publication. Most researchers provide access to their recent technical writings through pointers from their web home page. In fast-moving fields, other researchers commonly build upon such work, without waiting for formal review and publication.
Our attempt here is to rethink the review paradigm from scratch. If information dissemination is cheap and quick, additional costs and delays imposed by the reviewing process quickly become burdensome. Moreover, there is no reason to restrict what is disseminated when the cost of disseminating everything is close to zero. Of course, human capacity to read and absorb information is not scaling with changes in communications technology. So there remains an important role for a review mechanism. The idea of ACM-SIGMOD Digital Review is to provide readers with help in choosing what to read without in any way limiting what they have access to. Think of it like a movie or restaurant review column: as a consumer you may choose to obtain guidance from such reviews, you will learn how to calibrate the opinion of specific reviewers, and you will decide whose reviews to trust. As a restaurateur you first hope that your restaurant is reviewed at all, and then you hope that the review is good. Note that customers have access to all restaurants, irrespective of the reviews. Furthermore customers have access to diverse opinions when there is a difference of opinion -- a restaurant awarded two stars by Michelin could be panned by Zagat. In contrast, traditional journals and conferences merge the review and access functionality so that readers of the journal or conference proceeding have access only to works that have a positive consensus review.
HOW IT WORKS :
- Anyone may review any publicly available material, such as any free and public web page. Documents not available online, or available only for a fee, may also be reviewed, but only if they appear in a "standard" forum, covered by the Trier DBLP bibliography. In addition to documents, other technical material may also be reviewed, including software, demos, presentations, and bibliographies.
- Reviews are completely public, as also the reviewer's identity. Anyone may view any review, along with the name of the reviewer. A password-secured registration process is used to authenticate reviewer identity.
- No documents are submitted to the
ACM-SIGMOD Digital Review for review. The reviewer chooses what to review. In consequence, some hot material may have multiple reviews. The vast majority of research output will never be recognized with a review.
- ACM-SIGMOD
Digital Review links to a URL for the reviewed matter when possible. In the future, the DBLP bibliography will have reverse links from article headers to reviews in ACM-SIGMOD Digital Review.
- We intend in the future that authors be notified, if possible, when new reviews of their work are submitted. Authors would then have an opportunity to respond to the reviewer. We intend to experiment with different mechanisms to encourage such "interactive reviewing". Where possible, we would like such interaction to be public, even though private interaction is also encouraged.
- Readers may of course read any material of their choice. To help make this choice, readers may read any reviews of their choice in
ACM-SIGMOD Digital Review. A search form allows readers to select based on properties of the review, such as reviewer identity and rating given, and also based on properties of the reviewed material, such as title, author, date, and subject area.
- Since any one can write a review, it is expected that reviews may have a high variance in their quality and reliability. To assist readers in determining the credibility of reviews, we maintain a board of accredited reviewers comprising most senior leaders in the database research field. Readers may choose to pay as much weight to this accreditation as they wish.
- Program Committee members for major database conferences, such as SIGMOD, PODS, VLDB, ICDE, PDIS, and EDBT, will automatically have "guest editor" privileges from the conference submission deadline through a few weeks after the PC meeting. We will urge PC members to write positive reviews for any papers they read and liked, particularly if the paper eventually did not get accepted to the conference in question. (Assuming, of course, that the conference submission is publicly available, such as through CoRR). Such guest editor reviews would be so identified explicitly, and may be used for querying purposes. Guest Editors, like regular Editors, may write negative reviews if they wish (for papers that were accepted over their objections), or reviews of any other papers that they happened to read even if these were not submitted to the conference in question.
- To encourage discussion with regard to the hot topics of the day,
ACM-SIGMOD Digital Review maintains a threaded bulletin board as an on-line discussion forum.
- A "front page" highlights reviews or articles of particular interest. While readers are free to read any reviews of their choice, the front page directs them to a few of particular interest. Thus, the front page operates as a "meta-review" of sorts, directing users to specific collections of reviews that are interesting for some reason.
SOME BENEFITS :
The new paradigm has benefits all around, to authors, to readers, and to reviewers. We consider each in turn.
Benefits to Authors
- Immediate dissemination of ideas. No delays on account of slow reviews or long publication queues. A paper is available to the public as soon as it is written. Each review is available to the author and to the public as soon as it is written (we do not have a specified set of reviewers with a need to wait for the last one of them). Readership can be attracted to the work as soon as favorable reviews appear. Thus, there is the opportunity to gain quick attention to seminal or controversial work.
- The author exercises judgement on what to publish – ultimately this self-policing will lead to more efficient maintenance of quality than the current journal system of minimally satisfying nit-picky reviewers.
- The quality of reviews is improved. In today’s system, it is often the case that an author gets a review of barely a few lines, or gets a review that clearly shows that the reviewer missed the point of the paper. Since reviews are signed in our system, there is an incentive for reviewers to do a good job.
Benefits to Readers
- Reviews help create a public assessment system, available to readers. This is contrast to current conference and journal selection mechanisms where the end-result is visible to everyone, but the reasoning is not and the reviews are not. This additional information may be of value. In particular, where there is diversity of opinion, the reader gets to know, and can benefit from this knowledge. In contrast, traditional conference program committees spend the bulk of their effort resolving such differences, often imperfectly, and then expose only the final decisions.
- Since reviews are signed, the reader can decide how much store to set by each review, based both upon the reviewer and the content of the review itself.
- The reader’s choice is not abrogated by some editor – the reader has access to a wide variety of material. For instance, one may choose to read even mediocre papers in one’s own area of specialization, while restricting oneself only the most outstanding papers in other sub-areas.
Benefits to Reviewers
- Papers for review are not assigned by an editor – the reviewer can choose to review exactly what he or she wishes. In particular, these are likely to be papers or technical reports that the reviewer would read any way.
- The review is written to address the big picture and discuss the importance of the work being reviewed. In short, the review has the potential for significant influence. All too often, in the traditional process, reviewers are forced to spend most of their energy on issues of detail.
- Since reviews are public, good reviewers build up a public reputation. As such, there is a reward for writing good reviews. In contrast, the traditional review system operates in secrecy, and only an editor gets to acknowledge reviews that display extraordinary insight or effort.
Of course, with any new effort, there are dangers. The primary concern is the lack of support for anonymous reviews. This makes it much harder to write negative reviews. (Every review entered at present, except one, has a rating of either "positive" or "superb". The sole stand out is a review with a "neutral" rating. No reviews have negative ratings whatsoever.) In itself, this is not necessarily a bad thing, as long as authors have other channels through which to obtain valuable negative feedback. There is also the fear of reviews being written in favor of friends or co-authors. A strict conflict-of-interest policy minimizes this danger. Finally, there is the possibility of reviewers being coerced, or colluding with authors, or other undesirable behaviors. We’ll just have to see how the sociology develops around this before taking any needed measures to avoid or mitigate. Overall, this is an experimental service that is guaranteed to produce value. The exact ways in which it will do so depends on what we as a community make of it.
RELATED EFFORTS :
There is no resource available to the database research community that bears much similarity to ACM-SIGMOD Digital Review. We point out some of the key differences below.
- CoRR: The Computing Research Repository is an on-line repository of technical documents.
ACM-SIGMOD Digital Review is not a repository, but rather a moderated or review-based guide to the vast amounts of information available. In fact, in large part, ACM-SIGMOD Digital Review functions as an optional front-end to CoRR.
The Uni-Trier Bibliography: This has become a critical research tool for members of the database community. However, its primary function is that of a bibliography (though more recently, electronic versions of some conference publications are also being made available). Our intention is to have cross-referencing between ACM-SIGMOD Digital Review and this bibliography so that a reader browsing the bibliography can pull up reviews for any paper seen, and also reviews can reference published articles recorded in the bibliography even if complete electronic versions are not available on-line.
SIGMOD DiSC and Anthology: These recent SIGMOD efforts make available in digital form vastly more material than ever before, and thereby make electronic access much more feasible. However, the focus of these efforts is to make available in electronic form material that is already available on paper and in traditional forums. ACM-SIGMOD Digital Review creates a new electronic review forum. Also, SIGMOD DiSC and Anthology comprise actual technical articles, just as CoRR does; ACM-SIGMOD Digital Review has reviews and pointers, but not the technical papers themselves.
ACM TODS and other scholarly publications: Play a central role in scholarship as we have it today. ACM-SIGMOD Digital Review is not a journal, even though it may in its own way play a role in electronic age scholarship.
SIGMOD Record: This SIG publication plays a central role in bonding the database research community. In a limited way, it provides a forum to comment about important trends or to identify critical pieces of work. However, this sort of forum is only a very small part of what SIGMOD Record stands for. ACM-SIGMOD Digital Review, in contrast, would focus entirely on reviewing scholarly work. There is no expectation that ACM-SIGMOD Digital Review would attempt to take over or imitate any of the central features of SIGMOD Record.
ACM Computing Reviews: This journal publishes reviews of published articles. The reviewers are typically junior people in the field, and the review typically serves more as an abstract written by a third party rather than as an evaluation of the work. Furthermore, in the print medium, it is usually too much trouble for someone browsing a journal to look up quickly what Computing Reviews has to say, or for someone to peruse Computing Reviews and then follow "pointers" in print to selected articles.
ACM-SIGMOD Digital Review differs in many significant ways. First, it is electronic, and therefore only a mouse-click away. A potential reader of a specific article can quickly call up its review, and a person browsing reviews can click to any full article of interest. (However, Computing Reviews is in the process of going on-line). Second, by having a significant core of senior leaders in the field as reviewers, ACM-SIGMOD Digital Review will have a high credibility with its audience. Third, there is no limit on the number of reviews for an article -- there may be none or several depending on how much the piece of work has raised interest. Thus, it is possible to get multiple points of view for important or controversial work, while avoiding the need to read reviews that have been written even though the reviewer did not have a whole lot to say. Fourth, there are no constraints of publication dates as in a journal -- reviews can appear as they are generated. One could get relatively immediate reviews, if some reviewer is quick. One could also get a history of reviews, if the importance of some article changes with time. Fifth, there is the possibility of a moderated discussion forum for works of particular controversy. Sixth, and finally, readers can query for articles that meet specified review criteria -- such as, articles that received a positive review from Mike Stonebraker.
CURRENT STATUS :
SIGMOD has generously supported the creation of this new venture, and ACM-SIGMOD Digital Review will be open for business by the time you read this, though it is not as of the time of this writing (Jan. 25, 2000). The Editor is H. V. Jagadish. Associate Editors are Alon Levy, Michael Ley, Yannis Papakonstantinou, and Nandit Soparkar. Several people have helped in building the system, but two in particular stand out. Zack Ives is the architect of the core software, and implementer of the review submission, storage and search facilities. Andrew Nierman is the implementer and maintainer of the front page, and the keeper of the on-line discussion forum. An advisory board, comprising Jim Gray, Joe Halpern, Alberto Mendelzon, and Rick Snodgrass, has given important formative direction to this effort.
ACM-SIGMOD Digital Review is archived as a "journal" in the Trier bibliography. Reviews are separated into "issues" based on the date written. The current "issue" keeps growing each time additional reviews are archived.
The home page for ACM-SIGMOD Digital Review is http://www.eecs.umich.edu/digital-review/
Please visit, and enjoy .
|
|
