Evaluation of Evaluation in Information Retrieval.

Tefko Saracevic: Evaluation of Evaluation in Information Retrieval. SIGIR 1995: 138-146

@inproceedings{DBLP:conf/sigir/Saracevic95,
  author    = {Tefko Saracevic},
  editor    = {Edward A. Fox and
               Peter Ingwersen and
               Raya Fidel},
  title     = {Evaluation of Evaluation in Information Retrieval},
  booktitle = {SIGIR'95, Proceedings of the 18th Annual International ACM SIGIR
               Conference on Research and Development in Information Retrieval.
                Seattle, Washington, USA, July 9-13, 1995 (Special Issue of
               the SIGIR Forum)},
  publisher = {ACM Press},
  year      = {1995},
  isbn      = {0-89791-714-6},
  pages     = {138-146},
  ee        = {db/conf/sigir/Saracevic95.html},
  crossref  = {DBLP:conf/sigir/95},
  bibsource = {DBLP, http://dblp.uni-trier.de}
}

BibTeX

Abstract

Evaluation is a major force in research, development and applications related to information retrieval (IR). This paper is a critical and historical analysis of evaluations of IR systems and processes. Strengths and shortcomings of evaluation efforts and approaches are discussed together with major challenges and questions. A limited comparison is made with evaluation in experts systems and Online Public Access Catalogs (OPACS). Evaluation is further analyzed in relation to the broad context and specific problems addressed. Levels of evaluation are identified and contrasted; most IR evaluations were concerned with the processing level, but others were conducted at the output, users and use, and social levels. A major problem is the isolation of evaluations at a given level. Issues related to systems under evaluation, and evaluation criteria, measures, measuring instruments, and methodologies are examined. A general point is also considered: IR is increasmgly embedded into many other applications, such as the Internet or digital libraries. Little evaluation in the traditional IR sense is undertaken in relation to these applications. The challenges are to integrate IR evaluations from different levels and to incorporate evaluation in new applications.

Copyright © 1995 by the ACM, Inc., used by permission. Permission to make digital or hard copies is granted provided that copies are not made or distributed for profit or direct commercial advantage, and that copies show this notice on the first page or initial screen of a display along with the full citation.

ACM SIGMOD Anthology

CDROM Version: Load the CDROM "Volume 2 Issue 3, SIGIR, DASFAA'97, OODBS'86" and ...

Windows: Click the letter of your CD drive
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
Mac: Click here
UNIX/LINUX: mount the CD and click on the path of your mount point:
/Anthology/An2-3 or /cdrom

DVD Version: Load ACM SIGMOD Anthology DVD 1" and ...

Windows: Click the letter of your CD drive
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
Mac: Click here
UNIX/LINUX: mount the DVD and click on the path of your mount point:
/Anthology/aDVD1 or /dvd

BibTeX

Printed Edition

Edward A. Fox, Peter Ingwersen, Raya Fidel (Eds.): SIGIR'95, Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Seattle, Washington, USA, July 9-13, 1995 (Special Issue of the SIGIR Forum). ACM Press 1995, ISBN 0-89791-714-6
Contents BibTeX

Online Edition: ACM Digital Library

Citation page

BibTeX

ACM SIGMOD Anthology - DBLP: [Home | Search: Author, Title | Conferences | Journals]