Distance-Based Outliers: Algorithms and Applications.

@article{DBLP:journals/vldb/KnorrNT00,
  author    = {Edwin M. Knorr and
               Raymond T. Ng and
               V. Tucakov},
  title     = {Distance-Based Outliers: Algorithms and Applications},
  journal   = {VLDB J.},
  volume    = {8},
  number    = {3-4},
  year      = {2000},
  pages     = {237-253},
  ee        = {db/journals/vldb/KnorrNT00.html},
  bibsource = {DBLP, http://dblp.uni-trier.de}
}

Abstract

This paper deals with finding outliers (exceptions) in large, multidimensional datasets. The identification of outliers can lead to the discovery of truly unexpected knowledge in areas such as electronic commerce, credit card fraud, and even the analysis of performance statistics of professional athletes. Existing methods that we have seen for finding outliers can only deal efficiently with two dimensions/attributes of a dataset. In this paper, we study the notion of DB (distance-based) outliers. Specifically, we show that (i) outlier detection can be done efficiently for large datasets, and for k-dimensional datasets with large values of k (e.g., k >= 5); and (ii), outlier detection is a meaningful and important knowledge discovery task.

First, we present two simple algorithms, both having a complexity of O(k N²), k being the dimensionality and N being the number of objects in the dataset. These algorithms readily support datasets with many more than two attributes. Second, we present an optimized cell-based algorithm that has a complexity that is linear with respect to N, but exponential with respect to k. We provide experimental results indicating that this algorithm significantly outperforms the two simple algorithms for k <= 4. Third, for datasets that are mainly disk-resident, we present another version of the cell-based algorithm that guarantees at most three passes over a dataset. Again, experimental results show that this algorithm is by far the best for k <= 4. Finally, we discuss our work on three real-life applications, including one on spatio-temporal data (e.g., a video surveillance application), in order to confirm the relevance and broad applicability of DB outliers.

Key Words

Copyright © 2000 by Springer, Berlin, Heidelberg. Permission to make digital or hard copies of the abstract is granted provided that copies are not made or distributed for profit or direct commercial advantage, and that copies show this notice along with the full citation.

Online Edition (Springer)

ACM SIGMOD Anthology

• Windows: Click the letter of your CD drive
  A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
• Mac: Click here
• UNIX/LINUX: mount the CD and click on the path of your mount point:
  /Anthology/An5-2 or /cdrom

• Windows: Click the letter of your CD drive
  A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
• Mac: Click here
• UNIX/LINUX: mount the DVD and click on the path of your mount point:
  /Anthology/aDVD2 or /dvd

References

[1]

Rakesh Agrawal, Tomasz Imielinski, Arun N. Swami: Mining Association Rules between Sets of Items in Large Databases. SIGMOD Conference 1993: 207-216 BibTeX

[2]

Andreas Arning, Rakesh Agrawal, Prabhakar Raghavan: A Linear Method for Deviation Detection in Large Databases. KDD 1996: 164-169 BibTeX

[3]

...

[4]

Jon Louis Bentley: Multidimensional Binary Search Trees Used for Associative Searching. Commun. ACM 18(9): 509-517(1975) BibTeX

[5]

Leo Breiman, J. H. Friedman, R. A. Olshen, C. J. Stone: Classification and Regression Trees. Wadsworth 1984, ISBN 0-534-98053-8
BibTeX

[6]

Stefan Berchtold, Daniel A. Keim, Hans-Peter Kriegel: The X-tree : An Index Structure for High-Dimensional Data. VLDB 1996: 28-39 BibTeX

[7]

...

[8]

Soumen Chakrabarti, Sunita Sarawagi, Byron Dom: Mining Surprising Patterns Using Temporal Description Length. VLDB 1998: 606-617 BibTeX

[9]

...

[10]

Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu: A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. KDD 1996: 226-231 BibTeX

[11]

...

[12]

Usama M. Fayyad, David Haussler, Paul E. Stolorz: KDD for Science Data Analysis: Issues and Examples. KDD 1996: 50-56 BibTeX

[13]

...

[14]

...

[15]

Antonin Guttman: R-Trees: A Dynamic Index Structure for Spatial Searching. SIGMOD Conference 1984: 47-57 BibTeX

[16]

...

[17]

...

[18]

Joseph M. Hellerstein, Elias Koutsoupias, Christos H. Papadimitriou: On the Analysis of Indexing Schemes. PODS 1997: 249-256 BibTeX

[19]

...

[20]

Theodore Johnson, Ivy Kwok, Raymond T. Ng: Fast Computation of 2-Dimensional Depth Contours. KDD 1998: 224-228 BibTeX

[21]

...

[22]

Edwin M. Knorr, Raymond T. Ng: A Unified Notion of Outliers: Properties and Computation. KDD 1997: 219-222 BibTeX

[23]

Edwin M. Knorr, Raymond T. Ng: Algorithms for Mining Distance-Based Outliers in Large Datasets. VLDB 1998: 392-403 BibTeX

[24]

Raymond T. Ng, Jiawei Han: Efficient and Effective Clustering Methods for Spatial Data Mining. VLDB 1994: 144-155 BibTeX

[25]

Franco P. Preparata, Michael Ian Shamos: Computational Geometry - An Introduction. Springer 1985, ISBN 3-540-96131-3
BibTeX

[26]

...

[27]

Hanan Samet: The Design and Analysis of Spatial Data Structures. Addison-Wesley 1990
BibTeX

[28]

Sunita Sarawagi, Rakesh Agrawal, Nimrod Megiddo: Discovery-Driven Exploration of OLAP Data Cubes. EDBT 1998: 168-182 BibTeX

[29]

Paul E. Stolorz, Hisashi Nakamura, Edmond Mesrobian, Richard R. Muntz, Eddie C. Shek, Jose Renato Santos, J. Yi, Kenneth W. Ng, S.-Y. Chien, Carlos R. Mechoso, John D. Farrara: Fast Spatio-Temporal Data Mining of Large Geophysical Datasets. KDD 1995: 300-305 BibTeX

[30]

...

[31]

...

[32]

Tian Zhang, Raghu Ramakrishnan, Miron Livny: BIRCH: An Efficient Data Clustering Method for Very Large Databases. SIGMOD Conference 1996: 103-114 BibTeX