TimeEval

Evaluation Tool for Anomaly Detection Algorithms on Time Series.

See TimeEval Algorithms for algorithms that are compatible to this tool. The algorithms in that repository are containerized and can be executed using the DockerAdapter of TimeEval.

If you use TimeEval, please consider citing our paper.

📖 TimeEval's documentation is hosted at https://timeeval.readthedocs.io.

Features

• Large integrated benchmark dataset collection with more than 700 datasets
• Benchmark dataset interface to select datasets easily
• Adapter architecture for algorithm integration
  • DockerAdapter
  • JarAdapter
  • DistributedAdapter
  • MultivarAdapter
  • ... (add your own adapter)
• Large collection of existing algorithm implementations (in TimeEval Algorithms repository)
• Automatic algorithm detection quality scoring using AUC (Area under the ROC curve, also c-statistic) or range-based metrics
• Automatic timing of the algorithm execution (differentiates pre-, main-, and post-processing)
• Distributed experiment execution
• Output and logfile tracking for subsequent inspection

Installation

TimeEval can be installed as a package or from source.

⚠️ Attention!

Currently, TimeEval is tested only on Linux and macOS and relies on unixoid capabilities. On Windows, you can use TimeEval within WSL. If you want to use the provided detection algorithms, Docker is required.

Installation using pip

Builds of TimeEval are published to PyPI:

Prerequisites

• python >= 3.9, < 3.13

  ⚠️ Attention!

  A dependency of TimeEval prevents us from supporting Python versions >= 3.13: prts is not updated and depends on NumPy<2.0.0. However, there is no NumPy version below 2.0.0 that supports Python>=3.13.

• pip >= 20

• Docker (for the anomaly detection algorithms)

• (optional) rsync for distributed TimeEval

Steps

You can use pip to install TimeEval from PyPI:

pip install TimeEval

Installation from source

tl;dr

git clone [email protected]:TimeEval/TimeEval.git
cd timeeval/
conda create -n timeeval python=3.9
conda activate timeeval
pip install .

Prerequisites

The following tools are required to install TimeEval from source:

• git
• Python > 3.9 and Pip (anaconda or miniconda is preferred)

Steps

1. Clone this repository using git and change into its root directory.

2. Create a conda-environment and install all required dependencies:

   conda create -n timeeval python=3.9
   conda activate timeeval
   pip install .

3. If you want to make changes to TimeEval or run the tests, you need to install the development dependencies with: pip install ".[ci]". The optional extra dependencies ".[dev]" contains additional dependencies for the notebooks and scripts packaged with TimeEval.

Usage

Example script:

from pathlib import Path
from typing import Dict, Any

import numpy as np

from timeeval import TimeEval, DatasetManager, Algorithm, TrainingType, InputDimensionality
from timeeval.adapters import FunctionAdapter
from timeeval.algorithms import subsequence_if
from timeeval.params import FixedParameters

# Load dataset metadata
dm = DatasetManager(Path("tests/example_data"), create_if_missing=False)


# Define algorithm
def my_algorithm(data: np.ndarray, args: Dict[str, Any]) -> np.ndarray:
    score_value = args.get("score_value", 0)
    return np.full_like(data, fill_value=score_value)


# Select datasets and algorithms
datasets = dm.select()
datasets = datasets[-1:]
# Add algorithms to evaluate...
algorithms = [
    Algorithm(
        name="MyAlgorithm",
        main=FunctionAdapter(my_algorithm),
        data_as_file=False,
        training_type=TrainingType.UNSUPERVISED,
        input_dimensionality=InputDimensionality.UNIVARIATE,
        param_config=FixedParameters({"score_value": 1.})
    ),
    subsequence_if(params=FixedParameters({"n_trees": 50}))
]
timeeval = TimeEval(dm, datasets, algorithms)

# execute evaluation
timeeval.run()
# retrieve results
print(timeeval.get_results())

Citation

If you use TimeEval in your project or research, please cite our demonstration paper:

Phillip Wenig, Sebastian Schmidl, and Thorsten Papenbrock. TimeEval: A Benchmarking Toolkit for Time Series Anomaly Detection Algorithms. PVLDB, 15(12): 3678 - 3681, 2022. doi:10.14778/3554821.3554873

@article{WenigEtAl2022TimeEval,
  title = {TimeEval: {{A}} Benchmarking Toolkit for Time Series Anomaly Detection Algorithms},
  author = {Wenig, Phillip and Schmidl, Sebastian and Papenbrock, Thorsten},
  date = {2022},
  journaltitle = {Proceedings of the {{VLDB Endowment}} ({{PVLDB}})},
  volume = {15},
  number = {12},
  pages = {3678--3681},
  doi = {10.14778/3554821.3554873}
}