Update files and add run_benchmark.py

Author: habibirani

README.md

@@ -3,50 +3,117 @@
 <!-- TITLE -->
 # Positional Encoding Benchmark for Time Series Classification
 
-This repository provides a comprehensive benchmark for evaluating different positional encoding techniques in Transformer models, specifically for time series classification tasks. The project includes implementations of several positional encoding methods and Transformer architectures to test their effectiveness on various time series datasets.
-
-
-
-<!-- DESCRIPTION -->
-## Description
-
-  
-This project aims to analyze how positional encodings impact Transformer-based models in time series classification. The benchmark includes both fixed and learnable encoding techniques and explores advanced approaches like relative positional encoding. The project evaluates performance on a diverse set of datasets from different domains, such as human activity recognition, financial data, EEG recordings, etc.
-
-
-## Positional Encoding Methods:
-  - Absolute Positional Encoding (APE)
-  - Learnable Positional Encoding (LPE)
-  - Relative Positional Encoding (RPE)
-  - Temporal Pseudo-Gaussian augmented Self-Attention (TPS)
-  - Temporal Uncertainty Positional Encoding (TUPE)
-  - time Absolute Positional Encoding (tAPE)
-  - efficient Relative Position Encoding (eRPE)
-  - Stochastic Positional Encoding (SPE)
-
+This repository provides a comprehensive evaluation framework for positional encoding methods in transformer-based time series models, along with implementations and benchmarking results.
+
+Our work is available on arXiv: [Positional Encoding in Transformer-Based Time Series Models: A Survey](https://arxiv.org/abs/2502.12370)
+
+## Models
+
+We present a systematic analysis of positional encoding methods evaluated on two transformer architectures:
+1. [Multivariate Time Series Transformer Framework (TST)](https://github.com/gzerveas/mvts_transformer)
+2. Time Series Transformer with Patch Embedding 
+
+
+
+### Positional Encoding Methods
+We implement and evaluate eight positional encoding methods:
+
+| Method | Type | Injection Technique | Parameters |
+|--------|------|-------------------|------------|
+| Sinusoidal PE | Absolute | Additive | 0 |
+| Learnable PE | Absolute | Additive | L×d |
+| RPE | Relative | MAM | 2(2L-1)dl |
+| tAPE | Absolute | Additive | Ld |
+| eRPE | Relative | MAM | (L²+L)l |
+| TUPE | Rel+Abs | MAM | 2dl |
+| ConvSPE | Relative | MAM | 3Kdh+dl |
+| T-PE | Rel+Abs | Combined | 2d²l/h+(2L+2l)d |
+
+Where:
+- L: sequence length
+- d: embedding dimension
+- h: number of attention heads
+- K: kernel size
+- l: number of layers
+
+## Dataset Characteristics
+
+| Dataset | Train Size | Test Size | Length | Classes | Channels | Type |
+|---------|------------|-----------|---------|----------|-----------|------|
+| Sleep | 478,785 | 90,315 | 178 | 5 | 1 | EEG |
+| ElectricDevices | 8,926 | 7,711 | 96 | 7 | 1 | Device |
+| FaceDetection | 5,890 | 3,524 | 62 | 2 | 144 | EEG |
+| MelbournePedestrian | 1,194 | 2,439 | 24 | 10 | 1 | Traffic |
+| SharePriceIncrease | 965 | 965 | 60 | 2 | 1 | Financial |
+| LSST | 2,459 | 2,466 | 36 | 14 | 6 | Other |
+| RacketSports | 151 | 152 | 30 | 4 | 6 | HAR |
+| SelfRegulationSCP1 | 268 | 293 | 896 | 2 | 6 | EEG |
+| UniMiB-SHAR | 4,601 | 1,524 | 151 | 9 | 3 | HAR |
+| RoomOccupancy | 8,103 | 2,026 | 30 | 4 | 18 | Sensor |
+| EMGGestures | 1,800 | 450 | 30 | 8 | 9 | EMG |
 
 ## Dependencies
 - Python 3.10
+- PyTorch 2.4.1+cu121
 - NumPy
-- SciPy
-- Matplotlib
-- Seaborn
-- Pandas
 - Scikit-learn
-- tsai
-- PyTorch
-
-
-## Installation
+- CUDA 12.2 
 
-To install and run the Positional Encoding Benchmark, follow these steps:
+## Clone and Installation
 
 ```bash
+# Clone the repository
 git clone https://github.com/imics-lab/positional-encoding-benchmark.git
 cd positional-encoding-benchmark
+
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate  # Linux/Mac
+# or
+.\venv\Scripts\activate  # Windows
+
+# Install dependencies
 pip install -r requirements.txt
+
+# Run benchmark with default config
+python examples/run_benchmark.py
+
+# Or with custom config
+python examples/run_benchmark.py --config path/to/custom_config.yaml
 ```
 
+## Results
+
+Our experimental evaluation encompasses eight distinct positional encoding methods tested across eleven diverse time series datasets using two transformer architectures.
+
+### Key Findings
+
+#### 1. Sequence Length Impact
+- **Long sequences** (>100 steps): 5-6% improvement with advanced methods
+- **Medium sequences** (50-100 steps): 3-4% improvement
+- **Short sequences** (<50 steps): 2-3% improvement
+
+#### 2. Architecture Performance
+- **TST**: More distinct performance gaps
+- **Patch Embedding**: More balanced performance among top methods
+
+#### 3. Average Rankings
+- **SPE**: 1.727 (batch norm), 2.090 (patch embed)
+- **TUPE**: 1.909 (batch norm), 2.272 (patch embed)
+- **T-PE**: 2.636 (batch norm), 2.363 (patch embed)
+
+### Performance Analysis
+
+#### Biomedical Signals (EEG, EMG)
+- TUPE achieves highest average accuracy
+- SPE shows strong performance
+- Both methods demonstrate effectiveness in capturing long-range dependencies
+
+#### Environmental and Sensor Data
+- SPE exhibits superior performance
+- TUPE maintains competitive accuracy
+- Relative encoding methods show improved local pattern recognition
+
 <!-- CONTRIBUTING -->
 ## Contributing
 Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
@@ -58,5 +125,10 @@ Please make sure to update tests as appropriate.
 ## Citation
 
 ```bibtex
-TBD
+@article{irani2025positional,
+  title={Positional Encoding in Transformer-Based Time Series Models: A Survey},
+  author={Irani, Habib and Metsis, Vangelis},
+  journal={arXiv preprint arXiv:2502.12370},
+  year={2025}
+}
 ```

environment.yml

@@ -25,6 +25,7 @@ dependencies:
   - pandas
   - scikit-learn
   - matplotlib
+  
   # Uncomment the below to add pip dependencies if necessary"
   # - pip:
     # - mydep==1.0

src/config/config.yalm

@@ -0,0 +1,19 @@
+model:
+  type: 'batch_norm'  # or 'patch_embed'
+  encoding: 'tupe'    # ['pe', 'lpe', 'rpe', 'tape', 'erpe', 'tupe', 'convspe', 't-pe']
+  input_timesteps: 100
+  patch_size: 16
+  embedding_dim: 128
+  num_layers: 4
+  num_heads: 8
+  dim_feedforward: 256
+  dropout: 0.1
+
+training:
+  batch_size: 32
+  learning_rate: 0.001
+  epochs: 100
+  device: 'cuda'
+
+dataset:
+  name: 'Sleep'

src/encodings/__init__.py

@@ -122,18 +122,6 @@ def get_relative_positions(
         x - y, bidirectional, num_buckets, max_distance
     )
     return relative_positions    
-
-def get_pos_encoder(pos_encoding):
-    if pos_encoding == 'fixed':
-        return FixedPositionalEncoding
-    elif pos_encoding == 'learned':
-        return LearnedPositionalEncoding
-    elif pos_encoding == 'tape':
-        return tAPE
-    elif pos_encoding == 'absolute':
-        return AbsolutePositionalEncoding
-    else:
-        raise ValueError(f"Unknown positional encoding type: {pos_encoding}")
 
 
 
@@ -204,3 +192,17 @@ def __init__(self, d_model, num_variables):
     def forward(self, x, variable_idx):
         variable_embed = self.variable_embedding(variable_idx)
         return x + variable_embed.unsqueeze(0)
+
+
+
+def get_pos_encoder(pos_encoding):
+    if pos_encoding == 'fixed':
+        return FixedPositionalEncoding
+    elif pos_encoding == 'learned':
+        return LearnedPositionalEncoding
+    elif pos_encoding == 'tape':
+        return tAPE
+    elif pos_encoding == 'absolute':
+        return AbsolutePositionalEncoding
+    else:
+        raise ValueError(f"Unknown positional encoding type: {pos_encoding}")

src/attentions.py renamed to src/encodings/attentions.py

@@ -9,7 +9,7 @@
 from load_data import get_dataset
 
 import torch.optim as optim
-from time_series_transformer import TimeSeriesTransformer
+from src.models.TimeSeriesTransformer import TimeSeriesTransformer
 from time_series_transformer_batchnorm import TSTransformerEncoder
 from utils import get_dataloaders
 

src/positional_encodings.py renamed to src/encodings/positional_encodings.py

@@ -1,7 +1,7 @@
 import torch
 import torch.nn as nn
-from patch_embedding_layer import TimeSeriesPatchEmbeddingLayer
 import math
+from src.encodings.positional_encodings import get_pos_encoder
 
 class TransformerBatchNormEncoderLayer(nn.Module):
     def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1, activation="gelu"):