Skip to content

sujin-1013/task-aware-DMO

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

20 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Task-Aware Dynamic Model Optimization for Multi-Task Learning

πŸ“„ IEEE Access 2023 β€” DOI: 10.1109/ACCESS.2023.3339793

This repository introduces the core ideas and implementation of
Task-Aware Dynamic Model Optimization (DMO), a memory-efficient multi-task learning (MTL) framework.


πŸ” Overview

Multi-task learning (MTL) often suffers from task interference and inefficient resource usage.
DMO (Dynamic Model Optimization) addresses these issues by:

  • Grouping tasks based on inter-task similarity (weight & loss correlations)
  • Dynamically allocating parameters based on task difficulty
  • Producing lightweight subnetworks, reducing parameters by over 80% on average

Figure: Overview of Task-Aware Dynamic Model Optimization (DMO)

🎯 Key Components

  • Task Similarity Measurement
    • Pearson correlation on weights (S_W)
    • Pearson correlation on losses (S_L)
  • Task Grouping
    • Graph-based maximal clique detection (Bron–Kerbosch)
  • Dynamic Parameter Allocation
    • Pruning by task difficulty
    • Merging into group-specific subnetworks

πŸ› οΈ Code Structure

Task-Aware-DMO/
β”œβ”€β”€ models/           # Backbone network, task-specific heads, pruning utilities
β”œβ”€β”€ train.py          # Training logic
β”œβ”€β”€ evaluate.py       # Evaluation and metrics
β”œβ”€β”€ config/           # Experimental settings
β”œβ”€β”€ results/          # Graphs and tables from the paper
β”œβ”€β”€ assets/           # Images and visual materials for README
└── paper/            # Original PDF paper for reference

πŸ§ͺ Datasets Used

  • CIFAR-10, STL-10, MNIST, USPS
  • CIFAR-100 (20-task version)
  • Visual Decathlon Challenge (10 diverse tasks)

πŸ“Š Result Summary

Dataset Group Param Reduction Accuracy (Avg)
CIFAR-10 + MNIST ~84% β­€ Comparable
CIFAR-100 (20 Tasks) ~85% ⬆️ Improved
Decathlon Challenge ~88% β­€ Slight Drop

πŸ”Ή Up to 97% reduction on USPS with minimal performance loss


πŸ§ͺ Visual Decathlon Challenge (Summary Table)

Method Avg Accuracy (%) # Parameters (M) Param Ratio
Single-task 74.69 101.68 1.00
Soft Param Sharing [7] 65.24 101.68 1.00
Cross-stitch [8] 76.33 203.36 2.00
NDDR [10] 59.76 252.86 2.49
TAPS [6] 59.11 130.50 1.28
DMO (Ours) 74.40 11.48 0.11

πŸ“‘ Visual Decathlon Table (From Paper)

Figure: Visual Decathlon Challenge results table (from IEEE Access paper)

πŸ§‘β€πŸ’» Author

Su-jin Choi (μ΅œμˆ˜μ§„)
Master of AI, Chung-Ang University
πŸ“§ [email protected]
πŸ“„ IEEE Access Paper

About

Task-Aware Dynamic Model Optimization for Multi-Task Learning (IEEE Access 2023)

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published