1.38.54

1.38.54 (2025-09-08)

Fix

• fix: Add dedicated display for RTEB benchmark results (#3089)

• feat - remove special filtering, keep zero-shot, keep borda rank

• feat - remove get_rteb_benchmark.py

• feat - delete get_rteb_benchmark.py;RTEB_BENCHMARK_ENTRIES changes

• feat -format

• Update mteb/load_results/benchmark_results.py

Co-authored-by: Roman Solomatin <[email protected]>

---

Co-authored-by: Roman Solomatin <[email protected]> (53f49ec)

Unknown

• Update tasks & benchmarks tables (32c9746)

• model: EmbeddingGemma 300M (#3129)

• model: EmbeddingGemma 300M

• Add license and revision (729f20a)

1.38.53

1.38.53 (2025-09-03)

Ci

• ci: Dataset check on new PR (#3103)

• add dataset check on new PR

• add extract datasets

• run as module

• update startswith

• update workflow name

• add GitPython

• export var

• same shell session

• address review comments

• add to docs to say what this script does

• add docs (6e8eba1)

Fix

• fix: add voyage quantization models (#3092)

• Adding quantization support

• Update mteb/models/voyage_models.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• Update mteb/model_meta.py

Co-authored-by: Roman Solomatin <[email protected]>

• Update mteb/model_meta.py

Co-authored-by: Roman Solomatin <[email protected]>

• Simplifying the quantization/output_dtype

• Update mteb/model_meta.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

---

Co-authored-by: Kenneth Enevoldsen <[email protected]>
Co-authored-by: Roman Solomatin <[email protected]> (9c7804c)

Unknown

• model: add Youtu-Embedding-V1 (#3115)

• add youtu models

• add a blank line

• fix the optional dependencies and lint the code

• remove unused dependencies and reformat

• revise prompt_type

---

Co-authored-by: springxchen <[email protected]> (652ff2b)

• Ci: test out GH models with welcoming new comers (#3112)

test out GH models with welcoming new comers (73a35e0)

1.38.52

1.38.52 (2025-09-01)

Fix

• fix: Allow closed datasets (#3059)

• • Added an include_private parameter to the get_tasks() function that defaults to False

  • This ensures that by default, tests only run on public datasets

  • Tests can explicitly set include_private=True when needed to test private datasets

  • Added is_public: bool | None = None field to TaskMetadata

  • The field is optional and defaults to None (treated as public)

  • Updated the is_filled() method to exclude is_public from required fields

  • Added documentation

• • Added an include_private parameter to the get_tasks() function that defaults to False

  • This ensures that by default, tests only run on public datasets

  • Tests can explicitly set include_private=True when needed to test private datasets

  • Added is_public: bool | None = None field to TaskMetadata

  • The field is optional and defaults to None (treated as public)

  • Updated the is_filled() method to exclude is_public from required fields

  • Added documentation

• Correcting due to comments

• Update mteb/abstasks/TaskMetadata.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• Update mteb/overview.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• Removing the not used filter_tasks_by_privacy function

• Correcting due to comments

• Correcting due to comments

• Correcting due to comments

• Removing the test case

• Rename the include_private parameter to exclude_private

• Rename the include_private parameter to exclude_private

• Add private tasks tests

• Add private tasks tests

• Update tests/test_tasks/test_private_tasks.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• Add private tasks tests

• Add private tasks tests

• Add private tasks tests

---

Co-authored-by: Kenneth Enevoldsen <[email protected]> (5844cc7)

Unknown

• model: Add ModelMeta for OrdalieTech/Solon-embeddings-mini-beta-1.1 (#3090)

• Add ModelMeta for OrdalieTech/Solon-embeddings-mini-beta-1.1

• Add training_datasets (common_corpus, fineweb, wiki_fr, private LLM-synth)

• Format with ruff + add loader per review

• Apply ruff format/fixes

• Update mteb/models/ordalietech_solon_embeddings_mini_beta_1_1.py

Co-authored-by: Roman Solomatin <[email protected]>

• Update mteb/models/ordalietech_solon_embeddings_mini_beta_1_1.py

Co-authored-by: Roman Solomatin <[email protected]>

• Register OrdalieTech/Solon-embeddings-mini-beta-1.1 in overview (ModelMeta + loader)

• Update mteb/models/ordalietech_solon_embeddings_mini_beta_1_1.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• fix import

• Add memory_usage_mb=808.0 and required fields to ModelMeta

• Fix 210 milions of parameters

---

Co-authored-by: Roman Solomatin <[email protected]>
Co-authored-by: Isaac Chung <[email protected]>
Co-authored-by: Kenneth Enevoldsen <[email protected]> (4774b74)

1.38.51

1.38.51 (2025-09-01)

Fix

• fix: Add @classmethod for @field_validators in TaskMetadata (#3100) (4012517)

Unknown

• Align task prompt dict with PromptType (#3101)

• align task prompt dict with PromptType

• use value instead of enum (7303c15)

1.38.50

1.38.50 (2025-09-01)

Fix

• fix: Updating the default batch size calculation in the voyage models (#3091) (5851c7a)

Unknown

• Combine Plots and Tables into a Single (#3047)

• feat - Combine Plots and Tables into a Single Tab #3009

• feat - Resize the plot to make it more readable

• feat - Remove the (radar chart)

• feat - Add a comment stating that it only shows the Top 5 models in the table.

• feat - adjust layout

• Update mteb/leaderboard/app.py

• format

---

Co-authored-by: Kenneth Enevoldsen <[email protected]>
Co-authored-by: Kenneth Enevoldsen <[email protected]>
Co-authored-by: Isaac Chung <[email protected]> (9586697)

• CI: Set upper limit for xdist version (#3098)

• Commentout bibtex formatting

• Remove -n auto

• get back bibtex

• try limiting versions

• revert coverage

• revert coverage

---

Co-authored-by: Isaac Chung <[email protected]> (17fa697)

1.38.49

1.38.49 (2025-08-28)

Fix

• fix: duplicate mteb multilingual variables (#3080)

• fix benchmark naming

• format

• lint (27be671)

• fix: Improving validate_task_to_prompt_name logs and error messages (#3079)

• Improving validate_task_to_prompt_name logs and error messages

• linter fixes

• Adding None prompts tests

• Update test_benchmark_sentence_transformer

• Update mteb/leaderboard/benchmark_selector.py

Co-authored-by: Roman Solomatin <[email protected]>

---

Co-authored-by: Roman Solomatin <[email protected]> (139fc73)

Unknown

• model: mdbr-leaf models (#3081)

• added MDBR leaf models

• fixed revision for mdbr-leaf-ir

• added model prompts

• updated training datasets

• fixed linting

• lotte task reference

---

Co-authored-by: Robin Vujanic <[email protected]> (e4c2a95)

• Update tasks & benchmarks tables (5bf303b)

• Move dev to dependency groups (#3088)

add dependency groups (cd14ef6)

1.38.48

1.38.48 (2025-08-27)

Fix

• fix: run ruff check on all files during ci (#3086)

• fix: run ruff check on all files during ci

• format (b46b633)

1.38.47

1.38.47 (2025-08-27)

Fix

• fix: Add beta version of RTEB related benchmarks (#3048)

• Add RTEB related benchmarks

• Add RTEB related benchmarks

• Correcting the task names in the RTEB benchmarks

• Update mteb/leaderboard/benchmark_selector.py

Co-authored-by: Roman Solomatin <[email protected]>

• Adding the CURE dataset to RTEB benchmarks

• Use the right language subset

• Fix broken finance icon URL in RTEB benchmarks

Replace broken libre-finance-dollar.svg with working libre-gui-price-tag.svg
Validated all icon URLs and confirmed accessibility compliance

• Add the rteb_benchmarks to the BENCHMARK_REGISTRY

• Add the rteb_benchmarks to the BENCHMARK_REGISTRY

• Add the rteb_benchmarks to the BENCHMARK_REGISTRY

• Add the rteb_benchmarks to the BENCHMARK_REGISTRY

• Add the rteb_benchmarks to the BENCHMARK_REGISTRY

• Add the rteb_benchmarks to the BENCHMARK_REGISTRY

• Add the rteb_benchmarks to the BENCHMARK_REGISTRY

---

Co-authored-by: Roman Solomatin <[email protected]> (1541318)

Unknown

• Fix the reference link for CoDi-Embedding-V1 (#3075)

Fix reference link (d2c3570)

1.38.46

1.38.46 (2025-08-25)

Fix

• fix: Update revision for qzhou models (#3069) (63a0c60)

1.38.45

1.38.45 (2025-08-25)

Ci

• ci: Add stale workflow (#3066)

• add stale workflow

• add permissions

• add bug label to bug issue template

• revert bug issue and only look at more info needed issues

• more accurate name

• override default (df719cc)

Fix

• fix: open_clip package validation (#3073) (1f9641a)

Unknown

• Correcting the JINA models with SentenceTransformerWrapper (#3071) (70724e7)