Merge pull request #564 from aurelio-labs/james/hybrid-router-example

feat: improved hybrid router doc

Author: jamescalam

docs/examples/hybrid-router.ipynb

@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Semantic Router: Hybrid Layer\n"
+    "# Hybrid Router\n"
    ]
   },
   {
@@ -107,7 +107,15 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now we initialize our embedding model:\n"
+    "Now we initialize our embedding models, we use a dense encoder from [OpenAI](https://platform.openai.com/) and a sparse encoder from [Aurelio](https://platform.aurelio.ai/). The `AurelioSparseEncoder` we use here provides a remote sparse encoder that can significantly improve routing accuracy when combined with dense embeddings.\n",
+    "\n",
+    "Semantic Router supports other _local_ sparse encoders like `TfidfEncoder` or `BM25Encoder`. Compared to these, the `AurelioSparseEncoder`:\n",
+    "\n",
+    "1. Doesn't require local fitting (training) on your dataset\n",
+    "2. Handles out-of-vocabulary words better\n",
+    "3. Works better with asymmetric retrieval (different encoding for queries vs. documents)\n",
+    "\n",
+    "We initialize both like so:"
    ]
   },
   {
@@ -117,16 +125,24 @@
    "outputs": [],
    "source": [
     "import os\n",
-    "from semantic_router.encoders import OpenAIEncoder, TfidfEncoder\n",
+    "from semantic_router.encoders import OpenAIEncoder, AurelioSparseEncoder\n",
     "from getpass import getpass\n",
     "\n",
+    "# get OpenAI API key from https://platform.openai.com/\n",
     "os.environ[\"OPENAI_API_KEY\"] = os.getenv(\"OPENAI_API_KEY\") or getpass(\n",
     "    \"Enter OpenAI API Key: \"\n",
     ")\n",
     "\n",
-    "dense_encoder = OpenAIEncoder()\n",
-    "# sparse_encoder = BM25Encoder()\n",
-    "sparse_encoder = TfidfEncoder()"
+    "dense_encoder = OpenAIEncoder(name=\"text-embedding-3-small\", score_threshold=0.3)\n",
+    "\n",
+    "# get Aurelio API key from https://platform.aurelio.ai\n",
+    "# use \"SRHYBRIDROUTER\" for free credits\n",
+    "os.environ[\"AURELIO_API_KEY\"] = os.getenv(\"AURELIO_API_KEY\") or getpass(\n",
+    "    \"Enter Aurelio API Key: \"\n",
+    ")\n",
+    "\n",
+    "# Using Aurelio's BM25 sparse encoder\n",
+    "sparse_encoder = AurelioSparseEncoder(name=\"bm25\")"
    ]
   },
   {
@@ -155,7 +171,10 @@
     "from semantic_router.routers import HybridRouter\n",
     "\n",
     "router = HybridRouter(\n",
-    "    encoder=dense_encoder, sparse_encoder=sparse_encoder, routes=routes\n",
+    "    encoder=dense_encoder, \n",
+    "    sparse_encoder=sparse_encoder, \n",
+    "    routes=routes,\n",
+    "    alpha=0.5  # Balance between dense (0) and sparse (1) embeddings\n",
     ")"
    ]
   },