SynapseML v1.1.0

We are excited to announce the release of SynapseML v1.1 marking a host of powerful new features introduced since the initial v1.0 release. SynapseML is an open-source library that aims to streamline the development of massively scalable machine learning pipelines. It unifies several existing ML Frameworks and new Microsoft algorithms in a single, scalable API that is usable across Python, R, Scala, and Java. SynapseML is usable from any Apache Spark platform with first class enterprise support on Microsoft Fabric.

Highlights

Microsoft Fabric	AI Functions	OneLake
Build and operationalize distributed ML with SynapseML in Fabric	Apply Pandas and Spark LLM transformations with one line of code	Automatically derive AI insights for unstructured data in OneLake
Build Your First Model	Explore AI Functions	Learn More

Hugging Face	Azure AI Foundry
Use open source models hosted on Hugging Face	Run Azure AI Foundry models in your notebook
Try an Example	View Notebook

More Hightlights

Spark 3.5 Support – In this version we transitioned to Spark 3.5 as our main Spark platform.

OpenAI Ecosystem – Comprehensive improvements including global parameter defaults, GPT-4 enablement, custom endpoints/headers, GPU-accelerated embeddings with KNN, and fine-grained control over model parameters (top_p, seed, responseFormat, temperature).

ML Innovation – HuggingFaceCausalLM transformer for distributed language model evaluation, custom embedder support, and synthetic difference-in-differences causal inference module.

Platform features – Spark Native OneLake support; MSI for Azure Storage; OpenAITranslate transformer.

AI Functions in Data Wrangler on Fabric – AI Functions built into Data Wrangler in Fabric allow you to apply LLM-powered operations to your dataframe without writing a single line of code.

New Features

Documentation 📚

• AI Functions
• AI Powered Transforms in OneLake
• Azure OpenAI for Big Data in SynapseML
• AI Functions in Data Wrangler
• AI Foundry
• Hugging Face

AI Functions ⚡

• Added support for AI Functions in Pandas (#1579613, #1585011, #1596611, #1509195, #1501185, #1494610, #1494951)
• Added support for AI Functions in PySpark (#1460790, #1572928, #1599735, #1439858, #1463533)
• Added support for async AI Functions execution (#1529058, #1523727)
• JSON response support & improved language validation. (#1551823, #1566189)
• Seed param for reproducible chat/completions (API 2024-10-21). (#1551883)
• Fuzzy case-insensitive matching for Classify. (#1515064)
• Add AI Functions Operations to Data Wrangler (#1590130, #1638257, #1718967, #1725101, #1730446)

Azure OpenAI 🌸

• Enhanced Model Parameters – Added top_p, seed, responseFormat, temperature, and subscription key support (#2410, #2329, #2324)
• GPT-4 Enablement – Full GPT-4 support in OpenAIPrompt (#2248)
• Custom Endpoints & Headers – Support for custom URL endpoints and HTTP headers (#2232)
• GPU-Accelerated Embeddings – OpenAI embeddings with GPU-based KNN pipeline (#2157)
• Embedding Dimensions Control – Configurable dimensions parameter for OpenAIEmbedding (#2215)
• Global Parameter Defaults – Centralized OpenAI parameter management with Python wrapper support (#2318, #2327)
• Updated OpenAI API version to 2024 (#2190)
• Updated OpenAIDefaults implementation (#2415)
• OpenAIPrompt bug fixes and improvements (#2334)
• Added responseFormat parameter to Chat Completion (#2329)
• Optimized getOptionalParams in HasOpenAITextParams (#2315)

OneLake 🌊

• Add Spark Native OneLake support (#1190687)

Machine Learning 🕸️

• HuggingFaceCausalLM – Transformer for evaluating language models on Spark clusters (#2301)
• Custom Embedder – Extensible custom embedding transformer support (#2236)
• Synthetic DiD – Synthetic difference-in-differences module for causal inference (#2095)

Azure AI Foundry 🔨

• AIFoundryChatCompletion – New transformer for Azure AI Foundry chat models (#2398)
• AI Foundry + OpenAI Prompt – Unified interface for OpenAI and Foundry deployments (#2404)

General ✨

• Add Spark 3.5 Support – Added full Spark 3.5 compatibility with new build variants (#2052)
• Python 3.11 Baseline – Upgraded to Python 3.11 as minimum version (#2193)
• Fabric Billing Integration – Enhanced Fabric Cognitive Service token for billing support (#2291)
• Fabric WSPL FQDN Selection – Configurable Fabric workspace FQDN endpoints (#2376)
• Added Bool input support for ONNX models ([#2130...

SynapseML v1.0.14

Changes:

• 123ead2 chore: bump to v1.0.14 (#2418)
• 1ceb3b1 chore: update OpenAIDefaults (#2415)
• 26220da feat: add aifoundry to openai prompt (#2404)
• aac2ed6 fix: fix error handling in networking layer (#2412)
• 7c2f22a feat: Add top_p and seed params to OpenAIDefaults (#2410)
• a9133aa noop commit to Stop library releases in build
• a394061 chore: Bump library to version to v1.0.13 (#2405)
• 85a6687 chore: fix propagation of fabric telemetry (#2403)
• 298c7ed chore: Add devcontainer configuration for vscode and Copilot (#2400)
• 873884d feat: Add AIFoundaryChatCompletion (#2398)

See More

• 83ebb5a Update README.md

This list of changes was auto generated.

SynapseML v1.0.14 Spark 3.5

v1.0.14-spark3.5

Enabling Synapse 3.5 tests

SynapseML v1.0.13

Changes:

• a394061 chore: Bump library to version to v1.0.13 (#2405)
• 85a6687 chore: fix propagation of fabric telemetry (#2403)
• 298c7ed chore: Add devcontainer configuration for vscode and Copilot (#2400)
• 873884d feat: Add AIFoundaryChatCompletion (#2398)
• 83ebb5a Update README.md

This list of changes was auto generated.

SynapseML v1.0.13-spark3.5

Enabling Synapse 3.5 tests

v1.0.12-spark3.5

Enabling Synapse 3.5 tests

SynapseML v1.0.12

Changes:

• 9c91148 chore: bump to v1.0.12 (#2397)
• e71fed8 chore: fix vw benchmarks (#2396)
• 2b23d7c chore: split deepLearning tests (#2387)
• 1639b14 fix: testing telemetry-properties header (#2375)
• 9e0fde4 chore: Fixing logic for polling in LRO tasks (#2389)
• b672aa4 chore: Updating codecov pipeline step (#2391)
• 84d7d65 feat: Support Choosing Fabric WSPL FQDN (#2376)
• ffa0383 docs: default dataTransferMode is streaming, not bulk (#2377)
• 04d9dc4 fix: auto-convert DateType/TimestampType to ISO8601 in Azure Search (#2381)
• 5bbbbc7 chore: fix failing openai tests (#2385)

See More

• 9af855e fix: support scoring profiles in Azure Search index parsing (#2383)
• 326988d chore: update sbt version to allow for amd local builds (#2384)
• 3865e71 fix: fix model checking logic (#2379)
• 1eec70d fix: fix bug where token cannot be acquired on system context (#2378)
• 141039b fix: add hf causal LM python tests, fix build (#2374)
• 6c95bf0 fix: add case for Python only envs (#2368)
• c1cef65 chore: limit adb concurrency (#2370)

This list of changes was auto generated.

SynapseML v1.0.11

Changes:

• 58b945f chore: fix yml (#2373)
• 2ec8db4 chore:fix release (#2372)
• 33b5a39 chore: bump to v1.0.11 (#2371)
• e12b0a1 fix: Adding trailing / to URLs set (#2364)
• 53b76a8 feat: Add HuggingFaceCausalLM Transformer for evaluating language models on cluster (#2301)
• 25b49cd chore: free up space in build (#2365)
• 26baa09 fix: Cannot Load LightGBM Model When Placed in a Spark Pipeline with Custom Transformers (#2357)
• b7971eb chore: Making unit tests green again (#2360)
• 57d15ec chore: Removing unnecessary logs (#2359)
• 6c9c9ce Add correct workspace ID for spark jobs (#2358)

See More

• a4dec08 fix: update openai compeletion doc, fix failed OpenAI test (#2348) [ #2351, #2353, #2354 ]
• 7878fec chore: streamline cache fix (#2354)
• 8787c82 chore: free up space on build machines prior to caching (#2353)
• b2f4080 Make sure library isnt released twice
• 0240895 chore: bump synapseml to v1.0.10 (#2351)
• 1b5df70 feat: Adding capability use Cognitive Service Language Service asynchronously for Summarization (#2342)
• bab6aed chore: fix github release yml (#2339)

This list of changes was auto generated.

SynapseML v1.0.11-spark3.5

chore: Adding Spark35 support

v1.0.10-spark3.5

chore: Adding Spark35 support