RTX 50 Series Blackwell Compatibility

[androsforever]

Hello, I'm on Win11 and have recently upgraded from an RTX 3080 TI to an RTX 5070 TI and unfortunately I am getting this error message:

NVIDIA GeForce RTX 5070 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_61 sm_70 sm_75 sm_80 sm_86 sm_90.
If you want to use the NVIDIA GeForce RTX 5070 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

warnings.warn(

I have downloaded the latest Nightly Pytorch from that link and still the same error.

[Kyrandranos]

[Kyrandranos]

I am now able to run text-generation-webui on my RTX 5080 and at first glance, the performance is already superior to what I was getting with my RTX 3080. Here is the full procedure that I used :

git clone https://github.com/oobabooga/text-generation-webui .\TextGen

Replace the following two lines in start_windows.bat, then continue setup :

set MINICONDA_DOWNLOAD_URL=https://repo.anaconda.com/miniconda/Miniconda3-py312_25.1.1-2-Windows-x86_64.exe

set MINICONDA_CHECKSUM=fb936987b769759fc852af1b2a0e359ac14620c2b7bea8a90c6d920f2b754c4a

cd .\TextGen

.\start_windows.bat

Answer "A"

Press "^C" once running

.\cmd_windows.bat

pip install --force-reinstall torch==2.7.0.dev20250307+cu128 torchvision==0.22.0.dev20250308+cu128 torchaudio==2.6.0.dev20250308+cu128 --index-url https://download.pytorch.org/whl/nightly/cu128 --extra-index-url https://download.pytorch.org/whl/nightly/cu128

pip install --force-reinstall llama-cpp-python --prefer-binary --extra-index-url=https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/AVX2/cu128

pip install exllamav2

git clone --recurse-submodules https://huggingface.co/Panchovix/xformers-windows-blackwell2.0-nightly.git

pip install .\xformers-windows-blackwell2.0-nightly\xformers-0.0.30+7cb59f0b.d20250226-cp312-cp312-win_amd64.whl
rm -fr .\xformers-windows-blackwell2.0-nightly

pip install -U --pre triton-windows

pip install --force-reinstall torch==2.7.0.dev20250307+cu128 torchvision==0.22.0.dev20250308+cu128 torchaudio==2.6.0.dev20250308+cu128 --index-url https://download.pytorch.org/whl/nightly/cu128 --extra-index-url https://download.pytorch.org/whl/nightly/cu128

pip install fastapi<=0.112.4 filelock==3.16.1 fsspec==2024.10.0 gradio<=4.44.1 gradio_client<=1.3.0 Jinja2==3.1.4 llvmlite==0.44.0 networkx==3.4.2 numba==0.61.0 numpy==*2.1.3 pillow<=10.4.0 starlette<=0.38.6

exit

.\start_windows.bat

[nan0bug00]

After trying the above instructions and failing to get things working, I decided to tinker until I figured it out and make it simple for others. I created a fork that works and simplifies the install process. It's Windows only for now but I should have Linux done later this week.

My changes:

• start_windows.batuses a Miniconda installer for Python 3.12
• one_click.py:
  • Sets up the environment in Python 3.12.
  • Installs Pytorch from the nightly cu128 index.
  • Will not 'update' your nightly cu128 pytorch to an older version.
• requirements.txt:
  • uses updated dependencies
  • pulls exllamav2/flash-attention/llama-cpp-python wheels that I built using nightly cu128 pytorch and Python 3.12 from my wheels repo.

The end result is that installing this is minimally different from using the upstream start_windows.bat - when you get to the part where you select your device, choose "A", and it will just install and work as normal. That's it. No manually updating pytorch and dependencies, no copying files over your regular install, no compiling your own wheels, no muss, no fuss.

It should be understood, but I'll just say it for anyone who needs to hear it:

• This is experimental. It relies on nightly pytorch. Things might break due to nightly pytorch updates if theres a long enough gap between pytorch nightly builds and my compiled wheels, and you may need to wait for me to recompile the wheels every now and then. I will do my best to keep things working until there is official Blackwell support.
• If you run into problems, report them on the issues page for my fork. DO NOT REPORT ISSUES FOR THE FORK IN THIS REPO.
• I am just one guy, I have a life, this is a hobby, and I'm not even particularly good at it. I'm doing my best, so if you run into problems, be kind.

https://github.com/nan0bug00/text-generation-webui

[Abattele]

GREAT JOB !! It works perfectly, you're a legend !! By the way, could you do something similar with ComfyUI (to make an installer for 5090) ? I know automatic1111 has a standalone version for blackwell, but it would be nice if comfyUi had some love too...

[Kyrandranos]

I made a Powershell scrip that roughly does the same thing for ComfyUI. It's not perfect but it should get you going. If you want it, send me a message in private.

[okazaki10]

try this #7011