Low Mixed Precision Performance

[fredlarochelle]

I am encountering some strange performance behavior on the A770. For example, taking the CIFAR-10 example in the documentation.

Using FP32, I get around 5.75s per epoch and, using BF16, I get around 6.2s per epoch. I also get the same exact performance with and without ipex.optimize().

Also, when I compare the performance with a Tesla T4 on Colab, in FP32, it runs each epoch in around 1s and, for FP16, around 0.25s. Wayy faster and the A770 has technically better specs...

Are the XMX engines being used on Arc GPUs? Yes #258?

Dunno, if it might be related, but I get the following warnings when running the example (EDIT I started going through the code in the repo, those warnings are not related to the current issue):

[/home/fred/.local/lib/python3.10/site-packages/intel_extension_for_pytorch/frontend.py:447](https://vscode-remote+ssh-002dremote-002b192-002e168-002e0-002e124.vscode-resource.vscode-cdn.net/home/fred/.local/lib/python3.10/site-packages/intel_extension_for_pytorch/frontend.py:447): UserWarning: For XPU device, the split master weight is unsupported for now, so temp to disable it
  warnings.warn("For XPU device, the split master weight is unsupported for now, so temp to disable it")
[/home/fred/.local/lib/python3.10/site-packages/intel_extension_for_pytorch/frontend.py:457](https://vscode-remote+ssh-002dremote-002b192-002e168-002e0-002e124.vscode-resource.vscode-cdn.net/home/fred/.local/lib/python3.10/site-packages/intel_extension_for_pytorch/frontend.py:457): UserWarning: For XPU device to save valuable device memory, temp to do optimization on inplaced model, so                     make inplace to be true
  warnings.warn(
[/home/fred/.local/lib/python3.10/site-packages/intel_extension_for_pytorch/frontend.py:464](https://vscode-remote+ssh-002dremote-002b192-002e168-002e0-002e124.vscode-resource.vscode-cdn.net/home/fred/.local/lib/python3.10/site-packages/intel_extension_for_pytorch/frontend.py:464): UserWarning: For XPU, the weight prepack and sample input are disabled. The onednn layout                     is automatically chosen to use
  warnings.warn(

Ubuntu 22.04 with 1.13.10+xpu.

[fredlarochelle]

I did some more digging around. It doesn't seem to be a problem with mixed-precision per se, more a performance problem in general.

Using the profiler, for FP32, I found that the aten: convolution_backward_overrideable operator is the most problematic. Just that operator takes 4.7s (80% of the time on the XPU) and, also, aten::convolution_overrideable, takes 0.9s (16%).

But if we compare again with a Tesla T4 on Colab, equivalent operators are faster on the A770, except for those two that are problematics.

Out of curiosity, I tried to run the same code on CPU and it took only 7.5s (compared to 5.75s on XPU), but the most surprising is that the torch_ipex:: convolution_backward operator on CPU was way faster, it took only 3.3s (compared to 4.7s on XPU)!

[asirvaiy]

Hi @fredlarochelle , Thanks for reporting. I have already faced this issue. We are working on it.
A few things to note and wanted to check.
always keep a few iterations for warm-up(at least 2).
Just wanted to check, are you using torch.xpu.synchronize() in xpu and torch.cuda.synchronize() in cuda?
When we calculate the latency using torch.xpu.synchronize() before start and end time, It really shows good numbers. Even though weights update part for the batch is taking more time due to aten: convolution_backward_overrideable

        torch.xpu.synchronize()
        btime = time()

        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        
        torch.xpu.synchronize()
        etime = time()

[asirvaiy]

Just checked, it's nothing to do with synchronize() and seems like not even aten: convolution_backward_overrideable

Each data batch from CPU to XPU is taking a significant amount of time. due to aten::copy_
Here : data= data.to("xpu:1")

[fredlarochelle]

Using torch.xpu.synchronize(), I get that it's a bit slower actually (around 6.2s, compared to 5.75s) (and that's after 3-4 iterations, where it's pretty stable).

Yeah, aten::copy_ is pretty slow too at around 0.9s (4x slower than Colab in my case, but that can depend on the CPU, memory, ...), but the main culprit is still aten: convolution_backward_overrideable.

[jingxu10]

That sample code just means to show usage of IPEX APIs, didn't mean to compare performance.
Would you try the training example with a large batch size?

[BA8F0D39]

@jingxu10
@fredlarochelle

As @fredlarochelle said aten::copy_ is extremly slow

The bottleneck seems to be at GPU to GPU transfer

Example code testing GPU to GPU memory copy

import torch
import torchvision

import intel_extension_for_pytorch as ipex

import time

sizes = [1000, 1000,1000,1000,1000,2000, 3000, 10000, 20000,30000,40000,42000,43000,44000,45000,]

for size in sizes:

    array0 = torch.rand(size, size, dtype=torch.bfloat16).to("xpu")
    array1 = torch.rand(size, size, dtype=torch.bfloat16).to("xpu")

    torch.xpu.synchronize()
    start = time.time()
    array0 = torch.clone(array1)
    torch.xpu.synchronize()
    end = time.time()
    transferrate = (size*size*16)/(end - start)
    datasize = (size*size*16)
    print("==========")
    print("Transfering " + str(datasize/8E9) + " GB")
    print("Bandwidth " + str(transferrate/8E9) + " GB/s")
    print("==========")


    torch.xpu.empty_cache()

It bottlenecks at 100 GB/s max transfer rate when the A770 bandwidth is 512 GB/s. Also, the most important thing is, transferring low amounts of data is extremely slow because small batches only achieves 30 GB/s which is 6% utilization of the maximum theoretical bandwidth of 512 GB/s.

==========
Transfering 0.002 GB
Bandwidth 29.746836879432625 GB/s
==========
==========
Transfering 0.008 GB
Bandwidth 50.45779248120301 GB/s
==========
==========
Transfering 0.018 GB
Bandwidth 71.29128611898017 GB/s
==========
==========
Transfering 0.2 GB
Bandwidth 80.71401905128451 GB/s
==========
==========
Transfering 0.8 GB
Bandwidth 91.80419151846785 GB/s
==========
==========
Transfering 1.8 GB
Bandwidth 102.16166711772665 GB/s
==========
==========
Transfering 3.2 GB
Bandwidth 103.13888713854288 GB/s
==========
==========
Transfering 3.528 GB
Bandwidth 102.9577837521917 GB/s
==========

[BA8F0D39]

Also matrix multiplication seems to be memory bandwidth limited

import torch
import torchvision

import intel_extension_for_pytorch as ipex

import time


sizes = [100,200,300,400,500,700,800,900,1000,2000, 3000,4000,5000,6000,7000,8000, 10000, 20000, 30000, ]

for i in range(10):
    for size in sizes:

        w = torch.rand(size, size, dtype=torch.bfloat16).to("xpu")
        x = torch.rand(size, size, dtype=torch.bfloat16).to("xpu")
        
        ops = 2*size**3   

        torch.xpu.synchronize()
        start = time.time()
        y = torch.mm(w,x)
        torch.xpu.synchronize()
        end = time.time()
        duration = (end - start)
        tflops = ops / duration / 10**12
        print("==========")
        print("Size " + str(size) + " by " + str(size))
        print("Performance " + str(tflops) + " TFLOPS")
        print("==========")


        torch.xpu.empty_cache()

TFLOPS in matrix multiplication steadily increases until 30000 by 30000 matrix and suddenly decreases. I am guessing matrix multiplication is limited by memory bandwidth and there are cache misses when fitting large matrices into the VRAM

==========
Size 100 by 100
Performance 0.011011431345646439 TFLOPS
==========
==========
Size 200 by 200
Performance 0.1333487885258964 TFLOPS
==========
==========
Size 300 by 300
Performance 0.4023397306761566 TFLOPS
==========
==========
Size 400 by 400
Performance 1.0155299684848484 TFLOPS
==========
==========
Size 500 by 500
Performance 0.9817501630740393 TFLOPS
==========
==========
Size 700 by 700
Performance 4.1549672471676296 TFLOPS
==========
==========
Size 800 by 800
Performance 4.508700568739496 TFLOPS
==========
==========
Size 900 by 900
Performance 5.3193192739425585 TFLOPS
==========
==========
Size 1000 by 1000
Performance 7.601463006346328 TFLOPS
==========
==========
Size 2000 by 2000
Performance 40.637242146577826 TFLOPS
==========
==========
Size 3000 by 3000
Performance 50.300903434917814 TFLOPS
==========
==========
Size 4000 by 4000
Performance 52.954898208148364 TFLOPS
==========
==========
Size 5000 by 5000
Performance 68.8153808348648 TFLOPS
==========
==========
Size 6000 by 6000
Performance 77.19263487094713 TFLOPS
==========
==========
Size 7000 by 7000
Performance 81.02188181087017 TFLOPS
==========
==========
Size 8000 by 8000
Performance 87.00238762927698 TFLOPS
==========
==========
Size 10000 by 10000
Performance 89.37010376841857 TFLOPS
==========
==========
Size 20000 by 20000
Performance 101.03734560466859 TFLOPS
==========
==========
Size 30000 by 30000
Performance 74.42093049111077 TFLOPS
==========

[fredlarochelle]

@BA8F0D39 Quick tip, when possible, creating the matrix on XPU will make everything run wayyy faster.

# Instead of
w = torch.rand(size, size, dtype=torch.bfloat16).to('xpu')
x = torch.rand(size, size, dtype=torch.bfloat16).to('xpu')

# Do this
w = torch.rand(size, size, dtype=torch.bfloat16, device='xpu')
x = torch.rand(size, size, dtype=torch.bfloat16, device='xpu')

For me, doing this, your TFLOPS program runs about 30x faster.

[BA8F0D39]

@fredlarochelle
Thanks for the tip. I was wrong to think torch.xpu.synchronize() synchronizes between CPU and GPU

Also, does your A770 GPU allow you to allocate bf16 memory above 8GB? My code crashes on the A770 if you try to allocate bf16 memory above 8GB. I have a A770 16 GB card.

[fredlarochelle]

No, like with CUDA, torch.xpu.synchronize() is to sync all the kernel on the GPU and wait for their completition.

And I can also confirm that I can't seem to allocate over 8GB at all on the A770 16GB (with fp32 and bf16) I get the following error RuntimeError: Native API failed. Native API returns: -997 (The plugin has emitted a backend specific error) -997 (The plugin has emitted a backend specific error)

[BA8F0D39]

@jingxu10
Does pytorch enforce memory limits to how much you can allocate and the memory bandwidth?

[gujinghui]

@jingxu10 Does pytorch enforce memory limits to how much you can allocate and the memory bandwidth?

No, PyTorch does not. It should be driver limits.