SOTA 3-bit quants #5196

Merged

ikawrakow merged 14 commits intomasterfrom

Jan 30, 2024

Commits on Jan 27, 2024

iq3_xxs: quantize/dequantize
Iwan Kawrakow
committed
iq3_xxs: CUDA dequantize works
Iwan Kawrakow
committed
iq2_xxs: tuning quantization
Iwan Kawrakow
committed
iq3_xxs: starting to look better
Iwan Kawrakow
committed
iq3_xxs: CUDA dot product
Iwan Kawrakow
committed
iq3_xxs: scalar and AVX2 dot products
Iwan Kawrakow
committed
iq3_xxs: ARM_NEON and Metal
Iwan Kawrakow
committed

Commits on Jan 28, 2024

iq3_xxs: slightly better grid points
Iwan Kawrakow
committed
Faster iq3_xxs and iq2_xs dot products on CUDA
Iwan Kawrakow
committed

Commits on Jan 29, 2024

iq3_xxs: add some quant mix
Iwan Kawrakow
committed
iq3_xxs: fix failing quantization test
Iwan Kawrakow
committed
iq3_xxs: hopefully fix ROCm
Iwan Kawrakow
committed
iq3_xxs: failing tests
Iwan Kawrakow
committed

Commits on Jan 30, 2024

Add IQ3_XXS to test-backend-ops
Iwan Kawrakow
committed