Increasing loss on ParametricUMAP training with Keras 3.7 on GPU #1211
Description
In testing various configurations for #1200 and #1180 I found some unusual behavior. I have access to two different environments at the moment (linux VM, mac with M2). All tests were done with the MNIST_Landmarks notebook.
When training a normal ParametricUMAP model for the first time the loss increases over time on Keras 3.7 (after a short initial period of decreating) but works just fine on 3.6. I don't have time to dig into the version differences now - I don't think this is related to the above linked issues but it could be.
| OS/Hardware | Python | Keras | tensorflow | torch | umap | Works? |
|---|---|---|---|---|---|---|
| Mac, M2 | 3.12.11 | 3.6 | 2.19.0 | 2.7.1 | 0.5.9.post2 | Yes |
| Mac, M2 | 3.12.11 | >=3.7 | 2.19.0 | 2.7.1 | 0.5.9.post2 | No (loss increases) |
| Mac, cpu | 3.12.11 | 3.6 | 2.19.0 | 2.7.1 | 0.5.9.post2 | Yes |
| Mac, cpu | 3.12.11 | >=3.7 | 2.19.0 | 2.7.1 | 0.5.9.post2 | Yes |
| Linux, cpu | 3.12.2 | 3.6 | 2.17.0 | 2.6.0 | 0.5.9.post2 | Yes |
| Linux, cpu | 3.12.2 | >=3.7 | 2.17.0 | 2.6.0 | 0.5.9.post2 | Yes |