NetCDF output failure

[ReadingClouds]

I have a very odd problem; we have some 'standard' idealised UM setups ([https://code.metoffice.gov.uk/trac/rmed/ticket/551]), originally setup at UM 12.0 and outputting NetCDF files. These work fine.

I am in the process of upgrading to UM 13.0; I have found a very reproducible problem. Suites run OK, but if I turn on the Smith cloud scheme, STASH output fails as soon as any data are written to output NetCDF (incidentally, I haven't yet tried reverting back to pp, but regular dumps also work fine. The error is:

****************** NetCDF_File Error Report ***************************
Problem with unit 17 filename is /home/d04/appc/cylc-run/u-do671/share//do671_1km_L75a_pr000.nc

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 60
? Error from routine: NC_PUT_VAR_REAL_1D
? Error message: NetCDF: Numeric conversion not representable : NF90_PUT_VAR
? Error from processor: 0
? Error number: 22
????????????????????????????????????????????????????????????????????????????????

As said, the model is running perfectly OK. If I turn on the bimodal cloud scheme, the error goes away.

This is a baffling problem - obviously, NetCDF i/o is unconnected with the choice of cloud scheme. I've tried disabling all cloud-related diagnostics to no avail.

Anyone got any ideas how to proceed?

[iboutle]

As I think you suspect, this is most likely due to their being a NaN or similar in one of the diagnostics being written, and switching between Smith & bimodal is presumably triggering that bad value to be present/not-present. The unknown is what diagnostic is causing the issue - not sure if there's any way to get this out of the netcdf writer. You said it still fails if you remove all "cloud-related" diagnostics - what about if you remove all diagnostics? Might be a question of adding/removing chunks at a time until you track down which one causes the issue.

[ReadingClouds]

Interestingly, I no longer get this error if I ask for 64 bit NetCDF files. I can now output all the diagnostics I had before.
This is a pragmatic solution, but at UM12.0 I could get 32 bit output. Since the runs should be bit reproducible (I haven't checked this, but they were at 12.0) I'd still have NaNs in the diags. I'm wondering if the scheme is generating numbers that are too small for 32 bit and they're not being converted correctly? Hard to track down as it would appear they're OK till they get to the file write.

[iboutle]

Yes, very much sounds like something >10^35 or <10^-35 and so it can't be sensibly converted to single precision in the file write. Have seen this before with the Richardson number or Obukhov length, which can have very extreme values (for sensible reasons!)