Float32 converted Float64 during reference creation

[yuanjianz]

Kerchunk Verison: 0.2.7
I am using kerchunk to convert my netcdf files on s3 to json references. The original netcdf files are composed of variables of mixed data type of float32 and float64. After converted to json and read through xarray, all float data are converted to float64. I wonder is it an expected behaviour?

[martindurant]

The output type must be consistent everywhere within a variable, sorry.

[yuanjianz]

Hi @martindurant, thanks for the reply. Sorry for the confusion. The netcdf files contain several variables and they are int, float and double. In the reference created, variables with float precision in the original netcdf are converted to double somehow.

Input netcdf:

	float PM25(time, lev, nf, Ydim, Xdim) ;
		PM25:_FillValue = NaNf ;
		PM25:fmissing_value = 1.e+15f ;
		PM25:grid_mapping = "cubed_sphere" ;
		PM25:long_name = "Particulate matter with radii < 2.5 um" ;
		PM25:regrid_method = "bilinear" ;
		PM25:standard_name = "Particulate matter with radii < 2.5 um" ;
		PM25:units = "ug m-3" ;
		PM25:valid_range = -1.e+15f, 1.e+15f ;
		PM25:vmax = 1.e+15f ;
		PM25:vmin = -1.e+15f ;
		PM25:coordinates = "lats lons" ;
	double Xdim(Xdim) ;
		Xdim:_FillValue = NaN ;
		Xdim:long_name = "Fake Longitude for GrADS Compatibility" ;
		Xdim:units = "degrees_east" ;
	double Ydim(Ydim) ;
		Ydim:_FillValue = NaN ;
		Ydim:long_name = "Fake Latitude for GrADS Compatibility" ;
		Ydim:units = "degrees_north" ;
	int nf(nf) ;
		nf:axis = "e" ;
		nf:grads_dim = "e" ;
		nf:long_name = "cubed-sphere face" ;
	double lons(nf, Ydim, Xdim) ;
		lons:_FillValue = NaN ;
		lons:long_name = "longitude" ;
		lons:units = "degrees_east" ;
	double lats(nf, Ydim, Xdim) ;
		lats:_FillValue = NaN ;
		lats:long_name = "latitude" ;
		lats:units = "degrees_north" ;
	double lev(lev) ;
		lev:long_name = "vertical level" ;
		lev:_FillValue = NaN ;
		lev:standard_name = "model_layers" ;
		lev:units = "layer" ;
		lev:positive = "down" ;
		lev:coordinate = "eta" ;
	int64 time(time) ;
		time:units = "days since 2019-07-01 00:00:00" ;
		time:calendar = "proleptic_gregorian" ;
...

output json and read through xarray:

	double PM25(time, lev, nf, Ydim, Xdim) ;
		PM25:_FillValue = NaN ;
		PM25:fmissing_value = 999999986991104. ;
		PM25:grid_mapping = "cubed_sphere" ;
		PM25:long_name = "Particulate matter with radii < 2.5 um" ;
		PM25:regrid_method = "bilinear" ;
		PM25:standard_name = "Particulate matter with radii < 2.5 um" ;
		PM25:units = "ug m-3" ;
		PM25:valid_range = -999999986991104., 999999986991104. ;
		PM25:vmax = 999999986991104. ;
		PM25:vmin = -999999986991104. ;
		PM25:coordinates = "lats lons" ;
	double Xdim(Xdim) ;
		Xdim:_FillValue = NaN ;
		Xdim:long_name = "Fake Longitude for GrADS Compatibility" ;
		Xdim:units = "degrees_east" ;
	double Ydim(Ydim) ;
		Ydim:_FillValue = NaN ;
		Ydim:long_name = "Fake Latitude for GrADS Compatibility" ;
		Ydim:units = "degrees_north" ;
	double lats(nf, Ydim, Xdim) ;
		lats:_FillValue = NaN ;
		lats:long_name = "latitude" ;
		lats:units = "degrees_north" ;
	double lev(lev) ;
		lev:long_name = "vertical level" ;
		lev:_FillValue = NaN ;
		lev:standard_name = "model_layers" ;
		lev:units = "layer" ;
		lev:positive = "down" ;
		lev:coordinate = "eta" ;
	double lons(nf, Ydim, Xdim) ;
		lons:_FillValue = NaN ;
		lons:long_name = "longitude" ;
		lons:units = "degrees_east" ;
	int nf(nf) ;
		nf:axis = "e" ;
		nf:grads_dim = "e" ;
		nf:long_name = "cubed-sphere face" ;
	int64 time(time) ;
		time:units = "days since 2019-07-01 00:00:00" ;
		time:calendar = "proleptic_gregorian" ;
...

[martindurant]

OK, that's probably not supposed to happen. It would be worthwhile to see what h5py thinks the data type is when you get to here. I'm not sure if there is any configuration we can pass to h5py or if the upcasting is happening elsewhere.