Since you state that you need a precision somewhere between 'float' and 'double': you can zero out any number of least significant bits in single- and double-precision floats. IEEE-754 floating point numers are represented binary roughly like seeefffffffff
, which represents the value
sign*1.fffffff*2^(eee).
You can zero out the least significant fraction (f) bits. For single-precision (32-bit) floats, there are 23 fraction bits of which you can zero out up to 22. For double-precision (64-bit), it's 52 and up to 51. (If you zero out all bits, then the special values NaN and +/-inf will be lost).
Especially if the data represents decimal values such as 1.2345, this will help in data compression. That is because 1.2345 cannot be represented exactly as a binary floating point value, but rather as 0x3ff3c083126e978d
, which is not friendly to data compression. Chopping off the least significant 24 bits will result in 0x3ff3c08312000000
, which is still accurate to about 9 decimal digits (in this example, the difference is 1.6e-9).
If you do this on the raw data, and then store the differences between subsequential numbers, it will be even more compression-friendly (via gzip) if the raw data varies slowly.
Here is an example in C:
#include <inttypes.h>
double double_trunc(double x, int zerobits)
{
// mask is e.g. 0xffffffffffff0000 for zerobits==16
uint64_t mask = -(1LL << zerobits);
uint64_t floatbits = (*((uint64_t*)(&x)));
floatbits &= mask;
x = * ((double*) (&floatbits));
return x;
}
And one in python/numpy:
import numpy as np
def float_trunc(a, zerobits):
"""Set the least significant <zerobits> bits to zero in a numpy float32 or float64 array.
Do this in-place. Also return the updated array.
Maximum values of 'nzero': 51 for float64; 22 for float32.
"""
at = a.dtype
assert at == np.float64 or at == np.float32 or at == np.complex128 or at == np.complex64
if at == np.float64 or at == np.complex128:
assert nzero <= 51
mask = 0xffffffffffffffff - (1 << nzero) + 1
bits = a.view(np.uint64)
bits &= mask
elif at == np.float32 or at == np.complex64:
assert nzero <= 22
mask = 0xffffffff - (1 << nzero) + 1
bits = a.view(np.uint32)
bits &= mask
return a