Single precision cube vs double precision: how to anticipate the effects ?

Hi everyone,

I'm currently reviewing several possibilities to optimize the size of our cubes. By default, everything was created as double precision cubes, but this might not be necessary.
After some testing, I found that the gap between single and double precision cubes is not always significant, however I find it difficult to anticipate the effect of the rounding.

In the official documentation, it is only written:

Floating-point numeric type (4 Bytes). Each cell occupies 4 bytes.
Numbers that need more than 4 bytes to be saved will berounded.

Can we have something more specific ? When I compare the input and output of the data reader, I cannot find a precise logic for the rounding. I suppose this is linked to a binary format rounding but any explanation is welcome

Etienne

Find more posts tagged with

Accepted answers

All comments

unknown

Hi Etienne CAUSSE,

Since most computer systems use floating point arithmetic, floating point error is hard to eliminate. The best we can do is minimize it by increasing the floating point precision. Certainly for larger cubes, increased precision does increase storage requirements. For most implementations, I tend to use double to reduce the possibility of floating point error.

For floating point of arithmetic, values aren't rounded from a source value to a target value. They are just enumerated into the closest floating point value to the desired number.

Helmut Heimann

Hi Etienne CAUSSE,

you might want to have a look into this document Double-precision floating-point format - Wikipedia , additionally.

It might help explaining rounding differences.

Kind Regards,

Helmut

Etienne CAUSSE

Thanks both for your help. Indeed the wikipedia articles help

A summary of the limits of the format might be useful in the documentation though, such as:

Precision limits on integer values[edit]
Integers in [-16777216,16777216] can be exactly represented
Integers in [-33554432,-16777217] or in [16777217,33554432] round to a multiple of 2
Integers in [-2^{26},-2^{25}-1] or in [2^{25}+1,2^{26}] round to a multiple of 4
....
Integers in [-2^{127},-2^{126}-1] or in [2^{126}+1,2^{127}] round to a multiple of 2^{103}
Integers in {[-2^{128}+2^{104},-2^{127}-1] or in [2^{127}+1,2^{128}-2^{104}] round to a multiple of 2^{127-23}
Integers larger than or equal to 2^{128}} or smaller than or equal to -2^{128} are rounded to "infinity".