decimal bug on data entry

Previous Member

Hi board members,

we use a single cube to enter distribution keys. In one example is just entered 0,24 and 0,76.

After a refresh i got the following values:

Same procedure works using 1,24 and 1,76.

I there any explanation for this behavior?

Best

Paul

Find more posts tagged with

Accepted answers

All comments

Daniele Di Lorenzo

Some suggestions:

Set the cube as double precision.

Or reduce the decimals visibility in the data view

Alexander Kappes

Dear Paul Stegmann,

problem is not really BOARD related, it´s general problem. Following link and explanation hopefully give you some more Information.

Like Daniele Di Lorenzo suggested using DOUBLE Cube brings you double precision concerning your fugures,

http://support.microsoft.com/kb/214118

This link to Microsoft Support site explains the issue as follows:

"The IEEE 754 standard is a method of storing floating-point numbers in a compact way that is easy to manipulate. This standard is used by Intel coprocessors and most PC-based programs that implement floating-point math.

IEEE 754 specifies that numbers be stored in binary format to reduce storage requirements and allow the built-in binary arithmetic instructions that are available on all microprocessors to process the data in a relatively rapid fashion. However, some numbers that are simple, nonrepeating decimal numbers are converted into repeating binary numbers that cannot be stored with perfect accuracy.

For example, the number 1/10 can be represented in a decimal number system with a simple decimal:
.1
However, the same number in binary format becomes the repeating binary decimal:
.0001100011000111000111 (and so on)
This number cannot be represented in a finite amount of space. Therefore, this number is rounded down by approximately -2.78E-17 when it is stored.

If several arithmetic operations are performed to obtain a given result, these rounding errors may be cumulative."

Hope it helps.

regards

Alexander Kappes

Previous Member

Thx - i'll switch to double.