| Normal or not? |
| Walrus | The code:[code]
Dim i As Single
For i = 0 To 3 Step 0.1
Debug.Print i
Next i
[/code]
And the result:
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8000001
0,9000001
1
1,1
1,2
1,3
1,4
1,5
1,6
1,7
1,8
1,9
2
2,1
2,2
2,3
2,4
2,5
2,6
2,7
2,799999
2,899999
2,999999
Normal or not?
|
| Eric Coleman | yes, that's normal. Data in computers is stored in a binary format.
Consider the integer conversion from binary to decimal. Each digit is a multiplier for the base. Example, the number 804 in decimal. What each digit means is that each number is a multiple for a certain value. The values in decimal, from right to left are 1, 10, 100, 1000, 10000. Or in power notation it's 10^0, 10^1, 10^2, 10^3, 10^4, etc. The numer 804 is the equation (4 * 10^0) + (0 * 10^1) + (8 * 10^2) + (0 * 10^3) + (0 * 10^4) .... The value of 0 for unwritten digits is implied.
Now consider binary, but instead of base 10 it is base 2. Each digit from right to left is a power of 2 instead a power of 10. 1, 2, 4, 8, 16, etc. The number 1011 is the sum of (1 * 2^0) + (1 * 2^1) + (0 * 2^2) + (1 * 2^3) + (0 & 2^4) + ....
If that makes any sense to you, then consider fractions. 1/10 in power notation is 10^-1. 1/100 is 10^-2. The value of PI would be 3,141..., in in decimal notation it would be (3 * 10^0) + (1 * 10^-1) + (4 * 10^-2) + (1 * 10^-3) ....
Positive powers, 10^(1,2,3 etc) are multiples of 10. Negative powers or divisions of 10. For binary numbers a fraction has to be represented as sum of the units of 1/2, 1/4, 1/16, 1/32, 1/64, 1/128. Or in power notation it would be 2^-1, 2^-2, 2^-3, 2^-4, etc.
The conversion from binary fractions to decimal fractions isn't easy and it is not always possible to have an accurate representation with the limited precision of a computer.
Hopefully you understand what numbers represent. However, the SINGLE and DOUBLE data types aren't exactly as I described. They're a little bit more complicated because they allow an arbitrary size within a limited precision. If you understand the stuff I described above, then grapsing floating point notation should be easy for you. |
| Walrus | I understand. I thought it had sth to do with binary format, but I've never thought about how fractions are stored.
Btw, I wrote a C++ equivalent, and this time the result was like you'd expect it to be. So how does C++ deal with this?
Thank you very much for your time! |
| Sion | Wow, I only thought that happend in C/C++. Stupid machines not able to store unlimited numbers for decimals! Bah! [;)] |