dim y_m as double
dim y as integer
y_m = 6.09
For i = 1 To 12
y_m = y_m + 0.01
y = y_m
If y_m = CDbl(y + 0.12) Then
y_m = y + 1
End If
Next
When y_m is 6.12 the if statement doesn't catch that fact!!!
The CDbl() function doesn't seem to help. It doesn't work whether or not I have the conversion function.
Single precision is even worse because adding 0.01 several times gets you not 6.10 and 6.11 but something crazy like 6.110002.
Brian
what is wrong with this?
NET PR
There is no guarantee in how the editor is showing the values and how the values are rounded.
Double might rounded to 7 decimals be 6.11000000 but it actually is 6.11000000001. Rounded output will display 6.11 but it is not exactly what it contains. The single value might be 6.1100001751, rounded to 7 decimal it is 6.1100002 and will be displayed as such.
For exact math do not use Double/Single but Decimal.
If you want to read more about floating point.
http://en.wikipedia.org/wiki/Floating_Point
Girardelli
Double and Single data types are floating point numbers, they usually do not contain the exact number but something very close. You might want to use Decimal instead.
You should never use floating point numbers if you expect precise numbers.
soconne
In the decimal number system, numbers like 1/3 or the square root of 2 cannot be represented exactly. Likewise in binary, numbers such as 0.1 cannot be represented exactly.
See the following article for more information. You should use the Decimal data type instead.
http://www.yoda.arachsys.com/csharp/floatingpoint.html
sunil_sg
In case you need to use the Sytem.Double, take a look at http://msdn2.microsoft.com/en-us/library/system.double.epsilon.aspx
SaloS
Well then there's something wrong with the VBEditor because when I put the cursor over variables, it shows exactly 6.11 not 6.1100002. The only time I see 6.1100002 is when the type is Single. I fully expect the code to work if it is a Double given what the VBEditor is showing me.
This is fcuked up.