This issue almost looks familiar, but I cannot recall for sure.
Surely someone can point out what I am missing…
I’m trying to do a simple math problem & store the answer in a decimal variable.
Here is my first attempt:
(The answer should be 1.25, not 1)
Second attempt trying to force a decimal:
Third attempt trying to force 10 decimals:
This looks so easy… What the heck am I missing?!?
Part Two…
Does anyone know the maximum number of decimal places webCoRE can spit out in a math equation? (I am really trying to do 100 / 8.04 million as precisely as possible… Hopefully 20 decimal places or more)
For example1:
1/3 will give you 0
However 1/3.0 will give you .3333333333333333
As far as part 2 goes:
Try:
format("%.5f", 1/3.0)
It will only return 5 decimal places
The number of decimal places may vary though, don’t know why.
Thanks man… That did the trick!! It was driving me bonkers!!
Apparently, the forced decimal can also be the first number (100.0 in this case)
I have done a little bit more testing, using slightly different code, and I was actually able to push it up to 21 decimals. In the following image, the only thing I changed between tests is the last number in the formula (yellow box). You can see when I tried using 24+ decimals, webCoRE actually returned incorrect data (red box)… Using 21 thru 23 decimals returned the exact same data as 21 (green box).
I think I will go with 21 decimals for the most precision.
