PDA

View Full Version : Precision or just how many decimal places are enough?



RPMiller
Mar 30th, '03, 02:25 PM
While examining Shrinker from CKC in HD I noticed that her stats require more decimal places than HD gives. Is there any chance of increasing the number of decimal places beyond 4, or is that the furthest that Java can go? I know that math was a consideration, but what about precision of that math? I am positive that the numbers are accurate even though it doesn't display.

Simon
Mar 30th, '03, 03:06 PM
Errr...Internally to HD, I'm using 64 bit floating point decimals.

The issue is entirely one of usability. The number of decimal places given for abilities like Shrinking are (at this point) fixed, regardless of the level of Shrinking taken. Four decimal places is enough for all but the most extreme cases....anymore and the majority of people will be put out in order to accomodate the one or two characters that are an exception.

For the time being, it's going to remain as-is....though as we move forward I may do some work to make it a bit more 'intelligent' in the decimal places, increasing the number of places as necessary.

Simon
Mar 30th, '03, 04:33 PM
I was thinking about this a bit and came up with the following solution:

The system will use four decimal places until the value drops below .0001, at which point it will switch over to scientific notation (e.g. 1.234E-6 = .000001234).

I'll have this included in Shrinking for the next update.

RPMiller
Mar 30th, '03, 07:46 PM
Originally posted by dsimon
I was thinking about this a bit and came up with the following solution:

The system will use four decimal places until the value drops below .0001, at which point it will switch over to scientific notation (e.g. 1.234E-6 = .000001234).

I'll have this included in Shrinking for the next update. That's very cool of you Dan. We all appreciate your willingness and abilities to make your wonderful program better even for those of us "retentive" types. ;)