Strange 'Math' Behaviour

[Wandering Soulstar]

I was debugging my current project and had a spot that was not working, basically a comparison that should have been resulting in a TRUE, was not. When I dug into the code here is what I found:

This following line was supposed to get the x-axis value of a PRIM_POS_LOCAL call and strip off what for me were the insignificant digits, for a comparison agains a constant I have in my code:

float test = (llFloor(pos.x * 10000))/10000;

Imagine my surprise when I examined the value of test after the above operation: 0.00000

If instead of doint the math operation all on one line as above, I instead have the following code, then test will end up with the value I expected:

float test = llFloor(pos.x * 10000);
test = test/10000;

Any explinations for this strange behaviour?

 

 

 

[Dora Gustafson]

llFoor() returns an integer so in the first example when you divide something less than 10000 with 10000 the result is 0
Say 980/10000 = 0

In the second case you convert to float before dividing so you get 980.0/10000 = 0.098
(10000 is automatically cast to a float in this case)

:smileysurprised::smileyvery-happy:

[Wandering Soulstar]

Dora,

The point is the llFloor does not return a value less than 1:

pos.x = 0.044634

(llFloor(pos.x * 10000)) = 446

So thus why does float test = (llFloor(pos.x * 10000))/10000 not result in test = 0.0446 since I declare test as a float and then divide one integer by another? It should be the same as the code that works, just doing it on the same line as opposed to two.

[Wandering Soulstar]

In any case .. thanks to Dora's comments I understand what I need to do, though it does seem counter intutive to me. If I make any of the following changes to the line of code, it works as expected:

 

• float test = ((float)llFloor(pos.x * 10000))/10000;
• float test = (llFloor(pos.x * 10000.0))/10000;
• float test = (llFloor(pos.x * 10000))/10000.0;

 

Basically .. need to include a float somewhere in the equyation and the result will be a float, otherwise it will come out as an integer.

 

[Dora Gustafson]

---

Wandering Soulstar wrote:

Dora,

The point is the llFloor does not return a value less than 1:

pos.x = 0.044634

(llFloor(pos.x * 10000)) = 446

So thus why does

float test = (llFloor(pos.x * 10000))/10000

not result in test = 0.0446 since I declare test as a float and then divide one integer by another? It should be the same as the code that works, just doing it on the same line as opposed to two.

---

Because, as I said: 446/10000 = 0;

446 and 10000 are both integers and the division is consequently an integer division

I see you found out how to force a floating point division by turning one or both numbers into floating point, so problem solved and we can all be happy

:smileysurprised::smileyvery-happy:

[Nova Convair]

The behavious is as expected.

Never mix up float and integer if you don't know what you are doing or if you don't know how the compiler will handle it.

Use float for the whole calculation and don't forget to write 10000.0 instead of 10000 since you don't know how the compiler treats that. The last operation on the result of the calculation can be a conversion.

As you want to remove the fraction part of your float, I see why you tried that but it doesnt work that way.

float test = (float)(llFloor(pos.x*10000.0)) / 10000.0;

That will work.

Another approach is:

string test = llList2String(llParseString2List( (string)pos.x ,["."],[]),0);

If you want to have pos.x as a string for output.

[Wandering Soulstar]

Thanks Dora .. I missread your initial response .. to fast on the keyboard on my end.

 

And Nova .. not trying to throw away the 'fraction' part .. retaher trying to keep only the first four digits of the decimal, any more that comes in is extra and actually screws up my later calculations/comparisons.