Floating point numbers are approximations of real numbers that can represent larger ranges than integers but use the same amount of memory, at the cost of lower precision. If your question is about small arithmetic or decimal conversion errors, please read the "learn more..." page linked below ...

learn more… | top users | synonyms

0
votes
0answers
22 views

numpy array division floating point error inconsistent behaviour?

To cut a long story short, I'm doing division of a numpy array of float32s which results in an overflow error for a single value. However, if I divide just the value by itself (not as an array), ...
1
vote
2answers
207 views

Is it possible to clear the FPU?

I'm using Delphi XE6 to perform a complicated floating point calculation. I realize the limitations of floating point numbers so understand the inaccuracies inherent in FP numbers. However this ...
-6
votes
0answers
43 views

Why 0.5==0.5 f is true while 0.1==0.1f false? [duplicate]

Please look at following codes: public static void main(String... args) { System.out.println(0.5==0.5f); } Output : true public static void main(String... args) ...
1
vote
0answers
38 views

VS2013 : compiler bug with float and /EHa + /fp:precise?

We just moved from VS2010 to VS2013 and I found a strange bug and I wonder it might be due to the compiler. Compiled with the command line cl ConsoleApplication1.cpp /EHa /fp:strict /O2 the following ...
0
votes
2answers
30 views

When working with floating point numbers, is it good to normalize between 0 and 1?

I have numbers in the range of let's say 1e10 and 1e11. Is it better to normalize those numbers to [0;1] before making any calculations and/or comparisons for the sake of accuracy? I wonder because I ...
0
votes
1answer
38 views

Why exactly 0.09f - 0.01f == 0.0800000057f? [duplicate]

I know that floating point variable stores the number in a sign-exponent-fraction format (as it's stated in the IEEE 754), it's never precise and I should probably never compare two floats without ...
-2
votes
0answers
29 views

Convert .33 to fraction 1/3 in PHP [duplicate]

The problem is that the input value is .33 and not .33333333 and so on to infinity. What I want is to recognize this is 1/3. And I want to do that in PHP, which doesn't have a fraction variable type ...
0
votes
1answer
39 views

Using int() with decimal numbers [on hold]

I'm using try for data validation (To prevent a string from being entered), as below: try: int(user input from raw_input) except ValueError: do something. However, this statement is ...
1
vote
2answers
33 views

Multiple dynamic float format specifier in C

It happened to me several times to work with float variables having to printf them out properly formatted with a certain number of decimal positions. I know that in C you can easily format float ...
3
votes
2answers
64 views

Subdividing square problems due to floating point precision errors

I have a problem (in JAVA) which I unable to solve due to floating point precision errors. I have an axis-aligned square class which is defined by specifying 2 points. In the constructor it is ...
1
vote
0answers
66 views

Python+numpy: a lucky floating point case?

I'm inverting a very simple matrix, and I stumbled in something which I don't understand. EDIT: In the original post I forgot to take care of the fact that numpy defaults to float64, hence the ...
0
votes
1answer
27 views

How to convert 32-bit binary to float

I want to perform IEEE 754 conversion from 32-bit binary to float in python. i have tried this import struct f = int('11000001101011000111101011100001', 2) print struct.unpack('f', struct.pack('i', ...
17
votes
2answers
286 views

Mathematical explanation of Decimal different representations as Double

I am not sure if this non-standard way of stating a Stack Overflow question is good or bad, but here goes: What is the best (mathematical or otherwise technical) explanation why the code: static ...
1
vote
1answer
36 views

Formatting a list of floats to 2 Decimal Points

I am trying to search though text to find Money EG £12.30 take the numbers and then sum them.I have managed to get to the point where I have a list of floats but I can't seem to get them to 2 decimal ...
1
vote
0answers
32 views

pragma STDC FENV_ACCESS ON is not supported

I tried to slightly modify the example from the article: #include <iostream> #include <cfenv> #pragma STDC FENV_ACCESS ON int main() { std::feclearexcept(FE_ALL_EXCEPT); //int r ...

15 30 50 per page