Questions tagged [precision]
For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].
4,942
questions
2
votes
1
answer
47
views
Check if root of cubic equation is complex or not?
I use this Cubic root implementation.
I have equation #1:
x³ -2 x² -5 x + 6 = 0
It gives me 3 complex roots ({real, imaginary}):
{-2, 7.4014868308343765E-17}
{1 , -2.9605947323337506E-16}
{3 , 2....
2
votes
5
answers
88
views
Do we need epsilon value for lesser or greater comparison for float value? [duplicate]
I have gone through different threads for comparing lesser or greater float value not equal comparison but not clear do we need epsilon value logic to compare lesser or greater float value as well?
e....
0
votes
0
answers
34
views
Trying to generate a series of decimal numbers in AP in Fortran 90 [duplicate]
I am trying to get series of numbers starting from zero by increasing it by 0.1 by coding it in Fortran 90. But I am not getting exact values. I read a lot of answers that it is floating point error. ...
1
vote
1
answer
25
views
How to perform binomial-coefficient and factorial calculation with more precision?
I was comparing the result of my following python calculation with Mathematica: https://www.wolframalpha.com/input?i=sum+%28500+choose+r+%29%28-1%29%5Er+%2F%28r%21%29+%2C+r%3D0+to+500
import numpy as ...
0
votes
0
answers
29
views
Postgres ERROR: cannot cast type double precision to date [duplicate]
I have a table with creation timestamps as decimals and am trying to convert to date data type. When I go to cast as date I get the following error message: "ERROR: cannot cast type double ...
1
vote
1
answer
31
views
Warp Matrix-Multiply functions - are single-precision multiplicands supported?
In the CUDA Programming guide, v11.7, section B.24.6. Element Types & Matrix Sizes, there's a table of supported type combinations, in which the multiplications are either sub-single-precision ...
1
vote
1
answer
14
views
Storing a positive floating point number in a known range in two bytes
I know that a number x lies between n and f (f > n > 0). So my idea is to bring that range to [0, 0.65535] by 0.65535 * (x - n) / (f - n).
Then I just could multiply by 10000, round and store ...
0
votes
1
answer
19
views
Representing numbers with IEEE754 with Round to Nearest Even
I'm currently learning about IEEE754 standard and rounding, and I have an exercise which is the following:
Add -325.875 to 0.546875 in IEEE754, but with 3 bits dedicated to the mantissa instead of 23.
...
4
votes
1
answer
67
views
Do FMA instructions of different CPUs have different intermediate accuracy? If yes, then how does a compiler equalize the floating-point behavior?
When I run fma-optimized horner-scheme polynomial computation (for cosine approximation), it makes 0.161 ulps error on FX8150 but 0.154 ulps on godbolt.org server despite the lack of -ffast-math (GCC)....
0
votes
0
answers
30
views
Reduce float precission to save memory in Python
I am preparing a code to create an array of 10^5 rows and 10^4 columns with Python. Each element of the array is a float (1.504096744947185, e.g.), which takes 24 Bytes, so the whole array would need ...
-2
votes
1
answer
56
views
Is it possible to calculate the value after overflow?
I understand that if 1 is added to INT_MAX it will overflow and we will get the negative maximum. It is connected in a cycle. In the code below can I calculate the value after overflow.
#include<...
0
votes
0
answers
10
views
Calculating confusion-matrix for Yolact in eval.py
I trained Yolact algorithm but I cannot calculate confusion-matrix for that in eval.py.
-1
votes
1
answer
56
views
Wrapping around very large angles to [0,2*PI] adds too big numerical error [closed]
In below sample, I try different methods to wrap an angle between 0 and 2 x PI. While doing this to compute cosine of same angles, both methods cause a different amount of numerical error compared to ...
1
vote
2
answers
72
views
Why is multiplication of numbers with floating point inside a loop isn't as precise as outside of it?
I was writing a program to calculate sin(x) with x in [0, 1], using Taylor series that include raising some of its terms to a certain power. Since I'm new to C, I decided to write my own pow() ...
-1
votes
1
answer
41
views
What is the smallest float number (the farthest from zero)
What is the smallest float number (the farthest from zero) that we can represent with precision ?
i googled the question but the answers Values are in floating point and denormalized number ..
So what ...
0
votes
1
answer
35
views
Java InputStream float value too big
I am struggling how to read float value from inputStream.
I tried before with BigDecimal, and it was not possible to get the correct number
Problem: im expecting V ~0.00 - 1.50, Later up to 55.00
...
1
vote
1
answer
25
views
compute probabilities from log scores
I'm using R and have the following task. Let's say I have access to log(a) and log(b), and I want to compute a/(a+b). Further suppose that log(a) and log(b) are very negative, e.g. -1000, then how do ...
3
votes
3
answers
103
views
Strange warning when calling sprintf with .* width specifier
For the following code:
https://godbolt.org/z/WcGf9hEs3
#include <stdio.h>
int main() {
char temp_buffer[8];
double val = 25.3;
sprintf(temp_buffer, "%.*g", sizeof(...
2
votes
2
answers
143
views
sprintf precision typecasting leading to compilation warning
I have the below C++ code which issues a compilation warning as shown below.
Case 1:
char temp_buffer[80];
double **data;
....
sprintf(temp_buffer, "%.*g", sizeof(temp_buffer), **data);
...
2
votes
0
answers
50
views
Check if number should be C++ float or C++ double using python
I am working on a simple C++ code generator created by excel parsing and I will need to distinguish between number type. I can deduce integer size (e.g. uint_32_t) using sys.getsizeof()
But How can I ...
2
votes
2
answers
57
views
Overwrite toFixed() with appropriate replacement to fix floating point error javascript
This is my attempt to fix the JavaScript toFixed() function...
Any input, ideas, corrections for possible errors are much appreciated!
Fix floating point inacurracy (example (35.355).toFixed(2) = 35....
1
vote
1
answer
41
views
SQL Server - Decimal values not precisely calculated
For the below calculation, the expected result is -0.0000671. But the result of the code is -0.00006800000.
declare
@V_A decimal (38,11) = 99.99329,
@V_B decimal (38,11) = 100,
...
0
votes
0
answers
9
views
is nDCG a precision-oriented measurement? Why?
is nDCG a precision-oriented measurement? Why?
1
vote
1
answer
107
views
What is correct way to compare 2 doubles values? [duplicate]
There are a lot of articles, PhD publications, books and question here on this topic, still my question will be pretty straightforward.
How can we improve this or workaround without writing dozens of ...
0
votes
0
answers
36
views
Big numbers problem while "odeint" numerical integration
I'm having some computational problems with the following code:
import numpy as np
from numpy import arange
from scipy.integrate import odeint
import matplotlib.pyplot as plt
from scipy.integrate ...
6
votes
3
answers
162
views
What is the minimum number of significant decimal digits in a floating point literal to represent the value as correct as possible?
For example, using IEEE-754 32-bit binary floating points, let's represent the value of 1 / 3. It cannot be done exactly, but 0x3eaaaaab produces the closest value to 1 / 3. You might want to write ...
0
votes
1
answer
55
views
Why is the IEEE754 double precision format only accurate to 15 digits?
I am currently learning about floating point representations. According to this website, here are the possible ranges for this representation. -1.79E+308 to -2.23E-308, 0 and 2.23E-308 to 1.79E+308. ...
4
votes
1
answer
111
views
C: Rounding floats
Could you please explain this code:
printf("Value1: %0.1f \n", (float)124/100); // = 1.2
printf("Value1: %0.1f \n", (float)125/100); // = 1.2
printf("Value1: %0.1f \n&...
-1
votes
0
answers
57
views
Doing precise calculations in python
I am performing the following calculation in a python program.
(dt*(b*h[k1]-(mu+b)*f[k1]))/(ms*mu + (ms+mu)*b)
But there seem to be some accuracy issues. As the dt, b, ms, mu, all these variables ...
-2
votes
0
answers
32
views
Is a Golang Float64/32 Interpreted Differently Between Code, Debugger & When Printed? [duplicate]
I'd like to understand why a float64 (or 32) value seems to be interpreted differently between what I write in code, what is printed and what my debugger shows (using vscode, when a variable holding ...
0
votes
1
answer
22
views
lodash _.sum method is giving odd result
I am using lodash version 4.17.15
Here when I am using its _.sum method, with certain values, I'm getting a bit odd result
Here are some examples
_.sum([5.32,1.25,1.3])
this is giving correct result #...
0
votes
0
answers
31
views
Pandas exponential power not computing as it should
I think I found an issue with floating point precision when using exponents in PANDAs. I had a list of numbers I was running through a 4th order polynomial, when I got to orders greater than 2 I ...
2
votes
2
answers
55
views
For loop in which the counter goes up to a value generated by Math.floor()
I'm a new web developer, been learning web dev for around 8-9 months. I recently became a mentor for new students in the bootcamp I graduated and I wanted to write a simple program to calculate all ...
1
vote
2
answers
143
views
Subtract extremely small number from one in C++
I need to subtract extremely small double number x from 1 i.e. to calculate 1-x in C++ for 0<x<1e-16. Because of machine precision restrictions for small enoughf x I will always get 1-x=1. ...
3
votes
1
answer
82
views
fread/fwrite introduces garbage values
Data file data.dat:
5625010350032.36719 5627008621379.12591 5628763999478.55791 5630383772880.98831 5632384688238.96095 5633992371569.87936 5635830220975.76879 5637713568911.67183 5639436594135.51215 ...
5
votes
1
answer
89
views
Is it ok that with fp:fast 3000.f/1000.f != 3.f?
I'm using MSVC 2019 v16.11.12.
When I tried compiling my code with /fp:fast instead of /fp:precise, my tests started failing.
The simplest case is:
BOOST_AUTO_TEST_CASE(test_division) {
float v = ...
0
votes
0
answers
49
views
Differential Equation in R and Python with one parameter that changes in time (odeint and deSolve) [solved]
I'm trying to "translate" a script I have in R that can solve a DE of a function that has a parameter changing in time. I've compared with the results that I obtained from a commercial ...
0
votes
1
answer
64
views
Pandas json conversion messing with float precision
After saving a DataFrame as a json string with to_json(), when the json is cast back to a DataFrame with read_json(), the initial and resulting DataFrame are not equals. I understood that there were ...
0
votes
0
answers
20
views
why am i getting more decimal places from subtraction? [duplicate]
was doing an algorithm question, and my code produced unexpected output. heres the code
const price = 3.26;
const cash = 100;
const cid = [
["PENNY", 1.01],
["NICKEL", 2.05],
...
0
votes
0
answers
35
views
Why does Python (3.10.2) divide extremely large/small numbers incorrectly? [duplicate]
So I am new to programming and am currently trying to make a program to calculate the amount of photons that are emitted from a metal by a specific wavelenght or frequency (photoelectric effect). The ...
0
votes
0
answers
87
views
Intel 8087 floating point numbers
Good morning,
I have to change the constant values "wired to code" for an old machine based on intel 8086 and intel 8087 coprocessor.
I have the source code, written in the RTL/2 language. ...
1
vote
1
answer
69
views
numpy.linalg.solve for Decimal in Python
Is there any way how to solve equations when a matrix A and a vector b is composed of decimal.Decimals?
My A:
array([[Decimal('-5266125828.168885444558615257'),
Decimal('11312418445....
0
votes
0
answers
15
views
How to use exact values of axes' using matplotlib in python [duplicate]
I have taken some graphics using matplotlib in python. But, y values cannot be understood easily. The picture is given below
enter image description here
How can I change these coordinate scripts. How ...
0
votes
2
answers
69
views
Find least significant digit in a double in Python
I have a lot of financial data stored as floating point doubles and I'm trying to find the least significant digit so that I can convert the data to integers with exponent.
All the data is finite, e.g....
-1
votes
3
answers
61
views
How can I remove empty space in float from a number I get from web scraping? Error: could not convert string to float: '1\xa0364'
in the code I try to take price data from website. This website use an empty space in the price and the float class raises a flag:could not convert string to float: '1\xa0364'
this code should extract ...
0
votes
2
answers
87
views
Looking for suitable datatype in C to handle two mathematical expressions
I have to calculate following two mathematical formula in C programming language.
result = 48^108 // 108 should be 103
result%143 =9
To calculate this two equation I need some data type whose range ...
0
votes
1
answer
54
views
Wrong floating point results when calculating in loops
I wrote the following C++ code in Visual Studio 2017 to evaluate floating point performance:
#include <iostream>
#include <Windows.h>
int main(void)
{
LARGE_INTEGER frequency; /...
0
votes
2
answers
28
views
Converting a double to a long, then back to double?
Does Converting a double to a long, then back to double, guarantees keeping the exact value to the left of the decimal point?
EDIT:
Working with C++: Conversion is as follows:
double d_var = func();
...
3
votes
2
answers
98
views
double precision in C++, 308 digits or 15 digits?
If double can represent value up to 3.4E308 (308 zeros), then why do we say that double stores only 15 digits? What is point of saying "ten power of 308" ?
2
votes
1
answer
43
views
How to emulate Excel's automatic rounding?
When Excel reports the results of a floating point calculation, it appears to round it to the correct precision based on the significant digits in the input numbers. But if I include the result in a ...