Questions tagged [precision]
For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].
4,531
questions
0
votes
1
answer
37
views
High Accuracy Plotting Julia
I need to plot a vector to check the smoothness after a time step. I am using BigFloats for my calculations as I need around 30 significant figures of precision. However when I plot the curve it ...
0
votes
0
answers
8
views
How to improve the precision of matrix calculation in Matlab or Python?
Now, I have a matrix S, and it satisfies the recursion relation:
S_{m,n}=(1+|α|^2)S_{m-1,n-1}+(m-1)(n-1)S_{m-2,n-2}+α(m-1)S_{m-2,n-1}+α*(n-1)S_{m-1,n-2}
S_{0,n}=(α*)^n and S_{n,0}=α^n
Obviously, when |...
0
votes
1
answer
75
views
Format a number to 8 or 16 chars without "e" with highest precision
I have the following python procedure.
Input is any number (from a calculation) and format, where format can be 'long' or 'short'
What i want to do:
If the format is short, i want the output to be the ...
0
votes
0
answers
8
views
In arangodb how to save decimal with trailing zeroes in order to keep the precision of the value
So if the user inputs 12.00000 I need to save the trailing zeroes 5 in this example.
I am evaluating arangodb. But there is no clear information about this.
I would use BigDecimal in Java to handle ...
0
votes
1
answer
90
views
Rounding errors when computing intersection between two lines in python
I'm attempting to create a Python script that identifies all the unique intersection points, eliminating duplicates, among all the lines passing through a subset of points of a regular grid in 2D or ...
0
votes
1
answer
39
views
Precision and recall for which class is calculated in GridSearchCV?
I'm running GridSearchCV on XGBoost model to tune some parameters. My target variable is binary. When I use, 'recall' or 'precision' as scoring metrics in cross validation, I get a single value. I'm ...
-2
votes
0
answers
75
views
Is this an issue with my code or the expected output?
I have to create a program that outputs weather data from a file.
my issue is with one of my functions PrintMonthWeatherSummary.
the month of June gives the wrong output.
void PrintMonthWeatherSummary(...
0
votes
1
answer
70
views
What's the real limit for NUMBER data type in Oracle?
I need help with understanding NUMBER data type, because 2 below statements from Oracle documentation seems to be for me mutually exclusive.
"The NUMBER data type stores zero as well as positive ...
1
vote
0
answers
91
views
Minimum number of digits for exact double-precision and the fmt='%.18e' of numpy.savetxt()
I want to save 64-bit (double-precision) floating point numbers to a text file in Python using NumPy's savetxt(). The default format is fmt='%.18e' which gives 19 significant decimal digits (e.g. f'{...
-1
votes
4
answers
107
views
Control the scale of a floating point number [closed]
Is there a way to control the scale of a floating point number in lisp:
instead of 89.833336 I would like my output to be 89.83
Searched high and low for an appropriate function in the HyperSpec - can'...
0
votes
0
answers
9
views
Is there fallback strategy if user device or browser does not support vertex shaders of specific precision and ThreeJS crashes?
Our team facing more often than before with problem that may be related to wider spread of some (mobile?) devices that are already support (small or large) subset of WebGL.
Even browser vendors think ...
4
votes
2
answers
119
views
Convert max integer to floating point without precision loss
I have a following snippet which I used to try to test if a precision loss occurs if I convert a max integer to a double:
#include <cstdint>
#include <limits>
#include <iostream>
#...
1
vote
2
answers
67
views
Change -0.67 floating point to fixed point
I want to write this (-0.67) in an appropriate 8-bit Qn.m format
how can I do that?
normally we have -1.67
which can be represented as
2-bits for the integer part and 4 bits for the fraction part
but ...
1
vote
2
answers
120
views
imshow plotting very large integers, but "dtype object cannot be converted to float"
I have the following code, plotting a function on a grid, where the function happens to have a very large integer value:
import matplotlib.pyplot as plt
from matplotlib.ticker import ScalarFormatter, ...
0
votes
0
answers
72
views
Anomaly when adding floating points of high precision [duplicate]
Consider the following example:
int main()
{
float x = 1.0f;
float a = 0.0000000000000001f; //16 floating point
std::cout<<std::fixed;
std::cout<< std::...
-1
votes
0
answers
83
views
Floating Point problem, choose epsilon after subtraction
There is a known problem with floating point inaccuracy. (How to choose epsilon value for floating point?)
This works most of the time. However, if a variable first contains a high number, i.e. 10000, ...
1
vote
1
answer
118
views
N-body simulation in C++ has great momentum conservation and huge energy deviation
I am using Verlet integration (more specifically, the last equation in the "Non-constant time differences" section in the Verlet integration Wikipedia page), with C++ and the Eigen library, ...
-1
votes
2
answers
57
views
Trouble comparing floats in Python
I know that this should be easier, but I've having issues comparing float values.
I have some very basic code that is doing a few things:
Any weights under 5 are added to light_pkgs.
Any weights over ...
0
votes
0
answers
18
views
Why the accuracy, precision and recall results for training and validation shows same value in each epoch in my deep learning model
I want to get the accuracy, precision and recall values, classification report for this problem.But I get same values for accuracy, precision and recall in each epoch. Also I get meager value for iou. ...
1
vote
1
answer
48
views
Is There any way to use min of 16 digit after decimal point to perform arithmetic operation in PHP
My actual value is: 0.990641327804487
but while doing php arithmetic operation it convert this number into 0.99064132780449
I have tried number_format function,
if i convert it into number_format it ...
0
votes
0
answers
29
views
numpy.loadtxt does not read in np.longfloat data for polynomial regression
I have a comma-separated file with values in scientific notation. I have tried reading in the data in the following two ways, but neither way actually seems to use the longdouble data type I specified....
0
votes
1
answer
43
views
PyTorch load_state_dict() does not load precise value
For simplicity say I want to set all params of a torch model to the constant 72114982 with this code
model = Net()
params = model.state_dict()
for k, v in params.items():
params[k] = torch.full(v....
0
votes
1
answer
79
views
Why is "(eps * 0.5) + 1" not greater than "1" in computer science?
I am studying Matlab, and I do not understand why (eps * 0.5) + 1 is not greater than 1.
eps
ans =
2.220446049250313e-16
fprintf('%.52f\n', eps);
0....
0
votes
2
answers
77
views
Under what condition will `str(float(val)) == val` NOT hold in Python? [duplicate]
For a string val variable like "1.2345", does assert str(float(val)) == val always hold?
I've already found it doesn't always hold
>>> str(float('1.2345'))
'1.2345'
>>> str(...
0
votes
1
answer
54
views
Double Exponential integrals in Julia - better precision
I'm trying to calculate integral with DoubleExponentialFormulas.jl in Julia using the function quadde. However, the relative error I'm getting is ~10^10 in comparing result with Mathematica from the ...
0
votes
1
answer
29
views
get float64 for numpy calculation
def chi2_distance(a,b):
d=1-(np.dot(a,b)/(norm(a,axis=1)*norm(b)))
return d
a and b are float32 and have max 8 decimals. but i want the calculation has 16 decimals accuracy.
a....
18
votes
1
answer
363
views
Java 17 vs Java 8 double representation
What is the reason of the difference of the same values when doing average between 2 different JVM's (Java 8 and Java 17)?
Is that because the floating point? Or has something else changed between 2 ...
0
votes
1
answer
39
views
Quick way to figure out 32-bit floating point representation and associated loss
I am dealing with real numbers from a given range (namely lontitudes and lagitudes) from a high-precision system which will give me many decimals (in the order of 15 usually, plus of course up to ...
0
votes
0
answers
48
views
Data type issues with pandas data structures for (really) big numbers
I am having issues with Python Pandas and some precision issues for a data set.
Here is the set of integers i am using as a random case found here: link
What I am trying to do is compute some ...
0
votes
1
answer
50
views
PIRLS step halving with cloglog link in lme4
I'm getting the PIRLS step-halving error using lme4::lmer() with family = binomial(link = cloglog). Seems possible to me that I've generated data for which the model below is too complex, which may be ...
0
votes
0
answers
38
views
float precision problem :calculating of volume [duplicate]
I wrote a code to calculate the volume of a cone
this is my code
#include <iostream>
using namespace std;
int main()
{
cout<<(float)1/3*3.1415926*2*2*5;
return 0;
}
and I wrote ...
0
votes
2
answers
60
views
Ensuring exact same coordinate returned by calculation
I have placed 4 smaller circles on a big circle at 0, 90, 180, 270 degree angle. I expect javascript to return exact same y for red and blue circle (as the shift is horizontal, y is the same but x ...
3
votes
2
answers
153
views
C# Formatting Floating Point Rounding/Accuracy Issue: N2 vs 0.00
In C# (.Net 7):
float a = 13082987520.00f;
Console.WriteLine(a);
Console.WriteLine($"{a:N2}");
Console.WriteLine($"{a:0.00}");
I get:
1.3082988E+10
13,082,987,520.00
13082990000....
0
votes
1
answer
95
views
With the Python, what is the decimal precision required for volume and price on Kraken's REST API?
I want to use Python to interact with the Kraken REST API (https://support.kraken.com/hc/en-us/articles/360025180232-REST-API-command-line-client-Python-#python). I'm US-based. I was trying to ...
0
votes
0
answers
12
views
Getting TensorFlow to use double precision for custom classes
Consider the following TensorFlow code, which does not use Keras. I would like TensorFlow to perform the calculations in double precision. For the function tfmult I was able to figure out how to pass ...
1
vote
3
answers
88
views
Without overflow and underflow, for a float number x, is x/4 always equals to x/2/2 (also for x*4 == x*2*2)?
For example, as I know, a float point x, which x/n^2 may not be equal to x/n/n, because in x/n/n, which would create an intermediate x/n first. For example:
document.write(1.23456789/49 == 1....
6
votes
4
answers
204
views
How to properly round to the nearest integer in double-double arithmetic
I have to analyse a large amount of data using Python3 (PyPy implementation), where I do some operations on quite large floats, and must check if the results are close enough to integers.
To exemplify,...
1
vote
2
answers
68
views
How to Improve Timing Precision in Python Script for DMX Control
I have a Python script that controls DMX lighting via the OLA library. The script reads color values from a CSV file and sends them to a DMX controller at regular intervals of 120ms.
I'm using time....
0
votes
0
answers
48
views
Converting double value in spark sql
I have a requirement in which I need to convert string to double in such a way using Spark SQL:
12.56 - 12.56
15.40 - 15.40
12 - 12
0 - 0
We do not need to round, only that whenever there is a whole ...
0
votes
1
answer
30
views
python ctypes precision
I am using ctypes to create a wrapper over a C++ library (.so). I ran into some trouble with the precision of an argument. When I pass math.pi as a double, I am only getting six decimal places of ...
0
votes
0
answers
19
views
MIPS: How to maintain precision when working with doubles or to round them to a specific number of decimal places?
I am writing a program in MIPS assembly (using MARS) that reads numbers from a .txt file and loads them into a vector. Some of these numbers are doubles. The program and its logic generally work, but ...
2
votes
1
answer
131
views
Machine values for arithmetic operations and castings in C++
I am new in C++ programming. I'm wondering what possible machine values are obtained as the results of the following simple operations in C++:
int a = 3;
double inv_a;
inv_a = (double) 1 / (double) ...
2
votes
2
answers
106
views
C# IEEE754 Rounding
Consider the following C# code...
double x = Math.Round(72.6d, 2, MidpointRounding.ToZero);
double y = Math.Round(82.6d, 2, MidpointRounding.ToZero);
x becomes 72.59 and y becomes 82.6.
But why? ...
0
votes
0
answers
18
views
Numerical inprecision in pytorch-lightning's Gradient Accumulation?
In my understanding of Gradient Accumulation (e.g. this post and this post) the result of computing with batch_size = x vs. accumulate_gradient = h, batch size = x/h should be the same.
So those three ...
0
votes
1
answer
56
views
When working with numbers of order e-325 'decimal' python module returns '0'
i'm trying implement a function that is expressed as a power series.
It is of the form:
The problem is related to the terms t_min and t_max on the numerator: when s becomes large, both terms start ...
0
votes
0
answers
18
views
CPU float32 precision differences between projects in pytorch
I understand that float32 per IEEE has precision issues when summing values on the order of 10^-7. I fully expect to see differences in computations between architectures, including GPUs. However, ...
1
vote
0
answers
77
views
Different floating point precision in Matrix multiplication in Python and C++
I am trying to do a very simple thing
take a square shape 2d array
take its transpose
multiply the array with its transpose
I am trying to perform the above steps in C++ and Python as shown in the ...
0
votes
0
answers
24
views
How to calculate macro, weighted Precision Recall values in tensorflow?
I am trying to get macro and weighted precision, recall values while training ResNet/ResNeXt models.
How I compile my model:
f1 =tfa.metrics.F1Score(num_classes=2, average='weighted')
model.compile (
...
2
votes
0
answers
172
views
F1 Score in tensorflow
While training the model, I am trying to get the f1 score of binary classification problem on ResNeXt and ResNet. When I use tf.keras.metrics.F1Score function I get the following error while I am ...
0
votes
0
answers
37
views
Will Boost's float128 type work on Visual C++? [duplicate]
This might be a stupid question, because why would they design it for a library if it's only for compilers that already support float128.
I just want to make sure before I download the library. Couldn'...