Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

Filter by
Sorted by
Tagged with
0 votes
1 answer
37 views

High Accuracy Plotting Julia

I need to plot a vector to check the smoothness after a time step. I am using BigFloats for my calculations as I need around 30 significant figures of precision. However when I plot the curve it ...
HenryS's user avatar
  • 21
0 votes
0 answers
8 views

How to improve the precision of matrix calculation in Matlab or Python?

Now, I have a matrix S, and it satisfies the recursion relation: S_{m,n}=(1+|α|^2)S_{m-1,n-1}+(m-1)(n-1)S_{m-2,n-2}+α(m-1)S_{m-2,n-1}+α*(n-1)S_{m-1,n-2} S_{0,n}=(α*)^n and S_{n,0}=α^n Obviously, when |...
Young Q's user avatar
  • 31
0 votes
1 answer
75 views

Format a number to 8 or 16 chars without "e" with highest precision

I have the following python procedure. Input is any number (from a calculation) and format, where format can be 'long' or 'short' What i want to do: If the format is short, i want the output to be the ...
Lumpi's user avatar
  • 2,783
0 votes
0 answers
8 views

In arangodb how to save decimal with trailing zeroes in order to keep the precision of the value

So if the user inputs 12.00000 I need to save the trailing zeroes 5 in this example. I am evaluating arangodb. But there is no clear information about this. I would use BigDecimal in Java to handle ...
Jyri Ahia's user avatar
0 votes
1 answer
90 views

Rounding errors when computing intersection between two lines in python

I'm attempting to create a Python script that identifies all the unique intersection points, eliminating duplicates, among all the lines passing through a subset of points of a regular grid in 2D or ...
Edoardo Serra's user avatar
0 votes
1 answer
39 views

Precision and recall for which class is calculated in GridSearchCV?

I'm running GridSearchCV on XGBoost model to tune some parameters. My target variable is binary. When I use, 'recall' or 'precision' as scoring metrics in cross validation, I get a single value. I'm ...
thileepan's user avatar
  • 631
-2 votes
0 answers
75 views

Is this an issue with my code or the expected output?

I have to create a program that outputs weather data from a file. my issue is with one of my functions PrintMonthWeatherSummary. the month of June gives the wrong output. void PrintMonthWeatherSummary(...
Jace .flochart's user avatar
0 votes
1 answer
70 views

What's the real limit for NUMBER data type in Oracle?

I need help with understanding NUMBER data type, because 2 below statements from Oracle documentation seems to be for me mutually exclusive. "The NUMBER data type stores zero as well as positive ...
drk's user avatar
  • 41
1 vote
0 answers
91 views

Minimum number of digits for exact double-precision and the fmt='%.18e' of numpy.savetxt()

I want to save 64-bit (double-precision) floating point numbers to a text file in Python using NumPy's savetxt(). The default format is fmt='%.18e' which gives 19 significant decimal digits (e.g. f'{...
jmd_dk's user avatar
  • 12.4k
-1 votes
4 answers
107 views

Control the scale of a floating point number [closed]

Is there a way to control the scale of a floating point number in lisp: instead of 89.833336 I would like my output to be 89.83 Searched high and low for an appropriate function in the HyperSpec - can'...
David N Richards's user avatar
0 votes
0 answers
9 views

Is there fallback strategy if user device or browser does not support vertex shaders of specific precision and ThreeJS crashes?

Our team facing more often than before with problem that may be related to wider spread of some (mobile?) devices that are already support (small or large) subset of WebGL. Even browser vendors think ...
Kote Isaev's user avatar
4 votes
2 answers
119 views

Convert max integer to floating point without precision loss

I have a following snippet which I used to try to test if a precision loss occurs if I convert a max integer to a double: #include <cstdint> #include <limits> #include <iostream> #...
Marko Popovic's user avatar
1 vote
2 answers
67 views

Change -0.67 floating point to fixed point

I want to write this (-0.67) in an appropriate 8-bit Qn.m format how can I do that? normally we have -1.67 which can be represented as 2-bits for the integer part and 4 bits for the fraction part but ...
Fatima Ismail's user avatar
1 vote
2 answers
120 views

imshow plotting very large integers, but "dtype object cannot be converted to float"

I have the following code, plotting a function on a grid, where the function happens to have a very large integer value: import matplotlib.pyplot as plt from matplotlib.ticker import ScalarFormatter, ...
D.R's user avatar
  • 117
0 votes
0 answers
72 views

Anomaly when adding floating points of high precision [duplicate]

Consider the following example: int main() { float x = 1.0f; float a = 0.0000000000000001f; //16 floating point std::cout<<std::fixed; std::cout<< std::...
Constantinos Glynos's user avatar
-1 votes
0 answers
83 views

Floating Point problem, choose epsilon after subtraction

There is a known problem with floating point inaccuracy. (How to choose epsilon value for floating point?) This works most of the time. However, if a variable first contains a high number, i.e. 10000, ...
Bojan Hrnkas's user avatar
  • 1,603
1 vote
1 answer
118 views

N-body simulation in C++ has great momentum conservation and huge energy deviation

I am using Verlet integration (more specifically, the last equation in the "Non-constant time differences" section in the Verlet integration Wikipedia page), with C++ and the Eigen library, ...
Malmel's user avatar
  • 13
-1 votes
2 answers
57 views

Trouble comparing floats in Python

I know that this should be easier, but I've having issues comparing float values. I have some very basic code that is doing a few things: Any weights under 5 are added to light_pkgs. Any weights over ...
Justin's user avatar
  • 101
0 votes
0 answers
18 views

Why the accuracy, precision and recall results for training and validation shows same value in each epoch in my deep learning model

I want to get the accuracy, precision and recall values, classification report for this problem.But I get same values for accuracy, precision and recall in each epoch. Also I get meager value for iou. ...
Achini Nisansala's user avatar
1 vote
1 answer
48 views

Is There any way to use min of 16 digit after decimal point to perform arithmetic operation in PHP

My actual value is: 0.990641327804487 but while doing php arithmetic operation it convert this number into 0.99064132780449 I have tried number_format function, if i convert it into number_format it ...
Arnab Sheikh's user avatar
0 votes
0 answers
29 views

numpy.loadtxt does not read in np.longfloat data for polynomial regression

I have a comma-separated file with values in scientific notation. I have tried reading in the data in the following two ways, but neither way actually seems to use the longdouble data type I specified....
javery's user avatar
  • 15
0 votes
1 answer
43 views

PyTorch load_state_dict() does not load precise value

For simplicity say I want to set all params of a torch model to the constant 72114982 with this code model = Net() params = model.state_dict() for k, v in params.items(): params[k] = torch.full(v....
Random Seed's user avatar
0 votes
1 answer
79 views

Why is "(eps * 0.5) + 1" not greater than "1" in computer science?

I am studying Matlab, and I do not understand why (eps * 0.5) + 1 is not greater than 1. eps ans = 2.220446049250313e-16 fprintf('%.52f\n', eps); 0....
m615's user avatar
  • 19
0 votes
2 answers
77 views

Under what condition will `str(float(val)) == val` NOT hold in Python? [duplicate]

For a string val variable like "1.2345", does assert str(float(val)) == val always hold? I've already found it doesn't always hold >>> str(float('1.2345')) '1.2345' >>> str(...
zyxue's user avatar
  • 8,040
0 votes
1 answer
54 views

Double Exponential integrals in Julia - better precision

I'm trying to calculate integral with DoubleExponentialFormulas.jl in Julia using the function quadde. However, the relative error I'm getting is ~10^10 in comparing result with Mathematica from the ...
Student Debil's user avatar
0 votes
1 answer
29 views

get float64 for numpy calculation

def chi2_distance(a,b): d=1-(np.dot(a,b)/(norm(a,axis=1)*norm(b))) return d a and b are float32 and have max 8 decimals. but i want the calculation has 16 decimals accuracy. a....
MJELVEH's user avatar
  • 83
18 votes
1 answer
363 views

Java 17 vs Java 8 double representation

What is the reason of the difference of the same values when doing average between 2 different JVM's (Java 8 and Java 17)? Is that because the floating point? Or has something else changed between 2 ...
omar mahameed's user avatar
0 votes
1 answer
39 views

Quick way to figure out 32-bit floating point representation and associated loss

I am dealing with real numbers from a given range (namely lontitudes and lagitudes) from a high-precision system which will give me many decimals (in the order of 15 usually, plus of course up to ...
Charles's user avatar
  • 1,030
0 votes
0 answers
48 views

Data type issues with pandas data structures for (really) big numbers

I am having issues with Python Pandas and some precision issues for a data set. Here is the set of integers i am using as a random case found here: link What I am trying to do is compute some ...
ex1led's user avatar
  • 429
0 votes
1 answer
50 views

PIRLS step halving with cloglog link in lme4

I'm getting the PIRLS step-halving error using lme4::lmer() with family = binomial(link = cloglog). Seems possible to me that I've generated data for which the model below is too complex, which may be ...
Michael Roswell's user avatar
0 votes
0 answers
38 views

float precision problem :calculating of volume [duplicate]

I wrote a code to calculate the volume of a cone this is my code #include <iostream> using namespace std; int main() { cout<<(float)1/3*3.1415926*2*2*5; return 0; } and I wrote ...
蔡天意's user avatar
0 votes
2 answers
60 views

Ensuring exact same coordinate returned by calculation

I have placed 4 smaller circles on a big circle at 0, 90, 180, 270 degree angle. I expect javascript to return exact same y for red and blue circle (as the shift is horizontal, y is the same but x ...
smpa01's user avatar
  • 4,086
3 votes
2 answers
153 views

C# Formatting Floating Point Rounding/Accuracy Issue: N2 vs 0.00

In C# (.Net 7): float a = 13082987520.00f; Console.WriteLine(a); Console.WriteLine($"{a:N2}"); Console.WriteLine($"{a:0.00}"); I get: 1.3082988E+10 13,082,987,520.00 13082990000....
Tom Charles Zhang's user avatar
0 votes
1 answer
95 views

With the Python, what is the decimal precision required for volume and price on Kraken's REST API?

I want to use Python to interact with the Kraken REST API (https://support.kraken.com/hc/en-us/articles/360025180232-REST-API-command-line-client-Python-#python). I'm US-based. I was trying to ...
Dave's user avatar
  • 15.1k
0 votes
0 answers
12 views

Getting TensorFlow to use double precision for custom classes

Consider the following TensorFlow code, which does not use Keras. I would like TensorFlow to perform the calculations in double precision. For the function tfmult I was able to figure out how to pass ...
NNN's user avatar
  • 551
1 vote
3 answers
88 views

Without overflow and underflow, for a float number x, is x/4 always equals to x/2/2 (also for x*4 == x*2*2)?

For example, as I know, a float point x, which x/n^2 may not be equal to x/n/n, because in x/n/n, which would create an intermediate x/n first. For example: document.write(1.23456789/49 == 1....
wcminipgasker2023's user avatar
6 votes
4 answers
204 views

How to properly round to the nearest integer in double-double arithmetic

I have to analyse a large amount of data using Python3 (PyPy implementation), where I do some operations on quite large floats, and must check if the results are close enough to integers. To exemplify,...
kleite's user avatar
  • 317
1 vote
2 answers
68 views

How to Improve Timing Precision in Python Script for DMX Control

I have a Python script that controls DMX lighting via the OLA library. The script reads color values from a CSV file and sends them to a DMX controller at regular intervals of 120ms. I'm using time....
Losbello's user avatar
0 votes
0 answers
48 views

Converting double value in spark sql

I have a requirement in which I need to convert string to double in such a way using Spark SQL: 12.56 - 12.56 15.40 - 15.40 12 - 12 0 - 0 We do not need to round, only that whenever there is a whole ...
codingprabhu's user avatar
0 votes
1 answer
30 views

python ctypes precision

I am using ctypes to create a wrapper over a C++ library (.so). I ran into some trouble with the precision of an argument. When I pass math.pi as a double, I am only getting six decimal places of ...
supvato's user avatar
  • 53
0 votes
0 answers
19 views

MIPS: How to maintain precision when working with doubles or to round them to a specific number of decimal places?

I am writing a program in MIPS assembly (using MARS) that reads numbers from a .txt file and loads them into a vector. Some of these numbers are doubles. The program and its logic generally work, but ...
despinxz's user avatar
2 votes
1 answer
131 views

Machine values for arithmetic operations and castings in C++

I am new in C++ programming. I'm wondering what possible machine values are obtained as the results of the following simple operations in C++: int a = 3; double inv_a; inv_a = (double) 1 / (double) ...
p d's user avatar
  • 33
2 votes
2 answers
106 views

C# IEEE754 Rounding

Consider the following C# code... double x = Math.Round(72.6d, 2, MidpointRounding.ToZero); double y = Math.Round(82.6d, 2, MidpointRounding.ToZero); x becomes 72.59 and y becomes 82.6. But why? ...
pmcilreavy's user avatar
  • 3,167
0 votes
0 answers
18 views

Numerical inprecision in pytorch-lightning's Gradient Accumulation?

In my understanding of Gradient Accumulation (e.g. this post and this post) the result of computing with batch_size = x vs. accumulate_gradient = h, batch size = x/h should be the same. So those three ...
Astrid Hofmann's user avatar
0 votes
1 answer
56 views

When working with numbers of order e-325 'decimal' python module returns '0'

i'm trying implement a function that is expressed as a power series. It is of the form: The problem is related to the terms t_min and t_max on the numerator: when s becomes large, both terms start ...
filidibava's user avatar
0 votes
0 answers
18 views

CPU float32 precision differences between projects in pytorch

I understand that float32 per IEEE has precision issues when summing values on the order of 10^-7. I fully expect to see differences in computations between architectures, including GPUs. However, ...
xaviersjs's user avatar
  • 1,609
1 vote
0 answers
77 views

Different floating point precision in Matrix multiplication in Python and C++

I am trying to do a very simple thing take a square shape 2d array take its transpose multiply the array with its transpose I am trying to perform the above steps in C++ and Python as shown in the ...
skm's user avatar
  • 5,173
0 votes
0 answers
24 views

How to calculate macro, weighted Precision Recall values in tensorflow?

I am trying to get macro and weighted precision, recall values while training ResNet/ResNeXt models. How I compile my model: f1 =tfa.metrics.F1Score(num_classes=2, average='weighted') model.compile ( ...
javeqx's user avatar
  • 21
2 votes
0 answers
172 views

F1 Score in tensorflow

While training the model, I am trying to get the f1 score of binary classification problem on ResNeXt and ResNet. When I use tf.keras.metrics.F1Score function I get the following error while I am ...
javeqx's user avatar
  • 21
0 votes
0 answers
37 views

Will Boost's float128 type work on Visual C++? [duplicate]

This might be a stupid question, because why would they design it for a library if it's only for compilers that already support float128. I just want to make sure before I download the library. Couldn'...
Steven Rosario's user avatar

1
2 3 4 5
91