18

python 3.4, windows 10, cython 0.21.1

I'm compiling this function to c with cython

def weakchecksum(data):
   """
   Generates a weak checksum from an iterable set of bytes.
   """
   cdef long a, b, l
   a = b = 0
   l = len(data)
   for i in range(l):
       a += data[i]
       b += (l - i)*data[i]

   return (b << 16) | a, a, b

which produces this error: "OverflowError: Python int too large to convert to C long"

I've also tried declaring them as unsigned longs. What type do I use to work with really large numbers? If it's too large for a c long are there any workarounds?

12
  • How about cdef long long?
    – szxk
    Commented Nov 12, 2014 at 3:10
  • 1
    same error, the number that is causing it is 2891688164113197 Commented Nov 12, 2014 at 15:19
  • Consider making a big integer class? It will be a linked list that contains digits Commented Nov 14, 2014 at 21:28
  • 1
    Yeah I considered that, I'm using cython to speed up some code and was hoping to avoid creating special classes that will increase overhead Commented Nov 14, 2014 at 23:17
  • signed long long on gcc, Ubuntu 14.04, is about 9.2e+18, from limits.h it is exactly define LLONG_MAX 9223372036854775807LL see gnu.org/software/libc/manual/html_node/…; unsigned long long is bigger. Yes, saw you are on windows but this should really bit about how many bits fit into hardware-level algorithms.
    – Paul
    Commented Nov 18, 2014 at 8:49

2 Answers 2

13

cython compiles pyx files to C, thus it depends on underlying C compiler.

Size of integer types in C varies on different platforms and operations systems, and C standard don't dictate exact implementation.

However there is de facto implementation conventions.

Windows for both 32 and 64 bit uses 4 bytes (32 bits) for int and long, 8 bytes (64 bits) for long long. The difference between Win32 and Win64 is size of pointer (32 bits for Win32 and 64 bits for Win64). (See Data Type Ranges] from MSDN).

Linux uses another model: int is 32 bits for both linux-32 and linux-64, long long is always 64-bit. long and pointers are vary: 32 bits on linux-32 and 64 bits on linux-64.

Long story short: if you need maximum capacity for integer type which doesn't changed on different platforms use long long (or unsigned long long).

The data range for long long is [–9223372036854775808, 9223372036854775807].

If you need numbers with arbitrary precision there is GMP library -- de facto standard for high-precision arithmetic. Python has wrapper for it called gmpy2.

5
+50

If you make sure that your calculations are in c (for instance, declare i to be long, and put the data element into a cdefed variable or cast it before calculation), you won't get this error. Your actual results, though, could vary depending on platform, depending (potentially) on the exact assembly code generated and the resulting treatment of overflows. There are better algorithms for this, as @cod3monk3y has noted (look at the "simple checksums" link).

3
  • 1
    So I added 'i' to the cdef statement and re-ran it. Still got the same error. Also tried cdef long long a, b, l, i Commented Nov 21, 2014 at 21:56
  • Try cdef long d ... (and in loop) d = data[i] or use then use d in the calculation (or use <<long>>data[i]). You can compile with a flag to look at the actual c code generated w/ python lines in comments -- you should see that the calculation is a straight c expression when there are no python variables involved. (note cast I cant spell right in markup for some reason :))
    – shaunc
    Commented Nov 22, 2014 at 1:25
  • perfect.. I had given up on an answer for this one. That did it. You're a genius. For those of you reading this in the future. I declared cdef long a, b, l, i, d, and changed every reference to data[i] to d. Commented Nov 22, 2014 at 5:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.