Questions tagged [python-internals]

How does Python work underneath the hood? Use for questions relating to (for instance) the design decisions made and the internal data structures and algorithms used.

Filter by
Sorted by
Tagged with
2 votes
1 answer
251 views

What is the `ExceptionTable` in the output of `dis`?

In python3.13, when I try to disassemble [i for i in range(10)], the result is as below: >>> import dis >>> >>> dis.dis('[i for i in range(10)]') 0 RESUME ...
Amir reza Riahi's user avatar
2 votes
0 answers
64 views

What does RESUME opcode actually do?

The documentation is not very informative (at least for me): opcode:: RESUME (context) A no-op. Performs internal tracing, debugging and optimization checks. The context oparand consists of two parts....
Amir reza Riahi's user avatar
40 votes
3 answers
3k views

Why list comprehensions create a function internally?

This is disassembly of a list comprehension in python 3.10: Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux Type "help", "copyright", "credits" or &...
Amir reza Riahi's user avatar
0 votes
1 answer
103 views

Which calls in Python may not call `__call__`?

The answer to my question may depend on the interpreter for the code although I'm not sure. If it does, then I would be happy to hear about any widely used Python interpreter, especially CPython ...
Takuo Matsuoka's user avatar
0 votes
0 answers
32 views

Performance of very large dictionaries when storing/loading via pickle

I have a very large dictionary (multiple millions) which will also be accesses multiple millions times. I was wondering about the low-level performances of such dicts because if they need to be ...
Daraan's user avatar
  • 1,827
3 votes
2 answers
150 views

Can't create Race Condition in Python 3.11 using multiple threads

I believe this is a difference in Python 3.10 and above from older versions. Could someone explain this? import threading import time counter = 0 lock = threading.Lock() def increment(): global ...
RohanAlmighty's user avatar
0 votes
1 answer
72 views

Why does Python recursion limit change depending on function?

I was testing stuff when I noticed that python's recursion limit doesn't seem to apply equally to all functions. I'm not sure why or how and couldn't find any documentation explaining this behavior. ...
scr's user avatar
  • 908
-1 votes
1 answer
154 views

Python Tuple vs List vs Array memory consumption

I've been reading Fluent code by Luciano Ramalho and in the chapter 'Overview of Built-in Sequences' when describing C struct behind float he states: ".. That's why an array of floats is much ...
Andrey's user avatar
  • 338
0 votes
1 answer
64 views

Why does my Python thread block the main thread unless I add a print or a sleep?

Does anyone know why running this code causes the script to hang in the thread, unless I uncomment the print, the sleep, or the "if" condition, or remove the try/except? My understanding is ...
Nat's user avatar
  • 43
4 votes
2 answers
123 views

Storage of floating point numbers in memory in Python

I know that Python maintains an internal storage of small-ish integers rather than creating them at runtime: id(5) 4304101544 When repeating this code after some time in the same kernel, the id is ...
Sebastian Wozny's user avatar
4 votes
1 answer
162 views

Why is set.remove so slow here?

(Extracted from another question.) Removing this set's 200,000 elements one by one like this takes 30 seconds (Attempt This Online!): s = set(range(200000)) while s: for x in s: s.remove(x)...
Kelly Bundy's user avatar
  • 24.8k
0 votes
1 answer
61 views

How to interpret the error message "Foo() takes no arguments" when specifying a class instance as base class?

The following code: >>> class Foo: pass >>> class Spam(Foo()): pass will of course raise an error message: Traceback (most recent call last): File "<stdin>", line ...
yixuan's user avatar
  • 385
2 votes
1 answer
170 views

Give an example/explanation of the closure parameter of the exec function

Can someone please explain to me when and how I would use the closure parameter of the exec function? https://docs.python.org/3/library/functions.html#exec The closure argument specifies a closure–a ...
Ben Ellis's user avatar
  • 173
2 votes
1 answer
328 views

Why sending a message to a web socket does not yield control to the event loop?

Consider the following code: main.py import asyncio import websockets async def echo(websocket): async for message in websocket: await websocket.send(message) print(message) ...
OES's user avatar
  • 55
8 votes
2 answers
763 views

Is list[str] an iterable?

Python 3.10 doesn't think so: Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:38:29) [Clang 13.0.1 ] \ on darwin Type "help", "copyright", "credits" ...
Fallible's user avatar
  • 257
1 vote
1 answer
114 views

How to clone a python class object? (not the instance but the class itself)

Imagine you have the following code: class A: pass NewA = ... # copy A NewA.__init__ = decorator(A.__init__) # but don't change A's init function, just NewA's I am looking for a way to change ...
Vahid Zee's user avatar
2 votes
1 answer
104 views

Why is the difference between id(2) and id(1) equal to 32?

>>> a = 1 >>> b = 2 >>> id(a), id(b), id(b) - id(a) (1814458401008, 1814458401040, 32) Is the memory address returned by id in bits or in bytes? Per the docs: The current ...
BPDev's user avatar
  • 475
4 votes
1 answer
414 views

What's the benefit of asyncio using weakrefs to keep track of tasks?

Python docs for asyncio.create_task state: Important: Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. ...
max's user avatar
  • 49.9k
57 votes
2 answers
6k views

Why is b.pop(0) over 200 times slower than del b[0] for bytearray?

Letting them compete three times (a million pops/dels each time): from timeit import timeit for _ in range(3): t1 = timeit('b.pop(0)', 'b = bytearray(1000000)') t2 = timeit('del b[0]', 'b = ...
Kelly Bundy's user avatar
  • 24.8k
1 vote
1 answer
95 views

Prevent Python Interpreter from Exiting if CTRL-D is pressed

I am running a script with python -i main.py. The script starts some C threads and python threads using threading module, then python code ends and it goes to a prompt. How can i prevent python from ...
Ahmed Eshra's user avatar
15 votes
1 answer
550 views

Are Python 3.11 objects as light as slots?

After Mark Shannon's optimisation of Python objects, is a plain object different from an object with slots? I understand that after this optimisation in a normal use case, objects have no dictionary. ...
Jorge Luis's user avatar
1 vote
1 answer
65 views

Local imports work in bundled PyInstaller app but in Python source

This issue has plagued me for the last few months, I need a more experienced opinion. We have a CLI Python application that uses a gRPC server to communicate with other backend services. Its ...
wisenickel's user avatar
30 votes
2 answers
2k views

yield from vs yield in for-loop

My understanding of yield from is that it is similar to yielding every item from an iterable. Yet, I observe the different behavior in the following example. I have Class1 class Class1: def ...
erzya's user avatar
  • 598
2 votes
1 answer
119 views

Python small integer cache: what's different when assigning multiple values?

I'm aware of the CPython implementation that holds a small integer cache in the [-5, 256] range, so I understand that a=2 and b=2 will refer to the same memory address (thus causing a is b to return ...
Floella's user avatar
  • 1,315
0 votes
1 answer
61 views

Why PyList_Append is called each time a list is evaluated?

I'm working with CPython3.11.0a3+. I added a break point at PyList_Append and modified the function to stop when the newitem is a dict. The original function: int PyList_Append(PyObject *op, PyObject *...
Amir reza Riahi's user avatar
13 votes
1 answer
788 views

How are small sets stored in memory?

If we look at the resize behavior for sets under 50k elements: >>> import sys >>> s = set() >>> seen = {} >>> for i in range(50_000): ... size = sys.getsizeof(s)...
wim's user avatar
  • 343k
2 votes
0 answers
79 views

Python objects implementation

I am studying the CPython's objects system implementation and I struggling to understand the differences between and purposes of the PyTypeObject and PyType_Type structs. At first sight, I thought ...
arthurr's user avatar
  • 21
1 vote
2 answers
588 views

Please explain to me how does Python interpreter executes modules written in C/C++?

I'm trying to understand how it works. I know that Python interpreter translates python source code to byte code representation for a virtual machine (Python interpreter is a virtual machine) and ...
Viacheslav Prokopev's user avatar
0 votes
1 answer
222 views

Where is the actual implementation of "__getattribute__"

I can't find the Python source code for __getattribute__. I looked in "object", and "type" classes, but I only see the function declaration. I don't see the actual definition.
user1187968's user avatar
  • 7,414
2 votes
1 answer
157 views

Why __slots__ isn't the default in Python?

I've been programming in Python for a long time, but I still can't understand why classes base their attribute lookup on the __dict__ dictionary by default instead of the faster __slots__ tuple. ...
Nicholas Obert's user avatar
0 votes
1 answer
475 views

Extract PI OSIsoft Monthly Interval in Python

I am trying to extract the sum of PI data from OSIsoft 10m (10 minute) data in a one (1) month interval using Python pandas. However, I either get an error from OSIsoft or Python when I choose the ...
user2100039's user avatar
  • 1,306
2 votes
1 answer
220 views

Is list comprehension implemented via map and lambda function? [duplicate]

According to the list comprehension doc and this question: squares = [x**2 for x in range(10)] is equivalent to squares = list(map(lambda x: x**2, range(10))) But does Python actually implement list ...
hellohawaii's user avatar
  • 3,074
0 votes
1 answer
139 views

taking square using "value**2" results causes an overflow while "value*value" is fine

Given the same input, x**2 gives an integer overflow while x*x works fine. I am not sure if this is because of the python's internal implementation of those operator or if this is a bug in the ...
anilbey's user avatar
  • 1,847
3 votes
1 answer
562 views

Hash of integers in python

I understand that hash of an immutable object is an integer representation of that object which is unique within the process's lifetime. Hash of an integer object is the same as the value held by the ...
Saad's user avatar
  • 87
2 votes
0 answers
230 views

How to print the memory structure of a Python data type?

I'd like to be able to produce a hex dump of the data structure of Python data types. The closest I can get right now is something like this: from ctypes import string_at a = [n for n in range(20)] ...
martin's's user avatar
  • 3,883
2 votes
1 answer
312 views

How can I access the weakref object of the class itself through the class?

As far as I know, __weakref__ is a descriptor defined in class, so that if it invoked from the instances of the class, it will give the weakref object: from weakref import ref class A: pass obj =...
S.B's user avatar
  • 13.8k
11 votes
1 answer
1k views

Where is the default behavior for object equality (`==`) defined?

According to the object.__eq__() documentation, the default (that is, in the object class) implementation for == is as follows: True if x is y else NotImplemented Still following the documentation ...
Manuel Selva's user avatar
  • 18.7k
3 votes
1 answer
158 views

What is Python 3 `str.__getitem__` computional complexity?

''' Set up ''' s= open("Bilion_of_UTF-8_chars.txt",encoding="UTF-8").read() ''' The following doesn't look like a cheap operation because Python3 `str`-s are UTF-8 encoded (EDIT: ...
Sir 's user avatar
  • 337
0 votes
1 answer
440 views

Native array.frombytes() (not numpy!) mysterious behavior

[I cannot use numpy so please refrain from talking about it] I (apparently naively) thought Python array.frombytes() would read from a series of bytes representing various machine format integers, ...
Bob Denny's user avatar
  • 1,304
7 votes
1 answer
161 views

How does CPython implement os.environ?

I was looking through source and noticed that it references a variable environ in methods before its defined: def _createenviron(): if name == 'nt': # Where Env Var Names Must Be UPPERCASE ...
user avatar
5 votes
0 answers
157 views

20% faster with unused variable? Why?

I'm doing a lot of benchmarks. I've never seen something like this. I'm stumped. Creating an extra global variable, not used at all, makes part of my code about 20% faster. Why? I'm benchmarking a ...
Pychopath's user avatar
  • 1,560
6 votes
1 answer
727 views

Does Python not reuse memory here? What does tracemalloc's output mean?

I create a list of a million int objects, then replace each with its negated value. tracemalloc reports 28 MB extra memory (28 bytes per new int object). Why? Does Python not reuse the memory of the ...
Kelly Bundy's user avatar
  • 24.8k
0 votes
2 answers
669 views

Why python property() function is assigned to a class variable and not an instance variable?

I'm learning about encapsulation and abstraction in python and i came across the property function and decorator. The common example is something like this. class Celsius(): def __init__(self, ...
Daniel Florez Cortes's user avatar
3 votes
0 answers
40 views

Why do sets behave unpredictably? [duplicate]

As explained in Can Python's set absence of ordering be considered random order?, the set structure is in arbitrary order. For example, the following code results in different output between runs. ...
jpf's user avatar
  • 65
1 vote
1 answer
429 views

Why is memory not freed in this case?

I have the following Python code: import os, psutil import numpy as np process = psutil.Process(os.getpid()) print(process.memory_info().rss) def append(x): x.append(np.random.normal(size=(1000,...
bumpbump's user avatar
  • 694
101 votes
3 answers
8k views

Can we make 1 == 2 true? [duplicate]

Python ints are objects that encapsulate the actual number value. Can we mess with that value, for example setting the value of the object 1 to 2? So that 1 == 2 becomes True?
Kelly Bundy's user avatar
  • 24.8k
3 votes
3 answers
1k views

How does Python interpreter actually interpret a program?

Take a sample program: c = 10 def myfunc(): print(c) myfunc() This prints 10 as expected, but if we look at another program: c = 10 def myfunc(): print(c) c = 1 myfunc() It says: ...
Keerthi Vasan's user avatar
3 votes
1 answer
717 views

Why does the float object behave differently with the "is" operator?

As far as I know cpython implementation keeps the same object for some same values in order to save memory. For example when I create 2 strings with the value hello, cpython does not create 2 ...
Amir reza Riahi's user avatar
28 votes
1 answer
5k views

Python threads difference for 3.10 and others

For some, simple thread related code, i.e: import threading a = 0 threads = [] def x(): global a for i in range(1_000_000): a += 1 for _ in range(10): thread = threading.Thread(...
Zada Zorg's user avatar
  • 2,778
1 vote
0 answers
115 views

Is it possible to get the source code of the current (multiline) statement using the stack frame?

I'm interested in being able to use code like this to obtain 3 or (3, 7) (the lines containing the current statement), however, f_lineno returns 6 because that is where the actual execution is ...
Bea Steers's user avatar

1
2 3 4 5
15