Questions tagged [python-internals]
How does Python work underneath the hood? Use for questions relating to (for instance) the design decisions made and the internal data structures and algorithms used.
738
questions
2
votes
1
answer
251
views
What is the `ExceptionTable` in the output of `dis`?
In python3.13, when I try to disassemble [i for i in range(10)], the result is as below:
>>> import dis
>>>
>>> dis.dis('[i for i in range(10)]')
0 RESUME ...
2
votes
0
answers
64
views
What does RESUME opcode actually do?
The documentation is not very informative (at least for me):
opcode:: RESUME (context)
A no-op. Performs internal tracing, debugging and optimization
checks.
The context oparand consists of two parts....
40
votes
3
answers
3k
views
Why list comprehensions create a function internally?
This is disassembly of a list comprehension in python 3.10:
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or &...
0
votes
1
answer
103
views
Which calls in Python may not call `__call__`?
The answer to my question may depend on the interpreter for the code although I'm not sure. If it does, then I would be happy to hear about any widely used Python interpreter, especially CPython ...
0
votes
0
answers
32
views
Performance of very large dictionaries when storing/loading via pickle
I have a very large dictionary (multiple millions) which will also be accesses multiple millions times.
I was wondering about the low-level performances of such dicts because if they need to be ...
3
votes
2
answers
150
views
Can't create Race Condition in Python 3.11 using multiple threads
I believe this is a difference in Python 3.10 and above from older versions. Could someone explain this?
import threading
import time
counter = 0
lock = threading.Lock()
def increment():
global ...
0
votes
1
answer
72
views
Why does Python recursion limit change depending on function?
I was testing stuff when I noticed that python's recursion limit doesn't seem to apply equally to all functions. I'm not sure why or how and couldn't find any documentation explaining this behavior.
...
-1
votes
1
answer
154
views
Python Tuple vs List vs Array memory consumption
I've been reading Fluent code by Luciano Ramalho and in the chapter 'Overview of Built-in Sequences' when describing C struct behind float he states:
".. That's why an array of floats is much ...
0
votes
1
answer
64
views
Why does my Python thread block the main thread unless I add a print or a sleep?
Does anyone know why running this code causes the script to hang in the thread, unless I uncomment the print, the sleep, or the "if" condition, or remove the try/except? My understanding is ...
4
votes
2
answers
123
views
Storage of floating point numbers in memory in Python
I know that Python maintains an internal storage of small-ish integers rather than creating them at runtime:
id(5)
4304101544
When repeating this code after some time in the same kernel, the id is ...
4
votes
1
answer
162
views
Why is set.remove so slow here?
(Extracted from another question.) Removing this set's 200,000 elements one by one like this takes 30 seconds (Attempt This Online!):
s = set(range(200000))
while s:
for x in s:
s.remove(x)...
0
votes
1
answer
61
views
How to interpret the error message "Foo() takes no arguments" when specifying a class instance as base class?
The following code:
>>> class Foo: pass
>>> class Spam(Foo()): pass
will of course raise an error message:
Traceback (most recent call last):
File "<stdin>", line ...
2
votes
1
answer
170
views
Give an example/explanation of the closure parameter of the exec function
Can someone please explain to me when and how I would use the closure parameter of the exec function?
https://docs.python.org/3/library/functions.html#exec
The closure argument specifies a closure–a ...
2
votes
1
answer
328
views
Why sending a message to a web socket does not yield control to the event loop?
Consider the following code:
main.py
import asyncio
import websockets
async def echo(websocket):
async for message in websocket:
await websocket.send(message)
print(message)
...
8
votes
2
answers
763
views
Is list[str] an iterable?
Python 3.10 doesn't think so:
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:38:29) [Clang 13.0.1 ] \
on darwin
Type "help", "copyright", "credits" ...
1
vote
1
answer
114
views
How to clone a python class object? (not the instance but the class itself)
Imagine you have the following code:
class A:
pass
NewA = ... # copy A
NewA.__init__ = decorator(A.__init__) # but don't change A's init function, just NewA's
I am looking for a way to change ...
2
votes
1
answer
104
views
Why is the difference between id(2) and id(1) equal to 32?
>>> a = 1
>>> b = 2
>>> id(a), id(b), id(b) - id(a)
(1814458401008, 1814458401040, 32)
Is the memory address returned by id in bits or in bytes? Per the docs:
The current ...
4
votes
1
answer
414
views
What's the benefit of asyncio using weakrefs to keep track of tasks?
Python docs for asyncio.create_task state:
Important: Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. ...
57
votes
2
answers
6k
views
Why is b.pop(0) over 200 times slower than del b[0] for bytearray?
Letting them compete three times (a million pops/dels each time):
from timeit import timeit
for _ in range(3):
t1 = timeit('b.pop(0)', 'b = bytearray(1000000)')
t2 = timeit('del b[0]', 'b = ...
1
vote
1
answer
95
views
Prevent Python Interpreter from Exiting if CTRL-D is pressed
I am running a script with python -i main.py. The script starts some C threads and python threads using threading module, then python code ends and it goes to a prompt. How can i prevent python from ...
15
votes
1
answer
550
views
Are Python 3.11 objects as light as slots?
After Mark Shannon's optimisation of Python objects, is a plain object different from an object with slots?
I understand that after this optimisation in a normal use case, objects have no dictionary.
...
1
vote
1
answer
65
views
Local imports work in bundled PyInstaller app but in Python source
This issue has plagued me for the last few months, I need a more experienced opinion. We have a CLI Python application that uses a gRPC server to communicate with other backend services. Its ...
30
votes
2
answers
2k
views
yield from vs yield in for-loop
My understanding of yield from is that it is similar to yielding every item from an iterable. Yet, I observe the different behavior in the following example.
I have Class1
class Class1:
def ...
2
votes
1
answer
119
views
Python small integer cache: what's different when assigning multiple values?
I'm aware of the CPython implementation that holds a small integer cache in the [-5, 256] range, so I understand that a=2 and b=2 will refer to the same memory address (thus causing a is b to return ...
0
votes
1
answer
61
views
Why PyList_Append is called each time a list is evaluated?
I'm working with CPython3.11.0a3+. I added a break point at PyList_Append and modified the function to stop when the newitem is a dict. The original function:
int
PyList_Append(PyObject *op, PyObject *...
13
votes
1
answer
788
views
How are small sets stored in memory?
If we look at the resize behavior for sets under 50k elements:
>>> import sys
>>> s = set()
>>> seen = {}
>>> for i in range(50_000):
... size = sys.getsizeof(s)...
2
votes
0
answers
79
views
Python objects implementation
I am studying the CPython's objects system implementation and I struggling to understand the differences between and purposes of the PyTypeObject and PyType_Type structs.
At first sight, I thought ...
1
vote
2
answers
588
views
Please explain to me how does Python interpreter executes modules written in C/C++?
I'm trying to understand how it works. I know that Python interpreter translates python source code to byte code representation for a virtual machine (Python interpreter is a virtual machine) and ...
0
votes
1
answer
222
views
Where is the actual implementation of "__getattribute__"
I can't find the Python source code for __getattribute__. I looked in "object", and "type" classes, but I only see the function declaration. I don't see the actual definition.
2
votes
1
answer
157
views
Why __slots__ isn't the default in Python?
I've been programming in Python for a long time, but I still can't understand why classes base their attribute lookup on the __dict__ dictionary by default instead of the faster __slots__ tuple.
...
0
votes
1
answer
475
views
Extract PI OSIsoft Monthly Interval in Python
I am trying to extract the sum of PI data from OSIsoft 10m (10 minute) data in a one (1) month interval using Python pandas. However, I either get an error from OSIsoft or Python when I choose the ...
2
votes
1
answer
220
views
Is list comprehension implemented via map and lambda function? [duplicate]
According to the list comprehension doc and this question: squares = [x**2 for x in range(10)] is equivalent to squares = list(map(lambda x: x**2, range(10)))
But does Python actually implement list ...
0
votes
1
answer
139
views
taking square using "value**2" results causes an overflow while "value*value" is fine
Given the same input, x**2 gives an integer overflow while x*x works fine.
I am not sure if this is because of the python's internal implementation of those operator or if this is a bug in the ...
3
votes
1
answer
562
views
Hash of integers in python
I understand that hash of an immutable object is an integer representation of that object which is unique within the process's lifetime.
Hash of an integer object is the same as the value held by the ...
2
votes
0
answers
230
views
How to print the memory structure of a Python data type?
I'd like to be able to produce a hex dump of the data structure of Python data types. The closest I can get right now is something like this:
from ctypes import string_at
a = [n for n in range(20)]
...
2
votes
1
answer
312
views
How can I access the weakref object of the class itself through the class?
As far as I know, __weakref__ is a descriptor defined in class, so that if it invoked from the instances of the class, it will give the weakref object:
from weakref import ref
class A:
pass
obj =...
11
votes
1
answer
1k
views
Where is the default behavior for object equality (`==`) defined?
According to the object.__eq__() documentation, the default (that is, in the object class) implementation for == is as follows:
True if x is y else NotImplemented
Still following the documentation ...
3
votes
1
answer
158
views
What is Python 3 `str.__getitem__` computional complexity?
''' Set up '''
s= open("Bilion_of_UTF-8_chars.txt",encoding="UTF-8").read()
'''
The following doesn't look like a cheap operation
because Python3 `str`-s are UTF-8 encoded (EDIT: ...
0
votes
1
answer
440
views
Native array.frombytes() (not numpy!) mysterious behavior
[I cannot use numpy so please refrain from talking about it]
I (apparently naively) thought Python array.frombytes() would read from a series of bytes representing various machine format integers, ...
7
votes
1
answer
161
views
How does CPython implement os.environ?
I was looking through source and noticed that it references a variable environ in methods before its defined:
def _createenviron():
if name == 'nt':
# Where Env Var Names Must Be UPPERCASE
...
5
votes
0
answers
157
views
20% faster with unused variable? Why?
I'm doing a lot of benchmarks. I've never seen something like this. I'm stumped. Creating an extra global variable, not used at all, makes part of my code about 20% faster. Why?
I'm benchmarking a ...
6
votes
1
answer
727
views
Does Python not reuse memory here? What does tracemalloc's output mean?
I create a list of a million int objects, then replace each with its negated value. tracemalloc reports 28 MB extra memory (28 bytes per new int object). Why? Does Python not reuse the memory of the ...
0
votes
2
answers
669
views
Why python property() function is assigned to a class variable and not an instance variable?
I'm learning about encapsulation and abstraction in python and i came across the property function and decorator. The common example is something like this.
class Celsius():
def __init__(self, ...
3
votes
0
answers
40
views
Why do sets behave unpredictably? [duplicate]
As explained in Can Python's set absence of ordering be considered random order?, the set structure is in arbitrary order. For example, the following code results in different output between runs.
...
1
vote
1
answer
429
views
Why is memory not freed in this case?
I have the following Python code:
import os, psutil
import numpy as np
process = psutil.Process(os.getpid())
print(process.memory_info().rss)
def append(x):
x.append(np.random.normal(size=(1000,...
101
votes
3
answers
8k
views
Can we make 1 == 2 true? [duplicate]
Python ints are objects that encapsulate the actual number value. Can we mess with that value, for example setting the value of the object 1 to 2? So that 1 == 2 becomes True?
3
votes
3
answers
1k
views
How does Python interpreter actually interpret a program?
Take a sample program:
c = 10
def myfunc():
print(c)
myfunc()
This prints 10 as expected, but if we look at another program:
c = 10
def myfunc():
print(c)
c = 1
myfunc()
It says: ...
3
votes
1
answer
717
views
Why does the float object behave differently with the "is" operator?
As far as I know cpython implementation keeps the same object for some same values in order to save memory. For example when I create 2 strings with the value hello, cpython does not create 2 ...
28
votes
1
answer
5k
views
Python threads difference for 3.10 and others
For some, simple thread related code, i.e:
import threading
a = 0
threads = []
def x():
global a
for i in range(1_000_000):
a += 1
for _ in range(10):
thread = threading.Thread(...
1
vote
0
answers
115
views
Is it possible to get the source code of the current (multiline) statement using the stack frame?
I'm interested in being able to use code like this to obtain 3 or (3, 7) (the lines containing the current statement), however, f_lineno returns 6 because that is where the actual execution is ...