How to make asyncio using all cpu cores - any other options than ProcessPoolExecutor?
I assume that asyncio can not break GIL limit (maybe I am wrong) so programs will be executed faster than treading version but will on one core.
I study some examples and I found that one way to do it is multiprocessing and ProcessPoolExecutor.
https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor
# 3. Run in a custom process pool:
with concurrent.futures.ProcessPoolExecutor() as pool:
result = await loop.run_in_executor(
pool, cpu_bound)
print('custom process pool', result)
That is nice but need "pickle" between processes so some overhead is required and some optimization of passed arguments to reduce "pickle" serialization.
Using this simple pattern above I wrote such test code (you can skip this code reading if you do not like it since it is same as before). BTW this the fastest solution of my problem with parsing files. This part of code not whole program.
def _match_general_and_specific_file_chunk(file_name):
with codecs.open(file_name, encoding='utf8') as f:
while True:
lines = f.readlines(sizehint=10000)
if not lines:
break
for line in lines:
general_match = RE_RULES.match(line)
if general_match:
specific_match = RULES[general_match.lastindex].match(line)
groups = list(specific_match.groups())
continue
async def _async_process_executor_match_general_and_specific_read_lines_with_limit_file_chunk():
loop = asyncio.get_event_loop()
with ProcessPoolExecutor() as pool:
futures = []
for file_name in get_file_names():
future = loop.run_in_executor(pool, _match_general_and_specific_file_chunk, file_name)
futures.append(future)
await asyncio.gather(*futures)
def async_process_executor_match_general_and_specific_read_lines_with_limit_file_chunk():
asyncio.run(_async_process_executor_match_general_and_specific_read_lines_with_limit_file_chunk())