Linked Questions

4 votes
1 answer
8k views

Differences between file.read(), file.readline() and iterating over the file object [duplicate]

I am new to computer science and am trying to create a function in python that will open files on my computer. I know that the function f.readline() grabs the current line as a string, but what ...
Korrupted Studios's user avatar
0 votes
2 answers
146 views

File to array in python [duplicate]

I'm reading a tutorial about python for very beginners and at some point the author define some ways to work with files. My doubt is related to memory management and file arrays. #open a file for ...
Guilherme's user avatar
  • 453
16 votes
2 answers
26k views

Using python ijson to read a large json file with multiple json objects

I'm trying to parse a large (~100MB) json file using ijson package which allows me to interact with the file in an efficient way. However, after writing some code like this, with open(filename, 'r') ...
Boubouh Karim's user avatar
4 votes
4 answers
2k views

Does for line in File read entire file

Does the following code read one line for each loop or does it read the entire file into memory first before beginning the iteration? for line in f: print(line) My intentions are to read a ...
SSS's user avatar
  • 2,444
1 vote
4 answers
2k views

How do I clean up empty lines in a configparser file?

I sometimes get an .ini file that looks like this after changing and deleting values: [Section A] x = 1 d = 2 [Section B] a = 3 Is there an easy way to keep it clean and remove those blank lines ...
Dima's user avatar
  • 497
2 votes
3 answers
788 views

Does it takes RAM to save a readlines array?

I am using the command lineslist = file.readlines() of a 2GB file. So, I guess it will create a lineslist array of 2GB or more size. So, basically is it the same as readfile = file.read(), which ...
GP92's user avatar
  • 415
2 votes
4 answers
1k views

Does high disk usage mean faster file read/write operations?

I'm writing a python script in which I read a big file ~5 GB line by line, make some modifications in each of the lines, and then write it to another file. When I use the function file.readlines() ...
noobcoder's user avatar
1 vote
1 answer
2k views

Python: iteratively add to dataframe

I have the following code: for state in state_list: state_df = pd.DataFrame() for df in pd.read_csv(tax_sample,sep='\|\|', engine='python', dtype = tax_column_types, chunksize = 10, nrows = ...
Mike's user avatar
  • 1,069
0 votes
0 answers
1k views

how to read a huge csv file using mmap in python?

I want to read csv file and perform some operation on that file. I'm created the program for my requirement but I'm not getting output because the file size is very large i.e. ~5GB. I'm using simple ...
ketan's user avatar
  • 2,864
0 votes
3 answers
533 views

Unable to split the sequence from a Fasta File

my fasta file has multiple sequences like the following >header1 MPANFTE GSFDSSG >header2 MDNASFS EPWPANA so I am writing a code to remove the headers and the output looks like this in a ...
KBnd's user avatar
  • 21
1 vote
1 answer
350 views

What is causing this socket to run really slow?

the following code checks for an open port and the socket seems to be running really slow: for line in f.readlines(): line = line.rstrip() with closing(socket.socket(socket....
user7834940's user avatar
0 votes
3 answers
136 views

Locate multiple keywords in lines using Python

I got a line like this : 20:28:26.684597 24:d5:6e:76:9s:10 (oui Unknown) > 45:83:r4:7u:9s:i2 (oui Unknown), ethertype 802.1Q (0x8100), length 78: vlan 64, p 0, ethertype IPv4, (tos 0x48, ttl 34,...
Zoro99's user avatar
  • 185
1 vote
3 answers
159 views

import dirty csv file with unwanted characters, strings

I would like to import csv files with pandas. Normally my data is given in the form: a,b,c,d a1,b1,c1,d1 a2,b2,c2,d2 where a,b,c,d is the header. I can easily use the pandas.read_csv here. However, ...
Anna's user avatar
  • 79
0 votes
2 answers
136 views

Problem with python memory, flush, csv size

After solving a sorting of a dataset, I have a problem at this point of my code. with open(fns_land[xx]) as infile: lines = infile.readlines() for line in lines: result_station.append(...
S.Kociok's user avatar
  • 169
-3 votes
1 answer
92 views

How to open a text file which has more than 500k lines, without using any iteration?

I am working with text files and looping over them, python works well with files of 10k to 20k lines, most of them are of that length, few text files are over 100k lines, where the code just stops or ...
Manas Jadhav's user avatar

15 30 50 per page