0

I'm supposed to read a large txt file in chunks and every word in chunk has to be processed. But some words can be cut into pieses. For instance:

text_in_file = 'some text in file to be processed'
text_in_file.read(15)

result will be 'some text in fi', 'le to be proces' and so on

Is there a way to find out whether word is cut and to join ending of previous chunk and beginning of a next one?

1
  • read one character at a time into a buffer of some sort, process the buffer when you hit a word boundary. The problem you are usually trying to resolve is having the whole file in memory at once, which is why you are reading it in chunks. Jan 24, 2015 at 15:50

1 Answer 1

0

Just read line by line, here's how: https://stackoverflow.com/a/8010133/3997052

This way you (probably) don't get "splited" words - depends on your file.

1
  • Thanks, but this is not exactly what I needed. Jan 28, 2015 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.