I'm trying to find a nice way to read a log file in real time using python. I'd like to process lines from a log file one at a time as it is written. Somehow I need to keep trying to read the file until it is created and then continue to process lines until I terminate the process. Is there an appropriate way to do this? Thanks.
4 Answers
Take a look at this PDF starting at page 38, ~slide I-77 and you'll find all the info you need. Of course the rest of the slides are amazing, too, but those specifically deal with your issue:
import time
def follow(thefile):
thefile.seek(0,2) # Go to the end of the file
while True:
line = thefile.readline()
if not line:
time.sleep(0.1) # Sleep briefly
continue
yield line
-
7Worth noting that this will skip any contents already in the log file, only printing "new" entries created after this iterator was created. Also that PDF really is a gold mine ;)– blentedSep 2, 2016 at 0:48
-
3What to do, if I watch rotating logfiles? stackoverflow.com/questions/44407834/…– jaromraxJul 25, 2018 at 12:26
-
-
-
Tested on a short file and it keeps running with no end. Did I do something wrong ?
with open(myfile, "r") as f : x = follow(f) for l in x: print(l)
– KennySep 24, 2020 at 16:00
You could try with something like this:
import time
while 1:
where = file.tell()
line = file.readline()
if not line:
time.sleep(1)
file.seek(where)
else:
print line, # already has newline
Example was extracted from here.
-
This seems to be working but it won't allow me to create objects or write to a database at the same time in my django app. I don't see an obvious reason for this; is there a simple fix?– AnonJul 20, 2010 at 19:37
-
I don't know. You should post some code in a separate question to get answers to this one, I guess. I don't see any reason not to get database updated if you place that code inside this one... Jul 20, 2010 at 21:16
-
Got this to work but I had to mess with the string a lot before I could get it to write to my database. Thanks.– AnonJul 21, 2010 at 18:23
-
3
-
This seems to get one byte at a time or depending on speed partial lines Jul 19, 2019 at 11:44
As this is Python and logging tagged, there is another possibility to do this.
I assume this is based on a Python logger, logging.Handler based.
You can just create a class that gets the (named) logger instance and overwrite the emit
function to put it onto a GUI (if you need console just add a console handler to the file handler)
Example:
import logging
class log_viewer(logging.Handler):
""" Class to redistribute python logging data """
# have a class member to store the existing logger
logger_instance = logging.getLogger("SomeNameOfYourExistingLogger")
def __init__(self, *args, **kwargs):
# Initialize the Handler
logging.Handler.__init__(self, *args)
# optional take format
# setFormatter function is derived from logging.Handler
for key, value in kwargs.items():
if "{}".format(key) == "format":
self.setFormatter(value)
# make the logger send data to this class
self.logger_instance.addHandler(self)
def emit(self, record):
""" Overload of logging.Handler method """
record = self.format(record)
# ---------------------------------------
# Now you can send it to a GUI or similar
# "Do work" starts here.
# ---------------------------------------
# just as an example what e.g. a console
# handler would do:
print(record)
I am currently using similar code to add a TkinterTreectrl.Multilistbox for viewing logger output at runtime.
Off-Side: The logger only gets data as soon as it is initialized, so if you want to have all your data available, you need to initialize it at the very beginning. (I know this is what is expected, but I think it is worth being mentioned.)