26

I have Perl script which appends a new line to the existing file every 3 seconds. Also, there is a C++ application which reads from that file.

The problem is that the application begins to read the file after the script is done and file handle is closed. To avoid this I want to flush after each line append. How can I do that?

2
  • In "basic" perl there is not flush function, but if You call binmode $filehandle it will set :raw format and (as a side affect) it will make a flush. It works nicely. Anyway if it is on un*x, You can create a named pipe as to the filesystem, and You can write to it directly.
    – TrueY
    Commented Apr 12, 2013 at 15:14
  • Make your C++ program read from a pipe (/dev/stdin if it's stupid enough to require a filename) and check that it can actually read one line at a time - if it uses block-buffered reading there is nothing you can do on the writing side. Commented Apr 17, 2013 at 10:13

11 Answers 11

33

Try:

use IO::Handle;
$fh->autoflush;

This was actually posted as a way of auto-flushing in an early question of mine, which asked about the universally accepted bad way of achieving this :-)

3
  • 1
    Thank you. But this doesn't help me. Even after flushing after each line, my C++ program can't read this lines, after they been insreted. It reads only after perl ends its job. Commented Dec 27, 2010 at 13:34
  • Now how do you flush STDOUT?
    – Chloe
    Commented Apr 18, 2018 at 0:48
  • 1
    @Chloe use $| = 1
    – talkloud
    Commented Sep 1, 2020 at 12:20
30

TL/DR: use IO::Handle and the flush method, eg:

use IO::Handle;
$myfile->flush();

First, you need to decide how "flushed" you want it. There can be quite a few layers of buffering:

  • Perl's internal buffer on the file handle. Other programs can't see data until it's left this buffer.

  • File-system level buffering of "dirty" file blocks. Other programs can still see these changes, they seem "written", but they'll be lost if the OS or machine crashes.

  • Disk-level write-back buffering of writes. The OS thinks these are written to disk, but the disk is actually just storing them in volatile memory on the drive. If the OS crashes the data won't be lost, but if power fails it might be unless the disk can write it out first. This is a big problem with cheap consumer SSDs.

It gets even more complicated when SANs, remote file systems, RAID controllers, etc get involved. If you're writing via pipes there's also the pipe buffer to consider.

If you just want to flush the Perl buffer, you can close the file, print a string containing "\n" (since it appears that Perl flushes on newlines), or use IO::Handle's flush method.

You can also, per the perl faq use binmode or play with $| to make the file handle unbuffered. This is not the same thing as flushing a buffered handle, since queuing up a bunch of buffered writes then doing a single flush has a much lower performance cost than writing to an unbuffered handle.

If you want to flush the file system write back buffer you need to use a system call like fsync(), open your file in O_DATASYNC mode, or use one of the numerous other options. It's painfully complicated, as evidenced by the fact that PostgreSQL has its own tool just to test file syncing methods.

If you want to make sure it's really, truly, honestly on the hard drive in permanent storage you must flush it to the file system in your program. You also need to configure the hard drive/SSD/RAID controller/SAN/whatever to really flush when the OS asks it to. This can be surprisingly complicated to do and is quite OS/hardware specific. "plug-pull" testing is strongly recommended to make sure you've really got it right.

19

From 'man perlfaq5':

$old_fh = select(OUTPUT_HANDLE);
$| = 1;
select($old_fh);

If you just want to flush stdout, you can probably just do:

$| = 1;

But check the FAQ for details on a module that gives you a nicer-to-use abstraction, like IO::Handle.

1
  • Another (convoluted, single-line) way of writing this is select((select(OUTPUT_HANDLE),$|=1)[0]) (taken from stackoverflow.com/questions/196754/… ). See other considerations about the benefits and problems of this in that thread.
    – msb
    Commented Aug 17, 2018 at 0:36
4

Here's the answer - the real answer.

Stop maintaining an open file handle for this file for the life of the process.

Start abstracting your file-append operation into a sub that opens the file in append mode, writes to it, and closes it.

# Appends a new line to the existing file
sub append_new_line{
    my $linedata = shift;
    open my $fh, '>>', $fnm or die $!; # $fnm is file-lexical or something
    print $fh $linedata,"\n"; # Flavor to taste
    close $fh;
}

The process observing the file will encounter a closed file that gets modified whenever the function is called.

4
  • 2
    Yep, it works. I used the principle today in a Perl script. Performance was not an issue - one line written every 16 seconds. Commented Jun 2, 2020 at 22:37
  • Perhaps make an experiment measuring the overhead under certain conditions? Commented Jun 2, 2020 at 22:39
  • In Perl, we optimize for programmer time, outside of things going quadratic. Commented Jun 3, 2020 at 15:11
  • this is ok only, when you update file rarely. but for often update, it is a wrong way. otherwise, you are using some text file working as a log. all lines is ended by "/n", then it is done autoflush line by line. then we agree, file must not be closed/opened many times. but....... sometime log file is very big. then you must do some procedure for logrotate. first aproach, is check file inode is the same. if is different, before write you must do close/open. otherwise you must check reaction for signal from logrotate, to switch to another file.
    – Znik
    Commented Oct 8, 2020 at 10:49
2

All of the solutions suggesting setting autoflush are ignoring the basic fact that most modern OS's are buffering file I/O irrespective of what Perl is doing.

You only possibility to force the commitment of the data to disk is by closing the file.

I'm trapped with the same dilemma atm where we have an issue with rotation of the log being written.

1
  • 6
    "You only possibility to force the commitment of the data to disk is by closing the file". Closing a file doesn't commit it to disk, the OS is still free to write-back cache it. The only guaranteed way is to call fsync() or your OS's equivalent the file descriptor. Otherwise it might be lost on OS crash or power loss. Commented Jul 19, 2013 at 7:51
1

To automatically flush the output, you can set autoflush/$| as described by others before you output to the filehandle.

If you've already output to the filehandle and need to ensure that it gets to the physical file, you need to use the IO::Handle flush and sync methods.

1

There an article about this in PerlDoc: How do I flush/unbuffer an output filehandle? Why must I do this?

Two solutions:

  • Unbuffer the output filehandler with : $|
  • Call the autoflush method if you are using IO::Handle or one of its subclasses.
0

An alternative approach would be to use a named pipe between your Perl script and C++ program, in lieu of the file you're currently using.

0

For those who are searching a solution to flush output line by line to a file in Ansys CFD Post using a Session File (*.cse), this is the only solution that worked for me:

! $file="Test.csv";
! open(OUT,"+>>$file");
! select(OUT);$|=1;  # This is the important line
! for($i=0;$i<=10;$i++)
! {
!    print out "$i\n";
!    sleep(3);
! }

Note that you need the exclamation marks at every begin of every line that contains Perl script. sleep(3); is only applied for demonstration reasons. use IO::Handle; is not needed.

0

The genuine correct answer is to use:-

$|=1; # Make STDOUT immediate (non-buffered)

and although that is one cause of your problem, the other cause of the same problem is this: "Also, there is a C++ application which reads from that file."

It is EXTREMELY NON-TRIVIAL to write C++ code which can properly read from a file that is growing, because your "C++" program will encounter an EOF when it gets to the end... (you cannot read past the end of a file without serious extra trickery) - you have to do a pile of complicated stuff with IO blocking and flags to properly monitor a file this way (like how the linux "tail" command works).

-1

I had the same problem with the only difference of writing the same file over and over again with new content. This association of "$| = 1" and autoflush worked for me:

 open  (MYFILE, '>', '/internet/web-sites/trot/templates/xml_queries/test.xml');
 $| = 1; # Before writing!
 print  MYFILE "$thisCardReadingContentTemplate\n\n";
 close (MYFILE);
 MYFILE->autoflush(1); # After writing!

Best of luck. H

2
  • 3
    afaik, closing the file do the flush. What is MYFILE->autoflush(1) supposed to do here ?
    – PypeBros
    Commented Jul 23, 2012 at 12:16
  • 1
    select(MYFILE);$|=1 ? Commented Aug 23, 2018 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.