0

I have a small UDP perl service that receives syslog data, fiddles with it, and sends it on it's way (over UDP) back to a syslog server that is running also on localhost. It works really well, but I was concerned that it might have been losing syslog events so tested it

Basically I pushed 100 "this is a test $NUM++" messages in, and sometimes 100 would come out - but once 99 came out (as measured by tcpdump running on the host running the perl script). This is on our production system where it's handling 500-1500 syslog msg/sec. As usual it works perfectly when it's only got test traffic, but under load I'm not sure.

tcpdump shows the 100 events coming in over eth0, but tcpdump showed the 99 coming out over lo. So maybe one never made it into the '$rcvSock', or maybe one got lost going out over lo via '$sndSock'

Basically the perl code is like below. However, the "#fiddling" bit does involve some pauses for DNS lookups, so there is some "read->block->write" going on. I don't think "Listen" is supported under UDP, so I can't be sure if the perl script is blocking-and-dropping the next receive while it's doing the "fiddling"?

Can anyone shed any light on where the loss could be occurring and how to get past it?

$rcvSock = IO::Socket::INET->new(
     LocalAddr => $hn,
     LocalPort => $rcvPort,
     Timeout => $timeout,
     Proto    => 'udp'
);

$sndSock = IO::Socket::INET->new(
     PeerAddr => $syslogSrv,
     PeerPort =>$syslogPort,
     Timeout  => $timeout,
     Proto    => 'udp',
     Blocking => 0
);

while (1) {
    $now=time;
    $rcvSock->recv($input,2560);
    $remote_addr=$rcvSock->peerhost();
    #fiddling occurs
    $sndSock->send("$input");
}
2

2 Answers 2

0

Thank you the SO_RCVBUF did the trick.

What is happening is that I am pushing in (say) 1000 syslog records/packets per sec, but the DNS queries I do pause the processing by 1sec/record. So this means after processing ONE record, there are now 999 records to process. After two seconds there's 1998. This isn't looking good...

Those packets can be queued by the OS according to SO_RCVBUF, which is by default (on Redhat) 212992 bytes. So assuming an average of 400 bytes per syslog record, that's a maximum of ~530 records queued up before the kernel starts dropping new packets. So I can increase the SO_RCVBUF 10 even 100-fold but it won't get around the fundamental issue of that big pause. However, in reality I'm talking about peak rates: there are moments when the records/sec drop right down, and a lot of syslog records don't require DNS lookups (ie I skip them). Also by caching the heck out of those DNS lookups, I can minimize the amount of time they are done, so 1000/sec could be 101/sec involving DNS, which in turn could be 99% cachable, leading to only 2-5/sec that need DNS lookups - and at that level a healthy cache will get you through the peak load issues

I am not a programmer, so doing this properly with input queues, asynchronous DNS lookups, etc are beyond me. But I do know iptables... So I'm intending to run several of these on different ports, and use iptables to round-robin-deliver incoming packets onto them, giving them async functionality without me needing to write a single line of code. That should solve this for the load levels I need to worry about :-)

Thanks!

-1

Try to increase SO_RCVBUF

IO::Socket::INET->setsockopt(SOL_SOCKET, SO_RCVBUF, ...)

sar is very helpful for purpose of Network Statistic investigation.

idgmerr/s The number of received UDP datagrams per second that could not be delivered for reasons other than the lack of an application at the destination port [udpInErrors].

$ sar -n UDP 5

08:18:12 AM    idgm/s    odgm/s  noport/s idgmerr/s
08:18:15 AM    121.33    121.33      7.67      0.00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.