0

We setup a distributed monitoring cluster with one Zabbix (3.4.7) Server and 8 Proxies:

Zabbix:

OS: Debian Stretch
CPU: 16*2.27GHz
RAM: 48GB
Disk: Raid1 10K (Non-SSD)


LogFile=/var/log/zabbix/zabbix_server.log
PidFile=/var/run/zabbix/zabbix_server.pid
DBName=zabbix
DBUser=zabbix
DBHost=127.0.0.1
DBPort=3307
LogFileSize=0
DBPassword=****
Timeout=4
AlertScriptsPath=/etc/zabbix/alert.d/
FpingLocation=/usr/bin/fping
LogSlowQueries=3000
Include=/etc/zabbix/zabbix_server.conf.d/*.conf
StartAlerters=10
StartPollers=80
StartPollersUnreachable=80
StartTrappers=20
StartPingers=30
StartEscalators=5
CacheSize=8G
StartDBSyncers=16
HistoryCacheSize=2048M
TrendCacheSize=256M
ValueCacheSize=10G
HistoryIndexCacheSize=2G
ExternalScripts=/etc/zabbix/alert.d/
SSHKeyLocation=/nonexistent/.ssh

Proxy:

OS: Debian Stretch
CPU: 15*2.5GHz
RAM: 6GB
Disk: Raid1 10K (Non-SSD)


Server=XXXX
Hostname=zbx-lte
LogFile=/var/log/zabbix/zabbix_proxy.log
LogFileSize=0
PidFile=/var/run/zabbix/zabbix_proxy.pid
SocketDir=/var/run/zabbix
DBName=zabbix
DBUser=zabbix
DBPassword=159753
ConfigFrequency=600
DataSenderFrequency=1
StartPollers=240
StartPollersUnreachable=80
StartTrappers=20
StartPingers=80
SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
CacheSize=1G
StartDBSyncers=16
HistoryCacheSize=2048M
HistoryIndexCacheSize=2G
Timeout=6
ExternalScripts=/usr/lib/zabbix/externalscripts
FpingLocation=/usr/bin/fping
LogSlowQueries=3000

We're monitoring near 1650 nodes (snmp, icmp, agent, ssh ,external scripts and external applications) by Zabbix

Since (about ) 2 month ago ,  we saw so many lags in non-icmp graphs  (Pic: 5.png) on specific zabbix proxy (zbx-lte in pics)

too many lags on all graphs of Zabbix Proxy
(source: sassan.co)

this graph is related to a device (with snmp v2).  (there are many other graph like this on this proxy) I capture network traffic of relevant zabbix proxy , For one of items  , zabbix proxy sends only 16 queries instead of 60 queries (with interval 1m for 1 hour) .

too many lags on all graphs of Zabbix Proxy
(source: sassan.co)

If I change this device to monitor by Zabbix Server or any other proxies everything work properly

It sounds there is a problem on this proxy htop of Zabbix Proxy
(source: sassan.co)

free mem of Zabbix Proxy
(source: sassan.co)

Please help me to find the root cause.

More Details
(source: sassan.co)

More Details
(source: sassan.co)

More Details
(source: sassan.co)

More Details More Details
(source: sassan.co)

More Details
(source: sassan.co)

More Details
(source: sassan.co)

2
  • Load average is quite high, considering the 16 CPUs. Could you check the iowait of the system and share a screenshot ? Commented Dec 27, 2018 at 13:10
  • @SimoneZabberoni top screenshot added Commented Dec 28, 2018 at 14:42

1 Answer 1

0

I got it!

Surprisingly , The answer is very funny!

Server address in zabbix_proxy was an FQDN address , Our performance issues was solved when I add it to /etc/hosts.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.