An audit, in IT terms, is an evaluation and an exhaustive examination of services, health, processes, connections, applications, practices for optimization and possible vulnerabilities in the server. It provides the best information to take action and patch up insecurities that are funded in this examination. This audit involves gathering information about attacks that arrive over the TCP/IP network.
Once we’ve known what an audit is, we are going to recognize the importance of making it one of our servers. This is because computer security has become quite important in every organization due to the high quality and correct functionally required by users or customers in all current web services.
It’s a process that verifies if certain standards are being accomplished and it detects those areas that need remediation or improvement. A security audit involves a systematic and a measurable technical assessment of how an organization’s security policy is employed at a specific site or location to find information that drives to the suspect infiltration, suspect user, injected bad code, and system infection.
First of all, in an intrusion Audit, we need to identify technical vulnerabilities in computers and networks, as well as weaknesses in policies and practices related to the operation of these systems, read logs, suspicious services and processes.
A system that has been compromised (“or hacked”) is not always notable, it usually runs in an invisible way, but we can recognize it by the following clues:
Some common practices to start the Forensics analysis are to review uncommon processes, review how many users is the site running, review uncommon Unix users, Review docroot and /tmp folder to upload folder from the application and look for suspicious files to analyze permissions. Additionally, you should check the emails received in the root inbox, to find the intruder. For this you must know how your actual Linux system works, which services are running, the structure of the application code and a good sense of your IT system architecture. Keeping this in mind and looking for discrepancies, uncommon activities will lead you to the root cause, vulnerability and intruder.
One thing we have to take in consideration is that intrusions are not always in the same way; Almost every attack is executed by using different methods. The way to remove an intruder and its tools may vary depending on the case.
Check users on Linux server with this command:
$ cut -d: -f1 /etc/passwd
This command will list your server users*.
*If there is someone that isn’t a system user like httpd, MySQL, Nginx, bin, mail, www-data, Syslog, man, etc. If there is another name that you do not recognize, you can remove it by typing the command. Make sure that the user you are removing isn’t really a system user.
$ sudo userdel user-that-you-find
To check processes, sockets and ports use a tool called Netstat and lsof. To see sockets and ports in actual, use:
$ netstat -ant
Same as previous, except that it gives you more details like PID and Program name:
$ netstat -plant
To see sockets and ports in actual use every 5 seconds:
$ netstat -n -o 5
To see what program and PID, are using sockets, open ports, files, scripts, etc. use:
$ lsof -i
If you noticed some “suspicious” processes, then you can see what files are executing with the next command:
$ lsof -p Program-PID
The site was infected with injected code PHP and was sending Spam. For this reason, the email domain was banned and blacklisted by Google, Bing, and Yahoo.
First I needed to check which files, script, sockets and ports were being used. I used “lsof” and then filtered the results with “grep” command:
$ lsof -Pni |grep -vE ':80|:22|named|dovecot|sendm|minis|mysql|pure'
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEphp 1525 gozer 4u IPv4 10972645 0t0 TCP 50.97.123.10:55993->23.21.148.239:443 (CLOSE_WAIT)sshd 1746 root 3u IPv4 10011 0t0 TCP *:30001 (LISTEN)sshd 1746 root 4u IPv6 10013 0t0 TCP *:30001 (LISTEN)/usr/loca 1987 root 5u IPv4 10627 0t0 TCP 127.0.0.1:783 (LISTEN)spamd 1998 root 5u IPv4 10627 0t0 TCP 127.0.0.1:783 (LISTEN)httpd 2048 root 5u IPv4 10869 0t0 TCP *:443 (LISTEN)httpd 2048 root 6u IPv6 10871 0t0 TCP *:443 (LISTEN)cpsrvd 2268 root 4u IPv4 14295 0t0 TCP *:2087 (LISTEN)cpsrvd 2268 root 6u IPv4 14298 0t0 TCP *:2086 (LISTEN)cpsrvd 2268 root 8u IPv4 14301 0t0 TCP *:2095 (LISTEN)cpsrvd 2268 root 9u IPv4 14302 0t0 TCP *:2082 (LISTEN)cpsrvd 2268 root 10u IPv4 14303 0t0 TCP *:2083 (LISTEN)cpsrvd 2268 root 11u IPv4 14304 0t0 TCP *:2096 (LISTEN)cpdavd 2357 root 7u IPv4 15073 0t0 TCP *:2078 (LISTEN)cpdavd 2357 root 8u IPv4 15074 0t0 TCP *:2080 (LISTEN)cpdavd 2357 root 9u IPv4 15075 0t0 TCP *:2077 (LISTEN)cpdavd 2357 root 10u IPv4 15076 0t0 TCP *:2079 (LISTEN)sshd 5769 root 3r IPv4 69662948 0t0 TCP 50.97.123.10:30001->189.219.232.49:52994 (ESTABLISHED)perl 12432 wherechristispre 20u IPv4 17614303 0t0 TCP 50.97.123.10:39331->194.54.81.162:43158 (CLOSE_WAIT)perl 12432 wherechristispre 22u IPv4 19126496 0t0 TCP 50.97.123.10:39331->194.54.81.162:40990 (CLOSE_WAIT)perl 12432 wherechristispre 24u IPv4 19269835 0t0 TCP 50.97.123.10:39331->194.54.81.162:60506 (CLOSE_WAIT)perl 12432 wherechristispre 26u IPv4 21012134 0t0 TCP 50.97.123.10:39331->194.54.81.162:54509 (CLOSE_WAIT)perl 12432 wherechristispre 28u IPv4 21244803 0t0 TCP 50.97.123.10:39331->194.54.81.162:53758 (CLOSE_WAIT)perl 12432 wherechristispre 30u IPv4 21288232 0t0 TCP 50.97.123.10:39331->194.54.81.162:34472 (CLOSE_WAIT)perl 12432 wherechristispre 32u IPv4 22234865 0t0 TCP 50.97.123.10:39331->194.54.81.162:59834 (CLOSE_WAIT)perl 12432 wherechristispre 34u IPv4 22791575 0t0 TCP 50.97.123.10:39331->194.54.81.162:54283 (CLOSE_WAIT)
As we can see, PHP and PERL scripts were being executed, so I needed to check if an IP address connected to the server was on the blacklist with an online tool called “mxtoolbox” (http://mxtoolbox.com).
With “lsof” I got IP 23.21.148.239 and IP 189.219.232.49, so I checked it on mxtoolbox.
I noticed that the IP was on the blacklist and it was executing a PHP script, so I had to see which documents were being executed.
$ lsof -p 1525
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEphp 1525 gozer cwd DIR 202,2 4096 92370 /home/gozer/public_htmlphp 1525 gozer rtd DIR 202,2 4096 2 /php 1525 gozer txt REG 202,2 27664289 2772504 /usr/bin/phpphp 1525 gozer mem REG 202,2 1965856 2771095 /usr/lib64/libcrypto.so.1.0.1ephp 1525 gozer mem REG 202,2 444184 2772290 /usr/lib64/libssl.so.1.0.1ephp 1525 gozer mem REG 202,2 7707967 2771651 /usr/lib64/libmysqlclient.so.18.0.0php 1525 gozer mem REG 202,2 157960 2771757 /usr/lib64/libpng12.so.0.49.0php 1525 gozer mem REG 202,2 157072 4112697 /lib64/ld-2.12.sophp 1525 gozer mem REG 202,2 1926520 4112698 /lib64/libc-2.12.sophp 1525 gozer mem REG 202,2 145896 4112641 /lib64/libpthread-2.12.sophp 1525 gozer mem REG 202,2 22536 4112402 /lib64/libdl-2.12.sophp 1525 gozer mem REG 202,2 47112 4112817 /lib64/librt-2.12.sophp 1525 gozer mem REG 202,2 599392 4112830 /lib64/libm-2.12.sophp 1525 gozer 5r DIR 202,2 4096 181107 /home/gozer/public_html/wp-content/plugins/supermenu/admin-deprecated/redux-framework/sample/patterns
In this case, we can see that the plugin “redux-framework” was being executed by the IP blacklisted, so it was very likely that the Plugin was infected. The best option was to remove and install it again, and not allowing it to reconnect to the server with an iptables rule.
$ iptables -A INPUT -s 23.21.148.239 -j DROP
Once I discovered this, I had to know what file was running with the next command where “1525” is the program PID.
$ cat /proc/1525/environFILENAME=/home/gozer/public_html/xmlrpc.phpSCRIPT_NAME=/xmlrpc.php
“xmlrpc” is very vulnerable and has to be continuously upgraded. It must have a good configuration to protect the .htaccess file with the next configuration on the end line:
<Files "xmlrpc.php">Order Allow,Denydeny from all
I recommend always to have the ultimate version.
Now, I had to see what was happening with Perl scripts, using lsof again.
$ lsof -p 16786
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEperl 16786 wherechristispre cwd DIR 202,2 4096 2 /perl 16786 wherechristispre rtd DIR 202,2 4096 2 /perl 16786 wherechristispre txt REG 202,2 13304 2770784 /usr/bin/perlperl 16786 wherechristispre mem REG 202,2 1488544 2794989 /usr/lib64/perl5/CORE/libperl.soperl 16786 wherechristispre mem REG 202,2 157072 4112697 /lib64/ld-2.12.soperl 16786 wherechristispre mem REG 202,2 1926520 4112698 /lib64/libc-2.12.soperl 16786 wherechristispre mem REG 202,2 145896 4112641 /lib64/libpthread-2.12.soperl 16786 wherechristispre mem REG 202,2 22536 4112402 /lib64/libdl-2.12.soperl 16786 wherechristispre mem REG 202,2 17520 4112783 /lib64/libutil-2.12.soperl 16786 wherechristispre mem REG 202,2 599392 4112830 /lib64/libm-2.12.soperl 16786 wherechristispre mem REG 202,2 113952 4112885 /lib64/libresolv-2.12.soperl 16786 wherechristispre mem REG 202,2 12776 4112745 /lib64/libfreebl3.soperl 16786 wherechristispre mem REG 202,2 116368 4112903 /lib64/libnsl-2.12.soperl 16786 wherechristispre mem REG 202,2 43392 4112749 /lib64/libcrypt-2.12.soperl 16786 wherechristispre mem REG 202,2 21056 2801666 /usr/lib64/perl5/auto/File/Glob/Glob.soperl 16786 wherechristispre mem REG 202,2 120008 2801701 /usr/lib64/perl5/auto/POSIX/POSIX.soperl 16786 wherechristispre mem REG 202,2 17976 2803461 /usr/lib64/perl5/auto/Fcntl/Fcntl.soperl 16786 wherechristispre mem REG 202,2 25624 2801884 /usr/lib64/perl5/auto/Socket/Socket.soperl 16786 wherechristispre mem REG 202,2 19336 2801685 /usr/lib64/perl5/auto/IO/IO.soperl 16786 wherechristispre mem REG 202,2 99154480 2772010 /usr/lib/locale/locale-archiveperl 16786 wherechristispre 0r CHR 1,3 0t0 3800 /dev/nullperl 16786 wherechristispre 1w CHR 1,3 0t0 3800 /dev/nullperl 16786 wherechristispre 2w CHR 1,3 0t0 3800 /dev/nullperl 16786 wherechristispre 3u IPv4 69906812 0t0 TCP *:39331 (LISTEN)
So I was able to see the modules, sockets, etc. You could do Forensic Analysis (TCT, The Coroner’s Toolkit) and see if Perl was replaced or if it is a rootkit doing a specifically scan in the found path with ClamAV or Sophos. Another option is re-installing Perl.
To get more information I executed a nmap scan with the next IP 194.54.81.162 using nmap tool.
$ nmap 194.54.81.162
Starting Nmap 6.47 ( http://nmap.org ) at 2015-12-27 20:33 CST
Nmap scan report for 162.81.54.194.static.server.ua (194.54.81.162) Host is up (0.19s latency). Not shown: 993 closed ports PORT STATE SERVICE 21/tcp open ftp 25/tcp filtered smtp 53/tcp open domain 80/tcp open http 88/tcp open kerberos-sec 5222/tcp open xmpp-client 5269/tcp open xmpp-server
As you can see, this scan provided me enough information about this IP. The IP had a PTR record (reverse DNS) which concludes, it was a server with domain “UA” that is the Internet country code for Ukraine. I didn’t need this connection, so it was necessary to block this IP address, with the next command:
$ iptables -A INPUT -s 194.54.81.162 -j DROP
To save IPTABLES rules use:
$ iptables-save
Now, I had to do a scan check manually on the document root, looking for suspicious code in PHP files. I had to find PHP scripts that used “eval (“ or “base64” functions. This is not always malware, but with the names and with the type of scripts I was able to identify the ones that were actually malware. You can use an online tool called VirusTotal.com to check the signatures in the code and know if it is malware. However, I searched “manually” to find eval values:
$ grep -lir “eval (”
And got this results:
Then, I opened every suspicious file. I opened this one with vim:
$ vim wp-content/themes/betheme/functions/img/items/dir.php.suspected.
And I discovered the first Malware. In this case, I had to delete the file using the command:
$ rm -f wp-content/themes/betheme/functions/img/items/dir.php.suspected
All the files that had a similar structure had to be deleted with the RM command.
Then, I had to find malicious signatures in the document root, so I have “malware signatures” saved for this cases. I checked all of them with grep command.
Next signature was contained by two PHP files:
$ grep "setTineout(1e); var defau1t_keyword = encodeURIConponent(document.tit1e); var se_referrer = eQFodeURIConponent(document.referrer); var host =encodeURIConponent(window.location.host)" -lir
Here the results:
wp-content/themes/betheme/header.php wp-content/themes/twentyfifteen/header.php
I had to open the file and find the string, in this case, I used Vim to open the result files and localize the malicious script.
I commented the script to ignore this line of code. Then I reviewed whether the page was loading properly, sometimes scripts call other script and the page shows errors.
So as I noticed that there was no problem, I proceeded to eliminate this malicious script in all files that were found.
Now, the files were disinfected. But the real problem wasn’t resolved yet. The infected server was using Exim. In this case, an effective way to find intrusion and email sent without authorization was to check logs files. I found this worthy information in Exim logs that was located in: /var/log/exim_mainlog and other logs created time before. I identified the email subjects that were sent by the malware by typing this:
$ awk -F"T="" '/<=/ {print $2}' /var/log/exim_mainlog-20151220 | cut -d" -f1 | sort | uniq -c | sort -n The subject “You Have 1 SnapF#ckMsg” was sent 1063 times, so I found the user that was sending this Spam and stopped it. By using this command, I was able to see the user:
$ grep "You Have 1 SnapF#ckMsg" /var/log/exim_mainlog-20151220 | sort -n | uniq -c | sort -n
Result:
1063 U=wherechristispre
Because of this, I got the IP that was sending the mails (I’m not writing the address here because it was the server’s IP and for privacy reasons). But with this, I noticed that the user was sending SPAM locally. The first step to fix this was to change all Email password accounts and then to localize the script that was sending SPAM.
I had to find it by checking the logs and filtered with the next command:
$ grep cwd /var/log/exim_mainlog-20151220 | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n I located the PHP script that was sending SPAM and with this command, I found the path script that was sending emails.
This script is a malicious one because there shouldn’t be any PHP script here. By opening the file, I found the malicious script that was sending SPAM.
And finally, I deleted it. You could use lsof to see if there are any more PHP scripts executing and discover other possible malicious scripts by using for example:
$ lsof /home/wherechrisispresent/public_html/wp-admin/css/midnight/alias57.php
As I said before, it is necessary to change all the email passwords, WordPress password, update the theme that is being used and deletes unused themes.
A host-based intrusion detection system (HIDS), works by collecting details about your computer’s file system and configuration. It then stores this information to reference and validate the current state of the system. If changes are found between the known-good state and the current state, it could be a sign that your security has been compromised. Use these softwares to detect and avoid intrusions:
1) Tiger:
Tiger is a security tool that can be used both as a security audit and as an intrusion detection system. It supports multiple UNIX platforms and it is free and provided under a GPL license. Unlike other tools, Tiger needs only of POSIX tools and is written entirely in shell language.
Functions:
2)Snort:
Snort is a free open source network and works as an intrusion of detection and prevention system. It is a tool that is widely used in TCP/IP traffic sniffers and analyzers. Through protocol analysis, content is searching and matching.
Functions:
When suspicious behavior is detected, Snort sends a real-time alert to syslog, a separate ‘alerts’ file, or in a pop-up window.
3) Suricata:
Suricata is a high-performance Network IDS, IPS and Network Security Monitoring engine.
Functions:
4) Bro:
In a way Bro is both, a signature and an anomaly-based IDS. Its analysis engine will convert traffic captured into a series of events. An event could be a user login to FTP, a connection to a website or practically anything.
Functions:
4) Chkrootkit:
Is a shell script that checks system binaries for rootkit modifications.
Functions:
5) Rkhunter:
Functions:
Information security is the most important practice around the internet network, more specific practices like audits, intrusion detection, hardening in the server, forensics analysis and pen testing because the services that our servers provide are public. We must make them safe, because if we aren’t, then we are vulnerable to be compromised, like in the example above. For this reason, an audit could avoid this uncomfortable situation in which our server has been hacked. A vital part of intrusion detection is gathering information, and detecting and preventing the system attack.
As you can see, in this case of intrusion audit, it is a process that involves many stages and in each one, you need to be careful not to lose any detail (to find information that drives to the suspect infiltration, suspect user, injected bad code, and system infection). The tools that are needed performance and intrusion audit may vary depending on each case. In this case, we started this Forensic analysis reviewing what was happening with the PHP code and why was sending spam. Using Isof was an excellent idea to then filtered the results and started checking for uncommon information. We hope this example helps you understand better what is an Intrusion Audit and how to start a forensic analysis.
Have you ever wondered how businesses easily process enormous volumes of data, derive valuable insights,…
Discover the steps for developing cloud applications, from costs to cloud app deployment
Imagine launching your product with just the core features, getting honest user feedback, and then…
When a tight deadline is non-negotiable, every second counts! Here’s how we developed and launched…
You may have considered hiring a nearshore software development company or services, but you still have doubts…
End-to-end project management goes as far back as you can remember. Every project in history, even…
