The tool described in this article is suitable for "black box testing" in a home environment. It can help you record everything that happens in the network. You can use it to detect network threats, or provide data to relevant experts for network forensic analysis.
If you need a solution in an enterprise environment, you can refer to [this article] of security onion.
Shopping list
1. Raspberry Pie 3 (shell + power + circuit board)
2. SanDisk class 10 microSD card 64GB (80mb / s)
3. Debian OS-Linux RaspbianLite
4. I use d-link1100-08p for network Gigabit switches or other devices that support port mirroring
5. Critical Stack API（Threat Intel/ IOCs）
6. Mailgun account or similar email service supporting alert / notification
Total value: about £ 75
Overview map
Introduction to key technologies
What is port mirroring?
Copying traffic data from one port to another (passive) will increase the operating load of the switch.
What is bro?
An IDS protocol analysis tool, you can use it as Wireshark's protocol analyzer, but it has no GUI interface and is faster.
What is netsniff?
The daemons for packet capture use AF packet to speed up packet capture.
What is Loki?
Yara based file scanning daemons. Some are similar to anti-virus products based on signature detection, but you can make your own detection rules.
What is critical stack?
A threat intelligence platform that you can link to through the API on raspberry pie.
What is team Cymru MHR?
A malware hash database where you can use the information to match the detected malware hash.
Start working
1. Brush the system into raspberry pie
2. Assign an IP to raspberry pie
3. Run bash script
4. Get it done
1、 Swipe raspbian into the microSD card
I use MacBook, so different platforms may have different methods. Users of other platforms can refer to [this tutorial] for more information.
Insert microSD card:
diskutil list
Disk number found:
diskutil unmountDisk /dev/disk<disk#from diskutil>
Brush the raspbian image into the microSD card 'disk':
sudo dd bs=1m if=image.imgof=/dev/rdisk<disk# from diskutil>
diskutil unmountDisk /dev/disk<disk#from diskutil>
2、 Configure network
Log in using the default configuration. User name: Pi, password: raspberry.
Set the IP of WLAN 0 (WiFi) for trusted management access:
sudo nano/etc/wpa_supplicant/wpa_supplicant.conf
network={
ssid="The_ESSID_from_earlier"
psk="Your_wifi_password"
}
sudo ifdown wlan0
sudo ifup wlan0
ifconfig wlan0
After you get a DHCP IP, you can use SSH to access this node. Next, leave eth0 as the mirror interface, which does not need to set the IP address.
sudo apt-get update && sudo apt-get-y install vim
sudo vim /etc/network/interfaces
Add the following code:
iface eth0 inet static
static ip_address=0.0.0.0
sudo ifconfig eth0 down && sudoifconfig eth0 up
Three, deployment
Download and run the bash script, which has been successfully tested on raspbian.
-The core components of the installer
-Configure network options (disable NIC offloading)
-Create services for each program
-Creating mail alerts using mailgun / ssmtp
-Configure cron tasks
[email protected]:~# sudo su -
[email protected]:~# apt-get install -y git
[email protected]:~# cd ~
[email protected]:~# git clonehttps://github.com/sneakymonk3y/foxhound-nsm.git
[email protected]:~# chmod +xfoxhound-nsm/foxhound.sh
[email protected]:~# ./foxhound-nsm/foxhound.sh
[email protected]:~ $ sudo tcpreplay -t -K -q--loop=10 --intf1=eth0 /opt/foxhound-1476541443.pcap
Actual: 1048570 packets (1050923190 bytes)sent in 87.62 seconds. Rated:11994102.0 bps, 91.51 Mbps, 11967.25 pps
Here are the broctl netstats data before and after sending:
[email protected]:/etc/network# broctl netstats
bro: 1476547903.768150 recvd=1951368 dropped=5408 link=1956776
[email protected]:/etc/network# broctl netstats
bro: 1476548144.248161 recvd=3012168 dropped=14608 link=3026776
For a home or lab environment, its performance is relatively good (only 10000 packets are discarded from a million packets). My bandwidth speed is 40mbps, which is enough for IDS system. If you still need to improve performance, it is recommended to consider replacing raspberry pie.
If you want more NIC perf tests, you can run the following commands on the server:
[email protected]:~$ sudo apt install iperf3
[email protected]:~$ iperf3 -s
-----------------------------------------------------------
Serverlistening on 5201
Then run on raspberry Pie:
[email protected]:~# apt install iperf3
[email protected]:~# iperf3 -c 10.0.0.7 -i 1 -t20
Connecting to host 10.0.0.7, port 5201
[ 4]local 10.0.0.180 port 38562 connected to 10.0.0.7 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 8.86 MBytes 74.3 Mbits/sec 0 89.1 KBytes
...
...
...
[ 4] 19.00-20.00 sec 9.26 MBytes 77.7 Mbits/sec 0 1.23 MBytes
- - - - - - - - - - - - - - - - - - - - - -- - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-20.00 sec 185 MBytes 77.5 Mbits/sec 139 sender
[ 4] 0.00-20.00 sec 184 MBytes 77.1 Mbits/sec receiver
iperf Done.
I assume that the bottleneck here is the microSD card, whose speed is about 80mb / s.
Bro Foundation
-The default storage path for all bro logs is / NSM / bro / logs/
-The default script path is / usr / local / bro / share / bro / site / bro scripts/
The log directory structure is roughly as follows:
[email protected]:/nsm/bro/logs/current $ ls-lash
total 6.9M
4.0K drwxr-xr-x 3 root root 4.0K Oct 15 16:11 .
4.0K drwxr-xr-x 5 root staff 4.0K Oct 1516:50 ..
4.0K -rw-r--r-- 1 root root 349 Oct 15 16:51 app_stats.log
4.0K -rw-r--r-- 1 root root 121 Oct 15 15:51 .cmdline
16K-rw-r--r-- 1 root root 14K Oct 15 16:30communication.log
2.9M -rw-r--r-- 1 root root 2.9M Oct 15 16:52 conn.log
16K-rw-r--r-- 1 root root 14K Oct 15 16:52dhcp.log
384K -rw-r--r-- 1 root root 379K Oct 15 16:52 dns.log
4.0K -rw-r--r-- 1 root root 345 Oct 15 15:51 .env_vars
1.2M -rw-r--r-- 1 root root 1.2M Oct 15 16:52 files.log
1.6M -rw-r--r-- 1 root root 1.6M Oct 15 16:52 http.log
4.0K -rw-r--r-- 1 root root 291 Oct 15 16:44 known_hosts.log
4.0K -rw-r--r-- 1 root root 327 Oct 15 16:34 known_services.log
12K-rw-r--r-- 1 root root 11K Oct 15 16:50notice.log
4.0K -rw-r--r-- 1 root root 5 Oct 15 15:51 .pid
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.communication
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.conn
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:01 .rotated.conn-summary
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.dhcp
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.dns
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.files
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.http
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.known_hosts
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.known_services
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.loaded_scripts
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.notice
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.packet_filter
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.software
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.ssl
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.weird
4.0K -rw-r--r-- 1 root root 18 Oct 15 16:00 .rotated.x509
4.0K -rw-r--r-- 1 root root 3.0K Oct 15 16:51 software.log
320K -rw-r--r-- 1 root root 314K Oct 15 16:52 ssl.log
4.0K -rw-r--r-- 1 root root 58 Oct 15 15:51 .startup
4.0K drwx------ 3 root root 4.0K Oct 15 15:51 .state
4.0K -rwx------ 1 root root 18 Oct 15 15:51 .status
4.0K -rw-r--r-- 1 root root 46Oct 15 15:51 stderr.log
4.0K -rw-r--r-- 1 root root 188 Oct 15 15:51 stdout.log
24K-rw-r--r-- 1 root root 17K Oct 15 16:52weird.log
416K -rw-r--r-- 1 root root 412K Oct 15 16:52 x509.log
You can use the head command to find the text field name:
[email protected]:$ head dns.log
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path dns
#open 2016-10-15-16-00-01
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected
#types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool
Use bro cut to resolve logs:
[email protected]:$ cat dns.log | bro-cut -D tsid.orig_h id.orig_p id.resp_h id.resp_p proto query answers TTLs
Basic report:
[email protected]:$ bro-cut query < dns.log |sort | uniq -c | sort -rn | head -n 10
tail -f dns.log | awk '{print $3, $7, $9}'
Note that only conn.log can enable maxmind geoip queries.
[email protected]:$ bro-cut resp_cc < conn.log| sort | uniq -c | sort -rn | head -n 10
755 US
524 RO
123 GB
49 NL
28 EU
25 IE
10 DE
7 ES
6 CA
I haven't made a dashboard yet. If you don't want to do it yourself, you can directly use VPS elk instance to store all logs in the cloud, or you can use elk / Splunk / graylog to store logs locally.
Foundation maintenance
To start / stop netsniff ng:
[email protected]:~ $ sudo service netsniff-ngstop
[email protected]:~ $ sudo service netsniff-ngstart
[email protected]:~ $ sudo service netsniff-ngstatus
On / off bro:
[email protected]:~ $ sudo -i broctl stop
[email protected]:~ $ sudo -i broctl start
[email protected]:~ $ sudo -i broctl netstats
[email protected]:~ $ sudo -i broctl status
Run Loki manually:
[email protected]:~ $ python /nsm/Loki/loki.py--noprocscan --dontwait --onlyrelevant -p /nsm/bro/extracted -l /nsm/Loki/log
r[email protected]:~ $ sudo -u critical-stackcritical-stack-intel list
Reference material
1. Deploy network security monitoring: http://www.appliednsm.com/
2. Network security monitoring practice: https://www.nostarch.com/nsm
3. Laika boss object scanning system: https://github.com/lmco/laikaoss
4. PassiveDNS：https://github.com/gamelinux/passivedns
5. D3js (graphical tool): https://d3js.org/
6. Graylog：https://www.graylog.org/
*Reference source: sneakymonkey, compiled by FB editor alpha? H4ck, reprinted from freebuf.com