self made attack deception defense system

The Department routinely shares technical articles every Wednesday, starting with the official account of millet safety center.

Summary

Towering pillars, once destroyed; water moon mirror image, no intention to come and go.

The name of the self-made deception defense system is jinghuashuiyue. Jinghuayueis the name of blue dye's soul cutting knife (the most powerful soul cutting knife in hallucination) in death. Its ability is to completely control the opponent's five senses.

This name is very suitable for attack deception defense system: to distinguish and separate the attacker's traffic from the normal requests, and seamlessly transfer it to the sandbox that forges the normal business and service, so that the attacker can play in the sandbox environment and record its detailed behavior.

Compared with traditional honeypots, jinghuashuiyue has the following advantages:

Zero false alarm (alarm only for internal or external attacks);
Protect the business system (the attacker will be transferred from the normal service to the sandbox environment in the first time, but the attacker does not know it);
It is convenient to obtain evidence and locate the attacker (the source and detailed attack behavior of the attacker will be recorded in the sandbox).

Technical framework

Jinghuashuiyue consists of four modules:

Agent, the agent deployed in the server, is used to obtain the access log of the user in real time and deliver it to the detection server. If it is a malicious attack, the traffic will be redirected to the sandbox. Currently supported services are: webftp sshrsyncmysqlredismongodb
WEB
FTP
SSH
Rsync
Mysql
Redis
Mongodb
Server, attack detection server, real-time detect the logs passed by the agent and judge whether it is an attacker, and dynamically and real-time maintain an attacker's source IP Policy for the agent;
Manager, a policy management server, provides agents and servers with the functions of policy, attack log statistics, and viewing
The sandbox consists of three parts: the virtual machine or docker installed with FTP, SSH, Rsync, mysql, redis and mongodb, which can send the access logs of these services to the web site log server cloned in the remote rsyslog in real time. The data sent by the sandbox through rsyslog is parsed and stored in the database for the manager to query and analyze
Virtual machines or dockers with FTP, SSH, Rsync, mysql, redis and mongodb installed can send access logs of these services to remote rsyslog in real time
Cloned web site
Log server, which parses the data sent by the sandbox through rsyslog and stores it for the manager to query and analyze

In order to facilitate deployment and high-performance requirements, in addition to the Web attack detection module developed by Lua, the rest of the components of jinghuashuiyue are all developed by go language (the Web attack detection module is developed based on openresty + Lua).

The realization of agent

Agent architecture

The agent supports four modes: honeypot, mirror, attack rebound and firewall.

$ ./client -h
Usage of ./client:
  -c string
        command, such as start or stop policy (default "start")
  -m string
        running mode, honeypot, unreal, back or firewall mode (default "honeypot")

The differences and functions of the four modes are as follows:

Different strategies will be applied at startup: honeypot will directly turn all traffic except white list IP and port to sandbox; jinghuashuiyue will only be judged by server as malicious attacker turning to sandbox without affecting normal request (attack deception defense mode); rebound mode is to return all attacker's traffic intact, similar to that in Jin Yong's novel "eight Tianlong novels" The firewall mode is to block the IP of the identified attacker.
Honeypot will direct all traffic except IP and port of white list to sandbox;
Jinghuashuiyue will only be judged by the server as a malicious attacker turning to sandbox, without affecting normal requests (attack deception defense mode);
The rebound mode is to return the attacker's traffic intact, which is similar to the star shift in Jin Yong's novel eight parts of Tianlong;
The firewall mode is to block the IP of the identified attacker.
When it is started in the mode of reflection, attack rebound and firewall, in addition to the policies obtained from the manager, four goroutines for service monitoring will be started in the agent. These goroutines will send the collected logs of vsftpd, Rsync, MySQL and redis services to the server in real time.

The overall workflow of the agent is as follows:

Judge the startup mode. If the startup parameter is unreal, it will be started in the honeypot mode by default;
Read the configuration parameters from the configuration file conf / app.ini. If it is honeypot mode, obtain the policy from the manager and apply it regularly according to the time interval specified in the configuration file. If it is the mirror moon mode, update the policy every 10 seconds, and start the monitoring goroutine of vsftpd, Rsync, MySQL and redis

For the detailed implementation of honeypot, please refer to the front-end part of the honeypot made by the author in the previous article, https://xsec.io/2016/7/8/how-to-develop-a-honeypot.html. This article only pastes some key codes.

The code structure of the project is as follows:

Log configuration of agent

The agent reports the collected data of non web applications to the server in real time through rsyslog (the Web attack detection will be explained later). The following three steps need to be done before deploying the agent:

Configure the log parameters of applications running in the server to write syslog to syslog, and write syslog to the specified file
Read out the log in the write file, convert it to syslog format, and then transfer it to the local syslog
Configure the rules of rsyslog to forward the log we need to analyze to the back-end server. The forwarding strategy of rsyslog is shown in the following figure:

The logs of SSH and mongodb can be written to syslog, and the forwarding policy of rsyslog can be configured directly.

The operation content of redis does not support writing to syslog. You need to connect to redis first, and then use the monitor command to monitor the execution of the redis instruction. After testing, you cannot monitor the config instruction. Moreover, when monitor is enabled, the performance will be reduced by half. Therefore, redis instruction monitoring service is not suitable for deployment in servers with high performance requirements.

Vsftpd, Rsync and MySQL services need to configure the application's support for logs. The following are the key configuration items:

// VsftpdLog的配置
xferlog_enable=YES
xferlog_file=/var/log/xferlog
xferlog_std_format=YES
dual_log_enable=YES
syslog_enable=YES
log_ftp_protocol=YES

// Rsync的配置
$ cat /etc/rsyncd.conf 
log file = /var/log/rsyncd.log

// mysql的配置
[mysqld]
bind-address = 127.0.0.1
general-log-file = /var/log/mysqld.log                                                                                           
general_log = 1

Just like the method of converting the logs of vsftpd, Rsync and MYSQL to syslog, they are always tail files and send the new logs to syslog. The code is as follows:

// "/var/log/vsftpd.log"
func MonitorVsftpd(logName string) {
    t, err := tail.TailFile(logName, tail.Config{Follow: true})
    if err == nil {
        for line := range t.Lines {
            l3, err := syslog.New(syslog.LOG_ERR, "vsftpd-server")
            defer l3.Close()
            if err != nil {
                log.Fatal("error")
            }
            l3.Info(line.Text)
            fmt.Println(line.Text)
        }
    }
}

The following is the code for converting the result of redis's monitor instruction to syslog:

func MonitorRedis(host, port, password string) {
    flag.Parse()
    conn, err := net.Dial("tcp", host+":"+port)
    if err == nil {
        conn.Write([]byte(fmt.Sprintf("auth %v\r\n", password)))
        conn.Write([]byte("monitor \r\n"))
        defer conn.Close()

        ch := make(chan []byte)
        eCh := make(chan error)

        l3, err := syslog.New(syslog.LOG_ERR, "redis-server")
        defer l3.Close()
        if err != nil {
            log.Fatal("error")
        }

        go func(ch chan []byte, eCh chan error) {
            for {
                // try to read the data
                data := make([]byte, 512)
                _, err := conn.Read(data)
                if err != nil {
                    eCh <- err
                    return
                }
                ch <- data
            }
        }(ch, eCh)

        ticker := time.Tick(time.Second)
        for {
            select {
            case data := <-ch:
                n := bytes.IndexByte(data, 0)
                s := string(data[:n])
                d := strings.Replace(s, "\r\n", "", -1)
                l3.Warning(d)
            case err := <-eCh:
                fmt.Println(err)
                break
            case <-ticker:
            }
        }
    }
}

Coding implementation of agent

The processing code of startup mode is as follows:


// start honeypot
func StartHoneypot(mode string) {
    for {
        p, err := GetPolicy()
        log.Println(p, err)
        if err == nil {
            run(p, mode)
        }
        if strings.ToLower(mode) == "honeypot" {
            time.Sleep(time.Second * time.Duration(setting.Interval))
        } else {
            time.Sleep(time.Second * 10)
        }
    }
}

// start agent
func Start(mode string) {

    if strings.ToLower(mode) != "honeypot" {
        go util.MonitorVsftpd(setting.VsftpdLog)
        go util.MonitorRsync(setting.RsyncLog)
        go util.MonitorMysql(setting.MysqlLog)
        go util.MonitorRedis(setting.RedisHost, setting.RedisPort, setting.RedisPass)

    }

    StartHoneypot(mode)
}

Different iptables policies will be applied in different modes. Some codes are as follows:


// set iptables
func SetIptables(policy Policy, mode string) {
    if strings.ToLower(mode) == "honeypot" {
        whiteIpPolicy := policy.WhiteIp
        // set white policy
        for _, whiteIp := range whiteIpPolicy {
            fmt.Println("/sbin/iptables", "-t", "filter", "-A", "WHITELIST", "-i", setting.Interface, "-s", whiteIp, "-j", "DROP")
            exec.Command("/sbin/iptables", "-t", "filter", "-A", "WHITELIST", "-i", setting.Interface, "-s", whiteIp, "-j", "DROP").Output()
        }

        fmt.Println("/sbin/iptables", "-t", "nat", "-A", "HONEYPOT", "-i", setting.Interface, "-p", "tcp", "-m", "multiport", "!", "--dports", strings.Join(policy.WhitePort, ","), "-j", "DNAT", "--to-destination", policy.Backend)
        ret, err := exec.Command("/sbin/iptables", "-t", "nat", "-A", "HONEYPOT", "-i", setting.Interface, "-p", "tcp", "-m", "multiport", "!", "--dports", strings.Join(policy.WhitePort, ","), "-j", "DNAT", "--to-destination", policy.Backend).Output()
        fmt.Println(ret, err)
    } else if strings.ToLower(mode) == "unreal" {
        blackIps := policy.BlackIp
        for _, blackIp := range blackIps {
            if strings.TrimSpace(blackIp) != "127.0.0.1" {
                fmt.Println("/sbin/iptables", "-t", "nat", "-A", "HONEYPOT", "-i", setting.Interface, "-p", "tcp", "-m", "multiport", "!", "--dports", strings.Join(policy.WhitePort, ","), "-s", blackIp, "-j", "DNAT", "--to-destination", policy.Backend)
                ret, err := exec.Command("/sbin/iptables", "-t", "nat", "-A", "HONEYPOT", "-i", setting.Interface, "-p", "tcp", "-m", "multiport", "!", "--dports", strings.Join(policy.WhitePort, ","), "-s", blackIp, "-j", "DNAT", "--to-destination", policy.Backend).Output()
                fmt.Println(ret, err)
            }

        }
    } else if strings.ToLower(mode) == "back" {
        blackIps := policy.BlackIp
        for _, blackIp := range blackIps {
            if strings.TrimSpace(blackIp) != "127.0.0.1" {
                fmt.Println("/sbin/iptables", "-t", "nat", "-A", "HONEYPOT", "-i", setting.Interface, "-p", "tcp", "-m", "multiport", "!", "--dports", strings.Join(policy.WhitePort, ","), "-s", blackIp, "-j", "DNAT", "--to-destination", blackIp)
                ret, err := exec.Command("/sbin/iptables", "-t", "nat", "-A", "HONEYPOT", "-i", setting.Interface, "-p", "tcp", "-m", "multiport", "!", "--dports", strings.Join(policy.WhitePort, ","), "-s", blackIp, "-j", "DNAT", "--to-destination", blackIp).Output()
                fmt.Println(ret, err)
            }

        }
    } else if strings.ToLower(mode) == "firewall" {
        blackIps := policy.BlackIp
        for _, blackIp := range blackIps {
            fmt.Println("/sbin/iptables", "-t", "filter", "-A", "WHITELIST", "-i", setting.Interface, "-s", blackIp, "-j", "DROP")
            exec.Command("/sbin/iptables", "-t", "filter", "-A", "WHITELIST", "-i", setting.Interface, "-s", blackIp, "-j", "DROP").Output()
        }
    }
    exec.Command("/sbin/iptables", "-t", "nat", "-A", "POSTROUTING", "-o", setting.Interface, "-j", "MASQUERADE").Output()
}

Attack detection of Web Services

The attack detection of web server is realized by using the backward proxy WAF. For the self-made WAF, please refer to my previous article "how to build a free cloud WAF for small and medium-sized enterprises", https://xsec.io/2016/8/23/how-to-development-a-free-cloud-waf.html

There are three processing modes after WAF detects the attack: 1. Jump to the specified url1. Output the customized HTML content. 1. The pattern of mirror flowers and moon. The attacker's request is not reversed to the normal backend, but to the cloned web business backend. The cloned web site can be forged to be the same as the real site, and the detailed log, such as access log, debug log and ORM, is opened Log, etc., connected to the desensitized test database.

After WAF detects the attack, the processing code for turning on the mode of "jinghuashuiyue" is as follows:

-- WAF response
function _M.waf_output()
    if config.config_waf_model == "redirect" then
        ngx.redirect(config.config_waf_redirect_url, 301)
    elseif config.config_waf_model == "jinghuashuiyue" then
        local bad_guy_ip = _M.get_client_ip()
        _M.set_bad_guys(bad_guy_ip, config.config_expire_time)
    else
        ngx.header.content_type = "text/html"
        ngx.status = ngx.HTTP_FORBIDDEN
        ngx.say(string.format(config.config_output_html, _M.get_client_ip()))
        ngx.exit(ngx.status)
    end
end

-- set bad guys ip to ngx.shared dict
function _M.set_bad_guys(bad_guy_ip, expire_time)
    local badGuys = ngx.shared.badGuys
    local req, _ = badGuys:get(bad_guy_ip)
    if req then
        badGuys:incr(bad_guy_ip, 1)
    else
        badGuys:set(bad_guy_ip, 1, expire_time)
    end
end

When using WAF admin to add back-end sites to be protected, two back-end addresses need to be added:

Normal back-end address
Back end address of clone business

As shown in the following figure (please do not care about the UI made by the security backend Engineer):

After the WAF management background generates the configuration file of a new site, it will call the waf.start 〝 jingshuishuiyu() function in each request. Waf.start 〝 jingshuishuiyu() will judge whether the user's type is good or bad. If it is bad, it will go directly to the back end of the clone.

Here is the code for WAF. Start 〝 jingshuishuiyu():

function _M.start_jingshuishuiyue()
    local host = util.get_server_host()
    ngx.var.target = "proxy_"
    if host and _M.bad_guy_check() then
        ngx.var.target = "unreal_"
    end
end

The following is the template file content when the WAF management end enters a new website and generates a configuration file for it:

upstream proxy_{{.site.SiteName}} { {{range .site.BackendAddr}}
        server {{.}} max_fails=3  fail_timeout=20s;
        {{end}} }

upstream unreal_{{.site.SiteName}} { {{range .site.UnrealAddr}}
        server {{.}} max_fails=3  fail_timeout=20s;
        {{end}} }

server  {
        listen       {{.site.Port}};
        ssl          {{.site.Ssl}};
        server_name  {{.site.SiteName}};
        client_max_body_size 100m;
        charset utf-8;
        access_log      /var/log/nginx/{{.site.SiteName}}-access.log;
        error_log       /var/log/nginx/{{.site.SiteName}}-debug.log {{.site.DebugLevel}};
        
        location ~* ^/ {
            access_by_lua 'waf.start_jingshuishuiyue()';
            proxy_pass_header Server;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Scheme $scheme;
            proxy_pass $scheme://${target}{{.site.SiteName}};
            }

            error_page  404              /index.html;
            error_page   500 502 503 504  /index.html;
        }

Implementation of server

Architecture of server

The main function of server is to start rsyslog service on port 514 of TCP and UDP, receive rsyslog data from each agent, and judge the start type of agent and whether each service is attacked.

If the data received is from honeypot mode, an alarm will be given according to the rules;
If the received data is transferred from the "jinghuayuemo" mode, it will detect whether it is an attack, whether it has been attacked several times, whether it is necessary to launch "jinghuayuemo" against the attacker, if it needs to launch, it will modify the strategy in the manager in real time, and the agent will get the new strategy in a few seconds, and transfer the attacker to the sandbox.

The server side supports horizontal expansion. As long as it can connect to the mail server, mongodb and redis on the back end are OK. For the implementation of honeypot, please refer to the previous article of the author, which only writes the attack detection part.

The project structure of the server is as shown in the figure:

As shown in the figure above, if it is not the honeypot log, the application log type will be determined and the corresponding threat detection module will be called. The detection methods are all based on the characteristics of attacks in regular matching, such as breaking password, Rsync column directory, uploading and downloading files, etc. Then record the attacker's IP address and the number of attacks to manger's redis.

The following is an example of vsftp password brute force cracking detection:

package check

import (
    "fmt"
    "regexp"
    "strings"
    "time"

    "gopkg.in/redis.v3"

    "xsec-honeypot/server/setting"
)

func CheckvsFTP(sysLog map[string]interface{}, redisClient *redis.Client) {
    content, _ := sysLog["content"].(string)
    client, _ := sysLog["client"].(string)
    hostname, _ := sysLog["hostname"].(string)
    re, _ := regexp.Compile(`.* \[pid .*] \[(.+?)\] FTP response: Client "(.+?)", "530 .*"`)
    ret := re.FindStringSubmatch(content)
    if len(ret) > 0 {
        src := ret[2]
        slice_dst := strings.Split(client, ":")
        dst := client
        if len(slice_dst) > 0 {
            dst = slice_dst[0]
        }
        fmt.Printf("Src:%v, User:%v, Client:%v, NodeName:%v\n", src, ret[1], client, hostname)
        k := fmt.Sprintf("app-%v-%v-%v", src, dst, dst)
        bRet, _ := redisClient.Exists(k).Result()
        if bRet {
            redisClient.HIncrBy(k, "times", 1)
        } else {
            redisClient.HSet(k, "times", "1")
            redisClient.Expire(k, time.Duration(setting.AlarmRecoveryTime)*time.Minute)
        }
    }
}

Implementation of manager

Manager's architecture

Manager is the management background of the whole system, supporting the following functions:

user management
Agent asset management
Honeypot strategy management
Display of attack log and alarm log
Provide policy pull service for agent and server

The code structure of the manager is as follows:

Code implementation of manager

When obtaining policies from manager, authentication is needed to prevent attackers from inadvertently acquiring all policies of honeypot or honeymoon, and then targeting to avoid attacking other systems.

The API interface to get the policy only supports the post method:

m.Group("/api/", func() {
        m.Post("/policy/", routers.PolicyJSON)
    })

The key in the agent and server configuration file must be the same as that in the manager. Otherwise, the policy cannot be obtained due to authentication failure, as shown in the following code:

func PolicyJSON(ctx *macaron.Context) {
    ctx.Req.ParseForm()
    timestamp := ctx.Req.Form.Get("timestamp")
    secureKey := ctx.Req.Form.Get("secureKey")
    log.Println(timestamp, secureKey)
    mySecureKey := util.MakeMd5(fmt.Sprintf("%v%v", timestamp, setting.AppKey))
    ret := models.APIData{}
    if mySecureKey == secureKey {
        policy, _ := models.ListPolicy()
        ret = models.APIData{Code: 0, Data: map[string]interface{}{"message": "Get policy successful",
            "value": policy}}
    } else {
        ret = models.APIData{Code: 1, Data: map[string]interface{}{"message": "Api signature verification failed",
            "value": make([]string, 0)}}

    }

    ctx.JSON(200, ret)

}

Every time the policy interface is acquired, the manager will find out the IP addresses of all attackers who satisfy the policy from redis, and then return them to the agent, which can launch a "mirror moon" mode against these attackers' IP addresses.


func ListPolicy() (policy []Policy, err error) {
    err = collPolicy.Find(nil).All(&policy)
    blackIp, err1 := ListRealTimePolicy()
    fmt.Println(blackIp, err1)
    if err == nil && err1 == nil {
        if len(policy) > 0 {
            policy[0].BlackIp = blackIp
        }
    }

    return policy, err
}

func ListRealTimePolicy() (blackIp []string, err error) {
    ret, err := RedisClient.Keys("app-*-*-*").Result()
    // fmt.Printf("%v,%v", ret, ret)
    for _, item := range ret {
        v := strings.Split(item, "-")
        fmt.Println(v, len(v))
        if len(v) > 0 {
            attacker := Attacker{}
            attacker.Src = v[1]
            attacker.Dst = v[2]
            attacker.Client = v[3]
            strTimes, _ := RedisClient.HGet(item, "times").Result()
            times, _ := strconv.Atoi(strTimes)
            if times > AuthFailuresTimes {
                blackIp = append(blackIp, attacker.Src)
            }
        }
    }
    return blackIp, err
}

In the above code, listrealtimepolicy() means to filter the attacker's IP from redis, and the listpolicy() method returns the honeypot policy together.

ListRealTimePolicy() ListPolicy()

Implementation of sandbox

Sandbox can be installed in virtual machine or encapsulated in docker. The data generated by sandbox can be recorded in the following 2 ways:

Use tcpdump to grab the package of the corresponding protocol in the host, then use dpkt to parse the pcap file, and then store it in the DB of the manager
Deploy the log analysis script similar to agent in sandbox, send the result to rsyslog, and then rsyslog will forward it to our server, which will be stored in dB of manager after receiving it

IMCAFS