Reading: 2661
In the round table forum of techworld 2017, we discussed the application of machine learning in network security. What are the specific practices of machine learning in security attack and defense? How to use machine learning to backtrack and forewarn security events? Do you understand machine learning, full flow threat analysis, attack chain and temporal association? After reading this article, you are not far from the old driver!
Not good at writing? Click the video link at the end of the article to feel the mysterious magic of machine learning at close range!
Current situation and development direction of network security
- The purpose of network attack has changed from hacker's dazzling Technology (virus, webpage tampering, DDoS) in the past to the organized and premeditated transnational network war (Ukrainian power plant, target, swift event) for profit.
- In the face of new network attacks, the stacking of devices has been unable to compete with it. More intelligent means are needed to deal with the complex attack methods of hackers.
- Machine learning is everywhere: movie box office prediction, general election box office analysis, unmanned car, go, etc.
The practice of machine learning in flow analysis
Detection of advanced threat
Gartner put forward a word this year about a new network traffic analysis solution.
Network traffic analysis solution, through monitoring network traffic, connections and objects, to find malicious signs of behavior. Enterprises looking for network-based methods to identify advanced attacks bypassing perimeter security should consider using traffic analysis techniques to help identify, manage, and classify these events.
——Top technology of information security in 2017, Gartner
The method of this analysis is to find out malicious behaviors through network traffic connections and objects. Although it's also through traffic, it's totally different from the way we used to grab traffic and match rules. This way to identify the high-level behavior of bypassing the perimeter security and some of the detection we call whether it's firewall or IDS perimeter protection.
Green Alliance Technology made a project at the end of last year. This project is also in the process of productization this year, which is to detect and discover malicious behaviors through full flow threat analysis. Here are several sources of data, such as rules, including current intelligence, and some sandbox detection. We integrate these data with the traffic between us to backtrack and mine the massive network traffic in a long time span.
Full flow threat analysis scheme
The scheme of full flow threat analysis aims at the difficult problems found in APT analysis at present. Using the related technology of big data processing to provide the ability to mine and analyze the massive network traffic data in a long time span; combining with machine learning, rule detection, Sandbox detection, intelligence analysis and other threat detection means, according to the attack chain model to correlate the threat behavior, a set of relatively perfect apt detection system is constructed to make the apt attack in a large time span Hit detection is possible.
Flow threat analysis process
The analysis process of traffic threat is as follows: firstly, monitor the network, and input all traffic into the rule engine after monitoring. One is the rule, and the other is the sandbox. These two methods are widely used. There is a very important link in the middle, which is the original data of traffic, which is equivalent to intercepting the original data of traffic, including some packet headers, etc. there are some contents about traffic occupation in the middle. After removing the packet headers and the original data, save them, and then combine the behavior of attack chain. Through our big data analysis platform, do some corresponding operations to obtain the whole The backtracking or speculation of the attack.
In this scheme, we use rule engine and sandbox to analyze traffic and detect unknown threats, and then aggregate traffic to our big data analysis platform. In addition, this platform can also access Threat Intelligence.
By integrating these methods into our big data analysis platform, we can get this kind of abnormal analysis and retrieve the corresponding access content. We can trace back through the original data of this traffic, such as what happened before and what impact it has on my current host. In this way, it is a good supplement to our previous rule-based security protection.
Botnet discovery based on DNS data
Now there is a very popular botnet - domain flux botnet. According to a seed, the new domain name is generated automatically by DGA algorithm. The life cycle of the domain name is usually very short. When an attacker wants to launch an attack, register the domain name for the C & C host in advance and wait for the chicken to connect. However, if we adopt the previous rules, it is difficult to form some rules to monitor these contents quickly. But if we promote the way of machine learning, we will summarize the characteristics of this botnet after capturing the traffic data. For example, the domain name is wonderful, bad pronunciation, unreasonable length and so on. After we have these performance features, we will form the important feature content in machine learning just mentioned. According to these features, machine learning can match the corresponding algorithm to mine the botnet.
Machine learning discovers worm propagation behavior
In this way, we also found the spreading behavior of worms. Now the general behavior of worms is very simple, except for random port scanning. After scanning, it is found that you have opened the corresponding port, and then you will implant some back doors and so on. This is the way of worms.
But for this kind of worm, we can only apply some special rules, some rules for a specific kind of worm. Through machine learning, we can first form some characteristics, such as worm propagation, such as radioactive propagation, specific ports, and the same number of packets sent. Finally, some iterative algorithms are used to find the propagation of the botnet.
Machine learning discovers covert channels
Some often use DNS for covert channel propagation. Because you know that we generally do not control DNS data. So you can spread a lot of information through DNS data. Therefore, DNS is a good way for attackers. The recursive resolution system of DNS ensures the stability of the channel and the reliability of the transmission, and the firewall basically does not intercept DNS packets.
After we do the corresponding analysis, we find the randomly registered domain name, including some data transmitted by the random domain name and so on. These information also form some corresponding detection characteristics. Including the length of domain name and TTL. According to these detection characteristics, this kind of hidden lane can be detected well. In this way, some features based on traffic capture can not be realized at the beginning. The machine learning method can be easily implemented.
The practice of machine learning in log analysis
Traditional log analysis
Traditional logs are basically single point. For example, IDS logs, missed scan device logs, firewall logs and some logs formed in the data transmission process of other network devices. If there is a problem, it can only be checked one by one, and the efficiency is extremely low.
Security analysis based on attack chain
Network attacks occur in stages, and can be interrupted by establishing an effective defense mechanism in each stage.
——Lockheed Martin
Just now we have seen various methods of analysis, or points of some security incidents. No matter in the investigation stage, tool preparation stage, attack utilization stage, back door installation stage, etc., what we see is an event point. Through this attack chain analysis method, some events can be classified in the following seven steps. Through this classification, some characteristics can be formed.
Attack chain provides an effective theoretical support for the analysis of hacker's attack behavior.
Event understanding model
The attack chain method may be able to generate a more meaningful way for us to analyze security events. But there is a very important technical point in this, which is the understanding of events. Now all kinds of manufacturers, whether green alliance or other friendly businesses, will have incidents. After the event broke out, it would be very difficult to analyze the log. The difficulty lies in how to understand these events. Only by understanding these events can we proceed to the next step of reasoning just mentioned.
As for the understanding of events, we have made an event understanding model here, which is divided into three steps.
First, merge based on aging time window. In a fixed time cycle, the generated security events are merged based on time, which is called aging of a time window.
Secondly, event reliability analysis based on intelligence. Based on the information, these merging events are analyzed to produce the credibility of the events.
Thirdly, event understanding based on event knowledge base.
Temporal correlation analysis
With the understanding of events, the next step is crucial, which is about temporal correlation. So called temporal association, let's give a very simple example. At 6 a.m., Xiao Ming is in the playground. Generally speaking, at 6 a.m., he is either running or exercising. Let's talk about the complexity. For example, at noon or 3 p.m., Xiao Ming is in the office. What is it? In the office, it must be work. For example, at 3 o'clock on weekends, what are you doing at home? That may be watching TV at home, or resting and so on. At this time, there is a process of reasoning. We can infer several behaviors according to time, weather, hobbies and so on, which is timing analysis.
Attack timing analysis -- data characterization
In this time series analysis, how do we characterize the security data? For example, I have some attack features, some logs and some events. Then I use some scenarios, such as time constraints. For example, event a can only occur after event B, and generate some security scenarios to describe the process of a topology attack, which is the security sequence.
Machine learning practice -- induction of attack process
Through the way of security sequence, we can form the classification of attack process. The so-called classification of attack process is based on the seven step method. First is the investigation action, for example, before the attack, we need to do some scanning and detection. Then, if a vulnerability is found, POC tools are needed, or some malicious files are modified. Then we need to plant some Trojans, and then exploit the loopholes and so on. It is equivalent to forming a series of methods based on attack sequence. Through this time sequence method, we can give security warning to security events, form some evasive schemes, or draw an attack behavior to effective attack events.
Attack process analysis
Specific event analysis methods: for example, one day I received a pile of attack logs, including the events of the day before yesterday, yesterday and today, forming some alarm events. Then through seven steps of attack chain, each behavior is matched. For example, a port scan was done on May 3, which is part of the investigation. It was cracked on May 6, drawing a sequence of attacks.
Analysis of actual safety scenarios
With timing, an attack scenario can be formed. When did the hacker invade, and then what did he do, such as scanning, brute force cracking, lifting rights, embedding Trojans and so on. After the installation of Trojans, there are some malicious attacks, whether botnet or DDoS attacks and so on.
Machine learning practice: early warning and backtracking of attack and defense scenarios
Then, based on machine learning, we can give an early warning of the possible future events for the events at a certain time point. Another important practice is backtracking. Find out what impact some events have on me, so as to make a better early warning of future events.
Future: threat warning and intelligent "anti" attack chain
In the future, it can realize reasoning based on attack scenarios and predict future security events. In addition, intelligent disassembly is carried out based on the event disposal library, corresponding to different equipment, and corresponding strategies are issued for all-round protection disposal.