On August 5-7, 2016, at Defcon CTF, a top information security competition held in Las Vegas, the United States, a machine CTF team named mayhem and 14 other top human CTF teams staged the first man-machine hacker battle in the field of information security, and once exceeded two human teams. The significance of this event in the field of artificial intelligence and information security is no less than Google alphago's victory over Li Shishi this year. It is one of the landmark events that machine intelligence began to deeply affect information security. The origin of mayhem begins with the CGC (cyber grand challenge) held by DARPA (Defense Advanced Research Projects Agency).

1、 Competition background

DARPA has always had the tradition of accelerating the application of science and technology by holding challenge competitions. CGC's idea comes from the super challenge successfully held in the past. For example, the goal of the first super challenge competition in 2004 is to promote driverless technology. From the first time that no team has completed the competition, the competitors have achieved automatic desert crossing (2005) and automatic crossing of complex urban road sections (2007); another well-known competition held by DARPA is the DRC (DARPA Robotics challenge) (2013).

CGC is a global network security challenge launched by DARPA in 2013. It aims to promote the development of automatic network defense technology, that is, real-time identification of system defects and loopholes, automatic completion of patching and system defense, and finally the realization of fully automatic network security attack and defense system. The team is made up of computers without any intervention. Therefore, CGC is a CTF competition between machines, and the goal is to promote a fully automatic network security attack and defense system.

2、 Introduction to the competition

The schedule of CGC is divided into two stages: the preliminary competition (CQE) and the final competition (CFE).

1. Competition time

Preliminaries: June 3, 2015

Final: August 4, 2016

Figure 1. Timing of CGC competition

2. Teams

The teams in the preliminary stage are divided into funded track and open track. Fund track is a team that submits project application to DARPA in advance and gets $750000 in funding. There are 7 teams, including university research team and enterprise team. Open track is a team that is open to the world and organized by civil liberties. There are nearly 100 teams, including at least 18 teams from Europe, Asia and other non North American regions. Open track has a traditional CTF team (disekt, shellfish, etc.) and a competition team funded by well-known security enterprises.

Before the competition, each team needs to develop a set of fully automatic network reasoning system (CRS), which can analyze and find the loopholes in the Linux binary program automatically, generate the verification code that can trigger the loopholes automatically, and repair the loopholes in the program automatically.

3. Competition questions and regulations

The contest question is developed by the organizer (DARPA) and designed for the difficulties of automatic vulnerability mining. There are two rehearsals of system automation before the CGC official preliminaries, which are used to debug the system automation level and connect with the host system.

Figure 2. CGC competition test questions

(1) Preliminaries

The title of the preliminary contest is 131 Linux binary programs (no source code) with known vulnerabilities. All programs have memory processing vulnerabilities. The types of vulnerabilities cover 53 different types of CWE (list of common defects). The preliminary contest is a process of online automatic analysis and asynchronous attack and defense verification. Within the specified 24 hours, the automatic analysis system of each team needs to automatically download the application program from the sponsor under the condition of no intervention, find the vulnerability by the analysis program, submit the attack input that can trigger the vulnerability, and submit the repaired reinforcement program. In this preliminary competition, 590 loopholes reserved by developers were successfully repaired by participating teams.

The organizer makes a cross attack and defense comparison between all the submitted attack inputs and the reinforced programs among the teams, and determines the ranking of the preliminary competition through the comprehensive evaluation of the success rate of attack and defense and the performance of the reinforced programs.

(2) Final

The challenge content of the final match is basically the same as the preliminary match, but the introduction of online real-time confrontation is an online real-time attack and defense process. After the final, the organizer will release new binary applications from time to time. Each team's system needs to analyze and repair the applications in real time, deploy the repaired programs, generate attack programs, and submit them to the organizer. Different from the preliminary competition, the final system has increased the network defense ability, CRS system can automatically generate IDS rules, and can also choose to attack the target. In addition, the attack input in the final stage is no longer POC, but actual exploit, that is, it can be directly used to obtain program control permission or disclose information. Each team's CRS system deploys defense measures at the software level and network level to protect its own programs from attack.

Through the comprehensive calculation of attack score, defense score, performance loss and function loss introduced by defense measures, the winning team is finally judged.

Figure 3. CGC final rules

After the two rounds of competition, the participating teams need to submit technical paper, which mainly describes the CRS used by the participating teams; after the preliminary competition, the organizer will also visit the CRS system of each participating team on the spot. DARPA takes the above content as an important reference for ranking.

4. Bonus distribution

Through the preliminary teams, the fund track team will no longer receive bonus, and the open track team will receive $750000.

In the final stage, the total bonus is 3.75 million US dollars, including 2 million US dollars for the champion, 1 million US dollars for the second place and 750000 US dollars for the third place. The bonus will be directly given to the corresponding ranks of the team.

5. Competition

Figure 4. CGC finals (7 competing supercomputers)

There are 104 teams in the CGC preliminary competition. Of the seven teams entering the final, three are from fund track and four are from open track, all of which are from North America.

Funded Track：

• Code Jitsu: it is composed of University of California Berkeley. Professor Song Xiaodong, the leader, is a Chinese and graduated from Tsinghua University. Zhang Chao, the technical leader, is a member of blue lotus team and graduated from Peking University. The leader of blue lotus team, Dr. Yang Kun (Changting), is also a core member of the team.
• Forallsecure: a company founded by a professor from Carnegie Mellon University, most of its members are from cylab of Carnegie Mellon University; the PPP team of the ever victorious army in CTF competition comes from this lab, which has always occupied the first place in CTF time.
• TechX: a team of grammatec (focused on program analysis) and the University of Virginia.

Open Track：

• CSDs: a two person team of professors and postdoctoral researchers from the University of Idaho, the only team to redevelop a set of tools.
• Deep red: members from Raytheon, a US defense company, have special research on memory.
• Disekt: a CTF team with some knowledge of VM system. The team was founded and guided by Professor Li Kang (blue lotus enlightenment tutor), who has more than ten years of experience in competition and team organization and served as xctf consultant expert.
• Shellfish: students from the University of California, Santa Barbara, are not only traditional CTF strong teams, but also advanced to this year's Defcon finals. The academic research on network security is also quite powerful. His team member fish is also a former member of blue lotus.

On August 5, 2016, DARPA officially announced the results:

No. 1: Mayhem system developed by forallsecure team from Carnegie Mellon University, won US $2 million prize;

2nd place: xandra system developed by the team of TechX from grammatec and the University of Virginia, won a prize of US $1 million;

No. 3: mechanical physics system developed by shellfish student team from the University of California won a prize of 750000 US dollars.

Then, at the formal invitation of Defcon CTF organizers, mayhem system competed with 14 other top human CTF teams in the world. On August 6, the mayhem program defeated two human hackers. Hacker robot can stand on Defcon CTF arena, which has created a new situation of automatic network attack and defense.

Figure 5. Defcon CTF final live competition ranking (August 6)

3、 Summary and thinking

1. China's gap is not optimistic

In the CGC network challenge, the leaders and some members of codejitsu and disket are Chinese. In this competition, the Chinese people were brilliant, but we also noticed that all the teams were from foreign universities or security companies, but there was no team from China. Although China has a strong CTF team (blue lotus, 0ops, etc.) and has made great achievements in the xctf League (Defcon CTF runner up in 2016), there is a big gap in the process automation. We should strengthen the corresponding ability improvement and talent training.

2. The rise of artificial intelligence

There have been several man-machine battles between dark blue and chess masters, Chinese Super Computing tiansho and chess masters, cognitive computing Watson and champion of intelligence competition, alpha dog and go masters. However, it is the first time to apply computer programs to the attack and defense of hackers. CGC network challenge is a new exploration for the information security attack and defense to enter the stage of artificial intelligence. The automatic defense mechanism will greatly reduce the time difference between attack and defense, which will be the development trend of future security.

In November 2015, Kevin Kelly, the spiritual godfather of Silicon Valley, published "necessity", believing that artificial intelligence is as important as the Internet and will bring a new revolution to human society. After 60 years of accumulation, the cold winter of artificial intelligence has passed. With the gradual maturity and development of the three cornerstones of artificial intelligence (parallel computing, deep learning and big data), artificial intelligence will inevitably develop with it.

In August 2016, Dr. Wu Jun's book "the age of intelligence" mentioned that the emergence of big data and machine intelligence will have a significant impact on technological development, business and society, revealing that human beings are at the beginning of a major change.

On May 26, 2016, Cheng Wei, founder of didi travel, asserted that the first half of the Internet era has ended, and the second half belongs to artificial intelligence, but there are only 20 years left.

The theme of the Defcon hacking conference held in August 2016 was also defined as Rise of the Machines.

Artificial intelligence will play a more and more important role. CGC competition urges researchers to further develop software repair robots, which can scan system defects or vulnerabilities faster and more effectively than human teams, and improve the ability to quickly repair these program defects in billions of lines of code. This helps strengthen the ability of infrastructure such as power lines and water treatment equipment to resist cyber attacks, and helps protect privacy as online personal devices grow. Author: arilel / CX / Muqq