Preface

This article focuses on the idea of a vulnerability verification framework, but does not explain the details of the internal implementation. This article is a little longer, please be patient.

demand

To do a vulnerability scanner, first of all, we need to understand the effect that the scan needs to achieve. What does it need? What needs to be supported?

Possible needs are as follows:

• Cross platform operation, which can be run in windows, Mac OS, Linux and other operating systems or controlled by Web
• Customized POC, exp multi language support, POC supports multiple development languages, such as Java, python, etc.
• Multiple operation modes, including multithreading, multiprocessing, distributed operation, etc
• Visualization, vulnerability scanning framework through visual operation
• Asset management, POC verification and exp utilization through the asset of detection results
• 。。。。。。

requirement analysis

The requirements of some vulnerability scanners are simply defined above, and the implementation of the requirements is best to select appropriate technologies according to the requirements.

Cross platform operation

There are many cross platform development languages, such as C, C + +, Java, python, go and so on.

The choice of development language needs to be considered according to the operation efficiency and development efficiency that the framework needs to meet.

If you need high running efficiency and high development efficiency, you can choose to use Java or Python and other languages. If you need to pursue high efficiency, you can use go language.

Java

Java programs actually run in Java virtual machine (JRE is a software implementation). Java virtual machine is similar to a simulated execution environment, and has different Java virtual machine implementations on different operating systems. However, these Java virtual machines follow a unified specification to interpret class files and class files The instructions in the file are converted to the instructions corresponding to the local operating system, so that the same class file can be realized. The class file can be converted to the corresponding instructions on the corresponding operating system through the Java virtual machine, that is, the cross platform nature of Java programs.

Python

Python is a cross platform scripting language. Python specifies a python syntax rule, and the interpreter that implements Python syntax becomes the interpreter of Python.

Custom POC

The essence of custom POC and exp is POC written based on template, which enables scanning framework to recognize and run. The composition of the template is generally divided into vulnerability basic information, POC verification information and exp utilization information.

Multiple operation modes

Multiple operation modes, including multi process, multi thread and distributed operation. If the scanner needs large-scale scanning and detection, it is recommended to use distributed nodes for operation. In general, multithreading and multiprocessing can meet most of the requirements.

Generally distributed is suitable for large-scale intranet or large-scale scanning detection.

visualization

At present, there are three popular schemes for the selection of visualization technology.

The first is to develop visualization program by using QT.

The second is to use electron or nw.js to develop visual applications. The famous fofa Pro client, goby, China ant sword and so on are developed based on this scheme. There are many programs developed based on this framework on electron's official website.

The third is to use the B / S architecture for development, the back-end control uses web for control, and the nodes use python, Java and other languages for development, among which bugscan, w8scan, and patrol are all based on the B / S architecture.

Electron

Electron is an open source library developed by GitHub to build cross platform desktop applications with HTML, CSS and Ja VASC ript. Electron does this by merging chromium and node. JS into the same runtime environment and packaging them as applications under Mac, windows and Linux systems.

Electron was developed in 2013 as a framework for building atom, a programmable text editor on GitHub. The two projects were opened in spring 2014.

At present, it has become a common development tool for open source developers, start-ups and established companies.

Nw.js

Nw.js is based on chromium and node.js. Nw.js uses web technology and node.js and its modules to develop desktop applications.

asset management

Asset management, which is used to manage the asset information of the scan detection results, and to associate the asset with the corresponding POC. Among them, asset management mainly contains asset identification rules, which are used to identify asset information. For example, rules are made to identify the asset by its title, body, certificate, header and banner.

Design

The overall framework is divided into four modules: concurrency engine, plug-in center, target center and vulnerability verification.

• Concurrency engine: it mainly provides the running mode of vulnerability verification framework, such as multithreading, multiprocessing, distributed and other running modes.
• Plug in center: it mainly provides the template of vulnerability POC, POC loading and POC management.
• Target center: target loading, target asset management, vulnerability management.
• Log management: manage all network requests and response information in the framework, and provide subsequent backtracking. Process framework operation process

Technological process

• Select the appropriate running engine, and generally use multithreading and multiprocessing for detection. If the detection of large enterprise intranet and large-scale scanning detection, distributed operation is more appropriate.
• Plug in loading center to load POC.
• Target loading, loading operation needs to detect the target, which can be loaded in the form of a single IP, IP segment and file.

• Detection and analysis are mainly divided into asset detection and vulnerability detection. Asset detection identifies asset information through asset identification rules, and vulnerability detection analyzes vulnerability based on identified asset information.

Concurrency engine

The concurrency engine is mainly the running mode selected by the vulnerability verifier when it is running. There are three running modes: multithreading, multiprocessing and distributed. The multithreading, multiprocessing and distributed running modes are used to ensure the efficiency of the vulnerability verifier.

Under normal circumstances, using multithreading and multiprocessing is suitable for most scenarios.

Distributed operation is used for intranet detection or large-scale network detection in large enterprises.

Plug-in Center

The plug-in center is composed of template center, plug-in loading center and plug-in management center.

The template center provides POC and exp template information, mainly including general template and programming language template.

Plug in center, which provides users with optional plug-ins to be tested for loading.

Plug in management center manages POC written by user.

Template center

Template center is mainly divided into general template, programming language template, two modules. Each of them is composed of plug-in basic information, POC information and exp information.

Plug in basic information consists of plug-in name (vulnerability name), type, product name, level, product home page, description, etc.

POC information consists of contract awarding agreement, contract awarding method, contract awarding content, logical judgment, etc.

Exp information consists of POC information and execution command.

Generally, the contract awarding protocol is HTTP, HTTPS, TCP, UDP, etc. the contract awarding methods of the protocol (part) include get, post, put, head, delete, options, trace, connect, etc. the logical judgment is made by "and or not".

Plug in loading Center

The plug-in is loaded by the user's selection method (full load, specified load).

• Get the basic information of the plug-in to be loaded from the plug-in database.
• Get the code information in the plug-in, and get the plug-in code information from the local git warehouse according to the information.
• Then load POC through the corresponding load service.

Plug in Management Center

The plug-in management center manages and writes plug-in information, and performs operations such as modifying, deleting, and querying plug-ins.

• Add / modify: add and modify the general template and programming language template based on the template center.
• Query: use the fields in the template center as the basis for multi-dimensional query to obtain plug-in information for subsequent modification and utilization.
• Delete: use the fields in the template center as the basis for multi-dimensional query and deletion.

Target center

The target center is mainly responsible for analyzing the three modules of target loading, asset management and vulnerability management.

• Target loading module: mainly used to load the target to be analyzed.
• Asset management: mainly used to analyze the asset information, identification rules and vulnerability information management of target detection.

The vulnerability plug-in information can be stored by using the database with GIT. The database stores the basic plug-in information. Git stores the specific code information of the plug-in. Git can be used to control the version of the plug-in and perform operations such as fallback.

Target loading

At present, loading can be done by loading single IP, IP segment and file import.

• IP segment loading: generates a list of detection targets in the IP segment.
• File loading: parses the imported file format, and loads the target through the corresponding format.

asset management

The asset management module is mainly used to manage and analyze the successful asset information, vulnerability management and identification rule management.

• Target asset: identify the asset by matching the information of the detection response with the identification rules. If the identification is successful, it is known asset, otherwise it is unknown asset. The unknown asset is convenient for subsequent rule entry to improve the identification rate of the asset.
• Vulnerability management: the result of detection through the identified assets and corresponding vulnerability detection plug-ins.
• Recognition rules: the recognition rules construction format can be used as recognition rules by detecting the response protocol, response content, header, service type and other information.

Log management

The log management module mainly manages the log information of the overall framework, including log recording, query and other operations.

• The log information includes request information (request URI, request mode (post, get, delete, etc.), request parameters, response information (response header, response content, etc.) and framework execution operations (asset detection, plug-in detection, etc.).
• Logging, you can record the network log information through the unified request center.
• Log query, through the query of IP, request URI, request mode, response content and other multi-dimensional queries, is convenient for subsequent vulnerability troubleshooting. summary

summary

Flow chart in the paper: https://github.com/0nise/sc ripts

This article describes a vulnerability detection framework that I have conceived. If there are any discrepancies and problems described in the article, please don't be stingy, please correct them.