preliminary assumption of docker network isolation

Posted by fierce at 2020-02-27

Docker network status

At present, docker network isolation is the weakest part of the containerization scheme. Neither the docker engine nor the major management systems such as mesos and k8s provide solutions. But if not isolated, there will be a certain risk, because if the flow of one container is too large, it will occupy the network card, causing other containers can not be used.

Is there any way to achieve network isolation in the current situation? In fact, there are flexible ways.

POC thinking and practice process

At present, there are many tools for limiting applications or network cards on Linux. In the process of practice, the following are compared:

yum install wondershaper wondershaper <interface> <download-rate> <upload-rate>

With the tool, starting from limiting the container network, you need to find the virtual network card vethxxx corresponding to the host through the container ID. But the problem is that neither the API nor the command of docker can find the container to map to the host virtual network card. Later, I found a way through the communication with my acquaintance Yun Laoxiao:

docker inspect <contianerid> /var/run/docker/netns/5b0e87d40fad nsenter --net=/var/run/docker/netns/5b0e87d40fad ethtool -S eth0 |grep peer_ifindex nsenter ethtool ip a|grep <veth_id> wondershaper <interface> <download-rate> <upload-rate>

Step 5: test the effect. For simplicity, the test idea is to generate a large file such as 1g on the host (DD if = / dev / zero of = hello. TXT BS = 1g count = 1), and then get the file viewing rate WGet http://xxxx:8090/hello.txt in the container (start a simple HTTP service through Python: Python - M simplehttpserver 8090)

dd if=/dev/zero of=hello.txt bs=1G count=1 wget http://xxxx:8090/hello.txt python -m SimpleHTTPServer 8090

The first and second steps here are actually an old way. In fact, you can use docker exec < container_id > ethtool - s eth0 | grep peer_ifindex -:)

docker exec <container_id> ethtool -S eth0 |grep peer_ifindex

Some pits encountered

/usr/lib/systemd/system/docker.service MountFlags=slave

Feasible overall plan

At this point, we can manually set the bandwidth of the container's network. However, in an integrated system (mesos / k8s), this manual method is not realistic, so it is necessary to design a scheme to simplify and automate it

Design ideas on mesos + Marathon:

Advantages and disadvantages of the scheme:

Ultimate solution

We still need some + marathon to provide more suitable services. Just like CPU and memory, network card should also be a kind of manageable resource


Vipshop will recruit Java Architect / Senior Engineer / Senior Engineer / docker engineer, etc. if you want, you can send it to me: [email protected]