practical experience of automation operation and maintenance of some small teams

It's pull mode

It's pull mode

It is easy to use text mode to configure, which is conducive to configuration versioning

It is easy to use text mode to configure, which is conducive to configuration versioning

There are too many plug-ins. There are basically ready-made plug-ins for monitoring

There are too many plug-ins. There are basically ready-made plug-ins for monitoring

I have to learn all three of them again. Why don't I learn one recommended by Google SRE?

I have to learn all three of them again. Why don't I learn one recommended by Google SRE?

Prometheus server is responsible for monitoring data collection and storage

Prometheus server is responsible for monitoring data collection and storage

Prometheus alert manager is responsible for alerting according to alert rules, and can integrate many alert channels

Prometheus alert manager is responsible for alerting according to alert rules, and can integrate many alert channels

The role of node exporter [1] is to read indicators from the machine, and then expose an HTTP service from which Prometheus collects monitoring indicators. Of course, Prometheus officially has a variety of exporters.

The role of node exporter [1] is to read indicators from the machine, and then expose an HTTP service from which Prometheus collects monitoring indicators. Of course, Prometheus officially has a variety of exporters.

├── environments/         
│   │
│   ├── dev/              
│   │   ├── group_vars/   
│   │   │   ├── all
│   │   │   ├── db
│   │   │   └── web
│   │   └── hosts         
│   │
│   ├── prod/             
│   │   ├── group_vars/   
│   │   │   ├── all
│   │   │   ├── db
│   │   │   └── web
│   │   └── hosts         
│   │
│   └── stage/            
│       ├── group_vars/   
│       │   ├── all
│       │   ├── db
│       │   └── web
│       └── hosts         
│

---
- hosts: all
  vars:
    jenkins_plugins:
      - blueocean
      - ghprb
      - greenballs
      - workflow-aggregator
    jenkins_plugin_timeout: 120
  pre_tasks:
    - include_tasks: java-8.yml
  roles:
    - geerlingguy.java
    - ansible-role-jenkins

Settings on the interface

Settings on the interface

Using jenkinsfile: a text file similar to dockerfile. Details: using a jenkinsfile [7]

Using jenkinsfile: a text file similar to dockerfile. Details: using a jenkinsfile [7]

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh './gradlew clean build'
                archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true
            }
        }
    }
}

Install the Ansible plug-in in Jenkins [8]

Install ansible plug-in in Jenkins [8]

Execute in jenkinsfile

Execute in jenkinsfile

withCredentials([sshUserPrivateKey(keyFileVariable:"deploy_private",credentialsId:"deploy"),file(credentialsId: 'vault_password', variable: 'vault_password')]) {
             ansiblePlaybook vaultCredentialsId: 'vault_password', inventory: "environments/prod", playbook: "playbook.yaml",
             extraVars:[
               ansible_ssh_private_key_file: [value: "${deploy_private}", hidden: true],
               build_number: [value: "${params.build_number}", hidden: false]
             ]
}

Ansible playbook is the pipeline syntax provided by Jenkins ansible plug-in, similar to manual execution: ansible playbook.

Ansible playbook is the pipeline syntax provided by Jenkins ansible plug-in, similar to manual execution: ansible playbook.

With credentials is the syntax of the credentials binding [9] plug-in, which is used to reference some sensitive information, such as the SSH key and ansible vault password needed to execute ansible.

With credentials is the syntax of the credentials binding [9] plug-in, which is used to reference some sensitive information, such as the SSH key and ansible vault password needed to execute ansible.

Some sensitive configuration variables are encrypted with ansible vault [10] technology.

Some sensitive configuration variables are encrypted with ansible vault [10] technology.

Basic monitoring

Basic monitoring

Upper Gitlab

Upper Gitlab

Upper Jenkins and integrate gitlab

Upper Jenkins and integrate gitlab

Using Jenkins to realize automatic compilation and packaging

Using Jenkins to realize automatic compilation and packaging

Ansible with Jenkins

Ansible with Jenkins

Construction of CMDB: we use ANSI ble CMDB [12] to automatically generate the current situation of all machines according to inventory

Construction of CMDB: we use ANSI ble CMDB [12] to automatically generate the current situation of all machines according to inventory

Release management: each stage of the release can be customized on Jenkins. Blue green publishing and other publishing methods can be implemented by modifying ansible script and inventory.

Release management: each stage of the release can be customized on Jenkins. Blue green publishing and other publishing methods can be implemented by modifying ansible script and inventory.

Automatic expansion and reduction: it can be realized by configuring Prometheus alarm rules and calling corresponding webhook

Automatic expansion and reduction: it can be realized by configuring Prometheus alarm rules and calling corresponding webhook

Chatops: actual battle of chatops [13]

Chatops: actual battle of chatops [13]

https://github.com/prometheus/node_exporter

https://github.com/prometheus/node_exporter

https://github.com/ernestas-poskus/ansible-prometheus

https://github.com/ernestas-poskus/ansible-prometheus

https://github.com/timonwong/prometheus-webhook-dingtalk

https://github.com/timonwong/prometheus-webhook-dingtalk

https://www.digitalocean.com/community/tutorials/how-to-manage-multistage-environments-with-ansible

https://www.digitalocean.com/community/tutorials/how-to-manage-multistage-environments-with-ansible

http://docs.ansible.com/ansible/latest/modules/consul_module.html

http://docs.ansible.com/ansible/latest/modules/consul_module.html

https://github.com/geerlingguy/ansible-role-jenkins

https://github.com/geerlingguy/ansible-role-jenkins

https://jenkins.io/doc/book/pipeline/jenkinsfile/

https://jenkins.io/doc/book/pipeline/jenkinsfile/

https://wiki.jenkins.io/display/JENKINS/Ansible+Plugin

https://wiki.jenkins.io/display/JENKINS/Ansible+Plugin

https://jenkins.io/doc/pipeline/steps/credentials-binding/

https://jenkins.io/doc/pipeline/steps/credentials-binding/

http://docs.ansible.com/ansible/2.5/user_guide/vault.html

http://docs.ansible.com/ansible/2.5/user_guide/vault.html

https://github.com/audreyr/cookiecutter

https://github.com/audreyr/cookiecutter

https://github.com/fboender/ansible-cmdb

https://github.com/fboender/ansible-cmdb

https://showme.codes/2017-10-08/chatops-in-action/

https://showme.codes/2017-10-08/chatops-in-action/

IMCAFS

practical experience of automation operation and maintenance of some small teams