practical experience of automation operation and maintenance of some small teams

Posted by barello at 2020-03-26

It's pull mode

It is easy to use text mode to configure, which is conducive to configuration versioning

There are too many plug-ins. There are basically ready-made plug-ins for monitoring

I have to learn all three of them again. Why don't I learn one recommended by Google SRE?

Prometheus server is responsible for monitoring data collection and storage

Prometheus alert manager is responsible for alerting according to alert rules, and can integrate many alert channels

The role of node exporter [1] is to read indicators from the machine, and then expose an HTTP service from which Prometheus collects monitoring indicators. Of course, Prometheus officially has a variety of exporters.

├── environments/        
│   │
│   ├── dev/              
│   │   ├── group_vars/  
│   │   │   ├── all
│   │   │   ├── db
│   │   │   └── web
│   │   └── hosts        
│   │
│   ├── prod/            
│   │   ├── group_vars/  
│   │   │   ├── all
│   │   │   ├── db
│   │   │   └── web
│   │   └── hosts        
│   │
│   └── stage/            
│       ├── group_vars/  
│       │   ├── all
│       │   ├── db
│       │   └── web
│       └── hosts        

- hosts: all
     - blueocean
     - ghprb
     - greenballs
     - workflow-aggregator
   jenkins_plugin_timeout: 120
   - include_tasks: java-8.yml
   - ansible-role-jenkins

Settings on the interface

Using jenkinsfile: a text file similar to dockerfile. Details: using a jenkinsfile [7]

pipeline {
   agent any
   stages {
       stage('Build') {
           steps {
               sh './gradlew clean build'
               archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true

Install ansible plug-in in Jenkins [8]

Execute in jenkinsfile

withCredentials([sshUserPrivateKey(keyFileVariable:"deploy_private",credentialsId:"deploy"),file(credentialsId: 'vault_password', variable: 'vault_password')]) {
            ansiblePlaybook vaultCredentialsId: 'vault_password', inventory: "environments/prod", playbook: "playbook.yaml",
              ansible_ssh_private_key_file: [value: "${deploy_private}", hidden: true],
              build_number: [value: "${params.build_number}", hidden: false]

Ansible playbook is the pipeline syntax provided by Jenkins ansible plug-in, similar to manual execution: ansible playbook.

With credentials is the syntax of the credentials binding [9] plug-in, which is used to reference some sensitive information, such as the SSH key and ansible vault password needed to execute ansible.

Some sensitive configuration variables are encrypted with ansible vault [10] technology.

Basic monitoring

Upper Gitlab

Upper Jenkins and integrate gitlab

Using Jenkins to realize automatic compilation and packaging

Ansible with Jenkins

Construction of CMDB: we use ANSI ble CMDB [12] to automatically generate the current situation of all machines according to inventory

Release management: each stage of the release can be customized on Jenkins. Blue green publishing and other publishing methods can be implemented by modifying ansible script and inventory.

Automatic expansion and reduction: it can be realized by configuring Prometheus alarm rules and calling corresponding webhook

Chatops: actual battle of chatops [13]