3rdstage's Wiki
Register
Advertisement

References

Ansible 2.7 "In the Light"

  • Inventory Parameters
    • ansible_connection, ansible_host, ansible_port, ansible_user, ansible_ssh_pass, ansible_ssh_private_key_file, ...
    • ansible_become_pass (ansible_sudo_pass, ansible_su_pass)

Ansible 2.6 "Heartbreaker"

Ansible 2.4

Commands

ansible-playbook
Option Description Remarks
--list-hosts
--list-tasks
--list-tags
--syntax-check
--connection=CONNECTION, -c CONNECTION
--limit=SUBSET, -l SUBSET
--tags=TAGS, -t TAGS
--ask-pass, -k

Ansible Container

Concepts

playbook = play+

play = (host+, task+)

task = (name, module, ...)

Directives

Scope Directive Description Remarks
Task environment A dictionary that gets converted into environment vars to be provided for the task upon execution.
when
become
with_items
with_subelements Subelements walks a list of hashes (aka dictionaries) and then traverses a list with a given (nested sub-)key inside of those records.
run_once Boolean that will bypass the host loop, forcing the task to execute on the first host available and will also apply any facts to all active hosts.
delegate_to
register
changed_when
failed_when

Modules

Module Description Remarks
debug Print statements during execution
assert Asserts given expressions are true
fail Fail with custom message
pause Pause playbook execution
stat Retrieve file or file system status
file Sets attributes of files
copy Copies files to remote locations
archive Creates a compressed archive of one or more files or trees zip, tar cvf, tar cvzf
unarchive Unpacks an archive after (optionally) copying it from the local machine unzip, tar xvf, tar xvzf
lineinfile Ensure a particular line is in a file, or replace an existing line using a back-referenced regular expression
replace Replace all instances of a particular string in a file using a back-referenced regular expression
template Templates a file out to a remote server
shell Execute commands in nodes /bin/sh, /bin/bash
command Executes a command on a remote node
expect Executes a command and responds to prompts
get_url Downloads files from HTTP, HTTPS, or FTP to node curl, wget
uri Interacts with HTTP and HTTPS web services and supports Digest, Basic and WSSE HTTP authentication mechanisms curl, weg
apt Manages apt-packages
apt_repository Add or remove an APT repositories in Ubuntu and Debian
apt_key Add or remove an apt key, optionally downloading it
debconf Configure a .deb package using debconf-set-selections.
yum Manages packages with the yum package manager
yum_repository Add or remove YUM repositories
rpm_key Adds or removes a gpg key from the rpm db
pip Manages Python library dependencies
systemd Controls systemd services on remote hosts systemd provides a system and service manager that runs as PID 1 and starts the rest of the system.
cron Manage cron.d and crontab entries
docker_container Manage the life cycle of docker containers
docker_image Manage docker images
git Deploy software (or files) from git checkouts
mysql_variables Manage MySQL global variables (Query / Set MySQL variables)
mysql_db Add or remove MySQL databases from a remote host
mysql_user Adds or removes a user from a MySQL database

Variables

Variable Description Remarks
ansible_connection Connection type to the host. ssh, smart, local, ...
ansible_host The name of the host to connect to
ansible_user The default ssh user name to use.

Magic Variables

Variable Description Remarks
hostvars
group_names a list (array) of all the groups the current host is in
groups a list of all the groups (and hosts) in the inventory
inventory_hostname the name of the hostname as configured in Ansible’s inventory host file
ansible_play_hosts the full list of all hosts still active in the current play
ansible_play_batch a list of hostnames that are in scope for the current ‘batch’ of the play
playbook_dir the playbook base directory
inventory_dir the pathname of the directory holding Ansible’s inventory host file
inventory_file the pathname and the filename pointing to the Ansible’s inventory host file

Facts

Category Fact Description Values Remarks
Common ansible_architecture "x86_64"
ansible_distribution "Ubuntu"
ansible_distribution_version "16.04"
ansible_distribution_major_version "16"
ansible_distribution_release "xenial"
ansible_hostname hostname
ansible_env ansible_env.HOME, ansible_env.PATH, ansible_env.PWD
ansible_interfaces ["lo", "eth0", "eth1", "eth2", "eth3", "docker0", ...] Array
ansible_os_family "Debian"
ansible_pkg_mgr "apt"
ansible_check_mode false
inventory_hostname name of the host as configured in Ansible's inventory host file (usually hosts.yml) "m001", "m002" not always same with ansible_hostname

Filters

Filter Description Applied to Syntax Remarks
json_query query a complex JSON structure and iterate over it using a loop structure an object JMESPath Specification
regex_replace replace text in a string with regex string Regex in Python 3
regex_search search a string with a regex string
select Filters a sequence of objects by applying a test to each object, and only selecting the objects with the test succeeding. a sequence of objects
selectattr Filters a sequence of objects by applying a test to the specified attribute of each object, and only selecting the objects with the test succeeding. a sequence of objects selectattr(attrName, test, value)
(test : equalto, ne, gt, ge, number, odd, ...)
reject Filters a sequence of objects by applying a test to each object, and rejecting the objects with the test succeeding. a sequence of objects
rejectattr Filters a sequence of objects by applying a test to the specified attribute of each object, and rejecting the objects with the test succeeding. a sequence of objects
map Applies a filter on a sequence of objects or looks up an attribute. This is useful when dealing with lists of objects but you are really only interested in a certain value of it. a sequence of objects map(attribute="attrName")
map(filterName, [arg1, arg2, arg3, ...)
attr Get an attribute of an object. an object
list Convert the value into a list.
flatten Flatten a list a sequence of objects flatten([levels=n])
sort Sort an iterable iterable sort(reverse=False, case_sensitive=False, attribute=None)
default, d Return predefined value for a undefined variable or a variable whose value is evaluated to false default(default_value=u'', boolean=False)
combine allows hashes to be merged hash, dictionary dict1|combine(combine(dict2, recursive=True)
to_json, to_nice_json
to_yaml, to_nice_yaml
mandatory
common.quorum.download.constellation|regex_replace('^.*/', '')

configDefault.outputs|selectattr("type", "eq", "logstash")|first

['ripple']|map('extract', common, ['tls', 'dn', 'c'])|list|first|default('KO')

Tests

Test Description Applied to Syntax Remarks
directory, file, link, abs, mount provide information about a path on the controller NOT for remote path
failed, succeeded, changed, skipped check the status of tasks
match, search match strings against a substring or a regex string
version compare a version number string version(compared_version, operator) <, lt, <=, le, >, gt, >=, ge, ==, =, eq, !=, <>, ne

Plugins

Category Plugin Description Remarks
Lookup fileglob Matches all files in a single directory, non-recursively, that match a pattern

Jinja2

  • Core Concepts
    • Variable
    • Filter (var|filter1|filter2|...|filtern)
    • Test
    • Expression
    • Statement

Initialization Parameters

Parameter Description Default Remarks
trim_blocks If this is set to True the first newline after a block is removed (block, not variable tag!) False
lstrip_blocks If this is set to True leading spaces and tabs are stripped from the start of a line to a block. False
keep_trailing_newline Preserve the trailing newline when rendering templates. False
newline_sequence The sequence that starts a newline. Must be one of '\r', '\n' or '\r\n'. '\n'

Operators

Category Name Operator Description Remarks
Math Addition +
Subtraction -
Multiplication *
Division /
Division to truncated integer // 20 // 7 == 2
Comparison Equal to ==
Inequal to !=
Greater than >
Lower than <
Logic And and
Or or
Not not
Misc In in if _port.protocol in ['http', 'https']
Is is Perform a test
Filter | Apply a filter
String concatenation ~

Readings

var|default('', true)
var or ''
var or {}
var or []

JMESPath

Expressions

Expression Syntax Description Remarks
Sub Expression expr1.expr2, expr1.* definitions.simpleTypes
Index Expression expr[n] properties[0], properties[3], properties[-3]
Slice Expression expr[i:j:k] properties[0:-1:2]
Hash Wildcard Expression * Return a list of the hash element’s values. Any subsequent expression will be evaluated against each individual element in the list
List Wildcard Expression [*] Return all the elements in a list. Any subsequent expressions will be evaluated against each individual element.
Multi-Select List Expression [ expr1, expr2, expr3 ... ] Extract a subset of elements as a list from a JSON hash
Multi-Select Hash Expression [ key1: expr1, key2: expr2, ...] Extract a subset of elements as a hash from a JSON hash
Filter Expression [? boolean expr ] Select JSON elements based on a comparison to another expression for each element in an array
Flatten Operator [] Merge sublists in the current result into a single list. Filter first and flatten last
Pipe Expressions expr1 | expr2 combines two expressions, separated by the | character
Or Expression expr1 || expr2 evaluate to either the left expression or the right expression [validators, trackers][] `[]`
And Expression expr1 && expr2 evaluate to either the left expression or the right expression
Not Expression ! expr negates the result of an expression
Literal Expressions `json-value` allows arbitrary JSON objects to be specified escape ` with leading \ : \`

Operators

Operator Name Remarks
== Equality
!= Inequality
< Less than
<= Less than or equal to
> Greater than
>= Greater than or equal to

Examples

hostvars.*.ripple.[validators, trackers][][].name

common|json_query('haproxy.systemd.LimitNOFILE' || `20000`)

"{:,d}".format(node|json_query('uptime || `0`'))

hostvars|json_query('*.prepare_validator_token.results[].stdout_lines[]')

network[*].{group: `nodes`, data: {id: name, name: name}}

rippled|json_query('config.ports[?name==`port_rpc`]|[0]')

host|json_query('haproxy.rippleProxies[?backends[?@.servers[?@.name==`' ~ rippled.name ~ '`]]]')|first or {}

Roles

Role Location Remarks
apparmor https://github.com/Oefenweb/ansible-apparmor Remove apparmor in Debian-like systems.

Readings

Templating

Filters / Tests

Collection data-type

String

Formatting

Security

Modules

Examples

Tips and Tricks

Print out variables and expressions using Ansible console and debug module

Get into console using ansible-console command and then use debug task

$ ansible-console -l m001

...

sshuser@all (1)[f:5]$ debug msg="{{ hostvars }}"

...

sshuser@all (1)[f:5]$ exit

$

Executing simple modules using ansible command without any playbook

$ ansible m001 -m ping

$ ansible m001 -m command -a 'cat /etc/environment'

$ ansible m003 -i inventories/product2/hosts.yml -c local -m debug -a 'msg="{{ ansible_connection }}"'

Executing play on localhost instead of remote machine

  • Need more comparison with dry run using -C or --check option
  • You can change ansible_user into local user when executing playbook with '-c local' option.
$ ansible-playbook -i inventories/prd/hosts.yml -t setupRippleNode -l p001 \
  -c local --flush-cache -e "ansible_user=tom"  plays/prepare-hosts.yml

Using Jinja statements inside play

  - name: Start constellation nodes
    vars:
      tls: "{{ item.constellation.tls|default(common.quorum.constellation.tls) }}"
      
      protocol: >-2
        {%- if tls == 'strict' -%}'http'{%- else -%}'http'{%- endif -%}
    shell: |
      ...

To build more complex object as an task scope variable, build script variable using statements then print out the script variable using expression at the last line

  - name: Generate validators file ('validators.txt')
    vars:
      pubKeys: |
        {%- set pubKeys = [] -%}
          {%- for lines in hostvars|json_query('*.extract_validator_public_key.results[].stdout_lines')|sort(attribute='0') -%}
            {%- if lines[0] in item.peers -%}
              {%- do pubKeys.append(lines[1]) -%}
            {%- endif -%}
          {%- endfor -%}
        {{ pubKeys }}
    template:
    ...

Safe escaping undefined variable

Use default filter.

when: not fabric.generate.crypto.skip|default(false)
with_items: favorites.movies|default([])
{% for member in members|default([]) %}
  {%- for message in (member.mailbox|default({})).messages|default([]) -%}
  ...
  {%- endfor -%}
{% endfor %}

Safe drill-down using json_query filter

If you want access deep descendant node from a safe node without escaping every step using default filter, you can use json_query filter which utilizes JMESPath internally.

- name: Generate CSR for TLS certificate if not exists
  openssl_csr:
    path: "{{ ansible_env.HOME }}/ripple/nodes/{{ item.name }}/tls/tls-server.csr"
    privatekey_path: "{{ ansible_env.HOME }}/ripple/nodes/{{ item.name }}/tls/tls-server.key"
    state: present
    force: false
    mode: 0600
    country_name: "{{ item|json_query('tls.dn.countryName')|default(common.ripple.tls.baseName.countryName, true) }}"
    state_or_province_name: "{{ item|json_query('tls.dn.stateOrProvinceName')|default(common.ripple.tls.baseName.stateOrProvinceName, true) }}"
    locality_name: "{{ item|json_query('tls.dn.localityName')|default(common.ripple.tls.baseName.localityName, true) }}"
    organization_name: "{{ item|json_query('tls.dn.organizationName')|default(common.ripple.tls.baseName.organizationName, true) }}"
    organizational_unit_name: "{{ item|json_query('tls.dn.organizationalUnitName')|default(common.ripple.tls.baseName.organizationalUnitName, true) }}"
    email_address: "{{ item|json_query('tls.dn.emailAddress')|default(common.ripple.tls.baseName.emailAddress, true) }}"
    common_name: "{{ item|json_query('tls.dn.commonName')|default(item.name, true) }}"
  when: true
  become: false
  with_items: "{{ (ripple|default({})).validators|default([]) }}"
  register: generate_validator_tls_csr
  tags: ['tls']

When using multi-select-list expression of JMESPath, an item in the resulted list can be null or nested. So, it should be filtered using select and flattened.

- name: Backup and clean artifacts including data, configuration, and logs
  var:
    timestamp: "{{ lookup('pipe', 'date date +%Y%m%d'T'%H%M%Z -u') }}"
  file:
    src: "{{ ansible_env.HOME }}/ripple/nodes/{{ item.name }}/"
    dest: "{{ ansible_env.HOME }}/ripple/nodes/{{ item.name }}_{{ timestamp }}/"
  when: common.ripple.flags.cleans
  with_items: "{{ ripple|default({})|json_query('[validators, trackers]')|select|flatten }}"

Filtering and drilling down the entire hostvars using selectattr and map filters

With 4 host variable files under host_vars directory

  • host_vars/m001.yml
quorum: 
  nodes:
    - name: node1
      host: "{{ inventory_hostname }}"
      type: permissioned
    - name: node2
      host: "{{ inventory_hostname }}"
      type: permissioned
  • host_vars/m002yml
quorum: 
  nodes:
    - name: node3
      host: "{{ inventory_hostname }}"
      type: permissioned
  • host_vars/m003yml
quorum: 
  portal:
  • host_vars/m004yml
grafana:

The following task would print out only 3 quorum nodes

  - name: Print out Quorum nodes
    debug:
      msg: "{{ item }}"
    with_items: "{{ hostvars.values()|selectattr('quorum', 'defined')|map(attribute='quorum')|selectattr('nodes', 'defined')|map(attribute='nodes')|sum(start=[])|list }}" 
    # when: (hostvars[item].quorum|default({})).nodes
    when: true

Filtering and drilling down the entire hostvars using json_query filter

You can navigate the entire hostvars in null safe way more conveniently using json_query filter.

  - name: Print out Quorum nodes
    debug:
      msg: "{{ hostvars|json_query('*.quorum.nodes')|flatten|sort(attribute='name')|reverse|list }}"
    when: true

Or you can use json_query more aggressively

  - name: Print out Quorum nodes
    debug:
      msg: "{{ hostvars|json_query('*.quorum.nodes[]')|sort(attribute='name')|reverse|list }}"
    when: true

Even more aggressively (may not work)

  - name: Print out Quorum nodes
    debug:
      msg: "{{ hostvars|json_query('*.quorum.nodes[] | sort_by([], &name)')|reverse|list }}"
    when: true


Another example

  - name: Print out Quorum nodes
    debug:
      msg: "{{ hostvars|json_query('*.ripple.validators[?name!=`v0`][]')|sort(attribute='name')|reverse|list|json_query('[*].config.ports[?name==`port_peer`][]') }}"
   when: true

Filtering and drilling down current host variable using json_query filter

{% for node in ripple|default({})|json_query('[validators, trackers][]') -%}
  {%- set rpcPort = node|json_query('config.ports[?name==`port_rpc`][]')|first -%}
[[inputs.exec]]
  name_override = "rippled"
  commands = ["bash /etc/telegraf/telegraf_exec_cmd_rippled.sh {{ rpcPort.ip }} {{ rpcPort.port }}"]
  interval = "10s"
  timeout = "5s"
  data_format = "json"
  json_string_fields = ["build_ver", "complete_ledgers", "server_state", "base_log_level"]
  [inputs.exec.tags]
    type="ripple"
    service = "{{ node.name }}"    

{% endfor -%}

Fine tuning line breaks and indentations for Jinja2 templates

Default setting for line breaks and trimming should be specified at the top of the template file

#jinja2:line_statement_prefix: '%', trim_blocks: False, lstrip_blocks: True, keep_trailing_newline: True

For simple loop without other nested conditionals or nested loops, use {% for ... -%} ... {% endfor %}

#jinja2:line_statement_prefix: '%', trim_blocks: False, lstrip_blocks: True, keep_trailing_newline: True
[validators]
{% for key in pubKeys -%}
  {{ key }}
{% endfor %}

Generating well indented and lined files

Generate files with trim_blocks disabled.

Sample

#jinja2:line_statement_prefix: '%', trim_blocks: False, lstrip_blocks: True, keep_trailing_newline: True
{#
For, detailed help, refer https://github.com/ripple/rippled/blob/1.0.0/cfg/rippled-example.cfg
#}
# Configuration for Ripple node named {{ item.name }}
{# 1. Server #}
[server]
{% for port in item.config.ports -%}
  {{ port.name }}
{% endfor -%}

{% for port in item.config.ports -%}
[{{ port.name }}]
port = {{ port.port }}
ip = {{ port.ip }}
  {% if port.admin is defined %}{{ 'admin = ' ~ port.admin }}{% endif %}
protocol = {{ port.protocol }}
  {% if port.ssl_key is defined %}{{ 'ssl_key = ' ~ port.ssl_key }}{% endif %}
  {% if port.ssl_cert is defined %}{{ 'ssl_cert = ' ~ port.ssl_cert }}{% endif %}
{% endfor -%}

{%- if item.config.rpc_startup is defined %}
[rpc_startup]
  {% for cmd in item.config.rpc_startup -%}
    {{ cmd|to_json }}
  {% endfor %}
{%- endif %}

{# [websocket_ping_frequency] #}

{# 2. Peer Protocol #}
{# [ips] #}

{# [ips_fixed] #}

[peer_private]
1

[peers_max]
{{ common.ripple.configDefault.peers_max }}

[node_size]
medium
  # https://github.com/ripple/rippled/blob/1.0.0/cfg/rippled-example.cfg
  - name: Generate validator configuration from template
    template:
      src: "{{ playbook_dir }}/../templates/rippled.cfg.j2"
      dest: "{{ ansible_env.HOME }}/ripple/nodes/{{ item.name }}/rippled.cfg"
      newline_sequence: '\n'
      mode: "u=rw,g=r,o=r"
    become: flase
    with_items: "{{ (ripple|default({})).validators|default([]) }}"
    tags: [genConfig]

Tips

Non-nested for without included if
  • Use {% for ... -%}...{% endfor -%}.
  • Add blank line before ending {% endfor -%} line, to add a single blank line after each iterated item.
[server]
{% for port in item.config.ports -%}
  {{ port.name }}
{% endfor -%}
{% for node in ripple|default({})|json_query('[validators, trackers][]') -%}
  {%- set rpcPort = node|json_query('config.ports[?name==`port_rpc`][]')|first -%}
[[inputs.exec]]
  name_override = "rippled"
  commands = ["bash /etc/telegraf/telegraf_exec_cmd_rippled.sh {{ rpcPort.ip }} {{ rpcPort.port }}"]
  interval = "10s"
  timeout = "5s"
  data_format = "json"
  json_string_fields = ["build_ver", "complete_ledgers", "server_state", "base_log_level"]
  [inputs.exec.tags]
    type="ripple"
    service = "{{ node.name }}"    

{% endfor -%}

Executing complex interactive only command using expect module

  - name: Make MariaDB more secure using 'mysql_secure_installation'
    expect: 
      command: mysql_secure_installation --defaults-file="/etc/mysql/conf.d/my_{{ item.name }}.cnf"
      responses:
        'Enter current password for root \(enter for none\): ': "\n"
        'Set root password\? \[Y/n\]': 'n'
        'Remove anonymous users\? \[Y/n\]': 'Y'
        'Disallow root login remotely\? \[Y/n\]': 'Y'
        'Remove test database and access to it\? \[Y/n\]': 'Y'
        'Reload privilege tables now\? \[Y/n\]': 'Y'
      timeout: 10
    no_log: true
    when: true
    become: true
    with_items: "{{ mariadb|json_query('[*]')|select|flatten }}"
    tags: ['initMariaDB']

Using multiple vault passwords

To use multiple vault passwords to encrypt strings

  • Define all the passwords with vault_identity_list in ansible.cfg
  • Specify the label of the password to use with --encrypt-vault-id option when calling ansible-vault encrypt_string command.

Example ansible.cfg

[defaults]
inventory = hosts.yml
gathering = smart
fact_caching = jsonfile
fact_caching_connection = ./fact_cache
fact_caching_timeout = 60
retry_files_save_path = plays/retry
ask_sudo_pass = True
ask_vault_pass = False
vault_identity_list = dev@vault-pass-test.sh, test@vault-pass-test.sh
jinja2_extensions = jinja2.ext.do,jinja2.ext.i18n

; https://docs.ansible.com/ansible/latest/plugins/vars/host_group_vars.html
[yaml_valid_extensions]
defaults = [u'.yml', u'.yaml']

Example vault-pass-test.sh

#!/bin/sh

echo 5678

The commandline to use dev password (enclosing single quotation marks are not part of password)

$ ansible-vault encrypt_string --encrypt-vault-id dev 'abcd'

Nontrivial template samples using both Jinja and JMESPath extensively

#jinja2:line_statement_prefix: '%', trim_blocks: False, lstrip_blocks: False, keep_trailing_newline: True

{
{% set nodes = [] -%}
{%- for host in hostvars|json_query('*') -%}
  {%- set rippleds = host|json_query('ripple.[validators, trackers][]')|d([], true) -%}
  {%- for rippled in rippleds -%}
    {%- set proxy = host|json_query('haproxy.rippleProxies[?backends[?@.servers[?@.name==`' ~ rippled.name ~ '`]]]')|first or {} -%}
    {%- do rippled.update({'proxy': proxy}) -%}
    {%- do nodes.append(rippled) -%}
  {%- endfor -%}
{%- endfor -%}
"rippleds" : {{ nodes|to_nice_json(indent=2) }}
}
Advertisement