On the last salt-stack post we saw a first step by step usage of file.blockreplace. On this post we'll study usage of state accumulators with file.accumulated. Accumulators are used to collect data on several states and let you use this data on other file states (Actually only the blockreplace and managed states).
You can find the examples used on this post on this github repository.
Why using accumulated accumulators?
Let's start by the needs. What sort of problems could be solved by accumulators?. In fact, the idea is to use states executed before a final state. And this state ordering can be solved using requisites or orders. On theses first states you collect data for later usage. Then, on the final state, you have a dictionary containing this collected data, the accumulator dictionary, and you can do what you want with it with jinja, or even without jinja in the blockreplace case. Note that you only collect data in the current highstate execution, so all accumulators must run (be included) if you need the data at the end.
You can use this system to record one or more lists of things likes users, services, addresses, configuration commands or wathever else you want, while running the states. And all theses states recording data will have to run before the one using that data for something (writing theses lists on a targeted file). Doing that you will avoid doing the write operation after each new record, you will wait for the final list before doing the write. This will be faster, but also easier to manage. Inserting, updating and removing data in a file, in one shot.
In this post I will use two examples, one for a managed file and one for a blockreplace edited file. The first example is an apache configuration file with several states adding some inputs inside. The second example will fill a list of DNS IP-name associations that should be added in a hosts file.
States ordering
As I said before states ordering is very important here. The final step must be final. And used states may be split upon several sls files. You can use the order order keyword to ensure the final order, this way:
1 # first.sls
2 STATEID1:
3 cmd.run:
4 name: echo STATEID1
5 order: 100
6
7 # second.sls
8 STATEID2:
9 cmd.run:
10 name: echo last
11 order: 900
12
13 # third sls
14 STATEID3:
15 cmd.run
16 name: echo STATEID3
17 order: 200
This will make a final execution order of:
1 echo STATEID1
2 echo STATEID3
3 echo last
But to do that you need an ordering schema for all you states. I do not find that very useful when you have a lot of states.
The second way of ordering states is using requisites. Using this method you can declare dependencies between states. But having a list of all the states that collects data into accumulators, and eediting this list on the state using the accumulator would be quite hard. The best thing here is to use the *_in form of the requisites, to declare the dependency from the dependent state. When adding a new state using the accumulator the dependency will have to be set in this state and not in the previous managed state entry.
So using require_in the previous example would be:
1 # first.sls
2 STATEID1:
3 cmd.run:
4 name: echo STATEID1
5 require_in:
6 - cmd: STATEID2
7
8 # second.sls
9 STATEID2:
10 cmd.run:
11 name: echo last
12
13 # third sls
14 STATEID3:
15 cmd.run
16 name: echo STATEID3
17 require_in:
18 - cmd: STATEID2
Be careful: the syntax for a requisite is:
1 <requisite>:
2 - <module>: <state id>
It is not:
1 <requisite>:
2 - <module>.<function>: <state id>
So it is not cmd.run: here but only cmd:. And as soon as you start using requisites you see that using meaningfull states ids and not names shortcuts to declare states is quite important, for clarity at least.
file.managed using file.accumulated
All examples are available on github here.
In this first example we'll use several states, the last state will install a managed file in /etc/apache2/sites-available/100-foo.example.com, it's an apache virtualhost file, in the debian way. The jinja template associated with this file will contain the basic rules, but we want to allow some other states to add instructions on this virtualhost. We'll see theses states later. We start by the end, with this state building a virtualhost file in a example_com_apache_virtualhost.sls file:
1 apache-install:
2 pkg.installed:
3 - pkgs:
4 - apache2
5
6 100_example_com_virtualhost:
7 file.managed:
8 - source: salt://files/apache_vhost
9 - name: /etc/apache2/sites-available/100-example.com
10 - user: root
11 - group: root
12 - mode: "0664"
13 - template: jinja
14 - defaults:
15 - docroot: /path/to/www
16 - servername: example.com
17 - require:
18 - pkg: apache-install
This file should be put somewhere on your state tree, in this example I will make the function calls as if it were on the root of this tree (like the top.sls file). But there's also a source template for the managed file, which is called salt://files/apache_vhost, this file should be present under the salt master tree, in the directory files (or alter the used path). This is this very simple basic virtualhost example template content:
1 # Main Virtualhost for {{ servername }}
2 <VirtualHost *:80>
3 ServerAdmin foo@example.com
4
5 DocumentRoot {{ docroot }}
6
7 ServerName {{ servername }}
8
9 LogLevel info
10
11 <Directory />
12 AllowOverride None
13 Order allow, deny
14 deny from all
15 </Directory>
16
17 <Directory {{ docroot }}>
18 Options FollowSymLinks
19 AllowOverride None
20 Order allow,deny
21 Allow from all
22 </Directory>
23
24 </VirtualHost>
Let's test this simple state:
1 $# salt-call -linfo state.sls example_com_apache_virtualhost
You should en up with two states in success and a /etc/apache2/sites-available/100-example.com file created.
Now let's say we want other states, an infinite list of other states, to be able to alter this file and add content either in the main Directory section or at the end of the file. This way other states could add some apache configuration (this is an example, another way of doing it could be apache's include directive).
Here comes the accumulator jinja variable. It's a dictionnary, with several keys. Each key of this dictionary is the result of one or more file.accumulated states. So you may have this variable (or not) and it may contain some keys with text data inside. Let's see how to use this on the jinja template (and at first, we known it's empty, we did not used any accumulated state yet).
1 # Main Virtualhost for {{ servername }}
2 <VirtualHost *:80>
3 ServerAdmin foo@example.com
4
5 DocumentRoot {{ docroot }}
6
7 ServerName {{ servername }}
8
9 LogLevel info
10
11 <Directory />
12 AllowOverride None
13 Order allow, deny
14 deny from all
15 </Directory>
16
17 <Directory {{ docroot }}>
18 Options FollowSymLinks
19 AllowOverride None
20 Order allow,deny
21 Allow from all
22
23 # Here any extra configuration settings if any:
24 {% if accumulator|default(False) %}
25 {% if 'extra-settings-example-virtualhost-maindir' in accumulator %}
26 {% for line in accumulator['extra-settings-example-virtualhost-maindir'] %}
27 {{ line }}
28 {% endfor %}
29 {% endif %}
30 {% endif %}
31
32 </Directory>
33
34 # Here any extra configuration settings if any:
35 {% if accumulator|default(False) %}
36 {% if 'extra-settings-example-virtualhost' in accumulator %}
37 {% for line in accumulator['extra-settings-example-virtualhost'] %}
38 {{ line }}
39 {% endfor %}
40 {% endif %}
41 {% endif %}
42
43 </VirtualHost>
If you run the state nothing should move, except maybe the two comments lines. The thing we need to do now is to feed this accumulator variable with file.accumulated states. On theses states the name of the state will match the accumulator key.
So, for this example, we will use a second sls file more-things-for-virtualhost.sls :
1 {# Include dependencies #}
2 include:
3 - example_com_apache_virtualhost
4
5 example-a-first-rewrite-rule:
6 file.accumulated:
7 - name: extra-settings-example-virtualhost-maindir
8 - filename: /etc/apache2/sites-available/100-example.com
9 - text: |
10 # this is an example of thing added in the middle
11 RewriteEngine On
12 RewriteCond %{REQUEST_FILENAME} -d
13 RewriteRule ^(.+[^/])$ $1/ [R]
14 - require_in:
15 - file: 100_example_com_virtualhost
16
17 example-some-icons-added:
18 file.accumulated:
19 - name: extra-settings-example-virtualhost
20 - filename: /etc/apache2/sites-available/100-example.com
21 - text: |
22 # this is an example of thing added at the end'
23 Alias /icons /path/to/icons>
24 <Directory /path/to/icons
25 Order allow,deny
26 Allow from all
27 </Directory>
28 - require_in:
29 - file: 100_example_com_virtualhost
30
31 example-another-thing:
32 file.accumulated:
33 - name: extra-settings-example-virtualhost-maindir
34 - filename: /etc/apache2/sites-available/100-example.com
35 - text: |
36 # this is another example of thing added in the middle
37 RewriteRule ^/cgi-bin/imagemap(.*) $1 [PT]
38 - require_in:
39 - file: 100_example_com_virtualhost
40 - require:
41 - file: example-a-first-rewrite-rule
42
43 example-another-thing-again:
44 file.accumulated:
45 - name: extra-settings-example-virtualhost-maindir
46 - filename: /etc/apache2/sites-available/100-example.com
47 - text: |
48 # this is another example of thing added in the middle
49 <FilesMatch ".(gif|jpe?g|png)$">
50 ExpiresDefault A2592000
51 </FilesMatch>
52 - require_in:
53 - file: 100_example_com_virtualhost
Now let's run this new sls:
1 $# salt-call -linfo state.sls more-things-for-virtualhost
You should get a nice diff showing you that all theses states added content on the right place:
1 +++
2 @@ -21,9 +21,41 @@
3 Allow from all
4 # Here any extra configuration settings if any:
5
6 +
7 +
8 + # this is an example of thing added in the middle
9 +RewriteEngine On
10 +RewriteCond %{REQUEST_FILENAME} -d
11 +RewriteRule ^(.+[^/])$ $1/ [R]
12 +
13 +
14 + # this is another example of thing added in the middle
15 +RewriteRule ^/cgi-bin/imagemap(.*) $1 [PT]
16 +
17 +
18 + # this is another example of thing added in the middle
19 +<FilesMatch ".(gif|jpe?g|png)$">
20 + ExpiresDefault A2592000
21 +</FilesMatch>
22 +
23 +
24 +
25 +
26 </Directory>
27
28 # Here any extra configuration settings if any:
29
30
31 +
32 +# this is an example of thing added at the end'
33 +Alias /icons /path/to/icons>
34 +<Directory /path/to/icons
35 + Order allow,deny
36 + Allow from all
37 +</Directory>
38 +
39 +
40 +
41 +
42 +
43 </VirtualHost>
You can see that some extra spaces were added by my jinja control commands. We can strip down those whitespaces with jinja's-. Instead of:
1 {% if accumulator|default(False) %}
2 {% if 'extra-settings-example-virtualhost-maindir' in accumulator %}
3 {% for line in accumulator['extra-settings-example-virtualhost-maindir'] %}
4 {{ line }}
5 {% endfor %}
6 {% endif %}
7 {% endif %}
Use:
1 {% if accumulator|default(False) -%}
2 {% if 'extra-settings-example-virtualhost-maindir' in accumulator -%}
3 {% for line in accumulator['extra-settings-example-virtualhost-maindir'] -%}
4 {{ line }}
5 {%- endfor %}
6 {%- endif %}
7 {%- endif %}
And to get the right number of spaces on the resulting file use the indent filter:
1 {{ line|indent(4) }}
file.blockreplace using file.accumulated
Now if you read the previous post on file.blockreplace you may wonder how to use it with accumulators. It will differ a little from the file.managed usage of accumulators.
With file.managed you have this accumulator jinja variable and several keys inside. With file.blockreplace you have nothing to do.
- If one accumulator is targeted on the same file (the same as the one targeted by the blockreplace), then the blockreplace
contentattribute will be filled with all the lines contained on this accumulator. Any data directly set in the content attribute is not loose, accumulator data is only added, but the content attribute is not required so it could also be empty. - If several accumulators are targeted on this file they will be merged, but if you use several blockreplace states on the same file the accumulators are merged using the requisites dependencies you've made and accumulators names.
This last sentence is maybe weird. We'll make an example to see it, but another way of saying that is that it's magical and it should do the things you think it should do (if not, make bug reports).
So in this example we'll reuse the last example of managing entries in an /etc/hosts file. But we will manage two blocks of edition. So we have theses two states in an hostsedit_acc.sls file:
1 test-etc-hosts-blockreplace-services-local:
2 file.blockreplace:
3 - name: /etc/hosts
4 - marker_start: "# BLOCK TOP : salt managed zone : local services : please do not edit"
5 - marker_end: "# BLOCK BOTTOM : local : end of salt managed zone --"
6 - show_changes: True
7 - append_if_not_found: True
8
9 test-etc-hosts-blockreplace-services-central:
10 file.blockreplace:
11 - name: /etc/hosts
12 - marker_start: "# BLOCK TOP : salt managed zone : central services : please do not edit"
13 - marker_end: "# BLOCK BOTTOM : central : end of salt managed zone --"
14 - show_changes: True
15 - append_if_not_found: True
Same blocks as in the previous guide on blockreplace. But there I removed the content attribute (it could work with a content attribute adding more static stuff, but I do not need it).
So now if I want to use file.accumulated to push some content in theses blocks I just need to do two things:
- target the blockreplace state id in a requisite to ensure my current state will run before
- target the same file (name attribute)
Quite simple, but here we have two managed blocks, my accumulated content will be set in the block targeted by my requisite.
Let's try it in a second sls file: hosts_data.sls, but you could split that on more files if everything gets included at the end:
1 {# Include dependencies #}
2 include:
3 - hostsedit_acc
4
5 hostadata1-external-google-dns:
6 file.accumulated:
7 - filename: /etc/hosts
8 - text: |
9 8.8.8.8 ns1.google.com
10 8.8.8.4 ns2.google.com
11 - require_in:
12 - file: test-etc-hosts-blockreplace-services-central
13
14 hostadata2-external-thing:
15 file.accumulated:
16 - filename: /etc/hosts
17 - text: "93.184.216.119 : www.example.com"
18 - require_in:
19 - file: test-etc-hosts-blockreplace-services-central
20
21 hostdata3-internal-stuff1:
22 file.accumulated:
23 - filename: /etc/hosts
24 - text: "127.0.0.1 foo bar foo.local.net bar.local.net"
25 - require_in:
26 - file: test-etc-hosts-blockreplace-services-local
27
28 hostdata4-internal-stuff2:
29 file.accumulated:
30 - filename: /etc/hosts
31 - text: |
32 127.0.0.1 db.local.net
33 127.0.0.1 http.local.net
34 127.0.0.1 foobar
35 - require_in:
36 - file: test-etc-hosts-blockreplace-services-local
Note that I did not use the name attribute in theses states. Using name I could name the dictionary key, or we could say I would set the accumulator name. I could use names, but using the same accumulator name with the four states, strange things would happen, data from all theses accumulators would be merged in the same accumulator name. So, either avoid name attributes or use it with different names if the targeted blockedit is different. And test your recipes :-)
Now run it with:
1 $# salt-call -linfo state.sls hosts_data
You should get theses two managed blocks in /etc/hosts, filled from several states:
1 # BLOCK TOP : salt managed zone : local services : please do not edit
2 127.0.0.1 foo bar foo.local.net bar.local.net
3 127.0.0.1 db.local.net
4 127.0.0.1 http.local.net
5 127.0.0.1 foobar
6
7 # BLOCK BOTTOM : local : end of salt managed zone --
8 # BLOCK TOP : salt managed zone : central services : please do not edit
9 93.184.216.119 : www.example.com
10 8.8.8.8 ns1.google.com
11 8.8.8.4 ns2.google.com
12
13 # BLOCK BOTTOM : central : end of salt managed zone --
Last words
Note that this example is based on the development github repository of salt. You may not be able to run theses examples on versions prior to 0.18.0. The multiple blockreplace case is one of the last fix added by @kiorky. If you use accumulators you may need to subscribe on this reload_module vs accumulated issue on github. It's a quite general issue, but this actually prevents using accumulators on states with too much distance, as you may loose the accumulated data if something restarted the minion while your states are running.
