reverse engineering things that predecessors left without any documentation and throwing them out the window because devops

Salt Grains are really just descriptions of various pieces of the operating systems’ information on the minion.

As SaltStack documentation states, they basically provide you with “grains of information”. This allows you to grab information about the minion such as CPU information, underlying hardware (provided to you by dmidecode), network interfaces, and operating system revisions.

But wait, there is more! They don’t just display OS information! Grains can also be set as custom grains and you can place any sort of information that you want. You can set environment variables, system owner info, host group information – basically anything that you feel that might be useful to use as part of your Salt formula logic that can be parsed in the pillars or state files.

Each minion comes with pre-defined grains that are generated upon minion setup. You can assign custom grains and keep them in /etc/salt/grains or your minion config file. There are also existing auto_sync reactors that you can use to sync custom grains from a pillar (key-value store) to minions upon authentication or minion startup.

Besides using minion ID for minion targeting (which in itself isn’t a bad idea in secure environments), you can use these grains to target hosts in your salt CLI commands, states, and pillars. This empowers the SaltStack engine to create configuration templates for various environments on your network and to only affect those that you want. It’s also great if you’re shoehorning SaltStack into an existing, highly segmented environment.

You can view existing grains for a minion with the following command:

With that in mind, you can specify a grain and its value and execute an execution module against a set of minions with that matching grain.

Here we will install a package called nmap-ncat on all systems that are from the Red Hat family.

With -P flag, you can use PCRE (perl-compatible regex) on grains as well. This is described further in SaltStack’s Compound Matchers documentation (a highly recommended read in itself):

You can utilize grains in pillars and states using the following Jinja templating syntax. Jinja is a templating engine that is default in Salt and various web frameworks (such as Flask). It uses python-like syntax but is a bit more limited in terms of conditional tests that it can perform (unless you write your own filters).

Grains are parsed into salt as a dictionary of values. So in python-speak, a grain will look like this:

Multiple grain values will be stored as a list inside of a dict. In this case, it’ll look like this:

So now comes a question, what if you want to execute some state against a minion which may have a list (of IP addresses) for its ‘ipv4’ grain? Checking the equality with ‘==’ (equal) sign will not work since the grain contains a list. Jinja renderer will start throwing errors. You can’t use ‘.startswith()’ or ‘.endswith()’ string methods either.

Well, you could check if a particular string is in a grain:

This should parse through the entire lists in ‘ipv4’ grain. You could also match on the first items in the ‘ipv4’ grain if you choose so by adding “[0]” at the end of the salt[‘grains.get’]() method:

Or you could loop through each item in the ‘ipv4’ grain. Beware that this might cause problems since state/pillar IDs must be unique. You might need to affix a suffix at the end of the key to keep it unique:

Iterating through lists of items in singular Salt grains does pose a challenge and each use case may have different requirements. Due to limits in Jinja, you may have to come up with a hackish solution that is hopefully pythonic and doesn’t introduce race conditions when deploying states/pillars.

As you can see, Salt grains are fun! They’re not just boring static blobs of information. They can really bring your SaltStack environment alive and you should feel encouraged to use them wherever you can.

At first Pillars were something that I could not wrap my head around due to confusing Salt documentation. Eventually the docs became better and with working in Salt extensively, the pillars finally kicked in. They are something that any experienced Salt user should be using as they open up more doors in your DevOps infrastructure.

Pillars are basically a secure key value store in Salt that can be used by Salt States. You can store confidential information and options in pillars instead of keeping them in configs and the pass this data to specifically assigned minions (which you can do in pillars/top.sls file).

There are two main ways of getting a value from a pillar in your config file:

The ‘default value’ in first example can be substitute with an actual default value that you want to be filled in case your pillar dict is not found. In the second example, Salt will throw out Pillar SLS errors in case the pillar is not found. Clearly when you can store default values, using the first example is better.

To assign pillars to specific minions, you can do the following:

In the above example, we are assigning “grains” pillar to all the hosts (via “*”). All hosts matching a grains PCRE (P@) compound matcher will get nginx and java pillars. Finally, any hosts that have a grain that matches “specialgrain_match” will get the “special_state” pillar.

Official Salt docs on installing Salt on Solaris are quite out of date so this article is meant to illustrate how you can install SaltStack minion package on Solaris 10 (x86 and SPARC) hosts. Just like the official docs, we will be using OpenCSW but Salt itself will be installed using PIP. We will need to install the following application stacks and their dependencies:

  • pkgutil
  • python2.7
  • py_pyzmq
  • py_m2crypto
  • py_crypto
  • py_pip
  • salt

Start off by installing the OpenCSW’s pkgutil application:

Continue reading ›

SaltStack has a pretty cool feature where you can manage your systems’ crontabs from a central location. You can keep them in state SLS files and look them up via salt cron exectuion module. With a little bit of extra work, you can also keep the crons in pillars and feed them to your systems (that’s something that I will certainly have to experiment with that later on).  Even with base functionality, this is a pretty cool thing that should help you with truly automating your environment to the point where you will rarely have to log into a server to perform management duties.

Let’s set up a basic crontab in Salt and distribute it to one of our minions just to get a basic idea how this all works.

Create a directory for your cron and an empty cron SLS file:

Add the following to your testcronjob.sls file:

This will execute “bash /opt/scripts/” command every 2 minutes. You can also specify standard UNIX timing values via:

By default, if you don’t specify a timing value, Salt will implement * for each value in the schedule.

You can implement the cron via:

And you can view it by:

You can find more information about managing crontabs in Salt below::

Multi-master SaltStack setup is quite easy to build out. There is no need for VIPs or DNS CNAMEs (though they can be implemented) and all of the functionality is handled by Salt.  This greatly simplifies everything and you don’t have to rely on external tools.

To have working masters, you need to keep the a couple of directories in sync. You may use clustering filesystems or rsync to do that. In this example we will use rsync which is more than enough. With some extra ingenuity, you can even automate this sync process to happen automatically.

This howto describes how to do this on CentOS, but the setup should be the same on any other OS (such as FreeBSD).

Continue reading ›