Configuration Management (with SaltStack)

Open development discussions

Moderator: Developer

Post Reply
ASX
Posts: 988
Joined: Wed May 06, 2015 12:46 pm

Configuration Management (with SaltStack)

Post by ASX »

[edit: kraileth]
We had an internal discussion about configuration management and if it would make sense to adopt it for GhostBSD. It comprises the technical aspects obviously, the usual passion for doing things right and a pinch of humor. Since it turned out to be pretty extensive, ASX suggested to make it public for anybody who might be interested in the topic (or to see what kind of jokes the team makes internally :twisted: ). I agree that this might as well be public and everybody is invited to participate in the discussion. Feel free to ask questions, too!

This post was the reply to myself posting a first take on a state file that utilizes iocage to create jails and the newly released rewrite caused me a bit of headache while doing so.
[/edit]



Honestly I'm not a lover of "wrapper tools" in general, I guess I would prefer to use the plain jails commands instead of ezjail, iocage and similar, however I'm not an expert on the matter.

It was my understanding that the iocage project is dead, instead its fork iocell is alivem or at least so I read on fbsd forum.

The other thing I don't like about this "do it me all" tools, is that they are fooled very easily: what if a 00_Synth.conf files already existed ion the target machine ?

What if on the same machine, because (synth) the FreeBSD was previously disabled ?

Beside that, yesterday I started to look into saltstack tutorials .... we will see. ;)
kraileth
Posts: 312
Joined: Sun Sep 04, 2016 12:30 pm

Re: SaltStack states

Post by kraileth »

ASX wrote:Honestly I'm not a lover of "wrapper tools" in general, I guess I would prefer to use the plain jails commands instead of ezjail, iocage and similar, however I'm not an expert on the matter.
Understand that. They offer convenience by hiding complexity but you're of course trading control for that. That's nice on the command line if you only want to quickly achieve simple things. For automation purposes I'd also prefer to remove unneeded "additional layers". The problem is that I know too little about jail management (I'm looking forward to the jails book that M.W. Lucas wanted to write sometime in the future!) and therefore using a wrapper is currently my only choice due to the lack of time to really dig into it. Yeah, I really should because a somewhat deep understanding never hurts - but there are simply so many other topics that need some love as well that I'm willing to make a compromise here...
It was my understanding that the iocage project is dead, instead its fork iocell is alivem or at least so I read on fbsd forum.
The old iocage (now refered to as "iocage legacy") is dead, yes. It was meant to be rewritten in go and because some people liked the shell approach, iocell was forked from it. The go port didn't happen, instead Kris Moore asked the author to go for Python instead because they want to use it over at iX for FreeNAS/TrueNAS to replace the old warden utility (and their product already comes with Python). So the new Python-based iocage is very much alive and they have some nice ideas. In the latest BSDNow show (before stepping down as co-host for time reasons) announced that they aim to go the "docker path" when it comes to simplicity. And I must admit that it would really help people who are not knowledgeable in networking and stuff to get their jails up and running if iocage manages the NAT part for you. That would be a nice, user-friendly feature. And once you understand the magic, you can always do without the tool.
The other thing I don't like about this "do it me all" tools, is that they are fooled very easily: what if a 00_Synth.conf files already existed ion the target machine ?

What if on the same machine, because (synth) the FreeBSD was previously disabled ?
That's not a problem per se. You have to plan for that. Salt offers a lot of functionality regarding files. Let me explain a bit of what this is all about (I wanted to do that anyway and after blocking some idiot I'm currently waiting for the load on a server to drop in the third value far enough to make the monitoring system happy again):

SaltStack works with .sls files written in YAML. They contain so-called states which define the desired state of some kind of resource on a system. One main benefit of this declarative approach over, say, some shell script which could do much of the same, is that you only tell Salt what to do but not how. This is extremely helpful in a heterogeneous environment when you can just say "ensure that htop is installed" and the tool knows to use pkg on FreeBSD, apt-get on Debian, emerge on Gentoo, ... This is not so important in our FreeBSD-only environment. However using some kind of configuration management allows you to centrally manage servers and do so even at a large scale. We'll soon have three machines. Certainly not too much to manage by hand. But let's assume we get two more or so - this will get tiresome quite fast. Even if you only use Salt to deploy a baseline configuration and choose to do everything else by hand, that's potentially a huge relief!

But what I've come to appreciate most is that this helps to keep servers in a consistent state. Something's not right? Run Salt. Now the system should behave correctly again (if the problem was due to something that a state was previously written for). And even if a system completely dies (say a VM with just one virtual drive). Not a problem: Clone a machine template, start up the fresh system, give it the correct hostname - and run Salt. Done. The resulting machine exactly matches the previous one and can assume the same role.

One requirement for that: DON'T do manual editing of the config files if you want to keep those changes! You can edit things manually ad-hoc or to debug things. But once you settle on a new configuration that you want to keep, you're expected to tell Salt (i.e. update the corresponding states). They will be kept in a Git repo (Salt can directly read them from there) which allows for one central point to store configuration that we can all pull from and push to. And it also automatically adds the benefit of version control for configuration management. Something's not right now? Take a look at the changelog and revert to a known good state. It's as easy as that! No more "###, that used to work yesterday before I changed something that I just can't remember the old value of!".

States consist of several things, the first one being an ID. The only requirement here is that it is unique which means that Salt is happy with using something like "aa", "ab", "ac", ... However it totally makes sense to use that ID to describe what the state is supposed to be. That way everybody who speaks English quickly gets an idea even with 0 Salt knowledge.

The next line (how I use it, it can be done differently) is telling Salt what the state is actually about. Does the desired state mean a file may need to be altered? A package installed? A command run? The important thing here is idempotence: A routine is idempotent when it it guaranteed to produce the same result no matter how often it is run. This is a hard requirement for Salt being able to be run again and again without messing up things that are already in the desired state. If you are making use of pre-defined internal states, this is nothing to worry about as it is being taken care of for you. Example: "Salt, please make sure that user asx is present on the system and can login via SSH". Salt knows how to do that. But it also recognizes if that's already the case and won't do anything in that case. A script on the other hand needs checks written for everything or, if it's a dump quick script, tries to add an existing user again, etc. When using Salt to run commands, you have to take care of the idempotence yourself, though. Fortunately Salt offers some friendly functions which make this easier.

In case of files, you'd use a state provided by "file". Which state to use depends on what you want to do. If I want to configure a system to allow users to use passwordless sudo, I can use file.uncomment to have Salt remove the comment sign before the "# %wheel ALL=(ALL) NOPASSWD: ALL" line that I know is in sudoers by default (I just have to write other states to depend on that ensure the user is present and in the wheel group and that sudo is installed). If I want to add anything to /etc/rc.conf, I'd probably use file.append. Salt is smart enough to only act if the desired line(s) is/are not already in the file. Often you'll probably use file.managed which allows for several things; you could define the file contents (which only makes sense for short files) or simply point Salt to an actual file that you want to be placed on the system. Salt has an internal file server for that purpose and the files can also reside in a git repo, thus be versioned as well. You also have a lot of choices: "Salt, please put that file there - but only if a file by that name is not already there!", "Salt, please compare that hash first to see if the file is already in the desired state", "Salt, please make sure that the file is owned by root and has permissions 0644", etc.

So about your examples: The state for Synth's configuration could be something like "Salt, this is the base configuration for Synth. Please put it there if it doesn't exist. If there already is a file of that name, leave it alone!". This is possible if Salt is to be used for one-time deployment only (a perfectly legit use and probably the right solution for your second example). However the better way (for the first case) is (IMO) to actually never modify the file on the server directly (or only do so temporarily for tuning purposes and such) but instead make a change on your local workstation's git clone of the configuration repo, commit, push and then run Salt. This is the cleanest way of doing administration. Every change is documented (by git). If something goes wrong, you can always go back to a known good state. If somebody who usually does something is not available (at work, on vacation, left the project , ...), somebody else can at least take a look at what has been done in this regard. This can also be very valuable.
Beside that, yesterday I started to look into saltstack tutorials .... we will see. ;)
SaltStack's Getting started is actually a pretty worthwhile read. Note however that Salt can potentially do a ### of a lot more than we'd probably ever need. I basically want to use it for configuration management. It can also do orchestration, agentless runs over SSH, offers the "event reactor" (which I did not even bother to take a closer look at, yet) and can even be used for sophisticated monitoring and things like that.

Well, the monitoring aspect could in fact be interesting for us some time in the future. Knowing when your services are down is always a good thing after all! ;) And since Salt knows what state the various system resources should be in, it can also be used to monitor those. In fact using the reactor I think some kind of "self healing" could even be implemented ("Hey, that machine's web jail is not running but my state says it should! Better try to start it up again!").

This whole topic is huge. However while I want to concentrate on a few rather simple things, I also wanted to let you know how much potential this thing actually has. And the basics are fortunately rather simple. I hope that I'll be able to convince you of the benefits of using a system like Salt. I won't give up easily, but if I really can't, I'll propose Puppet/Chef/Ansible! :lol: While I don't like especially Puppet much, it's still just so much better of doing stuff manually like in the 90's... So you've been warned. You better like Salt. :P :mrgreen:
ASX
Posts: 988
Joined: Wed May 06, 2015 12:46 pm

Re: SaltStack states

Post by ASX »

You must be looking for a career as book's author. :)

Actually I read the "getting started" part of saltstack, not completely, but enough to get an idea about what it is about.

But is seems you misunderstood the aim of my questions, which are not specific to synth, were just an exemplification of the fact that by "automating" such tasks you might overlook something else.

On a completely different industry, aviation, airplanes and pilots, there exists a similar issue, that in the latest years is becoming increasingly worrying: it happens when autopilot disengage (for whatever reason) and pilots sometimes are unable to recover the flight control.

A phrase (too often) recorded in the CVR (voice recorder) is: "what is doing now ?" referred to the airplane computer systems.

What it means, it that by relying too much on computer controlled flights, many pilots have lost the capacity to "fully" control the airplane, when that happens the final result is usually very unpleasant.

Now, automation is great thing, I agree, but losing control is not, that is what my worries are about;
secondarily, by the time you will be able to use saltstack properly, you may find it would have been faster and simpler to have done the task by hand.
We'll soon have three machines
We are not there yet ? :mrgreen:
kraileth
Posts: 312
Joined: Sun Sep 04, 2016 12:30 pm

Re: SaltStack states

Post by kraileth »

ASX wrote:You must be looking for a career as book's author. :)
How in the world were you able to guess that? :shock: No, seriously, I used to write fantasy stuff but I haven't found the time to do so for about 2 years now. However I like to explain things and to argue about them now and then. ;)
Actually I read the "getting started" part of saltstack, not completely, but enough to get an idea about what it is about.
That will most likely be sufficient. Getting the idea will let you form a decision. The implementation details are pretty irrelevant in that.
But is seems you misunderstood the aim of my questions, which are not specific to synth, were just an exemplification of the fact that by "automating" such tasks you might overlook something else.

On a completely different industry, aviation, airplanes and pilots, there exists a similar issue, that in the latest years is becoming increasingly worrying: it happens when autopilot disengage (for whatever reason) and pilots sometimes are unable to recover the flight control.

A phrase (too often) recorded in the CVR (voice recorder) is: "what is doing now ?" referred to the airplane computer systems.

What it means, it that by relying too much on computer controlled flights, many pilots have lost the capacity to "fully" control the airplane, when that happens the final result is usually very unpleasant.
Ok, got you. To be completely honest, I'm a bit of a conservative guy myself and maintain a healthy scepticism about a lot of "modern" approaches. I wouldn't feel safe in a self-driving car even if studies "prove" that it's actually safer. And friends call me a radical because to this day I have never owned a mobile phone (don't need one and don't want one). I also witness that the young generation often cannot drive anywhere without a route guidance system. This is horrible. But it does not mean that I condemn those navigation systems. My stance on that is: You may use them if you could do without them. You can read a map? Excellent, go forward, use the navigation system if you wish. Same thing e.g. for pocket calculators in school. I'm ok with them - once the students have understood enough of basic math that they could also solve their schoolwork without them.

I'm really glad that I started using computers in times of DOS. Most of my classmates back then got their first PC with Win98 - and when it didn't boot up and they had no mouse, they were screwed while I could fix things. There's a huge benefit to that. However we live in a complex world and I accepted that nobody can master every topic. I couldn't fix my car if something breaks, so in that case I rely on others. But I'm fine with that. Other people are bad with computers. That's ok.

Still I do agree with you that the tools we have today bear risks. I've seen people specialize in tools but having no understanding at what actually happens in the background. That's pretty scary. Or the typical (?) Windows admin - a lot of those guys are not even aware what a kernel is and what components an OS consists of. Now I don't think everybody has to be a skilled assembly hacker before being allowed to do administration work. Of course your point remains valid: Hiding complexity is convenient but potentially keeps people from gaining a (somewhat) deep understanding of whatever topic. However my answer to that is: That doesn't apply to everybody. Those who are lazy don't care in the first place (and that's probably ok). The others who show interest in what happens behind the scenes will still strive to lift the curtain - and of course they can.

Now, automation is great thing, I agree, but losing control is not, that is what my worries are about;
secondarily, by the time you will be able to use saltstack properly, you may find it would have been faster and simpler to have done the task by hand.
There's a lot of problems with doing things by hand some of which are:
  • It doesn't scale.
  • We're not Amazon, Google or Microsoft, sure. Three servers [i]can[/i] be managed by hand without too much trouble. But what if the project really takes off and we need more servers around the world? Or we're moving from one server to another one? If the states already exist, even a complex system with many roles can be rebuilt on the new machine within minutes. Another case: You're in Italy so I'd imagine that there wouldn't be too much difference in routing. Have you ever worked on a server in China? It's a pain. Always reminds me of my old 80286 where I could type faster than the PC could process the input... SSHing into the thing takes a moment and after you type in a command you need to wait again. I have no motivation whatsoever to type in a hundred commands remotely to do configuration... Writing the states locally and just running Salt and comming back to that tmux window twenty minutes later to see if everything is ok really feels like deliverance!
  • It demands a lot of resolve
  • Undocumented changes are... potentially problematic to say the least. Manually changed something on the server? Documenting things sucks, it's boring. But: [i]always[/i] force yourself to document it properly! You [i]won't[/i] remember exactly what you did in a month or two (if you [i]do[/i], hats off to you! :P). You also need to manually backup stuff because when you are experimenting and break things, you need to be able to quickly go back to a working state. Doing manual administration [i]right[/i] is also wasting time because if you compare it to the automated ways those are [i]self-documenting[/i].
  • It is error prone
  • Let's assume we're talking about [i]super admin[/i](tm) here who never makes any mistakes (everybody else does). Even then trouble is ahead if there's a second admin in team because if both are doing changes, they are bound to be stepping on each other's toe somewhen. Communication is essential to avoid that. This again takes time - and frankly speaking, it's hard enough to communicate everything to people in the same office... And since we're talking about an open source project with team members from around the globe who don't speak the same native language, live in different time zones and such, this [i]is[/i] a problem. Working on e.g. state files in the same repo makes it easy to take a look at what others did since you last changed something. Good commit messages help a lot.
  • You repeat yourself quite often
  • It's not easy to re-use anything except you start scripting things - which is a (more primitive) form of automation again. Let's assume that because of his own stupidity a team member, say kraileth, has lost the passphrase that he uses for his SSH key (I've been there...). I have to mail the new public key to somebody else and he has to log into every server, delete the old file and place the new one there. Perhaps on one machine he introduces a typo, placing the new key in authorizd_keys. We won't know until I find out that I cannot log in. The other way I'm pushing my new key to the git repo and ask whoever is available on IRC to just start a salt run on all servers. Done.
  • Subtle differences are quite likely
  • When you set up a new webserver after you haven't done that for over a year, it might end up providing the same functionality as the other one(s) in the end. Again it will take you more time to do because you don't do it regularly. If the new webserver is, say, for a development environment, even minimal differences are known to bite you when you transfer stuff that worked perfecty in [i]dev[/i] over to [i]prod[/i]. If on the other hand you built the webserver with configuration management - just assign the new server the "webserver" role and run salt to create an [i]identical[/i] setup.
Initially it takes longer to create proper states for Salt, that is true. But it's not that bad, really. And after working with configuration management for some years now (mostly my not exactly beloved Puppet, though), I can confirm that it starts paying off immediately in a big environment. However even for a small one like what we have, I expect it to pay off rather soon, too.
We'll soon have three machines
We are not there yet ? :mrgreen:
:P
ASX
Posts: 988
Joined: Wed May 06, 2015 12:46 pm

Re: SaltStack states

Post by ASX »

Wonder if this tread should be on the public side of the forum ... look like an interesting discussion.
ASX
Posts: 988
Joined: Wed May 06, 2015 12:46 pm

Re: SaltStack states

Post by ASX »

It doesn't scale.
I wish we had a scale problem, we don't.

For the other things, yes, your reasoning does have some merit.
And friends call me a radical because to this day I have never owned a mobile phone
Can't blame your friends. :o :o :o
kraileth
Posts: 312
Joined: Sun Sep 04, 2016 12:30 pm

Re: SaltStack states

Post by kraileth »

ASX wrote:Wonder if this tread should be on the public side of the forum ... look like an interesting discussion.
That's true, maybe somebody else cares as well. For the posts that don't contain code, I wouldn't mind to make them public. Haven't tried moderation utils in this forum, though, but I'd give it a try.
ASX wrote:
It doesn't scale.
I wish we had a scale problem, we don't.
Not currently, no. But who knows...?
And friends call me a radical because to this day I have never owned a mobile phone
Can't blame your friends. :o :o :o
Haha! Forgot for a moment that Italians have quite a special relationship to phones. :mrgreen: In my country it's usually seen as just a tool and today even quite some people wish they didn't need one. But hey, I cannot talk nearly as fast as your countrymen (and you, too, without any doubt) can, not even in my own language. Perhaps a good indication that I don't have as much to tell. Maybe that's a reason why I don't need a mobile! :lol:
ASX
Posts: 988
Joined: Wed May 06, 2015 12:46 pm

Re: SaltStack states

Post by ASX »

kraileth wrote:
ASX wrote:Wonder if this tread should be on the public side of the forum ... look like an interesting discussion.
That's true, maybe somebody else cares as well. For the posts that don't contain code, I wouldn't mind to make them public. Haven't tried moderation utils in this forum, though, but I'd give it a try.
Personally I have no problem with any thing I posted, however exercise your moderation skill and fill free to cut out or even delete irrelevant things.
Haha! Forgot for a moment that Italians have quite a special relationship to phones. :mrgreen: In my country it's usually seen as just a tool and today even quite some people wish they didn't need one. But hey, I cannot talk nearly as fast as your countrymen (and you, too, without any doubt) can, not even in my own language. Perhaps a good indication that I don't have as much to tell. Maybe that's a reason why I don't need a mobile! :lol:
True that Italy was a country where mobile phone has been loved, but that's history now.
As for the mobile phone, I can assure you I use it just as any other tools, only that I found it particularly useful at times, expecially when travelling, which I did a lot in the past.
Post Reply