Controlling Multiple Linux Servers Simultaneously
There's many things I do to make my job easier. As my team and I manage around 25 Linux servers and multiple Linux desktops as well, we try to automate as much as we can. Some of that includes things like:
One of my other favorite tools is ClusterSSH. ClusterSSH allows you to open several SSH sessions at the same time in separate windows, and issue the same commands in all of them at one time. For instance, if I need to run an emergency security package update on all of my servers at one time I can launch ClusterSSH, tell it to connect to a list of servers that I've predefined, then type the commands one time and have it replicated to all of the servers at the same time.
Easy to Install in Ubuntu
It's easy to install just run:
sudo apt-get install clusterssh
After it's installed, launch it once from your desktop menu, then close it. It will create a config directory at ~/.clusterssh that we need. Now create a file in there called "clusters" that looks something like this:
gedit ~/.clusterssh/clusters
# My clusters allservers databases webservers applicationservers databases db1 db2 192.168.1.52 webservers web1 web2 web3 applicationservers app1 app2 app3
You'll notice the first line creates a cluster called "allservers" that includes the databases, webservers and applicationservers clusters. You can put machines in a cluster by either DNS name or IP Address.
Now, launch clusterssh again from your desktop menu and pick "Add hosts or clusters" from the Hosts menu. You can enter any DNS name or IP address, or one of the four cluster names we just created. If you enter "allservers" it will open a shell to every machine in that config file. If you enter "database" it will just open SSH to db1, db2 and 192.168.1.52.
Using the interface
Once you are connected to multiple machines, you can click on a single SSH window and enter commands directly to that server, just like you'd expect. However if you want to send the same command to ALL open SSH windows, type it in the input box of the CSSH window. You'll notice your keystrokes happen on all windows at the same time. This is handy, because if you run a command to all servers but they don't all react the same way - for instance one of the servers asks for some sort of response and the others don't - you can then set focus to a single window and make the necessary commands that would only apply to that window.
Why this can be extremely handy
Let's say there's a security update to the glibc package "libc6" that's critical you roll out immediately. You're pretty sure that all of your servers have already been updated but you want to audit this quickly on all machines.
Simply launch ClusterSSH and connect to all of your servers at the same time. Once logged in, you could enter the following into the central input window:
dpkg --list | grep libc6
At a glance you can see if any of the servers have an out of date package for libc6. You can even fix the issue on any affected servers with the open shell without impacting all of the other servers.
That's a powerful tool for system administrators that you might consider adding to your arsenal.