Archives

All posts for the month September, 2009

V. ENERGY
1. Sun Tzu said: The control of a large force is the same principle as the control of a few men: it is merely a question of dividing up their numbers.
2. Fighting with a large army under your command is nowise different from fighting with a small one: it is merely a question of instituting signs and signals.

Energy can be considered another word for efficiency. When it becomes necessary to scale things up, there are two factors which will ensure you lose as little efficiency as possible.

The first factor is how you break the smaller scale model into separate tasks or functions. For example, you may have an application which currently utilizes one or two servers with each of those servers handling several roles. A common model is to have a Web/application server at the front with a database/storage server at the back. As you scale up this solution, you will need to split out the Web service, application (there may be sub-roles here), database and storage roles . As scale increases even more you will most likely end up with many load-balanced Web front end servers; a number of application servers to support the front end and any application sub-roles; and a database back end with query, engine and storage roles residing on separate hardware.

The second factor is how you co-ordinate the communications between the various servers involved. If you started with an optimized, efficient small scale model, then this is simply a matter of mimicking calls that used to be local as network calls to the correct processes on the remote servers. If, however, your small scale design was rife with inefficient calls, that will translate to great losses in efficiency when you try to send those calls across the wire to other servers.

Therefore, it is essential that you take time to maximize efficiency on your smaller scale implementations prior to attempting to scale them up. Make sure to design your application with modules that can be separated later. Pay close attention to the communications which take place between modules that could be split out to different servers later. With your application broken into logical, separate modules which transfer only a minimum amount of data to perform their tasks, scaling up will be much easier.

Studying for LPI certification this evening, and I had a fit of whimsy whilst playing with regular expressions. Random thoughts passing through my head resulted in the following combination of “memes”:

  • Swear words in the linux kernel source code.
  • Britney Spears’ song, “If You Seek Amy.”

So I downloaded the source code (linux-source-2.6.28, Ubuntu), unzipped it and ran the following grep command against it:

grep -ri "<fuck me>" *

Amazingly, despite the prevalence of the just the swear word itself (33 variations including fuck, fucked, fucking, fucker, etc in 2.6.28) in the kernel, there was only one hit:

fs/binfmt_aout.c:               /* Fuck me plenty... */

So I guess a more apt title for this post is, “If you seek Amy Plenty.”

IV. Tactical Dispositions
1. Sun Tzu said: The good fighters of old first put themselves beyond the possibility of defeat, and then waited for an opportunity of defeating the enemy.
2. To secure ourselves against defeat lies in our own hands, but the opportunity of defeating the enemy is provided by the enemy himself.
3. Thus the good fighter is able to secure himself against defeat, but cannot make certain of defeating the enemy.

This section brings to mind the common phrase, “low hanging fruit,” which is often used in business and political circles to refer to easy targets for adding new business or reforming policies. What is often missing from those discussions, however, is the question, “How high are we capable of reaching?”

When cutting the budgets of time, resources and money for a project, the goal can quickly get out of reach. As the time allowance shrinks, you leave less room for planning which can cause a higher margin of error — thus requiring more time to fix. As resources shrink, either more time will be needed or more resources will need to be brought in near the end — most likely at greater cost. As funding is removed, the ability to respond to unforeseen issues (hardware failures, natural disasters, personnel issues, etc.) is greatly diminished and will result in the need for emergency funding.

Keep in mind, too, that the height of the low hanging fruit is relative to your own capabilities and the capabilities of your competition. Do not take on a project which, once initiated, could easily be taken over by a more capable competitor. At the same time, be on the look out for smaller competitors who have taken on more than they can handle.

The fruit which hang low today may be out of reach tomorrow. Be prepared to reach as high as you can, but hold back the temptation to reach too high — even if the project is just within your range.

Sun Tzu wrote:

III. ATTACK BY STRATAGEM
18. Hence the saying: If you know the enemy
and know yourself, you need not fear the result of a
hundred battles. If you know yourself but not the enemy,
for every victory gained you will also suffer a defeat.
If you know neither the enemy nor yourself, you will
succumb in every battle.

We assume that we possess an accurate understanding of our own skills and capabilities. Many people, however, tend to overestimate their own performance, skills or capabilities. When they do, they make a fatal mistake which guarantees they will never see success.

Take some time out of your busy schedule to assess and re-assess your own skills and capabilities. An excellent way to do this is to review and update your current resume at least once a year. That will represent your view (however accurate) of yourself. Once you’ve updated this document, make sure to seek honest feedback from your colleagues. Hand them a red pen and request they be as brutally honest as they dare. Promise to take their feedback seriously and without retribution. Their honesty (should they wield it) will certainly be difficult to swallow and could possibly shatter your self esteem, but you will be better off for the experience.

Next, be sure to obtain as much information about potential projects and teams with which you work. If your skills do not benefit the team or will not advance the project, you should bow out. You will have only a 50/50 chance of success and may end up damaging your standing with others. Beware that some people may misrepresent the scope or requirements of a project. Be sure you always set up an exit path to avoid those sorts of situations.

With a solid grasp of your own capabilities and enough research prior to taking on any project, you will succeed far more times than you fail.

Here are some hints and tips for those who are new to using ssh/OpenSSH for Linux system administration. Most of these tips have come from my recent work with a large number of Linux servers hosted on a VMware ESXi 4.x server farm.

Password authentication VS ssh key authentication

  • If you are administering only a few systems on a closed network (i.e. accessible only locally or by a secure VPN connection), then password authentication is probably OK, but you should consider using ssh keys anyway.
  • If your network needs to allow ssh access directly from the Internet or you are administering a large number of systems, then you should definitely use ssh keys.

Ssh-agent, scripting and cron

  • ssh-agent can save you typing in the password to your ssh key every time you need it.
  • This site gives a good overview of ssh-agent and includes some code you can add to your .bash_profile script to ensure your keys get added upon login.
  • Although there are hack-ish ways to get ssh-agent and cron to work together, you are probably better off setting up special keys to use with scripts that must be called via cron. Just keep in mind that keys without passwords are a security risk.
  • If you cannot risk using keys without passwords, consider running those cron scripts locally on each system. Utilize shared file space or e-mail to collect the results.

Bash one-liners and ssh with ssh keys

  • I’ve become a fan of using bash “one-liner” scripts to keep abreast of server stats such as load averages, available patches and disk usage.
  • Keep an up-to-date list of hosts in a file called hostlist.
  • Run your one-liners while ssh-agent has your ssh keys cached.
  • Here’s a template one-liner which checks uptime on each host listed in the file hostlist:

for e in `cat hostlist`; do echo $e; ssh $e "uptime"; done

  • In the above example, you can replace uptime with just about any command which exists on the remote host.
  • You can also synchronize some of the configurations under /etc with the above by utilizing either scp or rsync instead of ssh in that one-liner.

Turn your one-liners into scripts

  • If you find yourself using the same one-liner over and over, it is time to save yourself some typing and turn it into a script.
  • I like to keep these sorts of scripts under ~/bin. I also like to add that to my $PATH and create a simlink ~/scripts.
  • Some one-liners are good candidates to be turned in to cron scripts. Just keep in mind the risks of using ssh keys without passwords, and include logic to detect conditions you want to monitor. For example, you can run /proc/loadavg through awk to isolate one of the three figures and send yourself an e-mail if that average is too high.