Occasionally I do something stupid. I know, I'm a freakin' genius, but sometimes I goof.

Today was one of those days. I was importing a test batch of data containing about 30,000 pages into a new system. I realized after triggering the import that I had a bug in my code so I killed the process and then deleted the temp data.

What I didn't realize was that the data had already started the import process on the new machine. On the importing test machine it's configured to notify via email if something goes wrong. All of those notifications have a threshold that says if it goes over 10 warnings in one batch to kill the process entirely.

All notifications except one.

Net result: Over 25,000 email notifications.

So, here's how to properly clear a specific set of emails from a Postfix queue:

for i in `mailq|grep 'sender@' |awk {'print $1'}|grep -v '@'`; do postsuper -d $i ; done

Replace "sender" above with the first part of the email address of the sender, such as www-data, root, spambomber, whatever.

This past week I stumbled on a little bug in Ubuntu Oneiric that would bring a PHP web server to it's knees.

There's a simple temporary fix, that requires editing a single file on an affected server. I decided to deploy the fix with Puppet, then I realized that since the file wasn't directly deployed by puppet that it wasn't as easy to manage that file.

The trouble was that I only wanted to change the file if the distribution is Oneiric and the file exists. The file exists if you install LAMP (Linux, Apache, MySQL, PHP), but there are certain instances where that file may be installed without Puppet being the one to install LAMP, so I didn't want to tie the fix to my LAMP setup script.

Rather than breaking my LAMP install into multiple parts and complicating things more I set out on a quest to find how to replace a file using Puppet, but only if that file exists to begin with on the target system.

The solution I found isn't ideal, as Puppet doesn't have a way to directly test for a file that it's not already managing as part of a recipe. Instead you use an exec command.

To keep the exec command from running every single time Puppet runs, I have it create a file called /root/fixedphpcron that lets the Puppet script know that it's already done this.

Here's a problem that has driven me crazy over the past year or so.

First, the Way it's Supposed to Work
In Ubuntu, you don't typically have a "root" user password. If a user needs to do something that requires administrator rights, they are placed in the "admin" group and when they try to say - edit the global network settings - it asks them for their password and uses "sudo" to do whatever they want.

Where it Goes Wrong
You'd think it decides this based on who has "sudoers" permission. Unfortunately, it doesn't. Enter LDAP authentication. If I have an LDAP group called "Domain Admins", and that group is in the "sudoers" file giving full access to root permissions, as far as anything in Ubuntu that uses the PolicyKit infrastructure is concerned, users in that group don't have administrator rights directly. It will ask those users for the system root account password, even though they could simply run the same command from a prompt prepended with "sudo" and it would work fine using their password.

How to fix it

I have a private, non-Internet connected server that runs an application. I point out that it's not Internet facing because it's not running in safe mode due to some of the things the app needs to do.

One of the things some calls will do is cause it to fire off a command line app using exec().

For some reason things started failing today, and I noticed the following in my logs:

PHP Warning:  exec(): Unable to fork [someapp] in somefile.php on line 177

My main symptom - none of the web requests to it were even hitting the Apache logs, and the server would not return a response at all, it would just close the connection.

It's strange, because this and 5 other servers just like it have been humming along just fine without a hiccup. In fact this particular server has an uptime of 386 days.

The resolution to the error? Restart Apache. I probably would have restarted the entire server, but I'm actually in the process of migrating that data to a new application server so there's no pressing need.

I usually don't take those stupid online tests. Well, okay sometimes I take them but never share the results because they're stupid. This one hit home though. And on the first try, too:

With a recent story that Apple and five e-book publishers are being warned by the DOJ that they may be investigated for fixing ebook pricing, I thought it warranted a look at the publishing models.

According to ehow.com, the average author receives an income of about $4,485 per book.

Total, on average.

If you run a firewall you know how important it is to block out the bad guys.

Many people have taken to blocking out certain country netblocks (say, China for some reason) at their firewall. Of course there are ways around this for the determined attacker, but it's a good first line of defense.

A better approach in security is to whitelist rather than blacklist. I try to always use a policy of "block everything, allow through certain things". You can never think of everything to block so a policy of "allow everything, block certain things" just isn't good practice.

The same is true of IP address blocks in my opinion. In my business there is absolutely no reason for anyone outside of the USA or Canada to talk to my servers. I'm sure there are millions of people in China (or Europe or Japan for that matter) who are really nice and would never try to attack or spam me, but why should I even give them the chance if there's no legitimate reason for them to contact my server?

Managing a blacklist of a huge volume in iptables is a problem. Even managing a small whitelist is a problem.

With today's digital devices using larger and larger memory cards, particularly with cameras it's a good idea to put new cards through the wringer before you use them. Better to find out before shooting your kid's birthday party that the card is bad, rather than finding out during, or worse, after.

With Linux, there's a free tool called "F3" that is just the ticket. It writes files to the card to fill it up entirely, then reads those files back to ensure that not only is it the right file, but it was in the same physical block on the card. This will tell you if you've found a counterfeit card or even one that just isn't very reliable.

Before I put a new 32 GB compact flash card in my camera and go shooting, I make sure the card holds up with this tool. It takes some time, but it's simple to use. Just make sure to read the README file in the archive.

I just love saying that... "Gluster Cluster". GlusterFS is a networked file system that can automatically provide redundancy and failover across multiple servers. Essentially it's RAID or mirroring for server storage. You don't want to use it for small files - like a web server - but for network storage in an enterprise environment.

Setting up Gluster is actually really simple. It has one requirement in Ubuntu which is that you install "nfs-common". I actually use Puppet to deploy the core GlusterFS system on servers and clients automatically.

Once a Gluster server cluster is running, a client mounts a share pretty much like any other network filesystem. The initial mount requires that the server in the mount command be available, but after the mount is made, that server could effectively go offline and data still be available to the clients through the other "bricks" in the file sharing system. A brick is a physical server that provides a GlusterFS share. It takes at least two bricks to provide a redundant Gluster filesystem.

Here's a little tip for you.

If you run Ubuntu and decide to make a quick backup of your home directory using cp -R -d, make sure to "eject" any network shares you mounted using the file browser.

I just found out that despite the -d (which doesn't go inside symbolic links but has no affect on nautilus mounted shares) it was trying to back up a mounted network share... that contained about 15 cd-rom images of various versions of Ubuntu. That might take a while.