Keeping your server alive with Monit

NOTE: This is super web/development/sysadmin stuff, casual non website people should probably check this instead

Lately GV has been pretty out of control and our server has been crashing way to regularly due to too many visitors or bots. I’ve been working to find all the little holes in the Apache/PHP/MySQL configurations that are causing the crashes when load gets high, but it’s impossible while you’re constantly putting out fires and restarting the servers manually.

monit logoI’ve been having frustrating fun with a tool called Monit that helps stop your server from completely crashing by watching it’s system stats and selectively restarting processes or executing whatever command you want. It installs pretty easily on Linux servers (I think it’s in both Yum for CentOS/RH and in apt for Debian/Ubuntu) and it uses text files similar to Apache to set up different status conditions and what to do. The configuration took me awhile to get right, but once the percentages were tuned based on watching it for awhile it has kept the server from crashing even once for more than a minute despite some record traffic related to our Mumbai coverage. If it weren’t for Monit I’d probably still be getting calls in the middle of the night saying the site was down.

You still need to find the bugs in your server configuration, or move to more powerful hardware (what we’re doing), but even if its annoying that the apache needs to be restarted every few minutes in order to not crash the server, its better than having it crash randomly when you’re not around. While you’re still tuning the system, you can have it email you based on certain conditions, so you can see how often a certain status is reached and determine whether a restart is necessary. The manual explains the functions pretty well and isn’t too long.

My advice if you’re setting it up

Setting the reset/exec levels

If you’re setting it up for the first time and you’re not having any problems at the moment you should be careful not to set the percentages too high, or the server might crash before it got that bad. I was giving a Memory/RAM max around 80%, but the remaining 20% didn’t seem to be enough to save the server, it was already to late. Here are my settings for our Apache webserver:

check system
if loadavg (1min) > 5 for 2 cycles then exec "/etc/init.d/httpd restart"
if loadavg (1min) > 7 for 1 cycles then exec "/etc/init.d/httpd restart"
if memory usage > 65% for 3 cycles then exec "/etc/init.d/httpd restart"
if memory usage > 75% then exec "/etc/init.d/httpd restart"

The ‘exec’ action is running the apache restart command directly, which will clear out all appache processes and restart them, freeing up RAM temporarily. I’m also running two levels of Load checking, which will measure the strain on the CPU. Together these cover a lot of situations that result in crashes, and there are two versions of each, one for bad situations that have gone on for awhile (“for 3 cycles” i.e. 3 minutes) and one for terrible situations that are seen even once (“for 1 cycles”, which is actually unnecessary to write).

You can actually also set up monitoring of specific processes like Apache or other servers, but its been a lot buggier (thinks the program’s not running when it is) for me than the raw server statistics, so use at your own risk.

Alternate Email Formatting for Monit

The default email format template that comes with Monit is pretty hard to read to the point where it’s kind of maddening to recieve messages from it. Luckily they offer a custom mail formatting api so you can make one that makes sense for you. The pieces they give you are a bit limiting but I worked out one that is very short and clear and should even work okay as an sms:

set mail-format {
subject: [$ACTION] $EVENT on $SERVICE
– – – – – – – – – – – – – – – – – –
Action: [$ACTION] at $DATE from $HOST


Which sends you emails like:

[exec] Resource limit matched for
‘’ mem usage of 71.9% matches resource limit [mem usage>65.0%]
– – – – – – – – – – – – – – – – – –
Action: [exec] at Fri, 28 Nov 2008 19:35:22 -0500 from


Which I think is a lot better than the default. Any Monit users out there with a good format I’d love to see what else you’ve come up with.

Awesome: Global Voices website being shown on CNN!

It’s too bad that GV only gets attention when something horrible happens in a part of the world we cover well (in this case the nightmare attacks in Mumbai), but it’s pretty thrilling to see the site I run and design actually being shown on CNN TV!

global voices website being shown on CNN, photo of a tv

click image for larger version

Thanks to Jules Rincon for her quick reflexes getting this photo of the tv for us all!

Google Groups Spam Uprising – Securing your list

Global Voices uses a lot of email lists to communicate, in fact as time goes on we realized that if anything Global Voices IS just a bunch of mailing lists. Yeah the product is a giant journalism website, but without the mailing lists there would be no posts. Just for fun here’s a graphic illustrating the detail with which we organize our mailing list communications:

gv mailing list graphic - small

(graphic by Solana Larsen, click to see bigger version)

Most of those lists run on the Google Groups service, which like everything Big Gee does is simple, elegant and feature rich, epecially in how it presents archives on the web in case you don’t keep them in your mailbox. It lets us keep using the fairly-archaic but very effective email list structure while also giving us the same opportunities as say a web-based forum or Drupal community.

Google Groups SPAM and how to stop it
In the last few days a lot of our extra groups have started getting spammed by sham user accounts that join your group and post about puffy nipples and see-through tops, something that never happened before. It seems that Google uses a captcha (“type in the letters to prove your a human”) to stop such spam accounts that used to work, but now the spammers have managed to solve it using robots and thus create all the accounts they want. (here’s an article about it with an annoying ad before you can read it)

So far it seems that only open-membership groups are effected, so if your group is invite only or you moderate your posts you shouldn’t have too many real problems. That said, at Global Voices we’re discovering that some of our open groups used to feel closed, and need to be secured now that the spam has started.

If you run a google group, log into the admin interface by going to and choosing your list, then go to GROUP SETTINGS > ACCESS (access is a tab at the top) and check the following settings:

  • Who can join? : If your group is small/exclusive enough then you might want to set this to “People can request an invitation to join”, that way you get an email before they join and you can check their posting history to see if they look like spammers. Each Google Groups user has a profile, and the spambots show their colors with the obviousness of their previous postings to other random groups.
  • Who can post messages? : If you are only using your list to send messages out and don’t actually have disucussions, set this to “Managers only”. It should definitely be set at least to Members only.
  • Message moderation: Choosing the “Messages from new members are moderated” option should help stop spammers. Unless you approve every new user account (time consuming and frustrating for users) the spambots will be able to create accounts, but in my experience so far they always post right away with some sex spam, so if you moderate just the new messages from each member you’ll avoid anyone else seeing the spam.
  • Save Settings: Dont’ forget to save!

I’ll try to update this post if my advice turns out not to work. Any other advice about securing GGroups from this stuff without completely locking them down is very welcome!

Speaking at Wordcamp San Francisco ’08

wordcamp san franciscoIn a fit of shortsighted kindness, the organizers of Wordcamp San Francisco (a conference/unconference about my favorite headache, WordPress) have invited++ me to come and speak about Global Voices and how we use WordPress to do the crazy thing we do.

I’ve heard only great things about past Wordcamps (they’re all over the place, there’s one in Toronto in October and a New York one on the same day!), and I’m deeply honored to be able to represent Global Voices and all the amazing people that make it a reality at the SF meeting. I can’t think of anywhere I’d rather be, let alone be the one speaking.

If anyone is in the area and into WordPress you should check it out, if I know you and you’re from SF drop me a line, I’ll be in town for a few days.

On Vacation : Jun 10 – Jul 4

Not that anyone would check here, but I will be out of town on vacation surrounding the Global Voices Summit in Budapest starting tomorrow and lasting about a month. I’ll be seeing Morrocco, Spain, Hungary and London (Note: NOT London, ON). Thanks to GV for the cross-atlantic travel opportunity, hopefully I’ll even post some photos if I see anything good out there.

And because i made this badge and love how it looks:

Global Voices Citizen Media Summit 2008 in Budapest

P.S. if you’re looking for some actual content check out Jim Kunstler’s Clusterfuck Nation, that’s what’s on my mind.

Global Voices Summarized in Podcast

Global Voices OnlineEthan Zuckerman Solana larsen

When I tell people about my job at Global Voices they tend to get confused, sometimes even after they’ve seen the site. Admittedly it’s a pretty complex piece of journalism, research and community all mixed up together. For those people I recommend this podcast from Open Source Radio, where the host Chris Lydon interviews two of my bosses, Ethan Zuckerman and Solana Larsen about how and why Global Voices does what it does. They make a great argument for Global Voices as an idea and as a reality, and it’s great to hear the voices of people I’m so used to dealing with through text only (There is no Global Voices office, except maybe our irc channel).

It’s kind of long so you might want to actually put it on your “pod” of choice.

Link to episode page. Direct MP3 link (71 min, 33MB).

Voices without Votes != splog

vwv-promo-125-square.gifAs part of my ongoing work for GlobalVoices, I recently designed and coded a GV-like site with the specific goal of covering foreign blog reactions to the 2008 U.S. presidential election, called Voices without Votes. The VwV site is a mix of original posts by GV authors about what people in other countries are saying about the elections and links to foreign posts about the elections along with the first ~50 words as a preview.

Despite what seems like a clearly good motivation and practice, some people have big issues with this, likening it to a pathetic practice called ‘splogging‘ (spam blogging, wikipedia link). In the WordPress IRC channel (a chatroom, #wordpress on some guys got pretty upset about the idea and I had a debate with them about copyright and aggregation of blogs (aggregation means getting content from varied sources and showing them toghether). This resulted in ‘markr’ posting a lengthy critique of VwV’s aggregation practices on his blog, HERE.

Most of his arguments stem from misconceptions about VwV or about copyright law, but some are sincerely just expressions of his dictatorial attitude towards content. I left the response below as a comment on the post, but am reprinting it here for posterity and for those who read my blog and not his (note: his full response is viewable on his page, only the parts I wanted to respond to are below. Skipped parts are mostly him telling the story i told above).

WARNING: This isn’t for my casual readers. It’s long, detailed, political and there are absolutely no lolcats anywhere to be found.

Continue reading “Voices without Votes != splog”

Going to India

I haven’t mentioned it here before but I’ve been working for the past six months or so on a website/project called Global Voices, doing a redesign and hacking the shit out of WordPress. It’s comprised of a group of international blog-fiends who each read just about everything that happens in their local blogospheres (like, all popular blogs in india) then file summaries and links to Global Voices. It’s a kind of filter that can give westerners a sense of what’s happening in the blogospheres they have no real access to, as well as giving links to the best in english blogs coming out other countries (it only covers “developing” areas under the assumption that technologically advanced areas have their perspectives covered adequately already). I’m working under Boris (also a web developer/designer), who hired me because I updated my flickr and was able to talk shit about semiotics with him.

7175261_586e74191e.jpgAll that to say that the maniacs who run Global Voices are having a conference in India (Delhi) and were nice enough to invite me as deputy-g33k, so I will be pretty damn gone from Montreal and probably the web for a month (I’m bumming around after the conference and stretching the free ride as much as I can). I’ll be checking my email and probably updating this site when I can, as well as putting up any good images my camera happens to find (Monkey Temple of Jaipur anyone?).

I hope everyone has a downplayed and non-commercial holiday season but don’t have faith in it actually happening, so just try to enjoy it despite the mall /shudder

(India photo cc Mosseby)