Steven Reimer

My new role at work requires the 24 hour monitoring of mission-critical Red Hat servers. I am looking for a Linux based smart phone that would provide at the least an SSH utility that would allow me to access these servers without having to carry a notebook and staying close to Internet access.

Any ideas?

A while ago I was given the responsibility of server maintenance. Not just any servers, server clusters. We are using DRBD 8 and Heartbeat for our clusters. When it's working, it's a thing of beauty, setting it up however, can be challenging.

I have been working with one of our new clients who had wanted to do their own server setup but couldn't get the DRBD disk syncing to work. I had initially helped them get it to the point where it was syncronizing over eth0 (eth1 was preferred) and then a couple days later nothing was working.

What I found out was that DRBD had become split-brained at one point and now both nodes would default to Secondary and disconnected states. The trick was to get them back together again. Fortunately, this was a new client so there was no worry about losing any data.

Through the usual Google searches I discovered that I needed to do the following:
1. Stop DRBD on the secondary
# /etc/init.d/drbd stop

2. Restart DRBD on the primary
# /etc/init.d/drbd restart

3. Make sure the primary is primary. In this case it would start as primary and then switch to secondary when starting.
# drbdadm primary [resource]

4. Invalidate the data on the secondary. This will force it do re-sync with the primary.
# modprobe drbd  -- we need the drbd module to be loaded
# drbdadm invalidate [resource]
# drbdadm adjust [resource]

At this point, /proc/drbd should give you a syncronization progress
[user@Primary ~]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:43:01
 0: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
    ns:78160576 nr:0 dw:0 dr:78160576 al:0 bm:4770 lo:0 pe:0 ua:0 ap:0 oos:395131820
    [==>.................] sync'ed: 16.6% (385870/462199)M
    finish: 10:28:23 speed: 10,436 (10,232) K/sec

That's what it took for me to fix my situation. It's possible you might not need the exact same solution, but maybe it will help.

We are back from our 10 day stint in Manitoba. It was really refreshing to be completely disconnected - no cell coverage or Internet access. We did things like race beater cars on a dirt track in the pasture, watch drag and mud bog races and drive tractors. Completely different than what has become my normal life. Part of me didn't want to come back to the city.

But here I am.

Check the gallery for pictures from the race. I hope to add more from the week soon.

Here's a tip I learned today while maintaining an old Feisty Fawn Ubuntu server. If you run apt-get install [anything] you will get a bunch of 404 "file not found" errors. This is because the repository is no longer being maintained. To avoid having to upgrade to get your repositories working again, you can make a small change to your sources.list file.

Edit /etc/apt/source.list and replace all "archive.releases.ubuntu.com" with "old-releases.ubuntu.com"

Run sudo apt-get update and you should be ready to roll!