We experienced a power outage today, caused by someone tripping the emergency power off relay to our server room. Unfortunately, emergency power off really means "power off" so our UPS did the right thing and completely cut power rather than fall back to battery backup.
It was a little bit stressful getting everything back up, but everything appears to be working fine now.
The one serious error message we ran into is the following, when bring our OpenLDAP server back up:
[root@ldap ldap]# /etc/init.d/ldap restart
Stopping slapd: [FAILED]
Checking configuration files for slapd: bdb_db_open: unclean shutdown detected; attempting recovery.
bdb_db_open: Recovery skipped in read-only mode. Run manual recovery if errors are encountered.
bdb(dc=math,dc=ohio-state,dc=edu): PANIC: fatal region error detected; run recovery
bdb_db_open: Database cannot be opened, err -30974. Restore from backup!
bdb(dc=math,dc=ohio-state,dc=edu): DB_ENV->lock_id_free interface requires an environment configured for the locking subsystem
backend_startup_one: bi_db_open failed! (-30974)
slap_startup failed (test would succeed using the -u switch)
[FAILED]
stale lock files may be present in /var/lib/ldap [WARNING]
Fortunately, the solution to this problem is easy enough. Just run slapd_db_recover -v in the Berkeley Database directory.
cd /var/lib/ldap
slapd_db_recover -v
Finding last valid log LSN: file: 4 offset 4818337
Recovery starting from [4][4815752]
Recovery complete at Wed Feb 6 15:33:42 2008
Maximum transaction ID 80000ba7 Recovery checkpoint [4][4818337]
After that, slapd should startup just fine.
[root@ldap lib]# /etc/init.d/ldap start
Checking configuration files for slapd: bdb_db_open: unclean shutdown detected; attempting recovery.
bdb_db_open: Recovery skipped in read-only mode. Run manual recovery if errors are encountered.
config file testing succeeded
[ OK ]
Starting slapd: [ OK ]
