The symptom was that attempts to reach the configuration webpages of this Untangle firewall router would just fail as if apache2 was not getting the connection. However, telnet to port 80 would sometimes work and sometimes fail to connect. (When it failed, the browser usually acted like a dns failure...)

The day before, Comcast (our local cablemodem provider) was here. At their request, our Comptroller attempted to "reset our box" by forcing power off at the switch at the Untangle box. After that, the box would power back on and route traffic just fine, but the configuration web pages were not available (and unknown if any of the other apps worked properly, but there were no complaints).

A lot of forum searching led me to a log indicating database corruption. (The top few lines were normal, the box was "reset" at about 12:35 and the problem shows up at about 12:40...)

Code:
root@corp-untangle#
~ #less /var/log/postgresql/postgresql-8.3-main.log

[...]
2011-06-06 01:40:17 EDT ERROR:  relation "settings.n_branding_settings" does not exist
2011-06-06 01:40:17 EDT STATEMENT:  SELECT company_name FROM settings.n_branding_settings
2011-06-06 01:40:29 EDT ERROR:  relation "reports_state" already exists
2011-06-06 01:40:29 EDT STATEMENT:
        CREATE TABLE reports.reports_state (
                last_cutoff timestamp NOT NULL)
2011-06-06 12:49:53 EDT LOG:  unexpected EOF on client connection
2011-06-06 12:51:27 EDT LOG:  database system was interrupted; last known up at 2011-06-06 12:46:45 EDT
2011-06-06 12:51:27 EDT LOG:  unexpected pageaddr 1/612A0000 in log file 1, segment 112, offset 2752512
2011-06-06 12:51:27 EDT LOG:  invalid primary checkpoint record
2011-06-06 12:51:27 EDT LOG:  unexpected pageaddr 1/611B6000 in log file 1, segment 112, offset 1794048
2011-06-06 12:51:27 EDT LOG:  invalid secondary checkpoint record
2011-06-06 12:51:27 EDT PANIC:  could not locate a valid checkpoint record
2011-06-06 12:51:27 EDT LOG:  startup process (PID 3309) was terminated by signal 6: Aborted
2011-06-06 12:51:27 EDT LOG:  aborting startup due to startup process failure
2011-06-06 12:54:25 EDT LOG:  database system was interrupted; [...]
[... above lines repeat every 3 minutes ...]
This post led me to most of the answer, but I had to switch to the postgres account to do it, so the commands were

Code:
root@corp-untangle#
~ #su postgres

postgres@corp-untangle:/root$  /usr/lib/postgresql/8.3/bin/pg_resetxlog -f /var/lib/postgresql/8.3/main
(a message appears indicating that the transaction log was reset)
postgres@corp-untangle:/root$  exit

root@corp-untangle#
~ #
After that, I needed to wait for about 5 minutes, during which postgresql restarted automatically, figured out my configuration, and then left me able to access the configuration web pages.

As expected, I don't have last night's reports and it'll be interesting to see what I have tomorrow in terms of reports (hopefully just a gap). My impatient but well-meaning Comptroller also now knows not to force power down on that box and how to shut it down properly.

Hope this helps someone... and hopefully this is the happy ending but I'll post back if it turns out that this "solution" was just temporary ...