Page 1 of 2 12 LastLast
Results 1 to 10 of 11
  1. #1
    Untangle Ninja sky-knight's Avatar
    Join Date
    Apr 2008
    Location
    Phoenix, AZ
    Posts
    26,554

    Default Exchange 2007 Trial by Fire!

    Ok so the story goes.. I found an issue with UT at the end of last week.. fixed that issue allowing greater throughput than previously possible and flooded the Exchange server behind said UT right into the dirt by Monday...

    The last 48 hours have been me learning how to recover a dead Exchange 2007 server from utter scratch...

    Let me tell you... whoever designed the systems Exchange 2007 has built in to do disaster recovery deserves a freaking medal!

    Ok.. more details on the server.

    It had a single 3 drive RAID 5 array to house the Exchange and Windows installations. The RAID controller was an Intel soft controller built into the SuperMicro server I was using. It simply can't keep up with the load Exchange places on it.. the unit failed.. lost 2 drives at once and there goes the server.

    So after installing a real RAID controller and getting the drives back online.. I reinstalled windows 2003 applied Sp3 and all the latest security updates and took the server in for install this morning.

    At this point I installed the remaining support services the server needed, IIS, DNS, and AD. (yes I know Exchange doesn't like being on an AD server but this is the second box and I don't run domains without a 2nd DC)

    Now the time comes to reinstall Exchange... I insert the DVD and it pops up and says something to the tune of "Hey dork, you're exchange is borked I ain't installing!" (if only programmers wrote errors this way...)

    So I looked into the error it gave me and I found this wonderful command switch for Exchange's setup.exe...

    Code:
    setup /mode:RecoverServer
    This lovely command does some serious magic...

    You see, it inspects Active Directory for an Exchange server with the same name as the server it is run on... when configuration data is found it uses this configuration data to do an unattended installation of Exchange based on that configuration!

    So 8 hours later, after a pile of DC repairs and whatnot to get the process to run completely I have an Exchange server that is configured exactly the way I left it minus the message store...

    A quick NT backup puts the store back in place.. some time with eseutil to rebuild a working database and I have a happy Exchange server again.

    So 48 hours to learn how and rebuild an Exchange server from utter ruin to the place it was just 8 hours before the crash.

    Don't get me wrong there is more to be done.. a lot of little things go wrong when this sort of thing happens... But for the first time in my career of working with Exchange I can honestly say I am HAPPY with the product.

    I still won't sell it to a SMB... WAY too much effort to repair for that but for Enterprise that has the budget... this thing really is the way to go.
    Rob Sandling, BS:SWE, MCP
    NexgenAppliances.com
    Phone: 866-794-8879 x201
    Email: support@nexgenappliances.com

  2. #2
    Master Untangler
    Join Date
    Aug 2008
    Posts
    939

    Default

    Use acronis for backup. On disaster, pop in the cd. Click restore all. Watch the status. Eject the CD when done. Server boots just like it was before the crash. Nothing more to-do.

    P.S. Unless you are restoring an AD Controller, then you need todo the quick registry trick to prevent USN rollback.

  3. #3
    Untangle Ninja sky-knight's Avatar
    Join Date
    Apr 2008
    Location
    Phoenix, AZ
    Posts
    26,554

    Default

    Acronis is an imager.. how is that going to protect an AD enabled Exchange server? Everything is time and date coded and if you miss a single time stamp the blasted services won't start.
    Rob Sandling, BS:SWE, MCP
    NexgenAppliances.com
    Phone: 866-794-8879 x201
    Email: support@nexgenappliances.com

  4. #4
    Untangler Captain Colonoscopy's Avatar
    Join Date
    May 2008
    Location
    Cedar Rapids, IA
    Posts
    49

    Default

    Good to know about that RecoverServer switch . . . I hope I NEVER have to use it.

    But, yeah, you CANNOT run Exchange 2007 on a crappy RAID controller. If you read up on the Exchange best practices they say you shouldn't even think of using SATA drives for the message stores, let alone a soft-troller. Glad to hear you got everything working again!
    Chuck Norris lost his virginity before his dad did.

  5. #5
    Untangle Ninja sky-knight's Avatar
    Join Date
    Apr 2008
    Location
    Phoenix, AZ
    Posts
    26,554

    Default

    Hehe tell that to Dell when they sell those SMB servers...

    In my case the Exchange box is a supermicro 2u server. It really is a nice hunk of hardware. The onboard soft RAID controller has a BIOS switch that can take it from Intel to Adaptec and back again.

    So on this iteration I've taken the RAID into adaptec mode... The install is running on a mirror as opposed to the RAID 5 it had. The mirror has 2 partitions one for windows and one for the message store. The other 2 drives I had are now freestanding to hold message store backups.

    So what I did was...
    1.) Reinstall Windows 2003 Server x64
    2.) Apply SP3 and security patches
    3.) join it to the domain
    4.) Install DNS, and IIS
    5.) Make sure AD site replication is working
    6.) Install RTP over HTTP proxy in the networking components
    7.) Run setup.exe /mode:RecoverServer /DoNotStartTransport to install Exchange 2007
    8.) Babysit the thing and restarted it several times as I cleared permissions issues, odd domain issues and what not to get it to 97% where the install dies...
    9.) The install died because the Exchange System Attendant service is stuck on starting..
    10.) Call Microsoft, balk at the over inflated support fees and hang up the phone.
    11.) Go watch 2 movies with my kids to decompress
    12.) Came back to take another crack and noticed the darn Exchange services all running...
    13.) Restored the Information Store backup
    14.) Ran eseutil /cc on both info stores to play the logs and get them ready to mount
    15.) Mount the info stores
    16.) Relocate the info stores to the new volume setup for them
    17.) Realize that Windows didn't think Exchange was installed so I can't apply patches
    18.) Hack the registry to allow Setup.exe /Mode:Upgrade to run and complete
    19.) Scratch my head wondering why OWA works for Firefox but not IE
    20.) Realized I forgot to update Exchange with security patches
    21.) Installed Update Rollup 3 for Exchange 2007 SP1
    21.) Noticed OWA is happy now
    22.) Find out the hardway that Update Rollup 3 breaks Entourage 2004 and 2008

    So here I am now...

    I know the server shouldn't be a DC but that is the only secondary server I have so DC stays.
    I know that SATA is a bad idea given the load but I have no budget nor time to upgrade to SAS.

    So I have a write caching disabled mirror running a very slow and stable Exchange 2007 server... and a group of annoyed Mac users that are stuck with OWA until Microsoft gets their heads out of their arses...

    So I think I have aged 10 years in this mess... yet it was oddly fun at the same time. There is just something cool about pulling a system this massive back from the brink. Now, let's NOT do it again....
    Last edited by sky-knight; 09-18-2008 at 08:08 PM.
    Rob Sandling, BS:SWE, MCP
    NexgenAppliances.com
    Phone: 866-794-8879 x201
    Email: support@nexgenappliances.com

  6. #6
    Untangler Captain Colonoscopy's Avatar
    Join Date
    May 2008
    Location
    Cedar Rapids, IA
    Posts
    49

    Default

    Quote Originally Posted by sky-knight View Post
    Hehe tell that to Dell when they sell those SMB servers...
    Yeah, Dell does have a tendency to try and sell people SATA drives in their servers . . . probably to stay cheap . . . oh well.

    I just copied that last post and saved it in my "just in case" folder of tips/tricks. Hopefully I will never have to do that, but you never know...

    I know how you feel about MS support. I had to call them about a failed CCR mailbox cluster at a school district I had setup a while ago. Apparently the backups had been failing for a while so the logs weren't getting cleared out. So their half-tarded "Desktop Supervisor" decided he would try and fix by manually deleting log files. So he did that and then noticed some time later that the active node had gone passive and the normally passive node had gone active. So he decided he didn't like that and forced node1 to go active and then all of a sudden no one could connect to their mailbox. Hilarity ensued. I get a call at 4pm on a Friday, and proceed to spend the next five hours getting everything back in order.....
    Chuck Norris lost his virginity before his dad did.

  7. #7
    Untanglit
    Join Date
    Aug 2008
    Posts
    24

    Default

    With respect to image backups, you should have a look ShadowProtect Server (www.storagecraft.com). Incremental image backups every 15 mins, and hardware independent restore (including restore to a VM).

  8. #8
    Master Untangler
    Join Date
    Aug 2008
    Posts
    939

    Default

    Quote Originally Posted by sky-knight View Post
    Acronis is an imager.. how is that going to protect an AD enabled Exchange server? Everything is time and date coded and if you miss a single time stamp the blasted services won't start.
    Acronis works flawless. It uses Microsoft VSS (shadow copy) to backup everything. Restoring is a sinch. Myself and my techs have restored Exchange and Active Directory many many many times with Acronis without failure. Just a bit more than a week ago we did a restore of our Exchange 2007 Server to new hardware. Not a single glitch.

    P.S. You can do mailbox level restores as well. Just mount up a backed up copy of the database in recovery. Then restore the mailbox of choice.

  9. #9
    Master Untangler
    Join Date
    Aug 2008
    Posts
    283

    Default

    Quick question. What was the problem you had that, when fixed, allowed greater throughput?

  10. #10
    zay
    zay is offline
    Master Untangler zay's Avatar
    Join Date
    Aug 2008
    Posts
    103

    Default

    I am a firm user of Backup Exec. Never do mailbox (brick) backups, they use up too much space. I had my share of exchange crash and restore issue. That eseutil is a really good utility. And, if you have trouble restoring some mailboxes, you can use the Exmerge tool to export the .pst then once you create a new mailbox for the user, you can then import the .pst back to the mailbox. Here is a link on how to use exmerge.
    http://www.exchangeinbox.com/article.aspx?i=88
    What does it profit you to gain the world and lose your soul?

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2