Page 1 of 2 12 LastLast
Results 1 to 10 of 19

Thread: Hacked?

  1. #1
    Master Untangler
    Join Date
    Aug 2008
    Posts
    112

    Default Hacked?

    One of my clients has an untangle installation.

    Everything seems normal on the box. Yesterday morning, he called saying can't receive email (entire office down). Web working fine.

    We had configured tcpdump to dump log files to a second disk on the hour (writes log to primary then moves it when finished).

    The log files stopped yesterday 2am, resumed, 3pm (corrupt file). Every log file prior to 2am I can read with wireshark. Every log file post 3pm is fine.
    syslog and auth log files stopped just past 2:30 yesterday. Resumed just after 3pm yesterday, as did email. Coincident with that was my log in to their email offsite, was when the email started again in the office.

    Disk is 60% free (80gb disk). Log file disk 27% free (160gb)

    untangle uptime shows last boot 16 days ago which I knowingly remember (in my client reports)

    no edits showing to passwd file since I last edited it. Only account is mine besides what's expected.

    root login disabled in sshd config files.

    Client has had previous issues that were unexplained in that personal (gmail/hotmail) and business email were posted to 3rd parties. No keyloggers found. Nothing unusual shown in logs in 1 month (+ 1 day to be specific since this went in with full tcpdump logging). Email provider shows only his IP accessing the mail until initial attack and until yesterday, no other successful logins except mine from outside their ip.

    Any suggestions on what to look for. Has this been compromised. Any suggestions of how to track what happened as I'm at a loss to explain it.

    Dave
    Dave Bour
    Desktop Solution Center
    Burlington, ON, Canada
    www.desktopsolutioncenter.ca
    905.381.0077 X501

  2. #2
    Untangle Ninja jcoehoorn's Avatar
    Join Date
    Mar 2010
    Location
    York, NE
    Posts
    1,878

    Default

    Wait - you've got a 117GB of nothing but log files?!? Time to clean house, I think.

    But, moving on. It seems to me like the easiest, safest, and ultimately quickest course of action (in terms of both the amount of time you spend on the issue and the amount of time until you can have this resolved for your client) is to take a configuration backup, reinstall untangle, and restore the backup. Should take no more than an hour or at most two and this way you'll be sure.
    Last edited by jcoehoorn; 12-01-2010 at 08:05 AM.
    Five time Microsoft ASP.Net MVP managing a Lenovo RD330 / E5-2420 / 16GB with Untangle 16.2 to protect 500Mbits for ~450 residential college students and associated staff and faculty

  3. #3
    Master Untangler
    Join Date
    Aug 2008
    Posts
    112

    Default

    the log files are the tcpdump that run constantly to provide an audit for this type of activity..not that it did me any good here. It's a full month's worth at this point. Depending on what's going on in the office, it typically does 250-350MB an hour during business hours. We do see a few blips to the gb range when someone tries downloading videos. Currently there's no web filtering, content filtering or anything. He simply wants logging of all activity so we can find what's going on. It captures the backup going out nightly too. The staff are not aware of the logging at this point. It's simply passive recording of everything crossing the internet/intranet boundary. To find the problem, we have to have a starting point and this was the best solution we had since nothing could be found to the original compromise of data, albeit someone has definitely upped the ante... or a system failure that I can't explain...just too many coincidences here.
    Dave Bour
    Desktop Solution Center
    Burlington, ON, Canada
    www.desktopsolutioncenter.ca
    905.381.0077 X501

  4. #4
    Untangle Ninja Mathiau's Avatar
    Join Date
    Feb 2008
    Location
    Costa Frickn' Rica
    Posts
    1,636

    Default

    maybe someone's computer in the office was compirmised


    personal (gmail/hotmail) and business email were posted to 3rd parties
    People always claim they didn't use X address outside of work or something, when you say posted to 3rd parties can you be more specific? They are being spammed?

    As you said he only wants logging so basically he has UT doing nothing thus leaving the network wide open for the desktop users to go anywhere they want and get infected with something.
    kv-2 | UT 11.0.1 | Dell R610 Server | Intel Xeon 2.8Ghz Quad Cores | 24Gb DDR3 ECC | 1 Intel QPort NIC | Integrated Broadcom QP | Dell Perc 4i | 6 x 73G 2.5 15k SAS raid 10 | 100mb/100mb | 30mb/30Mb

  5. #5
    Master Untangler
    Join Date
    Aug 2008
    Posts
    112

    Default

    the original compromise involved contents from his system and personal emails given to other people that had no business receiving the files. I was called in at that point to investigate. Nothing conclusive could be found on his or any other system in the office. There's no spamming outside of the usual junk coming into email.
    We've since run the gamut of detection (malware, keyloggers, antivirus, rootkits) on all systems in the office as well as started logging all activity.
    Given this one has personally targeted him based on the original distribution of information, operating on the basis that this is related, this wasn't a "drive by" hacking, nor bank / personal identify compromised ruling out typical information theft or compromise.
    His business is such that there's nothing propriety that could be gained by stealing outside of his business contacts/bookkeeping as anyone working with his suppliers has access to the same technical information.
    All indications as well as my suspicions point to an inside job from someone either paid off or disgruntled however, nothing to backup my suspicions at this point as anything I've found (or more specifically, what couldn't be found) related to the original compromise indicate such rather than external access.
    The economics of this wouldn't make him a target either.
    All that said, the skill to get into the untangle box, disable most functionality for 13 hrs, suspend all logging (syslog, auth log), my tcpdump scripts and nothing showing puzzles me. Email stopped processing however web activity worked.
    If I didn't know any better, I would have thought the hard disk failed (no indication of that) though that wouldn't explain the web working. Alternatively, hard disk full, again that's not the case. It could explain the logging issues but there would be evidence of that somewhere still.
    Bottom line, I'm stumped. I've several clients running tcpdump's to monitor similar environments, both larger and smaller and have NEVER seen anything like this except when I've filled disks and never caught the warning email in time.
    If however, I take the original compromise and separate it from this event, anything is possible to explain yesterday's incident. That said, I'm a strong believer in coincidences and the timing of these so close together would make me believe there's a relationship between the two.
    D.
    Dave Bour
    Desktop Solution Center
    Burlington, ON, Canada
    www.desktopsolutioncenter.ca
    905.381.0077 X501

  6. #6
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,747

    Default

    I'm not sure I understand the whole story, but I'm not sure I see any evidence of a compromise (at least not on the local network). Its probably worth looking into the outage.

    Are you asking about the untangle box itself?
    If you're investigating the untangle box itself just check your .zsh_history and 'last'
    99.9% of the time they're too lazy to clean up the log files.
    You'll also see evidence in your tcpdump traffic.
    I've never seen an untangle box compromised outside of people who enable ssh and set the password to "passwd" or "12345." Thats not because untangle is ZOMG-UBER-SECURE per-say, its just because there aren't really any services running to target.
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  7. #7
    Master Untangler
    Join Date
    Aug 2008
    Posts
    112

    Default

    I don't know what to think at this point. Not a single system inside has a ssh client installed.
    the switch is under video surveillance (as are the windows server, untangle server and modem). Of the 7 systems in the office, 2 are under direct surveillance and 4 others, you must pass through surveillance to get access to the system. The remaining system is in a work area at the back. Anything could be plugged into the network ports on 5 systems in there without obvious detection except though an arp or dhcp request. Arp -a only shows the expected connected systems though I don't know how long it caches it's data.
    last only shows today's 5 entries. Nothing prior. My own untangle shows only today's entry so I don't know what to expect differently. Even last -n 50 shows the same results.
    .zsh_history is interesting in that there's a hole from stuff I know I did a couple weeks ago to stuff I typed yesterday is missing to today, seems intact. This is now really puzzling.
    ssh is enabled for external access as that's how I get at it. There are no other accounts than mine on the system with a reasonably secure password that should at least trigger an auth.log error if it was being brute force attacked.
    As to tcpdump, that's my problem... the time in question is missing. there's 13 hrs that's just gone, from 2am to 3pm yesterday.
    bottom line, I still suspect it's been compromised though I haven't a clue how.
    next is why... what was the target. was this one of those china hacking attacks to gain a box to control you constantly hear about, or personal related to the client's past issues he had.
    this is definitely leaning me towards reloading the untangle and every system inhouse although that's highly disruptive and I still don't know what I'm looking for if anything inside is actually compromised.
    Confused or puzzled to say the least.
    D.
    Dave Bour
    Desktop Solution Center
    Burlington, ON, Canada
    www.desktopsolutioncenter.ca
    905.381.0077 X501

  8. #8
    Master Untangler wharfratjoe's Avatar
    Join Date
    Dec 2008
    Location
    Southern California
    Posts
    431

    Default

    first, if it were me, i would enable openvpn and disable external access for ssh. This way one has to vpn in first then is able to ssh to the UT box from inside the network only. Also limiting ssh access from one or two source ip's on the LAN only is also something I do.
    Jim.Alles likes this.

  9. #9
    Master Untangler
    Join Date
    Aug 2008
    Posts
    112

    Default

    will do. got open vpn now but still have it external access too. If only I could configure it to read by Mac addresses, rather than ip...then I'm guaranteed to lock it down securely as there's only two systems that are used.
    D.
    Dave Bour
    Desktop Solution Center
    Burlington, ON, Canada
    www.desktopsolutioncenter.ca
    905.381.0077 X501

  10. #10
    Master Untangler Big D's Avatar
    Join Date
    Nov 2008
    Posts
    719

    Default

    I have had an untangle HDD die and it ran perfectly fine until we tried to reboot the thing. The web interface and most the untangle feature did stop working mind you but the web access stayed up.

    If the drive has become what I would call flaky (it stopped working beat it till it starts to work again) it could very well explain what you saw. Although a 13 hour period for it to stop then start working again is long. Is this a single drive or RAID installation?

    You could use an RSA certificate key to lock down external SSH. It is way more secure than any straight password could be.
    Last edited by Big D; 12-01-2010 at 03:12 PM.
    The beatings shall continue until morale improves!

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2