Page 1 of 3 123 LastLast
Results 1 to 10 of 25
  1. #1
    Untangler
    Join Date
    Jun 2008
    Posts
    47

    Exclamation Strange SMTP Issue: Incoming Traffic Outages

    I've been talking to support about this but we've had no luck narrowing down the problem, so I figured I would throw it out on the forums.

    We had an Untangle VM in transparent bridge mode between our ASA and internal network for years (8-9 years), and we've been having regular SMTP outages for incoming mail to our Exchange servers. We've spent a lot of time trying to track this down and eliminating potential causes. I've gone so far as to gather PCAP's on all interfaces of all equipment from my ASA down to my Exchange servers. What I see are SMTP related packets coming in through the ASA, out of its internal interface and into the external interface of the Untangle server. However, the packets never make it to Untangle's internal interface, which means they don't make it to the virtual interface of my load balancers, nor to the interfaces of my Exchange servers (all of which has been verified with packet captures).

    Taking it a step further I created a bypass rule within Untangle for our outside SMTP probe (which we were using to troubleshoot), and when I did that the outages continued but no longer affected my SMTP probe's traffic. This was obviously a confirmation that the problem was with Untangle.

    I've tried powering off all of the Untangle components (including SPAM Blocker, Phish Blocker, and both Virus Blockers) to no avail. Untangle support had me try all kinds of different settings on those particular components with no affect. They even setup a totally separate rack/policy for some of the SMTP traffic, but they were still unable make any progress.

    At this point I'm thinking about building a new replacement Untangle VM but I figured I would throw this out there first.

    And one last peculiar thing I've noticed, outgoing traffic appears to be completely unaffected during the incoming mail outages.

  2. #2
    Untangle Ninja sky-knight's Avatar
    Join Date
    Apr 2008
    Location
    Phoenix, AZ
    Posts
    22,202

    Default

    Untangle's Spam Blocker will manifest this behavior if it's having trouble logging the traffic. I've observed this behavior many times in Untangle platforms with faulty drives. In a virtual environment it indicates either disk fault, or drive IO limitations.

    Egress traffic will eventually halt too if the disk IO falls too far behind, but SMTP seems to go first. It's strange to see how long a linux unit will plug away with a bad disk...

    But in your case you should be either increasing drive IO priority for Untangle, or moving it to faster disks.
    Last edited by sky-knight; 04-02-2018 at 05:33 PM.
    Rob Sandling, BS:SWE, MCP
    NexgenAppliances.com
    Phone: 866-794-8879 x201
    Email: support@nexgenappliances.com

  3. #3
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,096

    Default

    Well it sounds like you've narrowed it down pretty far.

    So the SYN packet never gets sent on the internal to your exchange servers. Does any response get sent out on the external, like a reset or icmp message? Is all your mail coming from a relay or just anywhere all over the internet? For the sessions that fully connect, what do they look like in reports?
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  4. #4
    Untangler
    Join Date
    Jun 2008
    Posts
    47

    Default

    @sky-night the VM sits on a Pure Storage all flash array which has more IOPS to spare than are being used, so I don't thinks it's an IOP bottleneck, but that does remind me that a year or so ago we used a tutorial here on the forums to increase the storage space of the VM to accommodate longer report retention, and I wonder if that caused any problems with the underlying system.

    @dmorris you are correct, during the outage the SYN packet(s) never make it out the internal interface of the Untangle VM and to the Exchange servers, and there's no response obviously. In the PCAP you'll see a SYN packet and then several re-transmissions, then another SYN packet with more re-transmissions, over and over again until the outage is over and then packets all of a sudden start making it through.

    There is no relay prior to mail coming to us, and when mail is flowing properly the packet transmissions look perfectly normal. I can send you some screenshots of a couple PCAP examples if you like.

  5. #5
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,096

    Default

    I can think of several reasons that might happen (like greylisting or the load limit kicking in) but none of those would happen if port 25 is bypassed.

    What do the reports show for that session?
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  6. #6
    Untangler
    Join Date
    Jun 2008
    Posts
    47

    Default

    Quote Originally Posted by dmorris View Post
    I can think of several reasons that might happen (like greylisting or the load limit kicking in) but none of those would happen if port 25 is bypassed.

    What do the reports show for that session?
    The reports simply show gaps in traffic, so no specific logs/sessions during the outages.

  7. #7
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,096

    Default

    Quote Originally Posted by soccerextreme View Post
    The reports simply show gaps in traffic, so no specific logs/sessions during the outages.
    Thats probably not a coincidence. There is probably some major issue during these times with the server and the SMTP thing is just a symptom of that issue.
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  8. #8
    Untangler
    Join Date
    Jun 2008
    Posts
    47

    Default

    So I'm thinking that I should build a new Untangle VM from scratch and restore my configurations to it. Thoughts?

  9. #9
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,096

    Default

    Yeah, you could. Or work with support to figure out whats wrong with the server. Something sounds very wrong.
    Often if its a VM there is little they can do if the machine "just stops" because it could be an issue or configuration problem in the hypervisor.
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  10. #10
    Untangle Ninja sky-knight's Avatar
    Join Date
    Apr 2008
    Location
    Phoenix, AZ
    Posts
    22,202

    Default

    Building a new VM might not solve the problem, but it should rule out any potential corruption of the install itself.

    But, don't discount that drive IO just yet. Is the mail server Untangle is protecting on the same set of disks? Because regardless of configuration a busy mail server, with an anti-spam engine that lives and dies by eavesdropping such as Untangle, on the same disks?!? That's how you create race conditions... I'm not saying that's what's happening, I'm just saying there be dragons here. IO load in these configurations is highly contingent on mail volume, and you may have blobs of SMTP traffic landing at once and it's becoming a DOS. These things aren't as clear when you run a virtual Untangle, if it was bare metal you'd see it far more easily.
    Last edited by sky-knight; 04-03-2018 at 10:59 AM.
    Rob Sandling, BS:SWE, MCP
    NexgenAppliances.com
    Phone: 866-794-8879 x201
    Email: support@nexgenappliances.com

Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2