Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 30
  1. #11
    w00t
    Join Date
    Aug 2016
    Location
    Boulder, CO
    Posts
    19

    Default

    Hi guys,

    I am very sorry for the recent upgrade event, so let me explain what happened. First, a response from our product team:

    Our NG Firewall 15.1.0 release caused an issue that affected a small number of appliances if the appliance was rebooted. The only resolution for these appliances was a full re-installation of the software. We have resolved this issue and provided the fix in a new 15.1.1 version.

    This issue was important enough that we felt we needed to provide an update outside of our usual process. To expedite the delivery of this fix and avoid further cases of this boot failure, we initiated an update to all systems that we identified as having the issue. The update is minor and should not have resulted in any outage or loss of connectivity. We apologize to any of our customers who experienced issues during this upgrade.
    All of that is accurate, but let me add some clarification from the engineering perspective.

    The recent 15.1.0 date release should have been 15.1.1 release. That was the error in our deployment...

    15.1.0 date release included an upstream update to GRUB due to a CVE published in July. However, Untangle modifies settings associated with GRUB, which were not changed, and were not included as part of the update. As a result, folks who upgraded from 15.1.0 (previously released) to 15.1.0 (recently released) got the GRUB update and NOT the setting update.

    The result is that the box would be bricked on reboot and requiring a reinstall. We identified the issue fairly quickly and stopped the automatic updates and prepared 15.1.1 release which was nothing but a version change. However, the version change would also push untangle settings that would properly configure GRUB (even though the settings themselves have not changed).

    However, now we were in a critical situation. Although neither upgrade required a reboot, if folks on the latest 15.1.0 would reboot or lose power, they would be in trouble. To make matters more complex, this only affected a portion of the 15.1.0 customers - if you updated to 15.1.0 from 15.0, there is no issue. If you installed 15.1.0 fresh, there is no issue, etc.

    After much internal discussion, we decided that we should force the update. This was not an easy decision… However, sending notifications (a lot of it to customer who are not affected), receiving increased support volume as a result, and leaving people at risk of a broken NGFW due to power failure or reboot was a higher risk than forcing the update.

    Again, I am very sorry for the inconvenience. I assure you we took steps to prevent this kind of failure from happening in the future.

  2. #12
    Untangle Ninja sky-knight's Avatar
    Join Date
    Apr 2008
    Location
    Phoenix, AZ
    Posts
    25,263

    Default

    All of the above makes perfect sense, but a simple shout out on these forums would have been sufficient notice as far as I'm concerned.

    I don't expect Untangle to hunt us all down, but you have a Facebook page, a Twitter Feed, heck you have a CHANGE LOG page in the wiki!

    The above should have been published, somewhere, anywhere... pick a place, stick to it. Those of us in the know will watch, and when the excrement elevates to the fan, as it will from time to time we resellers that are actually trying to do our jobs will take on the task of getting the word out.

    It's interesting that some units don't show availability of v15.1.1 yet... but I assume that won't take too long.
    Last edited by sky-knight; 09-10-2020 at 09:27 AM.
    LoneWolf, Jim.Alles and donhwyo like this.
    Rob Sandling, BS:SWE, MCP
    NexgenAppliances.com
    Phone: 866-794-8879 x201
    Email: support@nexgenappliances.com

  3. #13
    Untangle Ninja
    Join Date
    Feb 2016
    Posts
    1,130

    Default

    Quote Originally Posted by sky-knight View Post
    All of the above makes perfect sense, but a simple shout out on these forums would have been sufficient notice as far as I'm concerned.

    The above should have been published, somewhere, anywhere... pick a place, stick to it.
    Precisely.

    Timur, whether we’re mom-and-pop-shop network owners or something on up the scale, we deserve absolute up-front clarity, to the extent that’s possible, from our security software and hardware vendor. It’s not just inconvenient for us to be in the dark, it’s needless uncertainty exactly where we as small business owners should be operating with certainty, to the extent that you can give us certainty. Silence on active issues, however small their footprint, from Untangle is a disservice to us, in my opinion.

    I’m not interested in making a big deal out of this, so I’ve said all I want to say. Carry on.
    donhwyo and LoneWolf like this.

  4. #14
    Untangle Ninja Jim.Alles's Avatar
    Join Date
    Jul 2008
    Location
    Central PA
    Posts
    2,606

    Default

    Untangle:
    The update is minor and should not have resulted in any outage or loss of connectivity.
    It appears that a nailed-down SD-WAN instance re-connected in my OpenVPN log.

    NGFW 15-1-1.png

    Quote Originally Posted by sky-knight View Post
    All of the above makes perfect sense, but a simple shout out on these forums would have been sufficient notice as far as I'm concerned.

    The above should have been published, somewhere, anywhere... pick a place, stick to it.
    In this case, there was already a thread, and this was a solution to an issue presented in the thread. The interested parties would have been watching for that there.

    And beating a dead horse, I have a u25xw appliance, not a z-series as mentioned in the changelog.
    Code:
         Fixes grub rescue issue on z-series.
        Includes bugs fixes in date release Date_Changelog#15.1.0_build_2020-08-26
    SMH
    Last edited by Jim.Alles; 09-10-2020 at 12:59 PM.

  5. #15
    Untangler jcoffin's Avatar
    Join Date
    Aug 2008
    Location
    Sunnyvale, CA
    Posts
    9,201

    Default

    It can affect u-series depending when it was installed with the ISO / IMG.
    Attention: Support and help on the Untangle Forums is provided by
    volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  6. #16
    Untangle Ninja Jim.Alles's Avatar
    Join Date
    Jul 2008
    Location
    Central PA
    Posts
    2,606

    Default

    Quote Originally Posted by jcoffin View Post
    It can affect u-series depending when it was installed with the ISO / IMG.
    As always, thank you for communicating that to us effectively!
    ncksh and CMcNaughton like this.

  7. #17
    Untangle Ninja sky-knight's Avatar
    Join Date
    Apr 2008
    Location
    Phoenix, AZ
    Posts
    25,263

    Default

    I have none of the listed devices, yet apparently was impacted too.
    Rob Sandling, BS:SWE, MCP
    NexgenAppliances.com
    Phone: 866-794-8879 x201
    Email: support@nexgenappliances.com

  8. #18
    Untangle Ninja
    Join Date
    Jan 2011
    Posts
    1,322

    Default

    Not very cool. I had several sites where all the OpenVPN connections dropped for several minutes with no explanation, right in the middle of the afternoon. Why couldn't you at least respect the configured update TIME?
    Jim.Alles and LoneWolf like this.

  9. #19
    w00t
    Join Date
    Aug 2016
    Location
    Boulder, CO
    Posts
    19

    Default

    Sorry about that.
    The issue is that regular updates are triggered from the box itself, where the time configuration lives (it's a pull). In this case, we had push from the cloud. Collecting the configuration of each box and scheduling an update based on configuration is a more involved process, as you can image, and given our circumstances we were under time constraints.

  10. #20
    Untangle Ninja
    Join Date
    May 2008
    Posts
    1,335

    Default

    Would disabling these stop the push process?

    Connect to Command Center
    Allow secure remote access to support team

    Thanks

Page 2 of 3 FirstFirst 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2