Release 15.1.1 - ignored update windows?

[Timur]

Hi guys,

I am very sorry for the recent upgrade event, so let me explain what happened. First, a response from our product team:

Our NG Firewall 15.1.0 release caused an issue that affected a small number of appliances if the appliance was rebooted. The only resolution for these appliances was a full re-installation of the software. We have resolved this issue and provided the fix in a new 15.1.1 version.

This issue was important enough that we felt we needed to provide an update outside of our usual process. To expedite the delivery of this fix and avoid further cases of this boot failure, we initiated an update to all systems that we identified as having the issue. The update is minor and should not have resulted in any outage or loss of connectivity. We apologize to any of our customers who experienced issues during this upgrade.

All of that is accurate, but let me add some clarification from the engineering perspective.

The recent 15.1.0 date release should have been 15.1.1 release. That was the error in our deployment...

15.1.0 date release included an upstream update to GRUB due to a CVE published in July. However, Untangle modifies settings associated with GRUB, which were not changed, and were not included as part of the update. As a result, folks who upgraded from 15.1.0 (previously released) to 15.1.0 (recently released) got the GRUB update and NOT the setting update.

The result is that the box would be bricked on reboot and requiring a reinstall. We identified the issue fairly quickly and stopped the automatic updates and prepared 15.1.1 release which was nothing but a version change. However, the version change would also push untangle settings that would properly configure GRUB (even though the settings themselves have not changed).

However, now we were in a critical situation. Although neither upgrade required a reboot, if folks on the latest 15.1.0 would reboot or lose power, they would be in trouble. To make matters more complex, this only affected a portion of the 15.1.0 customers - if you updated to 15.1.0 from 15.0, there is no issue. If you installed 15.1.0 fresh, there is no issue, etc.

After much internal discussion, we decided that we should force the update. This was not an easy decision… However, sending notifications (a lot of it to customer who are not affected), receiving increased support volume as a result, and leaving people at risk of a broken NGFW due to power failure or reboot was a higher risk than forcing the update.

Again, I am very sorry for the inconvenience. I assure you we took steps to prevent this kind of failure from happening in the future.

[sky-knight]

All of the above makes perfect sense, but a simple shout out on these forums would have been sufficient notice as far as I'm concerned.

I don't expect Untangle to hunt us all down, but you have a Facebook page, a Twitter Feed, heck you have a CHANGE LOG page in the wiki!

The above should have been published, somewhere, anywhere... pick a place, stick to it. Those of us in the know will watch, and when the excrement elevates to the fan, as it will from time to time we resellers that are actually trying to do our jobs will take on the task of getting the word out.

It's interesting that some units don't show availability of v15.1.1 yet... but I assume that won't take too long.

[Sam Graf]

All of the above makes perfect sense, but a simple shout out on these forums would have been sufficient notice as far as I'm concerned.

The above should have been published, somewhere, anywhere... pick a place, stick to it.

Precisely.

Timur, whether we’re mom-and-pop-shop network owners or something on up the scale, we deserve absolute up-front clarity, to the extent that’s possible, from our security software and hardware vendor. It’s not just inconvenient for us to be in the dark, it’s needless uncertainty exactly where we as small business owners should be operating with certainty, to the extent that you can give us certainty. Silence on active issues, however small their footprint, from Untangle is a disservice to us, in my opinion.

I’m not interested in making a big deal out of this, so I’ve said all I want to say. Carry on.

[Jim.Alles]

Untangle:

The update is minor and should not have resulted in any outage or loss of connectivity.

It appears that a nailed-down SD-WAN instance re-connected in my OpenVPN log.

NGFW 15-1-1.png

All of the above makes perfect sense, but a simple shout out on these forums would have been sufficient notice as far as I'm concerned.

The above should have been published, somewhere, anywhere... pick a place, stick to it.

In this case, there was already a thread, and this was a solution to an issue presented in the thread. The interested parties would have been watching for that there.

And beating a dead horse, I have a u25xw appliance, not a z-series as mentioned in the changelog.

     Fixes grub rescue issue on z-series.
    Includes bugs fixes in date release Date_Changelog#15.1.0_build_2020-08-26

SMH

[jcoffin]

It can affect u-series depending when it was installed with the ISO / IMG.

[Jim.Alles]

It can affect u-series depending when it was installed with the ISO / IMG.

As always, thank you for communicating that to us effectively!

[sky-knight]

I have none of the listed devices, yet apparently was impacted too.

[johnsonx42]

Not very cool. I had several sites where all the OpenVPN connections dropped for several minutes with no explanation, right in the middle of the afternoon. Why couldn't you at least respect the configured update TIME?

[Timur]

Sorry about that.
The issue is that regular updates are triggered from the box itself, where the time configuration lives (it's a pull). In this case, we had push from the cloud. Collecting the configuration of each box and scheduling an update based on configuration is a more involved process, as you can image, and given our circumstances we were under time constraints.

[donhwyo]

Would disabling these stop the push process?

Connect to Command Center
Allow secure remote access to support team

Thanks