Page 1 of 6 123 ... LastLast
Results 1 to 10 of 57
  1. #1
    That Which Lurks Below
    Join Date
    Jul 2018
    Posts
    143

    Exclamation ETM Dashboard Issue

    Hi, everyone!

    Early the morning of Monday, August 8th, we became of an issue affecting ETM Dashboard. Some appliances aren't able to connect to it, which causes them to appear offline and to be unreachable via Remote Access. In some cases, this is affecting the ability to assign or verify subscriptions as well.

    Affected appliances should be passing/processing traffic and otherwise acting completely normally, but they may throw an admin alert indicating they're unable to connect to CMD or ETM Dashboard.

    We're hard at work getting it sorted out! This post will be updated with any further information.

    Update, 08 Aug 22, 12:25 PDT: We have restarted the ETM Dashboard servers and we are seeing appliances reconnecting. If yours was one of the units that went offline early this morning, it may not reconnect yet, but the majority of appliances (over 30,000 as of this writing) are reconnected to ETM Dashboard. Our engineers are still working on resolving the issue for the few affected appliances that remain unable to connect.

    Update, 10 Aug 22, 12:00 PDT: Micro Edge v4.0.1 has been added to our upgrade service, so you should be able to update those appliances in Settings > Upgrade in the Micro Edge UI. An image for 4.0.1 has also been added to ETM Dashboard's Downloads section.

    Update, 11 Aug 22, 08:10 PDT: NG Firewall v16.5.2 has been added to our upgrade service and is rolling out as of now. Units which have not been able to reconnect to ETM Dashboard will receive the update today; all remaining units will receive it beginning next week. (If you have already performed the workaround, your unit(s) will still receive the updated version, although nothing will change.) An image for 16.5.2 has also been added to ETM Dashboard's Downloads section.

    It is important to note that only existing installs on version 16.5.1 will receive this patch. If your NG Firewall is running an older version, you will need to upgrade before the patch becomes available to that unit. The issue does not affect connections to our upgrade service, so you should find the Upgrade Now button available in Config > Upgrade.

    Final update! 19 Aug 22, 07:23 PDT: Update rollout is complete. NG Firewall 16.5.2 and Micro Edge 4.0.1 are available to all Edge Threat Management appliances.

    This update resolves the issue, so this forum post is now closed. We thank you again for your patience and understanding during this process!
    Last edited by gravenscroft; 08-19-2022 at 10:41 AM. Reason: final update
    dashpuppy and hpaunet like this.
    Græme Ravenscroft • Technical Marketing Engineer
    ('gram', like the unit of measurement)
    he/him
    Please don't reboot your NGFW.
    How can we make Arista ETM products better?

  2. #2
    Master Untangler
    Join Date
    Mar 2021
    Posts
    160

    Default

    Yep all but 1 Appliance (MicroEdge appliance) is showing Offline in my Dashboard and Im getting appliance offline emails... I just tried connecting into a client via Cmd Center remote access and cant.. This is super frustrating

    @dashpuppy this is the very reason I told you not having Wireguard VPN access to the webadmin gui would be a problem the other day when we were discussing locking down the webadmin gui to the Dashboard access only!! Here we are a few days later, I need to get into a clients appliance and cant because Cmd Center is down... Grrrr. Ill have to rethink this strategy.

  3. #3
    Master Untangler
    Join Date
    Mar 2021
    Posts
    160

    Default

    Looks like its slowly coming back... 16 back online... 9 still showing disconnected... of course the one I need to get into is still down.

  4. #4
    w00t
    Join Date
    Aug 2016
    Location
    Boulder, CO
    Posts
    27

    Default

    Hi everyone,

    Well.. I wish I was writing to you all under better circumstances, but sadly not today.

    As many of you noticed, command and control functionality was broken as of last night in Command Center for many of you. After an investigation, we found the reason and it is quite bad - the certificate we use for communicating with our C&C servers was generated on 8/7/2012 and expired yesterday.

    Unfortunately, certificate details were hard-coded into the script used to connect to our C&C servers. We were able to update those keys on appliances that were still connected. However, those that lost the connection will not be able to re-establish it until we issue a patch release for both NGFW and MicroEdge.

    We are working hard on getting the patch release available as soon as possible for NGFW, MicroEdge, and WAF.

    If you have a box that is still disconnected and you are able to get to it via SSH and are comfortable with the CLI, you can do the following.


    1. Log into the box via SSH
    2. Update the script:
    On NGFW run:
    wget -O /usr/bin/pyconnector https://www.untangle.com/util/1offs/pyconnector && chmod +x /usr/bin/pyconnector

    On MicroEdge run:
    wget -O /usr/bin/pyconnector https://www.untangle.com/util/1offs/pyconnector_mfw && chmod +x /usr/bin/pyconnector

    3. Restart the appliance

    The above will download a patch script with updated certificate information, and restarting the box will force it to reconnect to our C&C servers.

    I am extremely sorry for the inconvenience this has caused for many of you and we are taking steps to make sure this will not happen again in the future.

    Best,
    Timur

  5. #5
    Untangler
    Join Date
    Jul 2017
    Posts
    39

    Default

    If I can log into the box locally through the UI can I still run the script?

  6. #6
    Master Untangler johndball's Avatar
    Join Date
    Apr 2008
    Location
    Virginia
    Posts
    179

    Default

    I opened a ticket on this issue and the CSR redirected me here. This is exactly what I'm experiencing. However, I noticed that the certificate being used is invalid... very invalid. Not sure if that is contributing to the issues at hand or what, but this should raise some flags. This is the SSLlabs report from cmd.untangle.com and, looking at my network inspection gear, my Untangle device is not trusting this new certificate which was issued yesterday for cmd.untangle.com which is the frontend URL for the C&C.

    ETM cmd.untangle.com certificate.PNG

    ETM cmd.untangle.com certificate 2.PNG
    --
    "I have often regretted my speech, never my silence." - Xenocrates
    "Untangling" since 2007
    https://www.johndball.com

  7. #7
    That Which Lurks Below
    Join Date
    Jul 2018
    Posts
    143

    Default

    Quote Originally Posted by DolphinComputers View Post
    If I can log into the box locally through the UI can I still run the script?
    Yes, it doesn't have to be via SSH. You just need to access the NGFW's command line; how you do so is immaterial.
    Græme Ravenscroft • Technical Marketing Engineer
    ('gram', like the unit of measurement)
    he/him
    Please don't reboot your NGFW.
    How can we make Arista ETM products better?

  8. #8
    Untangler
    Join Date
    Jul 2017
    Posts
    39

    Default

    Also, I still cannot add new appliances, I have 3 firewalls going in this week but cannot finish the installs because the appliance can't connect to the ETM, is there a work around for this? Frustrating I cannot do anything without the ETM.

  9. #9
    That Which Lurks Below
    Join Date
    Jul 2018
    Posts
    143

    Default

    Quote Originally Posted by DolphinComputers View Post
    …is there a work around for this?
    [information superseded by official patch rollout]
    Last edited by gravenscroft; 08-11-2022 at 09:27 AM.
    Græme Ravenscroft • Technical Marketing Engineer
    ('gram', like the unit of measurement)
    he/him
    Please don't reboot your NGFW.
    How can we make Arista ETM products better?

  10. #10
    w00t
    Join Date
    Aug 2016
    Location
    Boulder, CO
    Posts
    27

    Default

    I assume this is a MicroEdge device? The certificate update took place, but there is still an issue with the script. We will remedy that today hopefully, with an update. If you pull the script per my instructions, it should work.

Page 1 of 6 123 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2