Page 1 of 2 12 LastLast
Results 1 to 10 of 12
  1. #1
    Master Untangler dmor's Avatar
    Join Date
    Jun 2009
    Posts
    668

    Default Upgrade from 13.2.1 to 14.0 to 14.1 to 14.2 breaking custom kernel modules

    Hello,

    I have a support ticket related to this, but I think I have finally gotten to the bottom of it (sort of) and wanted to share.

    Last year we deployed a SuperMicro SoC platform using the new Intel C3000 platform. Had issues with the NICs not being recognized and documented my journey to resolving that here:

    https://forums.untangle.com/hardware...-platform.html


    I had to build the custom kernel modules from the driver source downloaded from Intel.

    I was under the impression that these kernel modules would continue to be used unless Untangle did a major kernel upgrade. It is Untangle's practice (generally) to not push out the new kernels to existing deployed firewalls. Instead they make it available, but leave the old kernel on each deployed firewall as the default in order to avoid potential hardware compatibility issues.

    By disabling auto-upgrades in the Untangle GUI, I believed I would be safe from any changes that would mess up the box.

    We deployed it as
    13.2.1 which at the time had kernel 4.9.0-6

    We later upgraded it to
    14.0 which brings kernel 4.9.0-7
    (I do not think we ever rebooted it when installing this update, whereas the wiki says we are supposed to)

    We later upgraded it to
    14.1 which does *not* bring a new kernel (to my knowledge)

    Yesterday on-site I upgraded it to
    14.2 which brings kernel 4.9.0-8


    I have 2 major issues on my hands here which I need to find a solution for:
    (and explanation for, so I can fully understand what's going on)
    1. Why is the system defaulting to the new kernel each time these are installed & the next reboot occurs?
    2. Anyone have any ideas on how to make the kernel modules I've built be persistently functional across these minor patched kernel versions, so I don't have to build & install the kernel modules again each time?



    So basically the system is booting into the new kernels by default. I know this is not supposed to be the case. At least not generally. This is specified here:
    https://wiki.untangle.com/index.php/14.0.0_Changelog
    and in various other places & has been the standard practice from Untangle for the past couple of years (in my experience).

    None of the release notes beyond v14.0 mention a kernel update/patch.

    However, this Jira ticket gives the impression that Untangle is auto-switching us to the newer kernel?:
    https://jira.untangle.com/browse/NGF...3Aall-tabpanel


    I'm not sure why these kernel changes are not being put in the release notes. Perhaps Untangle considers them trivial.

    If someone can please shed some light on this situation, I'd greatly appreciate it.

    We were under the impression that each time we installed these updates, the system would keep booting the old kernel and we wouldn't have hardware (kernel module) issues.

    However, yesterday while on-site I watched with my own eyes as I upgraded the system from 14.1 to 14.2 that upon reboot it defaulted to the newest kernel (4.9.0-8). I even took a photo of this with my phone as proof.

    Each time these minor kernel changes occur, we no longer have the kernel module for the NICs and end up with a broken firewall.


    Thanks in advance for any help here!!
    -
    Doug

  2. #2
    Untangler jcoffin's Avatar
    Join Date
    Aug 2008
    Location
    Sunnyvale, CA
    Posts
    7,941

    Default

    If you are installing custom modules in the kernel, I would turn off upgrades and start testing upgrades offline. We do this with our hardware to verify the kernels are compatible. It's the risks assumed if you are not using the stock kernel.
    Attention: Support and help on the Untangle Forums is provided by
    volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  3. #3
    Master Untangler dmor's Avatar
    Join Date
    Jun 2009
    Posts
    668

    Default

    Quote Originally Posted by jcoffin View Post
    If you are installing custom modules in the kernel, I would turn off upgrades and start testing upgrades offline. We do this with our hardware to verify the kernels are compatible. It's the risks assumed if you are not using the stock kernel.
    So for the sake of understanding, based on the procedure listed on my prior thread from last year, I'm actually installing the module into the new kernel? I was thinking I was building the kernel module outside of the kernel and somehow the kernel was loading it. But I know there are supposed to be some additional measures for that to work, such as /etc/modules (or the other way that calls it on demand). I'm not having to do those, so maybe that means the method I used did actually compile it into the kernel? If so, that wasn't my intention. I'm not an expert on compiling kernel modules, although this has worked well.

    (feel free not to answer the above question if you want, since it is of course way out of the scope of Untangle support).

    I think I'd rather have the module compiled outside the kernel and referenced such as through /etc/modules. I think that would be more resilient. What do you think?

    Also this still doesn't explain why these new kernels are being booted by default.

    Thanks so much for your help!!!
    -
    Doug

  4. #4
    Untangler jcoffin's Avatar
    Join Date
    Aug 2008
    Location
    Sunnyvale, CA
    Posts
    7,941

    Default

    I understand you wanted to get this motherboard working with Untangle. But any CLI changes are not supported and the device should have upgrades off so you can do your own testing with your custom code base.
    https://wiki.untangle.com/index.php/Console

    Terminal_warning2.png
    Attention: Support and help on the Untangle Forums is provided by
    volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  5. #5
    Master Untangler dmor's Avatar
    Join Date
    Jun 2009
    Posts
    668

    Default

    I understand and it is not Untangle's problem. I just would like to understand the situation with the system booting the new kernels. I believed that was not supposed to happen by default. I don't know if that's a bug/regression that Untangle devs should look into, or whether there's something else at play that I'm not understanding.

  6. #6
    Untangle Ninja sky-knight's Avatar
    Join Date
    Apr 2008
    Location
    Phoenix, AZ
    Posts
    23,455

    Default

    Untangle doesn't default to the new kernel on next reboot, you have to choose the new kernel manually. Old kernel modules should still be there with the old kernel. I cannot explain why your unit is updating the kernel on restart, that's not normal behavior.

    It may be that Untangle is doing some minor kernel changes automatically do deal with security problems.

    And no, there is no way to make things persistent, Debian either supports it or doesn't. The good news is, the younger the Debian is, the more likely it'll be supported in the future. But, this is the issue with kernel modules, they must be compiled against the specific kernel you're using. Any version change at all, and BOOM broken.
    Rob Sandling, BS:SWE, MCP
    NexgenAppliances.com
    Phone: 866-794-8879 x201
    Email: support@nexgenappliances.com

  7. #7
    Untangler jcoffin's Avatar
    Join Date
    Aug 2008
    Location
    Sunnyvale, CA
    Posts
    7,941

    Default

    Minor kernel changes will automatically upgrade to the new kernel on reboot 4.9.0-6 -> 4.9.0-7 -> 4.9.0-8. It's safe for stock Untangle OS as the built-in drivers have not changed.
    Last edited by jcoffin; 06-14-2019 at 12:12 PM.
    Attention: Support and help on the Untangle Forums is provided by
    volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  8. #8
    Master Untangler dmor's Avatar
    Join Date
    Jun 2009
    Posts
    668

    Default

    Quote Originally Posted by sky-knight View Post
    Untangle doesn't default to the new kernel on next reboot, you have to choose the new kernel manually. Old kernel modules should still be there with the old kernel. I cannot explain why your unit is updating the kernel on restart, that's not normal behavior.

    And no, there is no way to make things persistent, Debian either supports it or doesn't. The good news is, the younger the Debian is, the more likely it'll be supported in the future. But, this is the issue with kernel modules, they must be compiled against the specific kernel you're using. Any version change at all, and BOOM broken.
    Thanks Rob. I am in complete agreement with everything you and John are saying. But I can promise you the box picked the new kernel automatically yesterday when I was on-site. It was highly unexpected to me because I know it's not supposed to. But I watched it happen upon first reboot after upgrading from 14.1 to 14.2.

    I'm just wondering if maybe Untangle does in fact switch to these new kernels automatically when we install the updates manually. Maybe they only withhold booting to new kernels, when it's like a major or minor kernel version, rather than a measly patch number...?

  9. #9
    Master Untangler dmor's Avatar
    Join Date
    Jun 2009
    Posts
    668

    Default

    Quote Originally Posted by jcoffin View Post
    Minor kernel changes will automatically upgrade to the new kernel 4.9.0-6 -> 4.9.0-7 -> 4.9.0-8. It's safe for stock Untangle OS as the built-in drivers have not changed.
    Ahhh. Thank you. That clears up everything. Now I know what we need to do with this particular box going forward to ensure this never happens again. My customer will also be happy to know we discovered the root cause and can assure them this won't recur.

    Thanks again.
    -
    Doug

  10. #10
    Untangle Ninja sky-knight's Avatar
    Join Date
    Apr 2008
    Location
    Phoenix, AZ
    Posts
    23,455

    Default

    Quote Originally Posted by dmor View Post
    Ahhh. Thank you. That clears up everything. Now I know what we need to do with this particular box going forward to ensure this never happens again. My customer will also be happy to know we discovered the root cause and can assure them this won't recur.

    Thanks again.
    -
    Doug
    Yeah, it seems like you're stuck on manual upgrade mode forever going forward until the Intel drivers work their way into the kernel. That will happen, the question is when. Honestly, I'm curious what NICs you have that v14 doesn't support, because the Intel umbrella there is pretty darned complete.
    Rob Sandling, BS:SWE, MCP
    NexgenAppliances.com
    Phone: 866-794-8879 x201
    Email: support@nexgenappliances.com

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2