If it's not handshaking then you've configured your client incorrectly, or you've not forwarded UDP 51820 to your Untangle.
And for the record, I did read your original post... several times. Did so twice again just now. Nothing in that post indicates to me you've actually tried to deploy this solution. What I interpret is theory crafting on the possibility.
And it most certainly IS possible. And once again if you can't handshake, then something is wrong with the UDP stream getting to Untangle. That could be a client configured to connect to the wrong IP address, a lack of a port forward on the upstream router that owns the appropriate IP address... something in that chain.
Now, if you can't route over the tunnel once its built! THAT's something else, and probably a lack of a static route to support the IP range wireguard is using.