Caching proxy?

[stlscott]

Some time ago there was someone from UT that commented on this and that due to the way in with the firewall & packet inspections work, a bunch of stuff would have to be re-written to support the Proxy firewall we all so desire.

So here's my solution.... in cases where I need a proxy firewall, I use Endian or Astaro. IN cases where I don't and my firewall is beefy enough (UT is quite bloated compared to others), I use UT. I love the product and have over 20 in the field... but it doesn't fit all the neads, unfortunately. However, I love how it does VPN so I use it a lot for that.

For solutions where I want to have UT and a Proxy server, I simply setup a separate Linux server (usually as a VM) and setup a Squid based proxy server on that, and then configure the firewall to only allow traffic on port 80 & 443 to come from the Proxy server... Dan's Guardian is a nice one to use.

Just my 2 cents...

Scott

Hey, I have tried to delve into this issue.

I would like to take a crack at developing this feature.

I would like to make it transparent, but allow the interface to opt in ports for the proxy. This way traffic can be routed specifically into the proxy based on port. If the proxy was put inline with the rest of the traffic flow would this not be possible while maintaining the untangle module ethic? I don't want to make this sound trivial, but if there has been any progress made I would love to see it and lend a hand. if not I would like to spend a few cycles seeing if there is an elegant way to accomplish this.

Any tips/suggestions from those in the know with untangle at how you would go about cracking this nut?

Thanks,
-Alex

[enderandrew]

I understand there is a level of complexity here. Yet the purpose of Untangle is to "untangle" as it were, the series of net appliances you had before and consolidate to one box. So when people suggest the best solution is to add another box in line rather than try to integrate within Untangle, I feel that goes against the spirit of this project.

Untangle appeals to me because of the single-box nature. I know what works. I can stick with existing solutions.

However many people do currently use proxy caches. The benefits are very real. I'm not sure how others are trying to trivialize this. Again, part of the design of Untangle is choice and flexibility. Not all modules have to be loaded. Untangle is supposed to offer a bevy of features and let each administrator choose what they want for themselves.

And while this feature would take some work to properly integrate, I find it disappointing that as the most requested bug on your bugzilla, it has sat ignored for years.

I'm looking at a product I can push to my clients and resell. Should I suggest Untangle support to my clients when Untangle does not appear to provide support for their existing users?

If the solution truly must be that squid/advproxy/another product must be in another box, why not prepackage a series of VMs that can be administered from one console then? It sure doesn't appear like Untangle wants to provide the solutions their customers are clammoring for.

[mdh]

I'm looking at a product I can push to my clients and resell. Should I suggest Untangle support to my clients when Untangle does not appear to provide support for their existing users?

Please name a service that Untangle offers which they do not provide support for. You are defining "lack of support" arbitrarily. If they elect to offer one, you can be sure that it would be supported.

If the solution truly must be that squid/advproxy/another product must be in another box, why not prepackage a series of VMs that can be administered from one console then? It sure doesn't appear like Untangle wants to provide the solutions their customers are clammoring for.

Untangle has products in the pipeline, products that are on roadmaps from Board of Directors and senior management, outstanding issues and enhancements that need to be dealt with, and a finite number of developers to do the job. If there were 10 extra coders sitting around with nothing to do, your statement would have a great deal of validity. Unfortunately, that's not the case here.

[enderandrew]

Please name a service that Untangle offers which they do not provide support for. You are defining "lack of support" arbitrarily. If they elect to offer one, you can be sure that it would be supported.

Supporting a project is more than simply telling people how to use your existing product. Untangle, Inc. has to allocate resources to managing the Bugzilla, these forums, and planning future development. These are all aspects of supporting Untangle as a project.

Invariably, there will always be a bugzilla issue with the most votes. Knock it off, and another replaces it. I'm not concerned that there is an often requested feature. I'm concerned that such a feature has a commanding lead in votes, and yet has been summarily ignored.

However, this has twice the votes of the next issue in line, and has been than way for years. That demonstrates that Untangle, Inc. has ignored community feedback via the forums and Bugzilla. They haven't properly planned on how to allocate resources to those aspects of the project. Either they have worked on the issue, and haven't responded back to users via the forums and Bugzilla in a satisfactory manner, or they haven't worked on the issue at all, which demonstrates a failure on their part to plan for future development.

This isn't a random feature request amonsgt the myriad of feature requests. This is the single most requested feature on the forums and bugzilla, that also happens to coincide with the supposed core goals of the project. It is also a standard, expected feature that all the competing products offer. Ignoring that for years is a failure in support.

Untangle has products in the pipeline, products that are on roadmaps from Board of Directors and senior management, outstanding issues and enhancements that need to be dealt with, and a finite number of developers to do the job. If there were 10 extra coders sitting around with nothing to do, your statement would have a great deal of validity. Unfortunately, that's not the case here.

Untangle has a major new release in beta that doesn't offer much in the way of new features. Minor point releases typically don't offer major features.

Untangle, Inc. as a company may lack the resources to achieve their goals. If that is the case, they could allocate more resources into community outreach to leverage community-driven opensource development. Browsing through the bugzilla, I'm not seeing much in accepted patches from outside development.

Every project has finite resources. Ignoring the largest requests of your users however does not count as great support, regardless of how finite those resources are.

Again, I'm not simply looking for a free, OSS product I can slap down. I have products like IPcop for that currently. I'm looking for a polished, streamlined, feature-rich product I can push to clients, rebrand, and resell.

For a moment I thought Untangle might be that product. It does not appear to be so.

[sky-knight]

enderandrew
Untanglit

Join Date: Nov 2009
Posts: 12

In short, you haven't been here long enough to see this process work. Prior to the caching proxy insanity, the most voted for and yelled about feature was multi-wan. The forums have always been in favor of the caching proxy, but multi-wan is in the product.

Why is multi-wan here and proxy not? Simple, we resellers yelled for multi-wan. This isn't a game of support, it's a game of sales.

The caching proxy is a nightmare and a half to implement in the current untangle realm. Go through that nightmare, for what? A debatable performance improvement?

Untangle doesn't do anything unless they feel that can do it well. At this point there is debate on whether or not this feature even has merit, despite what the zealots that drool on this idea may say.

Beyond that, bugzilla has never been what I would call... current. The Devs just don't put thier notes in there. So one day we'll wake up and see an announcement for the next big version and proxy may just be there.

7.0 introduced a new reports engine, 7.0.1 was released to address some massive bugs located in that engine and in other things. 7.1 so far as been more about addressing current issues and getting the product stable again before major new features are inserted.

Publicly, we have no idea what is planned for 7.2. But I can tell you from what I know of Untangle in the past. They are going to only roll a feature this big, by itself. Essentially, say 7.2 had the proxy component, it would be the "proxy release" meaning there won't be any other changes to the Untangle software other than bug fixes and that one new massive feature.

The reports module in 6.2 was a hopeless mess, now we have another mess in 7.0.1 but at least the platform the reports are built on is workable. We have to get that feature fixed before we can move forward.

One thing at a time...

[IHateShuttle]

I was just thinking about implementing a web cache today and I found this thread. I am bumping for if it were there I would use it. I am not here to make demands or slander.

This thread is tainted with the filth of nerd rage. Please show some respect. I challenge any hater out there to make squid work with untangle. If you want something done, do it yourself. It makes no sense to offend the great people at untangle. Especially if you use their product.

I have been happy with this software. Thank you.

[sky-knight]

No nerd rage... but you apparently hate shuttle...

As do I!

That said, we've had forum members get squid installed into Untangle and working. The trouble is... the way the untangle rack works traffic must be picked up on one interface, passed into the UVM, then dropped out another. By installing the proxy and making the changes to route traffic into the proxy and then back out again... you've essentially just replaced the UVM with the proxy.

In simpler terms, you can have the proxy, but none of the other untangle defenses with it.

I still maintain, that this is much much much much easier deployed independently. If you want to save hardware Untangle runs really well if configured properly within ESXi. An Untangle router with a proxy server behind it works. And, it works today.

[stlscott]

I've been using Untangle for 4-5 years now and have been "around" for quite some time. I've also been puting Linux into production environments since 1997. I was at the Linux World Expo when Untangle had their first booth there... While I realize that maybe the other guy hasn't been around that long, I have.

Now on to my comment..

While any of us geeks do really understand that there is a lot that goes into development, I think the real point that was being made here is that it's sad that this has been an ongoing request for so many years. This thread alone started in 2007 and it wasn't the first thread. I know, because I had posted in prior threads about this very issue. Yes, Wan Fail-over was a high demanding item on the list, Proxy Caching has been at the very top all along as well. Perhaps the issue is less about development and resources and more about a lack of communication from the Untangle team as to where this item sits on the development priority list and where in the process it is for development. Sometimes, something as simple as "It's our next project" or "12-18 months is what we think we can complete it in" is really enough. However, I have not found any dialog from the Untangle folks as to the status of this request.

The argument as to the negligible gain for this enhancement is nonsense since most other competing firewall solutions include this functionality. These other project groups (or corporations) wouldn't keep it in their products if customers didn't want it and didn't use it. While bandwidth for home users is often cheap, bandwidth for many businesses is still very expensive. I work in a business office park in St. Louis county (2.5 million people in the area) and there is no DSL or Cable service. The phone company & cable company are unwilling to invest in cheap Internet for this area since businesses will pay for the high priced services. We spend $1200 a month for 4.5mb total bandwidth between two ISP's, and this was the least expensive solution. There are many areas like this all across the country where bandwidth is still expensive and where Proxy Caching is still necessary to reduce overall bandwidth use. On a weekly basis, we average a 40% cache hit, that means that my bandwidth usage would be 40% higher if it wasn't for the cache. This is with an office of 50 people.

Lastly, if this project has run into resource issues, then ask the community to participate in the development of the Proxy addition. I realize that this means that you can't charge for the enhancement, if it was community driven and developed, but at least it would remove it from the requests, complaints, etc list and you could move on to other enhancements that could be charged for. The Untangle folks should at least consider making this addition a community (and free) one for this reason alone.

In short, you haven't been here long enough to see this process work. Prior to the caching proxy insanity, the most voted for and yelled about feature was multi-wan. The forums have always been in favor of the caching proxy, but multi-wan is in the product.

Why is multi-wan here and proxy not? Simple, we resellers yelled for multi-wan. This isn't a game of support, it's a game of sales.

The caching proxy is a nightmare and a half to implement in the current untangle realm. Go through that nightmare, for what? A debatable performance improvement?

Untangle doesn't do anything unless they feel that can do it well. At this point there is debate on whether or not this feature even has merit, despite what the zealots that drool on this idea may say.

Beyond that, bugzilla has never been what I would call... current. The Devs just don't put thier notes in there. So one day we'll wake up and see an announcement for the next big version and proxy may just be there.

7.0 introduced a new reports engine, 7.0.1 was released to address some massive bugs located in that engine and in other things. 7.1 so far as been more about addressing current issues and getting the product stable again before major new features are inserted.

Publicly, we have no idea what is planned for 7.2. But I can tell you from what I know of Untangle in the past. They are going to only roll a feature this big, by itself. Essentially, say 7.2 had the proxy component, it would be the "proxy release" meaning there won't be any other changes to the Untangle software other than bug fixes and that one new massive feature.

The reports module in 6.2 was a hopeless mess, now we have another mess in 7.0.1 but at least the platform the reports are built on is workable. We have to get that feature fixed before we can move forward.

One thing at a time...

[sky-knight]

The community has tried many times to implement the proxy, and has failed each time.

Also, in the last 5 years there has been a very large shift in the types of web content commonly accessed on the web. Web 2.0, means AJAX, AJAX means dynamic content. Dynamic content isn't cache-able. So yes, there is a massive debate as to the merit of a proxy. It isn't the drop in and go solution to save bandwidth that it used to be. Existing products are often built on old ideas and old ways of doing things. They have no real desire to pull a feature that exists because it tends to make users grumpy.

I see that in your case the proxy is worth the investment. However, in every case I've tried a proxy here the best I've seen is 5%. Even if I saw a 10% hit I'd probably use the darn thing... but in my case the proxies end up being more of a problem in and of themselves. Windows updates breaking at random, dynamic pages needing multiple refreshes to load, ajax components failing entirely...

And that cost for Internet is bloody ridiculous. I'd start leaning on my state legislature if I had to pay that kind of crap out here.

That and it appears the nature of the rack makes implementing a proxy feature very difficult.

[YeOldeStonecat]

I don't see why people want proxy server implemented in a UTM appliance. Firewalls need that CPU, why bog it down with the huge CPU load, RAM load, and massive hard drive load...of a proxy?

Yeah squid 'n stuff with other *nix distros..but they're lean mean fast distros like PFSense, where you can slap it on a honkin box and it'll perform well. But UT needs CPU, RAM, and some disk...it doesn't like to share, and it shouldn't share, it runs best when running on dedicated hardware (just looks at how it hates sharing in VMWare).

With most of the country on cheap fast broadband...apologies to the small percentage of those stuck in cup 'n string-ville...but, think about the dedicated manpower, salary, time, and effort for the UT team to implement this..and since Squid is free most people will demand it's part of the open source free package. Where's the business sense?

As an SMB consultant since way back in the dial up days, having done offices big 'n small, I'm still not a big fan of proxy/ISA. Haven't seen much of a benefit of it.

But for those demanding it....just think about what UT is, and thing about the logistics of it, and how cramming a proxy in the same box as UT would just.....be ...such an illogical choice. Talk about 2 things that should be kept separated.