Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 23
  1. #11
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    Quote Originally Posted by dmorris View Post
    syslog messages *are* events which add/delete/modify the content of the database. (well, the JSON representation of the events anyway)
    Thanks for the info,

    The documentation seems to imply slightly different: http://wiki.untangle.com/index.php/Reports#Syslog

    Reports supports the sending of all events via syslog messages
    Which is the information I was going off when I started looking in to this.

    Regardless of that fact, since the event viewer for the Firewall and Web Filter in the Untangle Web GUI displays the information about username and hostname against each web request, and that information is taken from the database, is it not possible to display the same information in the syslog messages for the events which update the database? The information must be there for the Web GUI to display it.

    I understand that you said I could just connect to the DB to get this information, but since the DB gets flushed out every x days because our Untangle appliance would become unstable if it were kept indefinitely I would need to clone the Untangle DB onto another DB just to keep the records for if someone requested information about someone's web usage from more than a few weeks back. While also collecting and processing the syslog messages (for compliance reasons we need a centralised logging solution) which are much less useful than they could be (since the information produced is disconnected from the user/hostname that generated them).

    Is there no possible way to get this information from the DB for each relevant syslog message? In other words, when the records in the database are changed, can't the syslog message reflect the username and other information that is also held in the database, not just the information that has changed?

    Thanks for your help...

  2. #12
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,486

    Default

    Reports supports the sending of all events via syslog messages
    syslog messages *are* events which add/delete/modify the content of the database. (well, the JSON representation of the events anyway)
    I'm not seeing it. These are both true.

    Syslog sends events.
    Events modify/delete/add content to the database.

    Maybe the cofusion in that you expect to see whats in the 'event logs' in the UI?
    Those view the fully denormalized contents of the database.
    For example, http_events stores all the web hits. HttpRequestEvent will create a new row, HttpResponseEvent will update that existing row, other events like Virus events may further modify that row.


    Regardless of that fact, since the event viewer for the Firewall and Web Filter in the Untangle Web GUI displays the information about username and hostname against each web request, and that information is taken from the database, is it not possible to display the same information in the syslog messages for the events which update the database? The information must be there for the Web GUI to display it.
    No because that information is in the database, not the events.

    Anything is possible. Untangle is open source and you are welcome to modify it.
    I think you'll find that replicating all information in all events will be a terrible idea. Doing a database lookup on each event to populate the missing information would probably cripple even relatively small deployments, but you are welcome to it.

    Ultimately you'll end up writing a way to convert events into sql so you can store the content of 'whats happening' in a database.
    Thats exactly what Untangle already does in its own database, which is why I would suggest you just look at that if you want a fully denormalized view.
    Last edited by dmorris; 04-21-2015 at 10:04 AM.
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  3. #13
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    Quote Originally Posted by dmorris View Post
    I'm not seeing it. These are both true.

    Syslog sends events.
    Events modify/delete/add content to the database.

    Maybe the cofusion in that you expect to see whats in the 'event logs' in the UI?
    Those view the fully denormalized contents of the database.
    For example, http_events stores all the web hits. HttpRequestEvent will create a new row, HttpResponseEvent will update that existing row, other events like Virus events may further modify that row.
    Nice one, that explains a lot, thank you. I completely misunderstood what I was looking at. Sorry about that.

    I think I have some reading to do...

  4. #14
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    My hopes with logstash was that I would be able to take the syslog messages and index them in elasticsearch, so that I would be able to report in kibana that users on each particular rack have had x web filter hits, y firewall hits, etc. Clearly that's not possible, because the syslog messages don't carry all of this information.

    So, would it be possible to do this from the information available in the syslog messages? I have a feeling I am getting out of my depth a bit...

  5. #15
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,486

    Default

    Quote Originally Posted by Rumbles View Post
    My hopes with logstash was that I would be able to take the syslog messages and index them in elasticsearch, so that I would be able to report in kibana that users on each particular rack have had x web filter hits, y firewall hits, etc. Clearly that's not possible, because the syslog messages don't carry all of this information.

    So, would it be possible to do this from the information available in the syslog messages? I have a feeling I am getting out of my depth a bit...
    Its in there, but not organized as you want.
    You can choose to reorganize it, which as you pointed out is a difficult task.

    Alternatively, you can use the data in the database, which is organized as you wish, but still must be pulled from postgresdb.

    Ultimately I don't know what to suggest as I'm not sure what your goals are.

    We're currently doing a lot of work on reports. Let me assure you if you are trying to implement your own reporting platform you are in for a ton of work. Thats the reality of it. I know every reporting platform claims to easily take your data and spit out beautiful metrics and graphs. Thats not reality. It takes a ton of work to do it and even more to make sure you do it in some scalable way.

    If you're just looking to get reports by rack, then I'd just wait on the report improvements.
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  6. #16
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    I need to be able to keep my reports for much longer than you currently support. How long are you planning on allowing users to keep reports for?

  7. #17
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,486

    Default

    I'm not sure.

    How long do you need to keep report data?

    btw, you can just set whatever retention time you want in the settings file. This isn't allowed in the UI because people just set it to huge numbers without understanding the implications. Then they ultimately have a bad experience and have no idea why. Thats not desirable for them or us.
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  8. #18
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    I was planning on keeping everything for at least 2 years.

    In elasticsearch you can keep your recent stuff indexed and immediately available for searching, then close records over a certain age. Those are still available for searching if you request it, but it takes longer.

    Since ELK is built for reporting it makes sense to use it for reporting, why reinvent the wheel when you don't need to.

  9. #19
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    Quote Originally Posted by Rumbles View Post
    I was planning on keeping everything for at least 2 years.

    In elasticsearch you can keep your recent stuff indexed and immediately available for searching, then close records over a certain age. Those are still available for searching if you request it, but it takes longer.

    Since ELK is built for reporting it makes sense to use it for reporting, why reinvent the wheel when you don't need to.
    Correction, closed records cannot be searched, but they can be manually re-opened. They stop using RAM when closed and just live on disc.

  10. #20
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    Hi,

    Just to update, I have decided to turn off syslog messages from my untangle server and will no longer be processing them with Logstash, there are a few reasons.

    First, rsyslog on Untangle cannot currently be encrypted. I have raised a bugzilla feature request for this, as in my environment I am not willing to have sensitive data sent across the network unencrypted.

    The syslog messages from Untangle are no use for trying to report on the functionality of Untangle, as they only show the updates to the database, without the database structure being mirrored in elasticsearch we cannot use this transactional data, as too much information is missing from the transactions to be of use for reporting.

    If the situation changes (or someone points out how wrong I am) I'm more than happy to revisit this, but I don't see reporting via ELK being possible at present.

    Thanks for the help everyone

Page 2 of 3 FirstFirst 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2