Page 1 of 3 123 LastLast
Results 1 to 10 of 23
  1. #1
    Untangle Ninja
    WebFooL's Avatar
    Join Date
    Jan 2009
    Location
    Sweden (Eskilstuna)
    Posts
    5,050

    Default POC Logstash/Kibana/Elasticsearch and Untangle

    So we have started to use Logstash/Kibana/Elasticsearch as a central log management tool and i am really impressed on what the platform can do.

    I started to think on how Untangle NG Admins really could use this tool in there management so i have a VM ready with a default installation that i am thinking of uploading somewhere and linking to this thread.

    But first do we have any Logstash/Kibana/Elasticsearch gurus here?
    If so would you like to chime in and create some good dashboards in the VM before i release it.

    Else i will try to refine the VM and instructions on how to "install" it during the weekend for release.

    If this is a success you should just have to configure the new VM as syslog server and it will start generating "realtime" logs. :-)

  2. #2
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    Hi,

    I can't claim to be an expert with logstash but I do use it frequently, I have configured a basic filter to process incoming syslog messages from untangle (although I have had some issues, which I have posted about). Currently I have had to modify it heavily to investigate the mentioned issue, but I can provide it if you think it would be of interest.

    How did you get on with creating dashboards for kibana focused on Untangle? I am currently just setting it up to store the logs, and haven't got to dashboards yet, so would be interested to see what you have!

    Thanks

  3. #3
    Untangle Ninja
    WebFooL's Avatar
    Join Date
    Jan 2009
    Location
    Sweden (Eskilstuna)
    Posts
    5,050

    Default

    I have to admit that it is on low priority at the moment.
    We have more critical systems that we are creating dashboard for.

    Next week I have a meeting with our Logstash guy will see how his nere future looks and if I can get him to do some sample dashboarda.

    If the community shows more of a intresse I think I can move it quicker but right now it is kind of just u and me :-)

  4. #4
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    haha, that's fair enough, I plan on making something myself in the future, but I also don't have much time to spend on it. Hopefully if I can get the data collection right, the dashboards can come later and I will have everything I need to get all the information I want (I am keeping pretty much all the data currently - I might change that down the line once I know what is the most useful).

    If you can give me an idea what you were hoping to get out of the data it might give me some ideas. I'm more than happy to share what I come up with (if anything)

    I'm currently using kibana 3, but might try out k4 beta some time soon, as it's meant to have some nice features on the way (like saving to csv) which will be needed as I expect web filter logs to be regularly searched and team managers will want evidence etc...

  5. #5
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    I've been looking in to this yesterday and today, I am a little stuck however. I am going to raise a support ticket about it to see if there is any way to change this, but I thought I would post here as well. I have found that when a user requests a web page the following syslog classes are generated, after each class I have noted the kind of useful information which is presented on each type of message:

    node.firewall.FirewallEvent - ruleId, blocked, flagged & sessionId
    node.http.HttpRequestEvent - Requested URI, websiteHost, sessionId, username & hostname
    node.webfilter.WebFilterEvent - blocked, flagged, category
    node.http.HttpResponseEvent
    node.classd.ClassDLogEvent - flagged, blocked, hostname, username

    All the events have one piece of information which links them together, every log entry has either TCP_93650930808539 or TCP93650930808539 near the start of the message, this appears to me at least identify those as messages generated from the same request.

    When I look at the web filter event logs in untangle however, there is a column for username, the same goes for firewall event log.

    I am wondering if there is any way for me to change the way Untangle produces syslog messages so that the username is displayed on all of these classes of syslog messages, that way I can search by username and easily see how many requests were blocked, then drill in to those and see what sites were blocked for that user.

    If that's not possible I would have to figure out a way to get these log entries associated with one another again, something I'm not sure how to do, and which will involve adding a lot of logic to my filtering, which I would rather not do if it can be avoided. Untangle clearly knows that this Firewall event relates to a certain user, but that information isn't reflected in the logs.

    Here is a pastebin of the 5 log entries I have been looking at as my example: http://pastebin.com/ZPcWbU38

  6. #6
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,486

    Default

    Quote Originally Posted by Rumbles View Post
    I am wondering if there is any way for me to change the way Untangle produces syslog messages so that the username is displayed on all of these classes of syslog messages, that way I can search by username and easily see how many requests were blocked, then drill in to those and see what sites were blocked for that user.
    No, that information is not in those events.

    The event logs show the data in the database. The events modify the contents of the database.

    It seems to me like you might just be better off accessing the database directly instead of trying to reconstruct it from the events.
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  7. #7
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    Quote Originally Posted by dmorris View Post
    No, that information is not in those events.

    The event logs show the data in the database. The events modify the contents of the database.

    It seems to me like you might just be better off accessing the database directly instead of trying to reconstruct it from the events.
    Thanks for the quick reply!

    However, I'm not quite sure I understand, you say the information is not in those events, but when you look at the event log in Web Filter and Firewall, that information is displayed, it just doesn't get reported on the same events in syslog.

  8. #8
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,486

    Default

    Quote Originally Posted by Rumbles View Post
    Thanks for the quick reply!

    However, I'm not quite sure I understand, you say the information is not in those events, but when you look at the event log in Web Filter and Firewall, that information is displayed, it just doesn't get reported on the same events in syslog.
    The event log looks at the contents of the database.
    You are looking at the events which add/delete/modify contents in the database.
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

  9. #9
    Untanglit
    Join Date
    Apr 2015
    Location
    Sheffield UK
    Posts
    26

    Default

    Quote Originally Posted by dmorris View Post
    The event log looks at the contents of the database.
    You are looking at the events which add/delete/modify contents in the database.
    Do you mean that the syslog messages only show events which add/delete/modify the content of the database?

  10. #10
    Untangle Junkie dmorris's Avatar
    Join Date
    Nov 2006
    Location
    San Carlos, CA
    Posts
    17,486

    Default

    Quote Originally Posted by Rumbles View Post
    Do you mean that the syslog messages only show events which add/delete/modify the content of the database?
    syslog messages *are* events which add/delete/modify the content of the database. (well, the JSON representation of the events anyway)
    Attention: Support and help on the Untangle Forums is provided by volunteers and community members like yourself.
    If you need Untangle support please call or email support@untangle.com

Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2