I've installed splunk in order to see what it's like and what it does. It looks very nice at first glance, but I'm having a hard time understanding its purpose.
I'm looking for a nice log analyzer. I have services I want to monitor and report on, which have different log formats. For example:
Flash Media Server Shoutcast Audio server
Can splunk be used to analyze statistics, like other analyzers out there do?
thank you Ricardo
First I would advise that you check out splunkbase.com as there are a bunch of "apps" there, which really are just chunks of splunk configuration.
Splunk can do just about anything you want, but it has to be configured to do so. Check out the search language http://www.splunk.com/base/Documentation/4.1.5/User/AboutSearch
Splunk has a pretty powerful search language which should be able to do what you want.
Splunk is a general purpose log collector, indexer, and search engine. It can read in and process practically any text-based log file from any application. Once a log file has been processed by Splunk, its contents are stored in the Splunk index in a format that is designed to help make free-form searches against this data quick and easy.
Similar to a web search engine, you can perform searches against the Splunk index for arbitrary words and phrases. If you know your log files have interesting messages that contain the word "walrus", but they are only interesting if they don't also contain the word "bucket" then you can perform a search on
walrus NOT bucket
and Splunk will give you back all of the log messages in the index that have the world "walrus" but not the word "bucket".
Splunk is also able to extract "fields" from a log message. A field is a specific substring of the message that has meaning in and of itself. Using an apache webserver logfile as an example, some of the fields in that log file include:
For Apache, Splunk is pre-configured out-of-the-box to extract these fields. Once a field is extracted, then you are able to use the search language to perform all manner of operations with them -- count them, sum them, search for specific ones, graph them over time (and others).
I've not used Flash Media Server or Shoutcast Audio server, so I don't know offhand what their log messages look like, what field extractions would make sense, or what the meaningful statistics for them are. Splunk may already have configuration for these out-of-the-box, or someone from Splunk or the Splunk community may have made a Splunk App to work with these log formats.
Splunk Apps usually include a series of field extraction definitions, and a bunch of predefined searches to populate dashboards and reports with meaningful information. But, even with an app you aren't limited to just the predefined stuff -- you can always create your own searches to answer the questions that the app's author may not have anticipated.
The difference between Splunk and application-specific logfile analyzers (like AWStats or Webalyzer) is that Splunk isn't limited to one log format or a fixed set of "statistics". You may have to do some work in defining field extractions and/or searches to come up with the answer you're looking for. But, the data and the tools to analyze are there.