How to report top-ten errors over a time range

[mwlarsen]

I need to produce a "top-ten" error report from log4j logs. Specifically, I need to sort the logs by error type/text over a specific time range, count the occurrence of each error type/text, then report the top-ten occurrences from most-repeated to least-repeated, and print the text of the first occurrence of each error. I can't find any examples of this kind of application of Splunk. Can someone help?

[the_wolverine]

index=yoursearch errorfield=* | bucket _time span=1d | stats count as COUNT by errorfield,_time | top 10 COUNT,errorfield by _time | sort - _time COUNT, errorfield | fields - count,percent

[msacks]

Lots of people need a solution to a problem like this. It's a common issue, and one that's very lightly documented and vague. Splunk should create some kind of quick start guide for searching and categorizing common things like app server messages and apache messages.

[msacks]

Both for Java exceptions and Apache error / access logs, respectively.

[farhadialex]

Agreed, this should be documented in a quick start guide.

[wkozaczuk]

Does anybody have good answer for this problem? I have been looking for a solution in Splunk to address almost exact same need for long time now. It is one of the most critical things I would like Splunk to do.

Ideally I want Splunk to AUTOMATICALLY categorize exception stack traces (Java, C#, etc) or errors in general so it can report for example top 10 most frequent ones in last 30 minutes. I know that you can manually find an exception stack trace event (or any other error) in Splunk UI and tell it to save it so it remembers it as known event type and have it categorize from this point on. However I have not found a way where Splunk can be configured to automatically do the categorization of stack traces.

And that would be extremely helpful in many scenarios. In dev to see most frequent exceptions that probably need to be looked or addressed. Same way during production outage it would helpful to know top 10 exceptions in order to troubleshoot.

Anyone?

[msacks]

Exactly, this is what we are asking for. It would be nice if this was done in the search UI, a clear document was created on how to do it, or an app created to do it.

[mwlarsen]

Thanks so much for your quick response! Unfortunately, that didn't produce any usable results for me. Let me provide an example of what I'm looking for.

Say I have an error in a log that looks like this, and it's repeated 300 times for a given time frame:

5/10/11 7:14:52.322 AM 2011-05-10 07:14:52,322 [asyncDelivery12] ERROR com.acme.klassified.business.ContractKlassifedImageImplementation
- There are error(s) in setContractKlassifiedImage for the payloads.
The Number of Payloads processed: total = 2 failed = 1.

The parts of that error I care about from a grouping perspective are:

ERROR com.acme.klassified.business.ContractKlassifedImageImplementation
- There are error(s) in setContractKlassifiedImage for the payloads.

I need the date/timestamps (5/10/11 7:14:52.322 AM 2011-05-10 07:14:52,322), things in braces ([asyncDelivery12]) and the entire "Number of payloads" line excluded, because they all have the potential to make an error message unique.

Say that I have another error in the same log that looks like this, and is repeated 200 times in the same given timeframe:

5/10/11 2:00:11.694 AM 2011-05-10 02:00:11,694 [Thread-43] ERROR com.siriusforce.tools.server.activator.ToolsActivatorImpl - Unable to register
GetJBossServerStackTrace MBean, exception message=com.siriusforce:service=ClusterStackTrace already registered.
javax.management.InstanceAlreadyExistsException: com.siriusforce:service=ClusterStackTrace already registered.
at org.jboss.mx.server.registry.BasicMBeanRegistry.add(BasicMBeanRegistry.java:761)
at org.jboss.mx.server.registry.BasicMBeanRegistry.registerMBean(BasicMBeanRegistry.java:225)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.server.MBeanServerImpl$3.run(MBeanServerImpl.java:1422)
at org.jboss.mx.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:1417)
at org.jboss.mx.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:376)
at com.siriusforce.tools.server.activator.ToolsActivatorImpl.start(ToolsActivatorImpl.java:37)
at com.siriusforce.plugin.server.ServerPluginInjector.invokePluginActivators(ServerPluginInjector.java:242)
at com.siriusforce.plugin.server.ServerPluginInjector.start(ServerPluginInjector.java:134)
at com.siriusforce.plugin.server.PluginSystemImpl.start(PluginSystemImpl.java:36)
at com.siriusforce.plugin.server.ServerPluginSystemLoaderImpl$1.run(ServerPluginSystemLoaderImpl.java:61)

The parts of that error I care about from a grouping perspective are:

ERROR com.siriusforce.tools.server.activator.ToolsActivatorImpl - Unable to register
GetJBossServerStackTrace MBean, exception message=com.siriusforce:service=ClusterStackTrace already registered.

I need the date/timestamps (5/10/11 2:00:11.694 AM 2011-05-10 02:00:11,694), things in braces ([Thread-43]) and any line preceded by the word "at" excluded, because they all have the potential to make the error unique.

Ultimately, I need to search for instances of the word "ERROR", and any error text excluding date/time stamps, things in braces, and stack trace elements, then group "like" messages together, count how many times each error shows up in a group, and produce a report that includes the totality of the first instance of each error:

TOP-TEN Report for 5/9/11 13:00:00 to 5/10/11 06:00:00

The following error occurred 300 times:

5/10/11 7:14:52.322 AM 2011-05-10 07:14:52,322 [asyncDelivery12] ERROR com.acme.klassified.business.ContractKlassifedImageImplementation
- There are error(s) in setContractKlassifiedImage for the payloads.
The Number of Payloads processed: total = 2 failed = 1.

The following error occurred 200 times:

5/10/11 2:00:11.694 AM 2011-05-10 02:00:11,694 [Thread-43] ERROR com.siriusforce.tools.server.activator.ToolsActivatorImpl - Unable to register
GetJBossServerStackTrace MBean, exception message=com.siriusforce:service=ClusterStackTrace already registered.
javax.management.InstanceAlreadyExistsException: com.siriusforce:service=ClusterStackTrace already registered.
at org.jboss.mx.server.registry.BasicMBeanRegistry.add(BasicMBeanRegistry.java:761)
at...

-----------------------edit-------------------------------

I think I understand what you're suggesting. You forgot to paste a link, but I believe the page you were referring to is:

http://www.splunk.com/base/Documentation/4.2.1/Data/Configureindex-timefieldextraction

I followed the directions as best I could and did the following, adding the entries at the bottom of the edited files:

vi transforms.conf

[topten]
REGEX = ERROR.*?\n
FORMAT = top_ten::"$1"
WRITE_META = true

vi props.conf

[log4jlog]
TRANSFORMS-topten = topten

vi fields.conf

[top_ten]
INDEXED = true

I restarted Splunk to pick up the changes. However, when I go to the search page, I don't see a new field in the left frame. If I do a search on a log and follow your suggestion to:

| timechart count by top-ten

It just lists the errors in reverse chronological order. It doesn't group them by error message content, or count how many times that content shows up in the logs. If I generate a report on the returned data, it the sourcetype isn't top_ten like I expected it to be. Can you explain where I'm going wrong? Thanks for your help.

[jbsplunk]

It would be better if you could edit your description of this issue vs adding this as another answer. I'll edit my response above with what I think you need to do.

[jbsplunk]

you can probably do this by appending the following to your search:

| timechart count by errorfield

The trick is, when you are trying to pull data out of error logs, you need to make sure the fields you are trying to reference exist, and contain the information you'd like them to contain. To do this, you'd set up index time field extraction, which you can learn about here:

http://www.splunk.com/base/Documentation/4.2.1/Data/Configureindex-timefieldextraction

There are some examples which are quite similar to what you are looking to accomplish.

[the_wolverine]

Although this does not provide the top 10 (or X) error fields by _time.