What are the best extraction methods for Java Stacktrace Errors?

[thomastaylor]

Hello everyone!

Currently, we are demoing the Splunk Enterprise trial and are here to ask what is the most efficient ways for examining large Java stack traces?

Here's an example of our stack trace:

    2018-06-07 09:55:15 ERROR ServiceBlankBlank:102 - Error 500 in method: blankblankmethodname
    Listener refused the connection with the following error:
    placehere, TNS:listener could not find protocol
    The Connection descriptor used by the client was:
    server:port/blabla
    java.sql.SQLException: Listener refused the connection with the following error:
    placehere, TNS:listener could not find available handler with matching protocol stack
    The Connection descriptor used by the client was:
    server:port/blabla
        at oracle.jdbc.driver...
        at oracle.jdbc.driver...
        at oracle.jdbc.driver...
        at oracle.jdbc.driver...
        at oracle.jdbc.driver...
        at oracle.jdbc.driver...
        at oracle.jdbc.driver...
        at java.sql.DriverManager...
        at java.sql.DriverManager...
        ...........

We have received this from a Universal Forwarder on a Linux server. Ideally, we would want to only send all the lines before the large indented stack trace to our Splunk instance so that unnecessary stack information is not included in our daily Splunk usage.

From this example, we would only want to keep the top portion above the "at... at... at..." for the purposes of our logging. To make this problem more complex, we may have different stack traces as well. Usually, the common similarity is the beginning of the "at..".

What are some ways that the folks from the community may handle this? We do not have the necessary access to the original source code, so we will not be able to implement the Splunk SDK method.

Can we remove the unnecessary log information from Splunk directly? Would the best way be to configure the inputs.conf / prop.conf to extract the necessary information and send it over? Should we use a third party program such as FluentD or logstash to parse this?

Any recommendations would be greatly appreciated!

[cpetterborg · Accepted solution]

Since you cannot do this in the universal forwarder, it will have to be done on the indexers. I would use a SEDCMD in the props.conf file, something like this:

[your-sourcetype-here]
SEDCMD-truncate-stacktrace = s/\s+at\s[\s\S]+//g

This will take everything from the first line that starts with at least one space followed by at followed by a space, clear to the end of the event and remove it from the raw event before it gets indexed. This assumes that you also have the proper line breaking, etc. for the sourcetype. Please be sure to get that done correctly.

[thomastaylor]

Thank you so much @cpetterborg ! After writing a simple event break regex, this worked like a charm.

[FrankVl]

That's what I would suggest too. using props and transforms with nullqueue routing doesn't work, since you only want to trash part of the event. While routing applies to entire events.

Never used it on stack traces, but I have seen the same solution getting applied to high volume windows events for stripping of the explanatory text sections from some of those. So I can confirm that concept works like a charm.

[thomastaylor]

Thank you for the input Frank! I'm trying out this method right now.

[sudosplunk]

Hello,

You should use a combination of props.conf and transforms.conf files to route the unwanted data to nullQueue so that it is not counted against your daily license volume. In this case, I would write the .conf files as below

In props.conf
[your_custom_sourcetype]
TRANSFORMS-routing=route_to_null_queue

In transforms.conf:
[route_to_null_queue]
REGEX = (?m)\s+at\s(oracle|java)
DEST_KEY = queue
FORMAT = nullQueue

More info is available at http://docs.splunk.com/Documentation/Splunk/7.1.1/Forwarding/Routeandfilterdatad

Let me know if this helps.

[thomastaylor]

For some reason, I still cannot get this working. I placed the settings in etc/apps/search/local exactly how you have described them. I also restarted the Splunk Enterprise instance and re-added the log directory. Is there more I should do?

Were these steps necessary after editing these two .conf files?

Thank you for your help.

[sudosplunk]

Can you please paste your configurations here.

[thomastaylor]

Thanks for your help!

In props.conf for my sourcetype:

[fbs]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Application
pulldown_type = 1
TRANSFORMS-routing=route_to_null_queue

In the transforms.conf (which I created in the directory):

[route_to_null_queue]
REGEX = (?m)\s+at\s(oracle|java)
DEST_KEY = queue
FORMAT = nullQueue

[FrankVl]

Routing won't work, as that applies to entire events, while you want to strip out just part of a (multiline) event. Best take a look at @cpetterborg's answer.