Exclude specific String from search

[dababi1234]

I am new to Splunk and would appreciate if anyone helps me on this. I would like to set up a Splunk alert for SocketTimeoutException from all sources. But I would like to exclude from the search if I have the following string "Exception in Client ABC service" in the server logs. This string is on a different line before the line java.net.SocketTimeoutException. For example, I get the following server logs:

01/15/2019 13:53:11,983 INFO stdout at java.lang.Thread.run(Thread.java:745)
01/15/2019 13:53:12,683 INFO stdout [default task-418] ERROR com.foo.ClientABC - Exception in Client ABC service ::
01/15/2019 13:53:12,683 INFO stdout java.net.SocketTimeoutException: connect timed out
01/15/2019 13:53:12,683 INFO stdout at java.net.PlainSocketImpl.socketConnect(Native Method)
01/15/2019 13:53:12,683 INFO stdout at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
01/15/2019 13:53:12,683 INFO stdout at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
01/15/2019 13:53:12,684 INFO stdout at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

My Splunk alert right now is:
index=XYZ "java.net.SocketTimeoutException"

I would like to get alerts for all other SocketTimeoutExceptions that don't contain the string Exception in Client ABC .

[dkeck]

Hi,

just add to your search NOT

index=XYZ "java.net.SocketTimeoutException"  NOT "Exception in Client ABC"

Or if you have a field with the value "Exception in Client ABC" you could add field!="Exception in Client ABC"

[dababi1234]

It doesn't have any effect if I include NOT "Exception in Client ABC". The problem is that, when I do index=XYZ "java.net.SocketTimeoutException", the search results doesn't show the text "Exception in Client ABC". But if I expand that, do Event Actions and do Show Source I see the line Exception in Client ABC which is causing the SocketTimeoutException.

My goal is to display all other causes for SocketTimeoutException, but I don't want the exception due to Exception in Client ABC to be shown. Does that make sense?

[prateeksawhney]

@dababi1234 

Were you able to find solution for this? I am facing similar issue.

[isoutamo]

Hi

It seems that in this case all those lines are separate events not a big one which contains the whole error message. If/when you need to do this, then you must first combine those individual lines (which belongs to same error message) to one event and then go through those which are not contains that special message.

I think that better way is correct ingestion phase so that all those lines create only one event and then you can do exclusion as proposed.

r. Ismo

[PickleRick]

Exactly. Unfortunately, this is java log/stacktrace which means that its ingestion is an ugly thing. Unfortunately, java logs are usually very ugly to ingest because they're not well-structured and have a very... "free" approach to what is in them.

So sometimes (not in this case though) you can distinguish stacktraces from the rest of the logs because they do not (again - not in this case) have a timestamp at the beginning of the line. But in such case as presented here - there's nothing obvious to build transaction upon. you could try to start it at the beginning of the stacktrace (for example on "\[default-task.*ERROR") but there's no good ending of the transaction. You'd have to dig in your logs a bit deeper to find where the entries end.

Unfortunately, the logs are "distorted" because not only they're are being sent/saved as multiline logs (which splunk can usually handle with some slight help  from good configuration, but also separate lines have additional timestamp lines (added by some intermediate "processor"?) which effectively split single event into multiple events.

It's tricky, especially without a context.