Splunk Search
Highlighted

How to troubleshoot why I'm missing events in my search results?

Communicator

Hi,

I have an issue with a search, that I also use as an alert, which is not finding current events:

alt text

So the search shows me the latest entry at 12/15/15 when in fact it also occurred last night 12/22/15.

2015 Dec 22 03:46:56:844 GMT -0500 BW.BusinessConnect-Interior_Server-2 Debug [bw.logger] BW-EXT-LOG-300002  {standardID=X12, errorInfoList={1={errorDataParent=CTB*AA*Mr. Simon          xxx-xxx-xxxx, errorSeverity=Normal, errorSegmentCount=11, errorData=Mr. Simon   xxx-xxx-xxxx, errorCategory=Rejecting,

As I see above in the log.

Splunk is not showing me any malfunctiong regarding the index, and is showing me events around the timeframe even within the same minute and even second. However, it didn't index this event.

I didn't change anything regarding the indexes or excluded anything since 12/15/15. So I am a little confused why it wouldn't find the string.

The only thing I noticed is that the log file that splunk searches is generating probably around 1000 lines per second. Might that be too much for Splunk to index?

Thank you,

Oliver

0 Karma
Highlighted

Re: How to troubleshoot why I'm missing events in my search results?

Influencer

When you say you have events around the missing one, are they all from the same BusinessConnect-Interior_Server.log ? Or do the events in the same timeframe come from different logs?

Do you get any events if you search the splunkd.log on the forwarder for "BusinessConnect-Interior_Server.log"?

Are the values for indextime and _time identical for events in the BusinessConnect-InteriorServer.log ?

0 Karma
Highlighted

Re: How to troubleshoot why I'm missing events in my search results?

SplunkTrust
SplunkTrust

Hi, you're doing a full text search on the data. "key=value" is not the same in Splunk as key=value. See this for more details: http://docs.splunk.com/Documentation/Splunk/6.3.1511/Search/Usethesearchcommand

1st off, you need to properly "handle" your events by breaking these multiline events into separate events. For this you will need props.conf settings. You'll need should line merge set to false, and a proper line breaker or break only before, etc. configured. Please read the docs on getting data into splunk. http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/IndexMulti-lineEvents

Once you've added a line breaker and set should line merge to false, you'll then need to extract this field called "errorCategory".

This way your search will look like this index=tibco errorCategory=Rejecting versus index=tibco "errorCategory=Rejecting". If you do what I say... you'll find it solves your problem.