topic Re: Events not found, even though dashboard says there are in Splunk Search

Events not found, even though dashboard says there are

gregwilliams — Thu, 05 Jul 2012 15:05:33 GMT

I have a universal forwarder pulling in a log file from a linux server. It has been working just fine up until the other day. The Summary dashboard shows events are coming in, however when I search for anything related to the source, sourcetype, anything within the event, I get "No Matching Events found". I can see events from other servers no problem. Nothing in the infrastructure nor Splunk config has changed. I am an admin, so I should be seeing everything. Any ideas?

Re: Events not found, even though dashboard says there are

Ayn — Thu, 05 Jul 2012 15:31:30 GMT

Perhaps it's writing to another index than the one(s) you're searching?

Re: Events not found, even though dashboard says there are

gregwilliams — Thu, 05 Jul 2012 16:07:29 GMT

That actually helped me figure it out. Apparently Splunk is reading the timestamp on the logs differently now and putting them in completely different days. So for example 07/05-08:25:43 is being read as May 8 8:25:43. It's ignoring month. I'll try to figure out how to change that. Thanks for the help.

Re: Events not found, even though dashboard says there are

sowings — Thu, 05 Jul 2012 16:31:27 GMT

Check TIMESTAMP_FORMAT in props.conf.

Re: Events not found, even though dashboard says there are

gregwilliams — Thu, 05 Jul 2012 16:39:57 GMT

I added this in my props.conf but without luck.

[sourcetype]
TIME_FORMAT=%m/%d-%h:%m:%s

The event comes in as:

07/05-10:01:01

Re: Events not found, even though dashboard says there are

gregwilliams — Thu, 05 Jul 2012 17:12:43 GMT

I was able to change the timestamp being produced by the program and Splunk now identifies it correctly. It's very odd however that it would just stop reading the correct timestamp from one minute to the next when nothing changed in the program, the logs themselves nor splunk

Re: Events not found, even though dashboard says there are

gregwilliams — Thu, 05 Jul 2012 17:15:10 GMT

Problem solved. The problem was Splunk was reading the log timestamp incorrectly. What's odd however is that the log timestamp format never changed, but Splunk was just now unable to read it correctly. I was able to configure the program sending the log to include the year instead of just the date.