Splunk Search

Events not found, even though dashboard says there are

gregwilliams
Path Finder

I have a universal forwarder pulling in a log file from a linux server. It has been working just fine up until the other day. The Summary dashboard shows events are coming in, however when I search for anything related to the source, sourcetype, anything within the event, I get "No Matching Events found". I can see events from other servers no problem. Nothing in the infrastructure nor Splunk config has changed. I am an admin, so I should be seeing everything. Any ideas?

Tags (1)
0 Karma
1 Solution

gregwilliams
Path Finder

Problem solved. The problem was Splunk was reading the log timestamp incorrectly. What's odd however is that the log timestamp format never changed, but Splunk was just now unable to read it correctly. I was able to configure the program sending the log to include the year instead of just the date.

View solution in original post

0 Karma

gregwilliams
Path Finder

Problem solved. The problem was Splunk was reading the log timestamp incorrectly. What's odd however is that the log timestamp format never changed, but Splunk was just now unable to read it correctly. I was able to configure the program sending the log to include the year instead of just the date.

View solution in original post

0 Karma

gregwilliams
Path Finder

I was able to change the timestamp being produced by the program and Splunk now identifies it correctly. It's very odd however that it would just stop reading the correct timestamp from one minute to the next when nothing changed in the program, the logs themselves nor splunk

0 Karma

gregwilliams
Path Finder

I added this in my props.conf but without luck.

[sourcetype]
TIME_FORMAT=%m/%d-%h:%m:%s

The event comes in as:

07/05-10:01:01

0 Karma

sowings
Splunk Employee
Splunk Employee

Check TIMESTAMP_FORMAT in props.conf.

0 Karma

gregwilliams
Path Finder

That actually helped me figure it out. Apparently Splunk is reading the timestamp on the logs differently now and putting them in completely different days. So for example 07/05-08:25:43 is being read as May 8 8:25:43. It's ignoring month. I'll try to figure out how to change that. Thanks for the help.

0 Karma

Ayn
Legend

Perhaps it's writing to another index than the one(s) you're searching?

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!