Getting Data In
Highlighted

Any idea why a particular sourcetype would stop showing data after 10/31/16 23:59:59?

Engager

Here are some pieces of info that may be relevant:

  • The sourcetype in question shows no data after midnight on October 31st when searching
  • Setup: 1 Splunk server, no replication or anything, 40 nodes (4 in question) reporting to splunk
  • Splunk server version: 6.5.0 (installed on 10/25/16)
  • Splunk server has more than 20% free disk on all drives
  • All other sourcetypes on the splunk server are working, even sourcetypes that dump into the same index as the broken sourcetype
  • No changes have been made on the splunk server or the nodes
  • The metrics.log on the splunk server shows data coming in for that sourcetype from the 4 nodes in question
  • The splunk forwarder logs on each of our nodes shows that it is sending data into the splunk server using the sourcetype
  • Each node in question is reporting data to the splunk server on additional sourcetypes and they are searchable in splunk

Any ideas would be greatly appreciated. I've restarted the splunk server (full windows reboot) as well as restarting the splunk forwarder on the nodes. It seems like it's date related but none of the other sourcetypes seem to be affected.

0 Karma
Highlighted

Re: Any idea why a particular sourcetype would stop showing data after 10/31/16 23:59:59?

Splunk Employee
Splunk Employee

My wild stab in the dark, based on the horrors of experience, is that the props.conf does not have timestamp format explicitly set, and maybe what we want to be Nov 1, 2016, is showing up in January 11th, 2016?

Can you share your props config for this sourcetype?

View solution in original post

Highlighted

Re: Any idea why a particular sourcetype would stop showing data after 10/31/16 23:59:59?

Engager

mmodestino,

You're exactly right! The time format in the logs is dd-mm-yyyy and it would appear that splunk is expecting mm-dd-yyyy. I am a bit puzzled as to why it chose November to do this (as opposed to July or October or whatever). Maybe it's a different default in splunk 6.5.0 versus the 6.4.x and previous. When I look back in January I see interlaced data for 11-01-2016 and 01-11-2016.

From props.conf:

[application_log]
NO_BINARY_CHECK = 1
maxDist = 75
pulldown_type = 1
REPORT-myname=applicationlogmap

[applicationlogmap]
DELIMS="|"
    FIELDS="TimeStamp","ThreadID","LogLevel","ClassName","LogHash","Msg"

I'll read up on how to specify the time/date format and get this resolved. Thanks so much for taking a wild stab! I'll mark your answer as accepted as soon as I verify this works.

0 Karma
Highlighted

Re: Any idea why a particular sourcetype would stop showing data after 10/31/16 23:59:59?

Splunk Employee
Splunk Employee

My pleasure!

If you share a sample event I can assist with the config.

0 Karma
Highlighted

Re: Any idea why a particular sourcetype would stop showing data after 10/31/16 23:59:59?

Engager

That would be awesome. Here's a sample line:

04-11-2016 09:15:29,482|http-apr-8080-exec-3| INFO|ConfigFilters||sessionManagement/template?name=portal/Login built in 0.0151612 seconds ().
0 Karma
Highlighted

Re: Any idea why a particular sourcetype would stop showing data after 10/31/16 23:59:59?

Splunk Employee
Splunk Employee

Add Data Wizard FTW!

[ tbourne ]
SHOULD_LINEMERGE=false 
NO_BINARY_CHECK=true
TIME_FORMAT=%d-%m-%Y %H:%M:%S,%f
TIME_PREFIX=^
TZ=UTC
MAX_TIMESTAMP_LOOKAHEAD=30

This should clean up your timestamping and get you some perf gains. Best practice is to set linebreaker and time formatting params explicitly on all sourcetypes, as making splunk auto discover makes it work harder and ensures you avoid this pain in the future.

http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Configuretimestamprecognition

ADD DATA WIZ FTW!

0 Karma