Getting Data In
Highlighted

How to troubleshoot why data is only getting indexed in Splunk for 1 hour every day with no interval specified in inputs.conf?

Path Finder

Hi,

We have an issue with Splunk getting data into indexes. We are getting data only during one hour (12.00 AM to 12.59 AM) every day. We have not specified any interval though in inputs.conf.

Can you please advise why it is restricting indexing to this one hour?

Please note that we have data in log files, verified our Universal forwarders side.

Thanks

0 Karma
Highlighted

Re: How to troubleshoot why data is only getting indexed in Splunk for 1 hour every day with no interval specified in inputs.conf?

Builder

Could you post your inputs.conf configuration? Do you see on Data Summary the data is being indexed?

0 Karma
Highlighted

Re: How to troubleshoot why data is only getting indexed in Splunk for 1 hour every day with no interval specified in inputs.conf?

Path Finder

Hi Below is in inputs.conf.

[monitor:///inpu/server*/logs/ca/data/]
disabled = 0
sourcetype = app:fp_ca
index = imdc
a

yes, we could see in datasummary the data is available from 12.00 AM to 12.59 AM.

Thanks
Sarath

0 Karma
Highlighted

Re: How to troubleshoot why data is only getting indexed in Splunk for 1 hour every day with no interval specified in inputs.conf?

Builder

Do you see any messages on splunkd.log?

0 Karma
Highlighted

Re: How to troubleshoot why data is only getting indexed in Splunk for 1 hour every day with no interval specified in inputs.conf?

Path Finder

No, we don't see any errors or message

0 Karma
Highlighted

Re: How to troubleshoot why data is only getting indexed in Splunk for 1 hour every day with no interval specified in inputs.conf?

Influencer

Can you post a few log samples? Have you tried searching for a known missing log over all time? In the future? (earliest=now latest=+1mon)

0 Karma
Highlighted

Re: How to troubleshoot why data is only getting indexed in Splunk for 1 hour every day with no interval specified in inputs.conf?

Path Finder

Below are sample logs

07-14-2016 00:00:09.430 -0700 INFO  ClientSessionsManager:Listener_AppEvents - Received count=3 AppEvents 
07-14-2016 00:00:09.702 -0700 INFO  PubSubSvr - Subscribed: channel=tenantService/handshake/reply/sgplu803/164E6DE8-9406-48ED-87D3-72BE00EFCC3E
0 Karma
Highlighted

Re: How to troubleshoot why data is only getting indexed in Splunk for 1 hour every day with no interval specified in inputs.conf?

Champion

Can you see the index time?
・・・| eval indextime=strftime(_indextime,"%Y/%m/%d %H:%M:%S")|table _time indextime

Or format of this log can be confirmed?

0 Karma
Highlighted

Re: How to troubleshoot why data is only getting indexed in Splunk for 1 hour every day with no interval specified in inputs.conf?

Path Finder
_time and Index time for my data: Indexing is stopped exactly around 1.00 AM 

50 Per Page Format  Preview Prev 1 2 3 4 5 6 7 8 9 ... Next
_time                                    indextime
2016-07-15 00:59:59.665 2016/07/15 01:00:00
2016-07-15 00:59:59.665 2016/07/15 01:00:00
2016-07-15 00:59:54.652 2016/07/15 00:59:55
2016-07-15 00:59:54.652 2016/07/15 00:59:55
2016-07-15 00:59:49.642 2016/07/15 00:59:50
0 Karma
Highlighted

Re: How to troubleshoot why data is only getting indexed in Splunk for 1 hour every day with no interval specified in inputs.conf?

Splunk Employee
Splunk Employee

Hello splunker9999, have you confirmed there are logs on the instance with timestamps outside of (12.00 AM to 12.59 AM) ?

0 Karma