Getting Data In

How to troubleshoot why data is only getting indexed in Splunk for 1 hour every day with no interval specified in inputs.conf?

splunker9999
Path Finder

Hi,

We have an issue with Splunk getting data into indexes. We are getting data only during one hour (12.00 AM to 12.59 AM) every day. We have not specified any interval though in inputs.conf.

Can you please advise why it is restricting indexing to this one hour?

Please note that we have data in log files, verified our Universal forwarders side.

Thanks

0 Karma

phadnett_splunk
Splunk Employee
Splunk Employee

Hello splunker9999, have you confirmed there are logs on the instance with timestamps outside of (12.00 AM to 12.59 AM) ?

0 Karma

splunker9999
Path Finder

Yes confirm, Today also we got event from 12.00 to 12.59 AM

0 Karma

phadnett_splunk
Splunk Employee
Splunk Employee

Hello Splunk9999, sorry for the confusion, I meant the actual log file you are monitoring.

0 Karma

HiroshiSatoh
Champion

Can you see the index time?
・・・| eval indextime=strftime(_indextime,"%Y/%m/%d %H:%M:%S")|table _time indextime

Or format of this log can be confirmed?

0 Karma

splunker9999
Path Finder
_time and Index time for my data: Indexing is stopped exactly around 1.00 AM 

50 Per Page Format  Preview Prev 1 2 3 4 5 6 7 8 9 ... Next
_time                                    indextime
2016-07-15 00:59:59.665 2016/07/15 01:00:00
2016-07-15 00:59:59.665 2016/07/15 01:00:00
2016-07-15 00:59:54.652 2016/07/15 00:59:55
2016-07-15 00:59:54.652 2016/07/15 00:59:55
2016-07-15 00:59:49.642 2016/07/15 00:59:50
0 Karma

twinspop
Influencer

Can you post a few log samples? Have you tried searching for a known missing log over all time? In the future? (earliest=now latest=+1mon)

0 Karma

splunker9999
Path Finder

Below are sample logs

07-14-2016 00:00:09.430 -0700 INFO  ClientSessionsManager:Listener_AppEvents - Received count=3 AppEvents 
07-14-2016 00:00:09.702 -0700 INFO  PubSubSvr - Subscribed: channel=tenantService/handshake/reply/sgplu803/164E6DE8-9406-48ED-87D3-72BE00EFCC3E
0 Karma

gfreitas
Builder

Could you post your inputs.conf configuration? Do you see on Data Summary the data is being indexed?

0 Karma

splunker9999
Path Finder

Hi Below is in inputs.conf.

[monitor:///inpu/server*/logs/ca/data/]
disabled = 0
sourcetype = app:fp__ca
index = imdc_a

yes, we could see in datasummary the data is available from 12.00 AM to 12.59 AM.

Thanks
Sarath

0 Karma

gfreitas
Builder

Do you see any messages on splunkd.log?

0 Karma

splunker9999
Path Finder

No, we don't see any errors or message

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!