Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

[itprdgetinsured]

Splunk Forwarder metrics log on application node :

metrics.log:05-19-2017 13:09:07.625 -0500 INFO  Metrics - group=per_source_thruput, series="/var/log/app/ms-starr/starr.log", kbps=0.005355, eps=0.096773, kb=0.166016, ev=3, avg_age=0.333333, max_age=1
metrics.log:05-19-2017 13:09:07.625 -0500 INFO  Metrics - group=per_source_thruput, series="/var/log/app/ms-starr/start.log", kbps=0.005355, eps=0.096773, kb=0.166016, ev=3, avg_age=0.333333, max_age=1
splunkd.log:05-18-2017 23:46:14.621 -0500 INFO  TailReader - Batch input finished reading file='/var/log/app/ms-starr/start.log'

Splunk Enterprise (receiver) log :

[root@600277-ITIAD-BASTION1 splunk]# grep -i starr.log splunkd.log
05-19-2017 12:54:36.773 -0500 INFO  WatchedFile - Will begin reading at offset=5534528 for file='/mnt/phixdr-app5/app/ms-starr/starr.log'.
05-19-2017 12:54:36.864 -0500 INFO  WatchedFile - Will begin reading at offset=106038 for file='/mnt/phixdr-app5/app/ms-starr/ms-starr.log.2017-05-19'.

Problem - starr.log is still not getting splunked.

[woodcock · Accepted solution]

If they got indexed and they are no longer there then they must have been either hidden with the delete command (highly unlikely) or expired a natural death. To check for the latter, use this search:

index=_internal sourcetype=splunkd bucketmover | rex "[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db[\/\\\]db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+" | rex "db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+.*?[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db" | stats max(oldestTime) AS oldestTime BY indexname | eval retentionDays=(now()-oldestTime)/(60*60*24)

[woodcock]

So which was it?

[itprdgetinsured]

Latter. 🙂

[woodcock]

I figured.

[ddrillic]

The following might help - I can't find my data!

[aakwah]

Hello,
Check if logs are indexed in the default index (index=main).

Try to list all sources and corresponding index with this query:

index=* | stats count by source, index

Regards

[itprdgetinsured]

Thanks woodcock and aakwah.

I tried woodcock's query but still dont see starr.log.

This is what I have on server :

[root@824611-PHIXPRD-APP7 ms-starr]# date
Fri May 19 17:30:53 CDT 2017
[root@824611-PHIXPRD-APP7 ms-starr]# ls -ltr starr.log
-rw-r--r-- 1 app app 463520 May 19 17:30 starr.log

And this is all I see in source when I query on UI : (starr.log is missing)

/var/log/app/ms-starr/ms-starr.log.2017-05-17 1,900 48.346%
/var/log/app/ms-starr/starr.log.1 1,298 33.028%
/var/log/app/ms-starr/starr.log.2 667 16.972%
/var/log/app/ms-starr/ms-starr.log.2017-05-16 65 1.654%

[woodcock]

Run this search for All time (it MUST be All time😞

index=* source="*starr.log*"