topic Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web? in Getting Data In

Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

itprdgetinsured — Fri, 19 May 2017 20:29:14 GMT

Splunk Forwarder metrics log on application node :

metrics.log:05-19-2017 13:09:07.625 -0500 INFO  Metrics - group=per_source_thruput, series="/var/log/app/ms-starr/starr.log", kbps=0.005355, eps=0.096773, kb=0.166016, ev=3, avg_age=0.333333, max_age=1
metrics.log:05-19-2017 13:09:07.625 -0500 INFO  Metrics - group=per_source_thruput, series="/var/log/app/ms-starr/start.log", kbps=0.005355, eps=0.096773, kb=0.166016, ev=3, avg_age=0.333333, max_age=1
splunkd.log:05-18-2017 23:46:14.621 -0500 INFO  TailReader - Batch input finished reading file='/var/log/app/ms-starr/start.log'

Splunk Enterprise (receiver) log :

[root@600277-ITIAD-BASTION1 splunk]# grep -i starr.log splunkd.log
05-19-2017 12:54:36.773 -0500 INFO  WatchedFile - Will begin reading at offset=5534528 for file='/mnt/phixdr-app5/app/ms-starr/starr.log'.
05-19-2017 12:54:36.864 -0500 INFO  WatchedFile - Will begin reading at offset=106038 for file='/mnt/phixdr-app5/app/ms-starr/ms-starr.log.2017-05-19'.

Problem - starr.log is still not getting splunked.

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

woodcock — Fri, 19 May 2017 22:03:18 GMT

Run this search for All time (it MUST be All time😞

index=* source="*starr.log*"

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

aakwah — Fri, 19 May 2017 22:08:53 GMT

Hello,
Check if logs are indexed in the default index (index=main).

Try to list all sources and corresponding index with this query:

index=* | stats count by source, index

Regards

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

itprdgetinsured — Fri, 19 May 2017 22:37:12 GMT

Thanks woodcock and aakwah.

I tried woodcock's query but still dont see starr.log.

This is what I have on server :

[root@824611-PHIXPRD-APP7 ms-starr]# date
Fri May 19 17:30:53 CDT 2017
[root@824611-PHIXPRD-APP7 ms-starr]# ls -ltr starr.log
-rw-r--r-- 1 app app 463520 May 19 17:30 starr.log

And this is all I see in source when I query on UI : (starr.log is missing)

/var/log/app/ms-starr/ms-starr.log.2017-05-17 1,900 48.346%
/var/log/app/ms-starr/starr.log.1 1,298 33.028%
/var/log/app/ms-starr/starr.log.2 667 16.972%
/var/log/app/ms-starr/ms-starr.log.2017-05-16 65 1.654%

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

ddrillic — Sat, 20 May 2017 00:32:36 GMT

The following might help - I can't find my data!

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

woodcock — Sat, 20 May 2017 01:06:40 GMT

If they got indexed and they are no longer there then they must have been either hidden with the delete command (highly unlikely) or expired a natural death. To check for the latter, use this search:

index=_internal sourcetype=splunkd bucketmover | rex "[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db[\/\\\]db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+" | rex "db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+.*?[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db" | stats max(oldestTime) AS oldestTime BY indexname | eval retentionDays=(now()-oldestTime)/(60*60*24)

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

woodcock — Mon, 22 May 2017 19:09:50 GMT

So which was it?

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

itprdgetinsured — Mon, 22 May 2017 19:14:10 GMT

Latter. 🙂

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

woodcock — Mon, 22 May 2017 19:35:24 GMT

I figured.