Getting Data In
Highlighted

Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

New Member

Splunk Forwarder metrics log on application node :

metrics.log:05-19-2017 13:09:07.625 -0500 INFO  Metrics - group=per_source_thruput, series="/var/log/app/ms-starr/starr.log", kbps=0.005355, eps=0.096773, kb=0.166016, ev=3, avg_age=0.333333, max_age=1
metrics.log:05-19-2017 13:09:07.625 -0500 INFO  Metrics - group=per_source_thruput, series="/var/log/app/ms-starr/start.log", kbps=0.005355, eps=0.096773, kb=0.166016, ev=3, avg_age=0.333333, max_age=1
splunkd.log:05-18-2017 23:46:14.621 -0500 INFO  TailReader - Batch input finished reading file='/var/log/app/ms-starr/start.log'

Splunk Enterprise (receiver) log :

[root@600277-ITIAD-BASTION1 splunk]# grep -i starr.log splunkd.log
05-19-2017 12:54:36.773 -0500 INFO  WatchedFile - Will begin reading at offset=5534528 for file='/mnt/phixdr-app5/app/ms-starr/starr.log'.
05-19-2017 12:54:36.864 -0500 INFO  WatchedFile - Will begin reading at offset=106038 for file='/mnt/phixdr-app5/app/ms-starr/ms-starr.log.2017-05-19'.

Problem - starr.log is still not getting splunked.

0 Karma
Highlighted

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

Esteemed Legend

Run this search for All time (it MUST be All time😞

index=* source="*starr.log*"
0 Karma
Highlighted

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

Builder

Hello,
Check if logs are indexed in the default index (index=main).

Try to list all sources and corresponding index with this query:

index=* | stats count by source, index

Regards

0 Karma
Highlighted

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

New Member

Thanks woodcock and aakwah.

I tried woodcock's query but still dont see starr.log.

This is what I have on server :

[root@824611-PHIXPRD-APP7 ms-starr]# date
Fri May 19 17:30:53 CDT 2017
[root@824611-PHIXPRD-APP7 ms-starr]# ls -ltr starr.log
-rw-r--r-- 1 app app 463520 May 19 17:30 starr.log

And this is all I see in source when I query on UI : (starr.log is missing)

/var/log/app/ms-starr/ms-starr.log.2017-05-17 1,900 48.346%
/var/log/app/ms-starr/starr.log.1 1,298 33.028%
/var/log/app/ms-starr/starr.log.2 667 16.972%
/var/log/app/ms-starr/ms-starr.log.2017-05-16 65 1.654%

0 Karma
Highlighted

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

Ultra Champion

The following might help - I can't find my data!

0 Karma
Highlighted

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

Esteemed Legend

If they got indexed and they are no longer there then they must have been either hidden with the delete command (highly unlikely) or expired a natural death. To check for the latter, use this search:

index=_internal sourcetype=splunkd bucketmover | rex "[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db[\/\\\]db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+" | rex "db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+.*?[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db" | stats max(oldestTime) AS oldestTime BY indexname | eval retentionDays=(now()-oldestTime)/(60*60*24)

View solution in original post

0 Karma
Highlighted

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

Esteemed Legend

So which was it?

0 Karma
Highlighted

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

New Member

Latter. 🙂

0 Karma
Highlighted

Re: Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

Esteemed Legend

I figured.

0 Karma