Getting Data In

Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

itprdgetinsured
Observer

Splunk Forwarder metrics log on application node :

metrics.log:05-19-2017 13:09:07.625 -0500 INFO  Metrics - group=per_source_thruput, series="/var/log/app/ms-starr/starr.log", kbps=0.005355, eps=0.096773, kb=0.166016, ev=3, avg_age=0.333333, max_age=1
metrics.log:05-19-2017 13:09:07.625 -0500 INFO  Metrics - group=per_source_thruput, series="/var/log/app/ms-starr/start.log", kbps=0.005355, eps=0.096773, kb=0.166016, ev=3, avg_age=0.333333, max_age=1
splunkd.log:05-18-2017 23:46:14.621 -0500 INFO  TailReader - Batch input finished reading file='/var/log/app/ms-starr/start.log'

Splunk Enterprise (receiver) log :

[root@600277-ITIAD-BASTION1 splunk]# grep -i starr.log splunkd.log
05-19-2017 12:54:36.773 -0500 INFO  WatchedFile - Will begin reading at offset=5534528 for file='/mnt/phixdr-app5/app/ms-starr/starr.log'.
05-19-2017 12:54:36.864 -0500 INFO  WatchedFile - Will begin reading at offset=106038 for file='/mnt/phixdr-app5/app/ms-starr/ms-starr.log.2017-05-19'.

Problem - starr.log is still not getting splunked.

0 Karma
1 Solution

woodcock
Esteemed Legend

If they got indexed and they are no longer there then they must have been either hidden with the delete command (highly unlikely) or expired a natural death. To check for the latter, use this search:

index=_internal sourcetype=splunkd bucketmover | rex "[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db[\/\\\]db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+" | rex "db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+.*?[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db" | stats max(oldestTime) AS oldestTime BY indexname | eval retentionDays=(now()-oldestTime)/(60*60*24)

View solution in original post

0 Karma

woodcock
Esteemed Legend

If they got indexed and they are no longer there then they must have been either hidden with the delete command (highly unlikely) or expired a natural death. To check for the latter, use this search:

index=_internal sourcetype=splunkd bucketmover | rex "[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db[\/\\\]db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+" | rex "db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+.*?[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db" | stats max(oldestTime) AS oldestTime BY indexname | eval retentionDays=(now()-oldestTime)/(60*60*24)
0 Karma

woodcock
Esteemed Legend

So which was it?

0 Karma

itprdgetinsured
Observer

Latter. 🙂

0 Karma

woodcock
Esteemed Legend

I figured.

0 Karma

ddrillic
Ultra Champion

The following might help - I can't find my data!

0 Karma

aakwah
Builder

Hello,
Check if logs are indexed in the default index (index=main).

Try to list all sources and corresponding index with this query:

index=* | stats count by source, index

Regards

0 Karma

itprdgetinsured
Observer

Thanks woodcock and aakwah.

I tried woodcock's query but still dont see starr.log.

This is what I have on server :

[root@824611-PHIXPRD-APP7 ms-starr]# date
Fri May 19 17:30:53 CDT 2017
[root@824611-PHIXPRD-APP7 ms-starr]# ls -ltr starr.log
-rw-r--r-- 1 app app 463520 May 19 17:30 starr.log

And this is all I see in source when I query on UI : (starr.log is missing)

/var/log/app/ms-starr/ms-starr.log.2017-05-17 1,900 48.346%
/var/log/app/ms-starr/starr.log.1 1,298 33.028%
/var/log/app/ms-starr/starr.log.2 667 16.972%
/var/log/app/ms-starr/ms-starr.log.2017-05-16 65 1.654%

0 Karma

woodcock
Esteemed Legend

Run this search for All time (it MUST be All time😞

index=* source="*starr.log*"
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...