Getting Data In

Splunk forwarder is logging data but why is Splunk Enterprise not showing the data in Splunk Web?

itprdgetinsured
New Member

Splunk Forwarder metrics log on application node :

metrics.log:05-19-2017 13:09:07.625 -0500 INFO  Metrics - group=per_source_thruput, series="/var/log/app/ms-starr/starr.log", kbps=0.005355, eps=0.096773, kb=0.166016, ev=3, avg_age=0.333333, max_age=1
metrics.log:05-19-2017 13:09:07.625 -0500 INFO  Metrics - group=per_source_thruput, series="/var/log/app/ms-starr/start.log", kbps=0.005355, eps=0.096773, kb=0.166016, ev=3, avg_age=0.333333, max_age=1
splunkd.log:05-18-2017 23:46:14.621 -0500 INFO  TailReader - Batch input finished reading file='/var/log/app/ms-starr/start.log'

Splunk Enterprise (receiver) log :

[root@600277-ITIAD-BASTION1 splunk]# grep -i starr.log splunkd.log
05-19-2017 12:54:36.773 -0500 INFO  WatchedFile - Will begin reading at offset=5534528 for file='/mnt/phixdr-app5/app/ms-starr/starr.log'.
05-19-2017 12:54:36.864 -0500 INFO  WatchedFile - Will begin reading at offset=106038 for file='/mnt/phixdr-app5/app/ms-starr/ms-starr.log.2017-05-19'.

Problem - starr.log is still not getting splunked.

0 Karma
1 Solution

woodcock
Esteemed Legend

If they got indexed and they are no longer there then they must have been either hidden with the delete command (highly unlikely) or expired a natural death. To check for the latter, use this search:

index=_internal sourcetype=splunkd bucketmover | rex "[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db[\/\\\]db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+" | rex "db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+.*?[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db" | stats max(oldestTime) AS oldestTime BY indexname | eval retentionDays=(now()-oldestTime)/(60*60*24)

View solution in original post

0 Karma

woodcock
Esteemed Legend

If they got indexed and they are no longer there then they must have been either hidden with the delete command (highly unlikely) or expired a natural death. To check for the latter, use this search:

index=_internal sourcetype=splunkd bucketmover | rex "[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db[\/\\\]db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+" | rex "db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+.*?[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db" | stats max(oldestTime) AS oldestTime BY indexname | eval retentionDays=(now()-oldestTime)/(60*60*24)
0 Karma

woodcock
Esteemed Legend

So which was it?

0 Karma

itprdgetinsured
New Member

Latter. 🙂

0 Karma

woodcock
Esteemed Legend

I figured.

0 Karma

ddrillic
Ultra Champion

The following might help - I can't find my data!

0 Karma

aakwah
Builder

Hello,
Check if logs are indexed in the default index (index=main).

Try to list all sources and corresponding index with this query:

index=* | stats count by source, index

Regards

0 Karma

itprdgetinsured
New Member

Thanks woodcock and aakwah.

I tried woodcock's query but still dont see starr.log.

This is what I have on server :

[root@824611-PHIXPRD-APP7 ms-starr]# date
Fri May 19 17:30:53 CDT 2017
[root@824611-PHIXPRD-APP7 ms-starr]# ls -ltr starr.log
-rw-r--r-- 1 app app 463520 May 19 17:30 starr.log

And this is all I see in source when I query on UI : (starr.log is missing)

/var/log/app/ms-starr/ms-starr.log.2017-05-17 1,900 48.346%
/var/log/app/ms-starr/starr.log.1 1,298 33.028%
/var/log/app/ms-starr/starr.log.2 667 16.972%
/var/log/app/ms-starr/ms-starr.log.2017-05-16 65 1.654%

0 Karma

woodcock
Esteemed Legend

Run this search for All time (it MUST be All time😞

index=* source="*starr.log*"
0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...