Getting Data In

Will Splunk index old data present in a folder after integration date?

santhoshi
Explorer

I have a Prod Linux server where I have deployed Universal forwarder and monitoring few directories. Say like below. This is a new server and I am integrating for first time on say Sep 8th.

$ ls -la
rwxr-xr-x 5 yusufshakeel yusufshakeel 160 Sep 4 02:53 alarm_0409.log
rwxr-xr-x 8 yusufshakeel yusufshakeel 256 Sep 5 02:53 alarm_0509.log
rwxr-xr-x 2 yusufshakeel yusufshakeel 64 Sep 6 02:53 alarm_0609.log
rwxr-xr-x 1 yusufshakeel yusufshakeel 10 Sep 7 02:53 alarm_0709.log
rwxr-xr-x 2 yusufshakeel yusufshakeel 64 Sep 8 02:53 alarm_0809.log

This is a new server and I am integrating for first time on say Sep 8th. Now the integration is completed and Splunk starts monitoring.  I see only Sep 8th file got indexed and all the previous 4 logs (Sep 4 - Sep 7) didn't get ingested.

My question is,  will splunk not index older log files ;from the date it got integrated? Please help.

Labels (3)

inventsekar
SplunkTrust
SplunkTrust

Hi @santhoshi .. if Splunk can monitor only the logs after its integration date, the whole vision and mission of Splunk can be thrown under the bus!  no hard feelings, just kidding,..in general, Splunk has to monitor "everything".

Old or Current or Future ( at strange times, we may end up having file with future date as well.. linux is crazy! 😉 

1. Are you monitoring the whole directory level, or just a direct filename monitoring?!
2. on the UF, using the btool, when you list the inputs config, may we know what you see?!

3. did you see any errors on splunkd.log? 

4. maybe all files reached splunk fine and you were only searching for today's file. may we know your search query and time picker which you selected?

 

(PS - i have given around 350+ karma points so far, received badge for that, if an answer helped you, pls give a karma point!. we all should start "Learn, Give Back, Have Fun")

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It depends on your input definition.  Please share the relevant inputs.conf stanza.

---
If this reply helps you, Karma would be appreciated.
0 Karma

santhoshi
Explorer

hello, thanks for your reply. Inputs would be like this.

[monitor:///opt/nfmp/alarm_*.log]
disabled = false
sourcetype = alarm_log
index = proj_app
crcSalt = <SOURCE>
blacklist = (\.(tar|gz|bz2|tar.gz|tgz|tbz|tbz2|zip|z)$)

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...