topic Re: Will Splunk index old data present in a folder after integration date? in Getting Data In

Will Splunk index old data present in a folder after integration date?

santhoshi — Sat, 26 Sep 2020 15:05:50 GMT

I have a Prod Linux server where I have deployed Universal forwarder and monitoring few directories. Say like below. This is a new server and I am integrating for first time on say Sep 8th.

$ ls -la
rwxr-xr-x 5 yusufshakeel yusufshakeel 160 Sep 4 02:53 alarm_0409.log
rwxr-xr-x 8 yusufshakeel yusufshakeel 256 Sep 5 02:53 alarm_0509.log
rwxr-xr-x 2 yusufshakeel yusufshakeel 64 Sep 6 02:53 alarm_0609.log
rwxr-xr-x 1 yusufshakeel yusufshakeel 10 Sep 7 02:53 alarm_0709.log
rwxr-xr-x 2 yusufshakeel yusufshakeel 64 Sep 8 02:53 alarm_0809.log

This is a new server and I am integrating for first time on say Sep 8th. Now the integration is completed and Splunk starts monitoring. I see only Sep 8th file got indexed and all the previous 4 logs (Sep 4 - Sep 7) didn't get ingested.

My question is, will splunk not index older log files ;from the date it got integrated? Please help.

Re: Will Splunk index old data present in a folder after integration date?

richgalloway — Sat, 26 Sep 2020 16:55:16 GMT

It depends on your input definition. Please share the relevant inputs.conf stanza.

Re: Will Splunk index old data present in a folder after integration date?

inventsekar — Sat, 26 Sep 2020 19:51:52 GMT

Hi @santhoshi .. if Splunk can monitor only the logs after its integration date, the whole vision and mission of Splunk can be thrown under the bus! no hard feelings, just kidding,..in general, Splunk has to monitor "everything".

Old or Current or Future ( at strange times, we may end up having file with future date as well.. linux is crazy! 😉

1. Are you monitoring the whole directory level, or just a direct filename monitoring?!
2. on the UF, using the btool, when you list the inputs config, may we know what you see?!

3. did you see any errors on splunkd.log?

4. maybe all files reached splunk fine and you were only searching for today's file. may we know your search query and time picker which you selected?

(PS - i have given around 350+ karma points so far, received badge for that, if an answer helped you, pls give a karma point!. we all should start "Learn, Give Back, Have Fun")

Re: Will Splunk index old data present in a folder after integration date?

santhoshi — Mon, 28 Sep 2020 13:49:12 GMT

hello, thanks for your reply. Inputs would be like this.

[monitor:///opt/nfmp/alarm_*.log]
disabled = false
sourcetype = alarm_log
index = proj_app
crcSalt = <SOURCE>
blacklist = (\.(tar|gz|bz2|tar.gz|tgz|tbz|tbz2|zip|z)$)