How to import files that change names when they roll over to a new daily file

[tonyparreiro]

Hi,

I have a system which logs data into a file, once about 24 hours of logging occurs the file is renamed and a new file starts for the next lot of events.

Right now I'm only indexing the files after they are completed which is ok, except you can't do any queries on the most recent data (worst case 24hours).

Is there away/scheme/method to index the data from the file being written to so it can be queried and once it's renamed not to be indexed again?

Thanks,
Tony

[somesoni2]

Splunk already does what you seek. You should be monitor the file that is being continuously written into so that you get near real-time events in Splunk (there will be small lag till the data is indexed and made searchable). If your monitoring stanza specifically includes the live file, Splunk will monitoring as and when new data is written into it and will not monitor it once it's renamed/rolled over. (it keeps a handler to know which file it has already read). So just have your monitoring stanza pointing to your live file, something like this

[monitor:///your/log/folder/path/logfile_*_LIVE_*.log]
index=yourindex
sourcetype=yoursourcetype
...

[gcusello · Accepted solution]

Hi tonyparreiro,
you can index log using a jolly character

[monitor:///apps/logs/mylogs*.log]
sourcetype=my_sourcetype
index=my_index

or monitor the file before it's renamed.
In this second way, you continuously have logs and you can monitor them in real time.
if you don't use the option crcSalt=<SOURCE>, when your file is renamed it will not be reindexed.

Bye.
Giuseppe

[gcusello]

Ad I described, if you don't use crcSalt option Splunk doesn't read files after rotation .
Bye
Giuseppe

[tonyparreiro]

Monitoring the file before it's renamed will index all the data but once it's renamed it will index the new file will it not?

So there will be duplicate entries in the index. Or is it smart enough to know the records are the same and wont index them again?

I guess will just have to try something and see what happens.

thanks,
Tony

[tonyparreiro]

Thanks for the response gokadroid.

The problem is not selecting the file to index, the problem I'm trying to avoid is that if I index the file as it's being created when it's finished and renamed it will be indexed again and I'll end up with duplicate entries.

For instance, say we had the following set of files. I'm currently indexing everything except file 3. File 3 is the file that is currently being written to by the system.

file 3: >logfile_20170103100502_LIVE.log
file 2: >logfile_20170102100503_20170103100502.log
file 1: >logfile_20170101100501_20170102100503_.log
etc....

Once file 3 rolls over at the end of the period a new file is created and we end up with something like the following:

file 4: >logfile_20170104100501_LIVE.log
file 3: >logfile_20170103100502_20170104100501.log
file 2: >logfile_20170102100503_20170103100502_.log
file 1: >logfile_20170101100501_20170102100503_.log
etc....

At this point file 3 is indexed. I would like to index file 4 in this case as I would like to report on information in "real time" if I can. as well as not double index any of the data. Also another problem would be that once the data is indexed the index file it belonged to will no longer exist so not sure what it would do then either.

Hope this clears up the problem.

Thanks.
Tony

[gokadroid]

If the name changes and the extension stays the same a good option might be to monitor the files having the same extension within a directory as follows, example, monitor all files *.log in directory /jfplogs/prod/CP/:

inputs.conf should have stanza as follows (or something silmilar)

[monitor:///jfplogs/prod/CP/*.log]
sourcetype=yourSourcetype
index=yourIndex

See below for other options to specify the file names as per your need:
http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/Specifyinputpathswithwildcards#Input_examples