topic inputs.conf configuration for files where content only sometimes changes in Getting Data In

inputs.conf configuration for files where content only sometimes changes

RobertRi — Mon, 17 May 2021 10:00:05 GMT

Hello Community!

I have a file which will be renewed once a day.

Often the output is the same as the output before.

So it happens that Splunk doesn't index the file and the content isn't available in a subsearch with a timefilter last24h.

So there are some config attributes like crcSalt to tellSplunk, it should read the file if the crc hash has changed, but in this time I think its worthless because the crcSalt is the same of the whole file.

So what can make sense is to, tell Splunk to use only the mtime of the file, but I haven't seen such a setting.

Do you have a hint, how I can read the file with same content, same crc and same size but an other timestamp.

Thanks

Re: inputs.conf configuration for files where content only sometimes changes

javiergn — Mon, 17 May 2021 10:13:50 GMT

Hi @RobertRi,

If the filename or path are different every day, even if the content is the same, you can use

crcSalt = <SOURCE>

(including the angle brackets)

In order to append the full directory path of the source file to the CRC.

That will make sure you are reading this file every single day.

If the file path or name don't change then is there anything you can do to modify the output, adding perhaps a timestamp to the raw data? Or alternatively, can you select different destination directories or append today's name to the file name in order to make it work with crcSalt=<SOURCE>?

Regards,

Re: inputs.conf configuration for files where content only sometimes changes

gcusello — Mon, 17 May 2021 10:13:15 GMT

Hi @RobertRi,

the easiest way to intervene is to change the name of the output file to read and use crcSalt = <SOURCE> (eventually using a scheduled script) but I don't know if this workaround is acceptable for you.

Ciao.

Giuseppe

Re: inputs.conf configuration for files where content only sometimes changes

RobertRi — Mon, 17 May 2021 10:13:53 GMT

Hi J.!

No, the path or filename doesn't change, only the mtime

Regards
Robert

Re: inputs.conf configuration for files where content only sometimes changes

javiergn — Mon, 17 May 2021 10:24:12 GMT

Hi,

Is there anything you can do to modify the output, adding perhaps a timestamp to the raw data? Or alternatively, can you select different destination directories or append today's name to the file name in order to make it work with crcSalt=<SOURCE>?

Alternatively, you could have a small cron job that uses awk to read that inputs.conf file and modifies the crcSalt every day for that particular stanza and then restarts splunk. Sounds a bit overengineering, but if you can't alter the way this way is written ...

Or following the same logic, a small job that appends the current time to the end of the file or moves the file somewhere else that is different every day.

Another option is to use btprobe from the CLI to force a reindex of the file:

splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file $FILE --reset

Example: https://community.splunk.com/t5/Archive/Use-btprobe-reset-to-re-index-multiple-files/m-p/313186

Re: inputs.conf configuration for files where content only sometimes changes

RobertRi — Mon, 17 May 2021 10:57:56 GMT

Ciao Giuseppe!

Grazie per la tua risposta!

I will follow this solution, that looks good for me!

Buona giornata!

Re: inputs.conf configuration for files where content only sometimes changes

RobertRi — Mon, 17 May 2021 10:59:36 GMT

Thanks J.!

I will follow your advice and try to modify the filename and have crcSalt in place!

Regards

Robert