Getting Data In

inputs.conf configuration for files where content only sometimes changes

RobertRi
Communicator

Hello Community!

I have a file which will be renewed once a day.

Often the output is the same as the output before.

So it happens that Splunk doesn't index the file and the content isn't available in a subsearch with a timefilter last24h.

 

So there are some config attributes like crcSalt to tellSplunk, it should read the file if the crc hash has changed, but in this time I think its worthless because the crcSalt is the same of the whole file.

So what can make sense is to, tell Splunk to use only the mtime of the file, but I haven't seen such a setting.

Do you have a hint, how I can read the file with same content, same crc and same size but an other timestamp.

Thanks

Labels (3)
0 Karma
1 Solution

javiergn
SplunkTrust
SplunkTrust

Hi,

 

Is there anything you can do to modify the output, adding perhaps a timestamp to the raw data? Or alternatively, can you select different destination directories or append today's name to the file name in order to make it work with crcSalt=<SOURCE>?

Alternatively, you could have a small cron job that uses awk to read that inputs.conf file and modifies the crcSalt every day for that particular stanza and then restarts splunk. Sounds a bit overengineering, but if you can't alter the way this way is written ...

Or following the same logic, a small job that appends the current time to the end of the file or moves the file somewhere else that is different every day.

Another option is to use btprobe from the CLI to force a reindex of the file:

splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file $FILE --reset

Example: https://community.splunk.com/t5/Archive/Use-btprobe-reset-to-re-index-multiple-files/m-p/313186

 

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @RobertRi,

the easiest way to intervene is to change the name of the output file to read and use crcSalt = <SOURCE> (eventually using a scheduled script) but I don't know if this workaround is acceptable for you.

Ciao.

Giuseppe

0 Karma

RobertRi
Communicator

Ciao Giuseppe!

Grazie per la tua risposta!

I will follow this solution, that looks good for me!

Buona giornata!

0 Karma

javiergn
SplunkTrust
SplunkTrust

Hi @RobertRi

If the filename or path are different every day, even if the content is the same, you can use

 

crcSalt = <SOURCE>

 

(including the angle brackets)

In order to append the full directory path of the source file to the CRC.

That will make sure you are reading this file every single day.

If the file path or name don't change then is there anything you can do to modify the output, adding perhaps a timestamp to the raw data? Or alternatively, can you select different destination directories or append today's name to the file name in order to make it work with crcSalt=<SOURCE>?

Regards,

J

 

0 Karma

RobertRi
Communicator

Hi J.!

No, the path or filename doesn't change, only the mtime

Regards
Robert

0 Karma

javiergn
SplunkTrust
SplunkTrust

Hi,

 

Is there anything you can do to modify the output, adding perhaps a timestamp to the raw data? Or alternatively, can you select different destination directories or append today's name to the file name in order to make it work with crcSalt=<SOURCE>?

Alternatively, you could have a small cron job that uses awk to read that inputs.conf file and modifies the crcSalt every day for that particular stanza and then restarts splunk. Sounds a bit overengineering, but if you can't alter the way this way is written ...

Or following the same logic, a small job that appends the current time to the end of the file or moves the file somewhere else that is different every day.

Another option is to use btprobe from the CLI to force a reindex of the file:

splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file $FILE --reset

Example: https://community.splunk.com/t5/Archive/Use-btprobe-reset-to-re-index-multiple-files/m-p/313186

 

0 Karma

RobertRi
Communicator

Thanks J.!

I will follow your advice and try to modify the filename and have crcSalt in place!

Regards

Robert

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...