Getting Data In

Forwarder only reads file if i save it, ignores when script saves it

LAcioffi
Explorer

Hello everyone,

I have Splunk Universal Forwarder running on a server watching a few files for changes. Log data is inserted at the end of the files every 5 minutes or so.
Up until a few days ago, all files were working and being correctly monitored. Today i noticed that a single file out of 10+ is not being monitored correctly.

When the script appends something to the file and closes it (thus updating the update date), the data doesn't arrive at the index. However, if i open the file, change anything and save it, all the data that should have arrived suddenly arrives.

This problem started out of the blue. I tried restarting the universal forwarder service, changing how the file is saved, deleting the file and letting the script re-create it, everything, but it still won't work.

Any ideas? Has this ever happened to anyone before?

P.S.: The file is open and closed explictly in my script. All other files do the same thing and work, only this one file is giving me trouble.

Thanks!

0 Karma

PavelP
Motivator

Hello @LAcioffi ,

this can be a problem with CRC calculation, so splunk doesn't notice that the file has been changed. This especially the case if the file has large header or/and is too small.

Run this command on the UF and compare the output after the file has been changed by the script:

./splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus

if you see that this command reports wrong status and size, then you need to change crcSalt or initCrcLength in your inputs.conf

Please let me know if it worked for you

0 Karma

LAcioffi
Explorer

Hey,

I don't have permissions to do that, but i checked the forwarder logs and the admin checked for error logs on index=_internal and we didn't find anything.

0 Karma

PavelP
Motivator

Hello @LAcioffi

there are no errors in such situations.

Can you post a redacted file?

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...