Universal Forwarder - Not Forwarding Data - Intermittent Issue

[madhav_dholakia]

Hi All,

We have a universal forwarder running on Windows Server which is sending data to our Splunk Instance in Cloud.

Below are some details of .conf files and logs:

inputs.conf

[default]
host = DB_DATA

[monitor://D:\ABC\DB_Monitoring\Cust]
disabled=0
index=rjsql
sourcetype = csv
crcSalt = <SOURCE>
time_before_close = 60

props.conf

[default]
NO_BINARY_CHECK=true
CHARSET=AUTO

[source::D:\ABC\DB_Monitoring\Cust\*.csv]
CHECK_METHOD = modtime

There are some files which are either 1) no being indexed at all 2) only headers are indexed - this doesn't happen with all the files, only some of them.

Logs from _internal (for file which has got only header indexed)

Tailing processer file status

btool output:

Can you please suggest what else I could check here and resolve this intermittent issue?

Thank you.

[VatsalJagani]

@madhav_dholakia - Your configuration looks correct to me.

I think you should observe how your system is overriding the data in the CSV files every time, how and when they are writing in the file, what is the size of the files, and how long it's taking to write a file.

Please monitor the above parameters on the host for files which are having this more than other files and compare. I think that should lead you to the root cause that is causing this intermittent issue.

I hope this helps!!!

[madhav_dholakia]

thanks, @VatsalJagani - these files are <10KBs and updated every 15 days/month. With a time_before_close param set, I don't think file writing will take more time looking at the file size.

[VatsalJagani]

Yeah, file size, I don't think should be a problem here then.