topic Re: Splunk Universal Forwarder only forwards one csv log in Getting Data In

Splunk Universal Forwarder only forwards one csv log

RecoMark0 — Thu, 08 Jan 2015 01:27:00 GMT

Hello,
I am having an issue with the universal forwarder, where only one csv log gets sent to the index. We have multiple servers with the forwarder installed, and each server has the following in the inputs.conf:

#DocView logs
    [monitor://\\$COMPUTERNAME\s$\Logs\Audit\*.docView.csv]
    disabled = 0
    followTail = false
    sourcetype = Doc View
    crcSalt = <SOURCE>
    ignoreOlderThan = 2d

I have ran "splunk list monitor" from the bin folder, and csv files in that folder are listed correctly, however, only one shows up in splunk when I search: sourcetype="Doc View", and it is always the first one alphabetically. The files are named: "ProjectA.docview.csv", "ProjectB.docview.csv", "ProjectC.docview.csv". ProjectA will always be the only result in splunk.

I do not think it is a KBPS issue either, as I have not seen the warning in splunkd for a few weeks.

I do not think it is a security issue as the files are created the same exact way from our system.

Question:
Is there anything else I can check, or logs I can look at to see what the issue is? Has anyone seen this issue before?

Thank you!

Edit: I do not need this log to be read as a csv file. It can be treated as a normal log with the csv extension.

Re: Splunk Universal Forwarder only forwards one csv log

tom_frotscher — Thu, 08 Jan 2015 09:23:29 GMT

Hi,

as a first step, take a look in the splunkd.log of the forwarders. Always a good starting point for investigations according to forwarders.

Re: Splunk Universal Forwarder only forwards one csv log

esix_splunk — Thu, 08 Jan 2015 09:33:26 GMT

Try adjusting your inputs for the following:

monitor://\\$COMPUTERNAME\s$\Logs\Audit\*.docView.csv

Change this to

monitor://\\$COMPUTERNAME\s$\Logs\Audit\*.docview.csv

I am not sure the case sensitivity is an issue on windows, however there might be something that its missing.

Also, I'd recommend keeping your sourcetypes as one word, without spaces. If you want to separate it, do something like

sourcetype=doc:view

Re: Splunk Universal Forwarder only forwards one csv log

RecoMark0 — Thu, 08 Jan 2015 19:53:59 GMT

the splunkd log is not really telling me anything, other than that the server is only trying to send one of the files, and not all of them. There are no warnings or errors in the log either 😞

Re: Splunk Universal Forwarder only forwards one csv log

eabrown2 — Fri, 09 Jan 2015 01:31:40 GMT

This appears to be an issue with the initCrc.

Without changing the length the csvs would not be picked up on rollover.

I tested this with adding to the inputs.conf:

initCrcLength=1024

All logs are grabbed now.

Re: Splunk Universal Forwarder only forwards one csv log

RecoMark0 — Fri, 09 Jan 2015 01:54:49 GMT

This seems to have done the trick, thank you. I tested it on one of the servers, and more than one csv is now coming through.

Re: Splunk Universal Forwarder only forwards one csv log

tom_frotscher — Fri, 09 Jan 2015 08:35:50 GMT

This looks like the csv's have a long header that is the same for all csv files. Then splunk reads the first bytes of the files and if they are the same, the file will not be processed, because splunk think it is the same file.