Getting Data In

Why is CSV data not getting parsed while being monitored on server with a universal forwarder?

Communicator

We have a remote server where some CSVs are stored and the directory set to be monitored by Splunk. Now, if I upload the same CSV locally to Splunk (indexer/deployment), it seems like parsing is working fine, but the same file in the remote directory is not parsing or extracting any fields. The props & transforms on the Splunk server (indexer).

inputs.conf

[monitor:///home/test/Report/Report*.csv]
sourcetype = new_test
index = test_index
crcSalt = <SOURCE>
disabled = false

props.conf:

[ndlp_test]
TRANSFORMS-ignoreHeader = ignoreHeader
INDEXED_EXTRACTIONS = csv
SHOULD_LINEMERGE = false
DATETIME_CONFIG = CURRENT
TIME_PREFIX = \w{3}\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w{3}\s\w{4}
MAX_TIMESTAMP_LOOKAHEAD = 8
REPORT-fields = new_test
pulldown_type = 1
KV_MODE = none
NO_BINARY_CHECK = 1

transforms:

[ndlp_test]
DELIMS = ","
FIELDS = "Field1","Timestamp","Content","Subject","Filename",

[ignoreHeader]
REGEX = ^Field1\,Timestamp\,Content,Subject
DEST_KEY = queue
FORMAT = nullQueue

Anything else I should be looking for?

Legend

In the inputs.conf, you specify the sourcetype as "newtest," but the sourcetype in props.conf is "ndlp_test" - props.conf does not define a sourcetype called "newtest".

In the props.conf, you specify the REPORTS field extract as "REPORT-fields = new_test", but there is no stanza named "new_test" - transforms.conf defines "ndlp_test".

You should not be using "INDEXED_EXTRACTIONS = csv" if you can extract the CSV fields at search time as you configured. Remove this line.

I don't know if these problems exist on both the local machine and the remote machine, but they would certainly cause problems. BTW, if by "remote machine" you mean a Universal Forwarder, then only the inputs.conf belongs on the remote machine.
If the remote machine is a Heavy Forwarder or Indexer, then all three files go on the remote machine.

Regardless of the type remote machine, props.conf and transforms.conf must always be on the local machine (indexer).

0 Karma

Communicator

Oh, I am sorry, Its just I wanted to rename the sourcetype before posting. Its all same in original settings. I checked the remote machine and saw the splunk config files where under root and not accessible manually. Could it be the deployment server not able to push the configs to that machine ?

Like I said, if Im uploading the csv to the indexer manually, its parsing perfectly. Its only splunk somehow cannot do the same when the csv is on the remote machine.

0 Karma

Legend

Did you remove the "INDEXED_EXTRACTIONS = csv" line from props.conf?

Where did you put each of the configuration files when you were trying to collect the input from the remote machine?

0 Karma

Communicator

Had to re-add "INDEXED_EXTRACTIONS = csv" to get it working.

0 Karma

Communicator

Yes, INDEXED_EXTRACTIONS = csv is removed. UF was reinstalled on the remote machine. I can see the configs are pushed by the deployment server to the UF (including inputs.conf). But the same parser which works when the CSV is uploaded to the indexer doesn't seems t have any effect whent he CSVs are on the UF installed machine.

0 Karma

Communicator

After couple of restarts and testing with new files, CSV seems to be parsing properly. Still not sure what actually did the trick. I tried the same on my test machine (Version 6.3) and that had no problems at all.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!