topic Re: How to edit inputs.conf on my Splunk forwarder to send CSV data for a second index? in Getting Data In

How to edit inputs.conf on my Splunk forwarder to send CSV data for a second index?

perfecto25 — Thu, 06 Apr 2017 20:08:26 GMT

Hello, I have an inputs.conf on my forwarder setup like this,

[monitor:///opt/jira-maestro/plugins/bintray_url/csv/*.csv]
index=bintray
sourcetype=csv

[monitor:///opt/jira-maestro/plugins/nessus/csv/*.csv]
index=nessus
sourcetype=csv

forwarder sends data for the 1st index, "bintray", but I cant get it to send for 2nd index "nessus"

I enabled DEBUG for Tailing Processor, getting tons of msg like this in splunkd.log

38915394548.tmp', does not match path='/opt/jira-maestro/plugins/nessus/csv' :Not a directory :Not a symlink
04-06-2017 16:07:04.657 -0400 DEBUG TailingProcessor -   Skipping itemPath='/opt/atlassian/jira/temp/imageio2771437074019475859.tmp', does not match path='/opt/jira-maestro/plugins/nessus/csv' :Not a directory :Not a symlink
04-06-2017 16:07:04.663 -0400 DEBUG TailingProcessor -   Skipping itemPath='/opt/atlassian/jira/temp/imageio1428026418037972330.tmp', does not match path='/opt/jira-maestro/plugins/nessus/csv' :Not a directory :Not a symlink

Not sure where else to troubleshoot. Spent entire day trying to get it to send data over.

Re: How to edit inputs.conf on my Splunk forwarder to send CSV data for a second index?

gjanders — Fri, 07 Apr 2017 00:47:50 GMT

Do you have any errors about file access permisisons or similar? Can the Splunk user read the files in the directory?

Try running:

splunk btool inputs list --debug

If that shows the information you expect just double check that the monitor information was printed on the startup of the forwarder.
Finally you might want to check the metrics log file and see if the log is mentioned (it might or might not mention the sourcetype/index/source depending on how busy the forwarder is), if it does then you might have an issue finding the data rather than an issue with the data getting indexed.

Good luck

Re: How to edit inputs.conf on my Splunk forwarder to send CSV data for a second index?

perfecto25 — Fri, 07 Apr 2017 18:06:39 GMT

Hello, I tried running,

splunk btool inputs list --debug

It shows correct syntax,

/opt/splunkforwarder/etc/apps/jira-maestro/local/inputs.conf               [monitor:///opt/jira-maestro/plugins/nessus/csv/report.csv]
/opt/splunkforwarder/etc/system/default/inputs.conf                        _rcvbuf = 1572864
/opt/splunkforwarder/etc/apps/jira-maestro/local/inputs.conf               disabled = false
/opt/splunkforwarder/etc/system/default/inputs.conf                        host = $decideOnStartup
/opt/splunkforwarder/etc/apps/jira-maestro/local/inputs.conf               index = nessus
/opt/splunkforwarder/etc/apps/jira-maestro/local/inputs.conf               sourcetype = csv

Also tried copying the csv file to some other location ,for example /opt/test

/opt/test/report.csv

created a new input.conf,

  [default]
  index = nessus 
  [monitor:///opt/test]
  whitelist = ^.*.csv
  sourcetype = csv
  disabled = false
  initCrcLength = 1048575
  crcSalt = /opt/test

Restarted forwarded, nothing gets sent to indexer, also tried modfying report.csv file to generate a change, using vim

04-07-2017 14:03:49.845 -0400 INFO  WatchedFile - Will begin reading at offset=45491 for file='/opt/test/report.csv'.
04-07-2017 14:03:49.849 -0400 INFO  WatchedFile - Resetting fd to re-extract header.
04-07-2017 13:53:06.534 -0400 WARN  FileClassifierManager - The file '/opt/test/.report.csv.swp' is invalid. Reason: binary
04-07-2017 13:53:06.534 -0400 INFO  TailReader - Ignoring file '/opt/test/.report.csv.swp' due to: binary
04-07-2017 13:53:10.667 -0400 WARN  FileClassifierManager - The file '/opt/test/.report.csv.swp' is invalid. Reason: binary
04-07-2017 13:53:10.667 -0400 INFO  TailReader - Ignoring file '/opt/test/.report.csv.swp' due to: binary
04-07-2017 13:53:13.984 -0400 INFO  WatchedFile - Will begin reading at offset=45491 for file='/opt/test/report.csv'.
04-07-2017 13:53:13.984 -0400 INFO  WatchedFile - Resetting fd to re-extract header.
04-07-2017 13:53:13.985 -0400 WARN  TailReader - Insufficient permissions to read file='/opt/test/.report.csv.swp' (hint: No such file or directory ,                            UID: 0, GID: 0).
04-07-2017 13:53:16.989 -0400 INFO  WatchedFile - Resetting fd to re-extract header.

also tried injecting a new column into csv to keep track of timestamp in format of "2017-04-07 11:38:53,008"

Nothing is being sent to indexer. Indexer splunkd log doesnt show anything coming in for this report.csv All permissions are splunk user + 644 on the report.csv file