topic Re: Help with monitor stanza for a csv file in Getting Data In

Help with monitor stanza for a csv file

vrmandadi — Tue, 29 Sep 2020 22:17:37 GMT

Below is the path of the csv file

/home/reports/8e20594b-282a-493e-ad9a-dc69e0ac676c.csv and I am using the monitor stanza as below

[monitor:///home/reports/*.csv]
recursive = true
index = main
sourcetype = rf

The sourcetype I made some props.conf changes

SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
KV_MODE=none
category=Structured
disabled=false
pulldown_type=true
TIMESTAMP_FIELDS=Timestamp
HEADER_FIELD_LINE_NUMBER=1

But I am only able to see one file

Re: Help with monitor stanza for a csv file

prakash007 — Tue, 04 Dec 2018 20:16:27 GMT

It's something to do with the file header, where splunk is thinking that it has already read the file..
you can either increase the initCrcLength or use crcSalt =

[monitor:///home/reports/*.csv]
recursive = true
index = main
sourcetype = rf

##defaults to 256bytes
initCrcLength = 1024

##if the above setting didn't work 
crcSalt = <SOURCE>

read this Splunk doc for more info...

http://docs.splunk.com/Documentation/Splunk/7.2.1/Admin/Inputsconf#MONITOR:

Re: Help with monitor stanza for a csv file

vrmandadi — Tue, 29 Sep 2020 22:17:40 GMT

I tried the above but it did not work.I removed the stanza recursive=true.Should I add it ? and is there way to re index the data after applying these changes

I tried the test command and below is the output

./splunk test sourcetype ///home/reports/*csv

Using logging configuration at /opt/splunk/etc/log-cmdline.cfg.
WARN FileClassifierManager - The file '///home/reports/*csv' is invalid. Reason: failed_stat
PROPERTIES OF ///home/reports/*csv
PropertiesMap: {CHARSET -> 'UTF-8' invalid_cause -> 'failed_stat' is_valid -> 'False' sourcetype -> 'unknown'}

Re: Help with monitor stanza for a csv file

prakash007 — Tue, 29 Sep 2020 22:17:42 GMT

Did you try crcSalt = ..??
using recursive depends on whether or not to monitor subdirectories..

recursive = <boolean>
* Whether or not the input monitors subdirectories that it finds within a 
  monitored directory.
* If you set this setting to "false", the input does not monitor sub-directories
* Default: true.

What's your directory structure in your monitor stanza..??
Can you run this commands to check the status of your file...

./splunk list monitor 
./splunk list inputstatus

Can you post a sample csv data, and also is there any specific reason to use HEADER_FIELD_LINE_NUMBER in props..??

Re: Help with monitor stanza for a csv file

vrmandadi — Tue, 29 Sep 2020 22:17:45 GMT

This is the latest error I got.Is it permission issue and I am doing this on my deployment server.so if I list the monitor it is all the splunk logs.I used the crc but does this re index the files or it applies to new files coming?

Using logging configuration at /opt/splunk/etc/log-cmdline.cfg.
WARN FileClassifierManager - Unable to open '///home/reports/.csv'.
WARN FileClassifierManager - The file '///home/reports/.csv' is invalid. Reason: cannot_open
PROPERTIES OF ///home/reports/*.csv
PropertiesMap: {CHARSET -> 'UTF-8' invalid_cause -> 'cannot_open' is_valid -> 'False' sourcetype -> 'unknown'}

Re: Help with monitor stanza for a csv file

vrmandadi — Tue, 04 Dec 2018 21:32:19 GMT

I checked the internal logs and below are the errors.The below logs are before the changes you have mentioned.Does the new files coming into splunk have the correct format and files coming in?

12-04-2018 12:11:13.581 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/reports/8e20594b-282a-493e-ad9a-dc69e0ac676c.csv'.

12-04-2018 12:11:13.579 -0500 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/reports/1bc778b1-47e5-4b72-bfed-798d84cddfd4.csv). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Re: Help with monitor stanza for a csv file

prakash007 — Tue, 04 Dec 2018 22:01:28 GMT

It won't reindex the same file again once it's indexed, you have to reset the checkpoint of a specific file or delete the fishbucket(this will reindex all data on that box)

https://answers.splunk.com/answers/46780/reset-splunkforwarder-to-re-read-file-from-beginning.html

Re: Help with monitor stanza for a csv file

vrmandadi — Tue, 29 Sep 2020 22:17:51 GMT

I got the data into Splunk but it is not breaking correctly.I initially done a testing through Web interface and it breaks correctly but does not break correctly through monitor stanza.Below is the props I used .The timestamp field in csv file is the below format

2018-11-23T04:17:30-05:00

Re: Help with monitor stanza for a csv file

ddrillic — Wed, 05 Dec 2018 00:34:57 GMT

@vrmandadi - have you placed the props.conf on the forwarder as well?

Re: Help with monitor stanza for a csv file

vrmandadi — Wed, 05 Dec 2018 01:10:39 GMT

Yes i did place it in the forwarder as well