Why is Splunk ignoring my files?

[ksextonmacb]

I have a folder set up on a Linux machine that a Splunk forwarder is monitoring. This folder is set up to receive FTP'd reports from our mainframe. At regular intervals, the mainframe sends a dozen reports to that folder over the course of a minute. The transfer process deletes the old files and places new, uniquely named files (name based on time of transfer, so no names conflict with old files).

For some reason Splunk only ever reads in one of them, if that. The Splunkd.log file logs no errors during this time.

I searched through previous answers and found that splunk may be considering the files as binary -- text encoding issues with files coming from the mainframe are common. However the recommended flag NO_BINARY_CHECK isn't helping.

My inputs.conf:

[monitor:///Znfs200g/Mainframe/splunk_sbss]
disabled = false
sourcetype = zos_sbss_report_source_type
crcSalt = <SOURCE>

My props.conf

[zos_sbss_report_source_type]
NO_BINARY_CHECK = true

Example file:

 BEGIN DATE AND TIME       20180308 @ 14:57:29:48

 AFTER DB6X LOAD           20180308 @ 14:57:29:49

 END OF ECSS RECORDS       20180308 @ 14:57:29:49

 END OF SBSS2 RECORDS      20180308 @ 14:57:29:49


 ===========3.5================
 END OF PROGRAM STATS FOR  20180308 @ 14:57:29:49

 DB6X-READ     =         0
 DB5A-READ     =         0
 ECSS IN       =         0
 SBSS 1 IN     =       176
 SBSS 2 IN     =         0
 ECSS OUT      =         0
 SBSS 1 OUT    =       176
 SBSS 2 OUT    =         0
 BAD RECS OUT  =         0
 B7A IN        =         0
 BL0 IN        =         0
 DSA IN        =         0
 DSB IN        =         0
 DSC IN        =         0
 DSM IN        =         1
 DSR IN        =         0
 XGF IN        =         7
 XGG IN        =         0
 XGH IN        =         0
 XGI IN        =        22
 XGJ IN        =       131
 XGL IN        =         0
 XHA IN        =         0
 XJE IN        =         3
 XJU IN        =         1
 XS2 IN        =         0
 XSA IN        =         5
 XSB IN        =         0
 XSC IN        =         0
 XSD IN        =         6
 XSK IN        =         0

[ksextonmacb · Accepted solution]

I figured it out; Splunk was putting the files at a different time than I expected, because the job FTPing the files from the mainframe wasn't generating new ones. So every file was from March 8th, when this started.

Just a dumb oversight on my part.

Sorry to waste other peoples' time.

[gcusello]

Hi ksextonmacb,
probably you forgot to insert * in you monitor

[monitor:///Znfs200g/Mainframe/splunk_sbss/*]

in this way, you're saying to Splun kto take a file called "splunk_sbss" located in a folder called "/Znfs200g/Mainframe" instead of many files (e.g. 2018-03-15-11-12-34.log) located in a folder called "/Znfs200g/Mainframe/splunk_sbss/".
Otherwqise you can use whitelist option.

Bye.
Giuseppe

[tiagofbmm]

If you check Splunk logging inputs, you see that there is no need for that:

[monitor:///opt/splunk/var/log/splunk]

No need for /* here, why would it differ there?

[p_gurav]

Hi,

Can you try adding this config in inputs.conf:

initCrcLength = <integer>
* This setting adjusts how much of a file the input reads before trying to
  identify whether it is a file that has already been seen. You might want to
  adjust this if you have many files with common headers (comment headers,
  long CSV headers, etc) and recurring filenames.
* CAUTION: Improper use of this setting will cause data to be re-indexed.  You
  might want to consult with Splunk Support before adjusting this value - the
  default is fine for most installations.
* Defaults to 256 (bytes).
* Must be in the range 256-1048576.

[tiagofbmm]

What would be the use of that if the user said: "The transfer process deletes the old files and places new, uniquely named files (name based on time of transfer, so no names conflict with old files)." ?

Wouldn't the CRCSalt in that case be enough?

[tiagofbmm]

Hey

This is just a file splunk_sbss?

[ksextonmacb]

no, it's a folder containing a dozen files. Is this going to be a "you forgot a trailing slash" type issue?

[tiagofbmm]

I don't think so, your syntax seems correct.

Your CRCSalt should take care of any issues with redundancy checks too...

Can you check the internal index for an ingestion of a specific file? At least an entry like

03-15-2018 12:32:55.317 +0000 INFO  Metrics - group=per_source_thruput, series="/opt/splunk/var/log/splunk/splunkd.log"

Just to see if splunk is actually verifying the file

[ksextonmacb]

These are the events I get from searching index=_internal group=per_source_thruput series=/znfs200g/mainframe/* around the time the files were available to read, on the host that runs the forwarder:

03-14-2018 16:03:43.705 -0400 INFO  Metrics - group=per_source_thruput, series="/znfs200g/mainframe/splunk_sbss/sbssreport20180314160052.sbss", kbps=0.12758116517915236, eps=0.09677267640255705, kb=3.955078125, ev=3, avg_age=0.3333333333333333, max_age=1

03-14-2018 16:03:43.705 -0400 INFO  Metrics - group=per_source_thruput, series="/znfs200g/mainframe/splunk_sbss/sbssreport20180314160045.sbss", kbps=0.11482304866123712, eps=0.09677267640255705, kb=3.5595703125, ev=3, avg_age=0.3333333333333333, max_age=1

03-14-2018 16:03:43.696 -0400 INFO  Metrics - group=per_source_thruput, series="/znfs200g/mainframe/splunk_sbss/sbssreport20180314160039.sbss", kbps=0.11227142535765408, eps=0.09677267640255705, kb=3.48046875, ev=3, avg_age=0.6666666666666666, max_age=2

03-14-2018 16:03:12.700 -0400 INFO  Metrics - group=per_source_thruput, series="/znfs200g/mainframe/splunk_sbss/sbssreport20180314160033.sbss", kbps=0.10972050637789256, eps=0.06451553174330289, kb=3.4013671875, ev=2, avg_age=0.5, max_age=1

03-14-2018 16:03:12.700 -0400 INFO  Metrics - group=per_source_thruput, series="/znfs200g/mainframe/splunk_sbss/sbssreport20180314160028.sbss", kbps=0.10972050637789256, eps=0.09677329761495433, kb=3.4013671875, ev=3, avg_age=0.3333333333333333, max_age=1

03-14-2018 16:03:12.696 -0400 INFO  Metrics - group=per_source_thruput, series="/znfs200g/mainframe/splunk_sbss/sbssreport20180314160021.sbss", kbps=0.10972050637789256, eps=0.09677329761495433, kb=3.4013671875, ev=3, avg_age=0.3333333333333333, max_age=1

03-14-2018 16:03:12.695 -0400 INFO  Metrics - group=per_source_thruput, series="/znfs200g/mainframe/splunk_sbss/sbssreport20180314160017.sbss", kbps=0.11227214606109936, eps=0.09677329761495433, kb=3.48046875, ev=3, avg_age=0.3333333333333333, max_age=1

03-14-2018 16:03:12.695 -0400 INFO  Metrics - group=per_source_thruput, series="/znfs200g/mainframe/splunk_sbss/sbssreport20180314160013.sbss", kbps=0.11992706511071978, eps=0.09677329761495433, kb=3.7177734375, ev=3, avg_age=0.3333333333333333, max_age=1

03-14-2018 16:03:12.695 -0400 INFO  Metrics - group=per_source_thruput, series="/znfs200g/mainframe/splunk_sbss/sbssreport20180314160010.sbss", kbps=0.10972050637789256, eps=0.09677329761495433, kb=3.4013671875, ev=3, avg_age=0.6666666666666666, max_age=2

03-14-2018 16:02:41.696 -0400 INFO  Metrics - group=per_source_thruput, series="/znfs200g/mainframe/splunk_sbss/sbssreport20180314160004.sbss", kbps=0.1097195401470167, eps=0.06451496360065753, kb=3.4013671875, ev=2, avg_age=0.5, max_age=1

03-14-2018 16:02:41.695 -0400 INFO  Metrics - group=per_source_thruput, series="/znfs200g/mainframe/splunk_sbss/sbssreport20180314155953.sbss", kbps=0.1097195401470167, eps=0.0967724454009863, kb=3.4013671875, ev=3, avg_age=0.3333333333333333, max_age=1

For reference, these are the files and their mod times:
-rw-rw-r-- 1 root other 68607 Mar 14 15:58 XSDReport20180314.sbss
-rw-rw-r-- 1 root other 3483 Mar 14 16:02 sbssReport20180314155953.sbss
-rw-rw-r-- 1 root other 3483 Mar 14 16:02 sbssReport20180314160004.sbss
-rw-rw-r-- 1 root other 3483 Mar 14 16:02 sbssReport20180314160010.sbss
-rw-rw-r-- 1 root other 3807 Mar 14 16:02 sbssReport20180314160013.sbss
-rw-rw-r-- 1 root other 3564 Mar 14 16:02 sbssReport20180314160017.sbss
-rw-rw-r-- 1 root other 3483 Mar 14 16:02 sbssReport20180314160021.sbss
-rw-rw-r-- 1 root other 3483 Mar 14 16:03 sbssReport20180314160028.sbss
-rw-rw-r-- 1 root other 3483 Mar 14 16:03 sbssReport20180314160033.sbss
-rw-rw-r-- 1 root other 3564 Mar 14 16:03 sbssReport20180314160039.sbss
-rw-rw-r-- 1 root other 3645 Mar 14 16:03 sbssReport20180314160045.sbss
-rw-rw-r-- 1 root other 4050 Mar 14 16:03 sbssReport20180314160052.sbss