topic Re: Issue on file monitoring using forwader in Getting Data In

Issue on file monitoring using forwader

ptrckjncbngn — Mon, 24 Feb 2020 10:28:20 GMT

i have these 2 directories being monitored by a forwarder. One i indexing and another is not. They have the same root folder

E:\FTP\BatFolder\Logs (Data is being ingested)
E:\FTP\BatFolder\CE\CSVtoSplunk (Data is not being forwarded)

All are just csv files

I am pretty sure i have correct props since its parsing the files coming from these 2 directories
I am also encountering this warning also on the _internal

02-24-2020 05:21:19.588 -0500 WARN AdminManager - Handler 'remote_monitor' has not performed any capability checks for this operation (requestedAction=edit, customAction="enable", item="E:\FTP\BatFolder\CE\CSVtoSplunk "). This may be a bug.

is anyone here experiencing same issue?

Re: Issue on file monitoring using forwader

gcusello — Mon, 24 Feb 2020 13:23:53 GMT

Hi @ptrckjncbngn,
could you share your inputs.conf and an example (one or two events) of both the sources?
I think that files in the folders are different, is it correct?

Ciao.
Giuseppe

Re: Issue on file monitoring using forwader

ptrckjncbngn — Wed, 30 Sep 2020 04:24:26 GMT

Here is my inputs.conf

[monitor://E:\FTP\Batch360\Logs]
disabled = 0
index = batch_monitoring
sourcetype = mainframe_logs

[monitor://E:\FTP\Batch360\UC4\CSVtoSplunk]
disabled = 0
index = batch_monitoring
sourcetype = uc4_logs

Re: Issue on file monitoring using forwader

gcusello — Tue, 25 Feb 2020 11:00:08 GMT

Try to insert in your inputs also the filenames, e.g.

[monitor://E:\FTP\Batch360\Logs\*.csv]
disabled = 0
index = batch_monitoring
sourcetype = mainframe_logs

[monitor://E:\FTP\Batch360\UC4\CSVtoSplunk\*.csv]
disabled = 0
index = batch_monitoring
sourcetype = uc4_logs

In addition, are files in the different folders different or the same?

Ciao.
Giuseppe

Re: Issue on file monitoring using forwader

ptrckjncbngn — Wed, 26 Feb 2020 07:05:51 GMT

I will try inputting the names. they are on same parent directory E:\FTP but they are on different sub folders

Re: Issue on file monitoring using forwader

ptrckjncbngn — Wed, 26 Feb 2020 07:09:31 GMT

putting file names is not working. will there be issue if they are on the same parent folder?

Re: Issue on file monitoring using forwader

gcusello — Wed, 26 Feb 2020 14:58:43 GMT

Hi @ptrckjncbngn,
parent folder isn't a problem.
there's a problem if the files are the same (at least the first 256 chars) because Splunk doesn't index twice the same file.
if this is the problem, try adding to both the stanzas crcSalt = <SOURCE>
Ciao.
Giuseppe

Re: Issue on file monitoring using forwader

ptrckjncbngn — Thu, 27 Feb 2020 05:16:43 GMT

the csv content of this directory is being forwarded to my splunk enterprise E:\FTP\Batch360\Logs. The mechanism here we are just overwriting the file. Meaning same file name all throughout but different content. no problem here

On this directory E:\FTP\Batch360\UC4\CSVtoSplunk there are 3 files not being forwarded. I am pretty sure each records are unique since there is a unique field there (runid). Please see sample logs below

Runid,Type,Name,Title,Agent,Status,Status Text,Activation,Start,End,Runtime

8926441,JOBS,JOB1,,GROOVY1,1900,ENDED_OK - ended normally,02-13-2020 05:44:54,02-13-2020 05:47:04,02-13-2020 05:47:05,00:00:01

8923603,JOBS,JOB2,Uiq Copy Gdva Apping Files,FTP2,1900,ENDED_OK - ended normally,02-13-2020 05:45:13,02

Ill try to add the crcSalt, but i don't think this will work since I tried this beforehand.

Re: Issue on file monitoring using forwader

broberg — Thu, 27 Feb 2020 10:04:13 GMT

Do you have correct timestamps on the logs?
Else you may index them in the feature or in a year waay back in time.