Issue on file monitoring using forwader

[ptrckjncbngn]

i have these 2 directories being monitored by a forwarder. One i indexing and another is not. They have the same root folder

E:\FTP\BatFolder\Logs (Data is being ingested)
E:\FTP\BatFolder\CE\CSVtoSplunk (Data is not being forwarded)

All are just csv files

I am pretty sure i have correct props since its parsing the files coming from these 2 directories
I am also encountering this warning also on the _internal

02-24-2020 05:21:19.588 -0500 WARN AdminManager - Handler 'remote_monitor' has not performed any capability checks for this operation (requestedAction=edit, customAction="enable", item="E:\FTP\BatFolder\CE\CSVtoSplunk "). This may be a bug.

is anyone here experiencing same issue?

[broberg]

Do you have correct timestamps on the logs?
Else you may index them in the feature or in a year waay back in time.

[gcusello]

Hi @ptrckjncbngn,
could you share your inputs.conf and an example (one or two events) of both the sources?
I think that files in the folders are different, is it correct?

Ciao.
Giuseppe

[ptrckjncbngn]

Here is my inputs.conf

[monitor://E:\FTP\Batch360\Logs]
disabled = 0
index = batch_monitoring
sourcetype = mainframe_logs

[monitor://E:\FTP\Batch360\UC4\CSVtoSplunk]
disabled = 0
index = batch_monitoring
sourcetype = uc4_logs

[gcusello]

Try to insert in your inputs also the filenames, e.g.

[monitor://E:\FTP\Batch360\Logs\*.csv]
disabled = 0
index = batch_monitoring
sourcetype = mainframe_logs

[monitor://E:\FTP\Batch360\UC4\CSVtoSplunk\*.csv]
disabled = 0
index = batch_monitoring
sourcetype = uc4_logs

In addition, are files in the different folders different or the same?

Ciao.
Giuseppe

[ptrckjncbngn]

I will try inputting the names. they are on same parent directory E:\FTP but they are on different sub folders

[ptrckjncbngn]

putting file names is not working. will there be issue if they are on the same parent folder?

[gcusello]

Hi @ptrckjncbngn,
parent folder isn't a problem.
there's a problem if the files are the same (at least the first 256 chars) because Splunk doesn't index twice the same file.
if this is the problem, try adding to both the stanzas crcSalt = <SOURCE>
Ciao.
Giuseppe

[ptrckjncbngn]

the csv content of this directory is being forwarded to my splunk enterprise E:\FTP\Batch360\Logs. The mechanism here we are just overwriting the file. Meaning same file name all throughout but different content. no problem here

On this directory E:\FTP\Batch360\UC4\CSVtoSplunk there are 3 files not being forwarded. I am pretty sure each records are unique since there is a unique field there (runid). Please see sample logs below

Runid,Type,Name,Title,Agent,Status,Status Text,Activation,Start,End,Runtime

8926441,JOBS,JOB1,,GROOVY1,1900,ENDED_OK - ended normally,02-13-2020 05:44:54,02-13-2020 05:47:04,02-13-2020 05:47:05,00:00:01

8923603,JOBS,JOB2,Uiq Copy Gdva Apping Files,FTP2,1900,ENDED_OK - ended normally,02-13-2020 05:45:13,02

Ill try to add the crcSalt, but i don't think this will work since I tried this beforehand.