Getting Data In

Issue on file monitoring using forwader

ptrckjncbngn
New Member

i have these 2 directories being monitored by a forwarder. One i indexing and another is not. They have the same root folder

E:\FTP\BatFolder\Logs (Data is being ingested)
E:\FTP\BatFolder\CE\CSVtoSplunk (Data is not being forwarded)

All are just csv files

I am pretty sure i have correct props since its parsing the files coming from these 2 directories
I am also encountering this warning also on the _internal

02-24-2020 05:21:19.588 -0500 WARN AdminManager - Handler 'remote_monitor' has not performed any capability checks for this operation (requestedAction=edit, customAction="enable", item="E:\FTP\BatFolder\CE\CSVtoSplunk "). This may be a bug.

is anyone here experiencing same issue?

0 Karma

broberg
Communicator

Do you have correct timestamps on the logs?
Else you may index them in the feature or in a year waay back in time.

0 Karma

gcusello
Legend

Hi @ptrckjncbngn,
could you share your inputs.conf and an example (one or two events) of both the sources?
I think that files in the folders are different, is it correct?

Ciao.
Giuseppe

0 Karma

ptrckjncbngn
New Member

Here is my inputs.conf

[monitor://E:\FTP\Batch360\Logs]
disabled = 0
index = batch_monitoring
sourcetype = mainframe_logs

[monitor://E:\FTP\Batch360\UC4\CSVtoSplunk]
disabled = 0
index = batch_monitoring
sourcetype = uc4_logs

0 Karma

gcusello
Legend

Try to insert in your inputs also the filenames, e.g.

[monitor://E:\FTP\Batch360\Logs\*.csv]
disabled = 0
index = batch_monitoring
sourcetype = mainframe_logs

[monitor://E:\FTP\Batch360\UC4\CSVtoSplunk\*.csv]
disabled = 0
index = batch_monitoring
sourcetype = uc4_logs

In addition, are files in the different folders different or the same?

Ciao.
Giuseppe

0 Karma

ptrckjncbngn
New Member

I will try inputting the names. they are on same parent directory E:\FTP but they are on different sub folders

0 Karma

ptrckjncbngn
New Member

putting file names is not working. will there be issue if they are on the same parent folder?

0 Karma

gcusello
Legend

Hi @ptrckjncbngn,
parent folder isn't a problem.
there's a problem if the files are the same (at least the first 256 chars) because Splunk doesn't index twice the same file.
if this is the problem, try adding to both the stanzas crcSalt = <SOURCE>
Ciao.
Giuseppe

0 Karma

ptrckjncbngn
New Member

the csv content of this directory is being forwarded to my splunk enterprise E:\FTP\Batch360\Logs. The mechanism here we are just overwriting the file. Meaning same file name all throughout but different content. no problem here

On this directory E:\FTP\Batch360\UC4\CSVtoSplunk there are 3 files not being forwarded. I am pretty sure each records are unique since there is a unique field there (runid). Please see sample logs below

Runid,Type,Name,Title,Agent,Status,Status Text,Activation,Start,End,Runtime

8926441,JOBS,JOB1,,GROOVY1,1900,ENDED_OK - ended normally,02-13-2020 05:44:54,02-13-2020 05:47:04,02-13-2020 05:47:05,00:00:01

8923603,JOBS,JOB2,Uiq Copy Gdva Apping Files,FTP2,1900,ENDED_OK - ended normally,02-13-2020 05:45:13,02

Ill try to add the crcSalt, but i don't think this will work since I tried this beforehand.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...