How to set up file monitoring for this folder structure?

[dreamfeeder]

Hi, I want to setup the file monitoring for all the files starting with "mq-" or "secs-" or "err-" in below directory:

/var/mtapps/ashl/logs/[folder A]/[folder B]/[folder C]/

because there are many folder A folder B and Folder C with different names so we setup as below:

but it doesn't work. all 3 kinda files doesn't go into the splunk server.

how can I resolve this? Thanks in advance!

[krishnarajapant]

Hi,

Can you please try with the below stanza.

[monitor:///var/mtapps/ashl/logs/.../.../.../(mq-* | secs-* | err-*)]

-Krishna Rajapantula

[dreamfeeder]

Hi Krishna,

Thanks for your reply. however if I setup like this, I believe I can only have 1 source type defined for all 3 kind of files. that is not what I want. I want to monitor them under 3 different source type.

[dreamfeeder]

I've just change the input.conf to below:

[monitor:///var/mtapps/ashl/logs/.../.../.../mq-*]

recursive=true

sourcetype = mqhist
index = automation
disabled=1

[monitor:///var/mtapps/ashl/logs/.../.../.../err-*]

recursive=true

sourcetype = hosterror
index = automation
disabled=0

[monitor:///var/mtapps/ashl/logs/.../.../.../secs-*]

recursive=true

sourcetype = secs
index = automation
disabled=1

if I enable all 3 soucetypes, none of files will go in. but if I enable only one, it works for that sourcetype.

how can I do in order to enable all 3 types???

[dineshraj9]

Can you try this in inputs.conf -

[monitor:///var/mtapps/ashl/logs/*/*/*/mq-*]
index = index_name
sourcetype = sourcetype_name
crcSalt=<SOURCE>

[monitor:///var/mtapps/ashl/logs/*/*/*/secs-*]
index = index_name
sourcetype = sourcetype_name
crcSalt=<SOURCE>

[monitor:///var/mtapps/ashl/logs/*/*/*/err-*]
index = index_name
sourcetype = sourcetype_name
crcSalt=<SOURCE>

[sbbadri]

Hi,

you can try this,

[monitor:///var/mtapps/ashl/logs/*/*/*/mq-*]
index = index_name
sourcetype = sourcetype_name1
crcSalt=
blacklist.1=secs-.*
blacklist.2=err-.*

[monitor:///var/mtapps/ashl/logs/*/*/*/secs-*]
index = index_name
sourcetype = sourcetype_name2
crcSalt=
blacklist.1=mq-.*
blacklist.2=err-.*

[monitor:///var/mtapps/ashl/logs/*/*/*/err-*]
index = index_name
sourcetype = sourcetype_name3
crcSalt=
blacklist.1=mq-.*
blacklist.2=secs-.*

Regards,
Badri Srinivas B

[sbbadri]

at the end is mq-, secs- and err-*

[dreamfeeder]

sorry for not making myself clear. yes, it can access. in the inputstatus, I can see it run through all folders for mq-* look up (strange thing is it only manage to run mq-, but not err- and secs-. Even for mq- files, I wait for more than 5 hours (set up at 9:30am today and now is 3:13pm) it only manage to process 1 file with 91.7% uploaded) this doesn't look correct to me. so I suspect it is because of the monitor string setup is not correct.

/var/mtapps/ashl/logs/Rorze/SorterRSC/RBWSA22200/mq-sRBWSA22200.log.tu

file position 24332126
file size 26533134
parent /var/mtapps/ashl/logs////mq-
percent 91.70
type open file

[dreamfeeder]

Hi Dineshraj9,

I want to monitor them under 3 different source type. is it possible to do that?

[dineshraj9]

Can you try having a single monitor for all -

[monitor:///var/mtapps/ashl/logs/.../.../.../(mq|secs|err)-*)]

Splunk should have read the events as and when the logs are written. Try checking internal logs if they print any errors and adding crcSalt(as shown above).

[dreamfeeder]

yes. this is how we setup, but it doesnt work. you can see my screenshot below.

[dineshraj9]

Can you check if there is any permission issue for these logs. Since splunk is unable to access them.

index=_internal sourcetype=splunkd host=<log_server> "var/mtapps/ashl/logs"

Also try running the command to check if these logs are monitored on the log server -

./splunk list monitor -auth admin:<password>