Getting Data In

How to set up file monitoring for this folder structure?

dreamfeeder
New Member

Hi, I want to setup the file monitoring for all the files starting with "mq-" or "secs-" or "err-" in below directory:

/var/mtapps/ashl/logs/[folder A]/[folder B]/[folder C]/

because there are many folder A folder B and Folder C with different names so we setup as below:

alt text

but it doesn't work. all 3 kinda files doesn't go into the splunk server.

how can I resolve this? Thanks in advance!

Tags (3)
0 Karma

krishnarajapant
Path Finder

Hi,

Can you please try with the below stanza.

[monitor:///var/mtapps/ashl/logs/.../.../.../(mq-* | secs-* | err-*)]

-Krishna Rajapantula

0 Karma

dreamfeeder
New Member

Hi Krishna,

Thanks for your reply. however if I setup like this, I believe I can only have 1 source type defined for all 3 kind of files. that is not what I want. I want to monitor them under 3 different source type.

0 Karma

dreamfeeder
New Member

I've just change the input.conf to below:

[monitor:///var/mtapps/ashl/logs/.../.../.../mq-*]

recursive=true

sourcetype = mqhist
index = automation
disabled=1

[monitor:///var/mtapps/ashl/logs/.../.../.../err-*]

recursive=true

sourcetype = hosterror
index = automation
disabled=0

[monitor:///var/mtapps/ashl/logs/.../.../.../secs-*]

recursive=true

sourcetype = secs
index = automation
disabled=1

if I enable all 3 soucetypes, none of files will go in. but if I enable only one, it works for that sourcetype.

how can I do in order to enable all 3 types???

0 Karma

dineshraj9
Builder

Can you try this in inputs.conf -

[monitor:///var/mtapps/ashl/logs/*/*/*/mq-*]
index = index_name
sourcetype = sourcetype_name
crcSalt=<SOURCE>

[monitor:///var/mtapps/ashl/logs/*/*/*/secs-*]
index = index_name
sourcetype = sourcetype_name
crcSalt=<SOURCE>

[monitor:///var/mtapps/ashl/logs/*/*/*/err-*]
index = index_name
sourcetype = sourcetype_name
crcSalt=<SOURCE>
0 Karma

sbbadri
Motivator

Hi,

you can try this,

[monitor:///var/mtapps/ashl/logs/*/*/*/mq-*]
index = index_name
sourcetype = sourcetype_name1
crcSalt=
blacklist.1=secs-.*
blacklist.2=err-.*

[monitor:///var/mtapps/ashl/logs/*/*/*/secs-*]
index = index_name
sourcetype = sourcetype_name2
crcSalt=
blacklist.1=mq-.*
blacklist.2=err-.*

[monitor:///var/mtapps/ashl/logs/*/*/*/err-*]
index = index_name
sourcetype = sourcetype_name3
crcSalt=
blacklist.1=mq-.*
blacklist.2=secs-.*

Regards,
Badri Srinivas B

0 Karma

sbbadri
Motivator

at the end is mq-, secs- and err-*

0 Karma

dreamfeeder
New Member

sorry for not making myself clear. yes, it can access. in the inputstatus, I can see it run through all folders for mq-* look up (strange thing is it only manage to run mq-, but not err- and secs-. Even for mq- files, I wait for more than 5 hours (set up at 9:30am today and now is 3:13pm) it only manage to process 1 file with 91.7% uploaded) this doesn't look correct to me. so I suspect it is because of the monitor string setup is not correct.

/var/mtapps/ashl/logs/Rorze/SorterRSC/RBWSA22200/mq-sRBWSA22200.log.tu

file position 24332126
file size 26533134
parent /var/mtapps/ashl/logs////mq-
percent 91.70
type open file

0 Karma

dreamfeeder
New Member

Hi Dineshraj9,

I want to monitor them under 3 different source type. is it possible to do that?

0 Karma

dineshraj9
Builder

Can you try having a single monitor for all -

[monitor:///var/mtapps/ashl/logs/.../.../.../(mq|secs|err)-*)]

Splunk should have read the events as and when the logs are written. Try checking internal logs if they print any errors and adding crcSalt(as shown above).

0 Karma

dreamfeeder
New Member

yes. this is how we setup, but it doesnt work. you can see my screenshot below.
alt text

0 Karma

dineshraj9
Builder

Can you check if there is any permission issue for these logs. Since splunk is unable to access them.

index=_internal sourcetype=splunkd host=<log_server> "var/mtapps/ashl/logs"

Also try running the command to check if these logs are monitored on the log server -

./splunk list monitor -auth admin:<password>
0 Karma
Get Updates on the Splunk Community!

Aligning Observability Costs with Business Value: Practical Strategies

 Join us for an engaging Tech Talk on Aligning Observability Costs with Business Value: Practical ...

Mastering Data Pipelines: Unlocking Value with Splunk

 In today's AI-driven world, organizations must balance the challenges of managing the explosion of data with ...

Splunk Up Your Game: Why It's Time to Embrace Python 3.9+ and OpenSSL 3.0

Did you know that for Splunk Enterprise 9.4, Python 3.9 is the default interpreter? This shift is not just a ...