Getting Data In

How to get all the log files not mentioned in inputs file from the same directory?

NanSplk01
Communicator

This is the inputs.file:: As you can see they all go to the same directory structure, but the last one is supposed to catch all the logs not beginning with the defined *_xxxxx_*.log so that general logs will be stored in Splunk as well.  How can I do this?

[monitor:///var/log/containers/*_ctisp1_*.log]
index = ctisp1
sourcetype = dks-ctisp1
followSymlink = true

[monitor:///var/log/containers/*_ocpprd_*.log]
index = ocpprd
sourcetype = dks-ocpprd
followSymlink = true

[monitor:///var/log/containers/*_custconnectp1_*.log]
index = custcontp1
sourcetype = custcontp1
followSymlink = true

[monitor:///var/log/containers/*_ocpnotifp3_*.log]
index = ocpnotifp3
sourcetype = dks-ocpnotifp3

[monitor:///var/log/containers/*_ocpcorep3_*.log]
index = ocpcorep3
sourcetype = ocpcorep3

[monitor:///var/log/containers/*_custcon2p3_*.log]
index = custcon2p3
sourcetype = custcon2p3

[monitor:///var/log/containers/*_custcon1p3_*.log]
index = custcon1p3
sourcetype=custcont1p3

[monitor:///var/log/containers/*_ctisap3_*.log]
index = ctisap3
sourcetype = dks-ctisap3

[monitor:///var/log/containers/*_ctisp1_*.log]
index = ctisp1
sourcetype = dks-ctisp1

[monitor:///var/log/containers/*_ivrp1_*.log]
index = ivrp1
sourcetype = dks-ivrp1

#[monitor:///host/containers/*/[a-f0-9]+-json.log$]
#index=dcp
#sourcetype=dner-logsiamanti-container-logs

#[monitor:///var/lib/docker/containers/*/[a-f0-9]+-json.log$]
#index=dcp
#sourcetype=diamanti-container-logs
[monitor:///var/log/containers/*_ocpnotifp3_*.log]
index = ocpnotifp3
sourcetype = ocpnotifp3

[monitor:///var/log/containers/*_ocpcorep3_*.log]
index = ocpcorep3
sourcetype = ocpcorep3

[monitor:///var/log/containers/*_custcon2p3_*.log]
index = custcon2p3
sourcetype = custcont2p3

[monitor:///var/log/containers/*_igridp2_*.log]
index = igridp2
sourcetype = dks-igridp2

## END of PROD

## Monitor all Diamanti logs
[monitor:///var/log/diamanti/.../*.log]
index=dcp
sourcetype = diamanti-system-logs

# Monitor Container logs
[monitor:///var/log/containers/*.log]
index=dcp
sourcetype = diamanti-container-logs

Labels (3)
0 Karma

smurf
Communicator

Hi,

you can add a blacklist. I am not sure if all your files could be easily matched with regex, but it is a possibility.

blacklist = <regular expression>
* If set, files from this input are NOT monitored if their path matches the
  specified regex.
* Takes precedence over the deprecated '_blacklist' setting, which functions
  the same way.
* If a file matches the regexes in both the deny list and allow list settings,
  the file is NOT monitored. Deny lists take precedence over allow lists.
* No default.

 

0 Karma

NanSplk01
Communicator

I will give this a try and see if I can use the blacklist

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @NanSplk01,

the problem is probably that there's an overlapping of inputs and usually Splunk index a log only one time.

I hint to blacklist "*_xxxxx_*.log" in the last two stanzas, in other words:

## Monitor all Diamanti logs
[monitor:///var/log/diamanti/.../*.log]
index=dcp
sourcetype = diamanti-system-logs
blacklist = *_xxxxx_*.log

# Monitor Container logs
[monitor:///var/log/containers/*.log]
index=dcp
sourcetype = diamanti-container-logs
blacklist = *_xxxxx_*.log

for more infos, you can see at https://docs.splunk.com/Documentation/Splunk/8.2.6/Admin/Inputsconf

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...