topic Re: Why is the regex in inputs.conf not working for monitoring my log files? in Splunk Search

Why is the regex in inputs.conf not working for monitoring my log files?

robertlynch2020 — Tue, 29 Sep 2020 13:33:57 GMT

I have the following file in multiple sub directories. I am trying to pick them up but the below is not working and i can't crack it.
The regex is good, but it just won't take them it... any help would be super... I am thinking something very small is wrong here.

-rw-rw-r-- 1 autoengine murex 4772 Apr 6 17:24 mxtiming_730010_dell427srv_121.log
-rw-rw-r-- 1 autoengine murex 4772 Apr 6 17:26 mxtiming_730018_dell427srv_504.log
-rw-rw-r-- 1 autoengine murex 4772 Apr 6 17:27 mxtiming_730022_dell427srv_531.log

[monitor:///net/dell427srv//data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/.../*.log]
disabled = false
host = RSAT_Campaign
index = mlc_live
whitelist = mxtiming_\d+_\w+_\d+.*\.log$
sourcetype = MX_TIMING

Re: Why is the regex in inputs.conf not working for monitoring my log files?

woodcock — Thu, 06 Apr 2017 16:29:18 GMT

Try this change:

 [monitor:///net/dell427srv//data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/*/]

This assumes only a single directory layer between QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART and your files. Leave everything else the same. Restart your forwarder's splunk instance.

Re: Why is the regex in inputs.conf not working for monitoring my log files?

gcusello — Tue, 29 Sep 2020 13:34:03 GMT

Hi,
Can you share sn example of not wanted files?
At a first sight you could insert part of filename in monitor
[monitor://net/dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/.../mxtiming_*.log]
Beware that there is a double slash in tour path.
Every way you could change your regex in this way
mxtiming_[^]+[^]+[^.]+.log
Bye.
Giuseppe

Re: Why is the regex in inputs.conf not working for monitoring my log files?

robertlynch2020 — Tue, 29 Sep 2020 13:34:06 GMT

I have files like below that i don't want to take in.
mxtiming_adaptposnbstoredpltables_20170306-093752167_44364646_6902.log
mxtiming_removecommodityfuturesindexplinstruments_20170306-093752167_222279393_6902.log

Cheers for you help

Re: Why is the regex in inputs.conf not working for monitoring my log files?

robertlynch2020 — Thu, 06 Apr 2017 16:52:48 GMT

Hi, Thanks, but these files could be multiple levels down from the main directory

Re: Why is the regex in inputs.conf not working for monitoring my log files?

robertlynch2020 — Thu, 06 Apr 2017 17:18:18 GMT

Hi No Luck, i tried this but nothing come out

[monitor://net/dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/.../*.log]
disabled = false
host = RSAT_Campaign
index = mlc_live
whitelist = mxtiming_[^]+[^]+[^.]+.log$  
sourcetype = MX_TIMING

Re: Why is the regex in inputs.conf not working for monitoring my log files?

DalJeanis — Thu, 06 Apr 2017 19:48:12 GMT

Can you please give the full path to a few of these? Feel free to alter directory names as needed for confidentiality, but we need to see the way the full path looks in order to check some things.

Re: Why is the regex in inputs.conf not working for monitoring my log files?

gcusello — Fri, 07 Apr 2017 07:58:49 GMT

Hi robertlynch2020,
sorry but I answered using my smartphone that has limited function keyboards!
regex isn't correct, try:

mxtiming_[^_]+_[^_]+_[^\.]+\.log$

Bye.
Giuseppe

Re: Why is the regex in inputs.conf not working for monitoring my log files?

robertlynch2020 — Fri, 07 Apr 2017 09:00:15 GMT

Thanks for your help on this , I think we are close.
To explain the issues more.

Wanted Files
Any sub directory of the main

[dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART]

dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_730010_dell427srv_121.log

Or 
dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/X/mxtiming_730018_dell427srv_504.log  
Or.
Etc..

mxtiming_730010_dell427srv_121.log
mxtiming_730018_dell427srv_504.log
mxtiming_730022_dell427srv_531.log

Unwanted Files
The files I don’t want are below and again they can also come into any subdirectory

mxtiming_commoditynearbyonindexhistoricaldata_20170306-093752167_1294331273_6902.log
mxtiming_commoditynearbyonindextofutures_20170306-093752167_1718781102_6902.log
mxtiming_datamartdatasetlabelstransfer_20170306-093752167_1714912538_6902.log
mxtiming_fillcommoditytimeunits_20170306-093752167_1066971732_6902.log
mxtiming_obsolete_typology_cleanup_in_stp_rights_20170306-093752167_1206801397_6902.log
mxtiming_new_stp_rights_migration_20170306-093752167_252516786_6902.log
mxtiming_adapt_warehouse_rebuild_20170306-093752167_1385637444_6902.log
mxtiming_updatepricingbookingpretraderouters_20170306-093752167_904493553_6902.log
mxtiming_collateralinterestopsobjectupgrade_20170306-093752167_1527129704_6902.log
mxtiming_refreshaccountssidata_20170306-093752167_421251909_6902.log

dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_adaptposnbstoredpltables_20170306-093752167_44364646_6902.log
Or 
dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/Z/A/mxtiming_removecommodityfuturesindexplinstruments_20170306-093752167_222279393_6902.log
OR .

…etc..

Re: Why is the regex in inputs.conf not working for monitoring my log files?

gcusello — Fri, 07 Apr 2017 09:15:46 GMT

Hi robertlynch2020,,
try this one, it should be correct (see https://regex101.com/r/8Mzm3g/1)

mxtiming_(?<ppp>\d*_[^_]*_\d*)\.log

Bye.
Giuseppe

Re: Why is the regex in inputs.conf not working for monitoring my log files?

robertlynch2020 — Fri, 07 Apr 2017 09:30:10 GMT

Thanks for your help on this , I think we are close.
To explain the issues more.

Wanted Files
Any sub directory of the main

[dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART]

 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_730010_dell427srv_121.log

 Or 
 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/X/mxtiming_730018_dell427srv_504.log    
 Or.
 Etc..

 mxtiming_730010_dell427srv_121.log
 mxtiming_730018_dell427srv_504.log
 mxtiming_730022_dell427srv_531.log

Unwanted Files
The files I don’t want are below and again they can also come into any subdirectory

 mxtiming_commoditynearbyonindexhistoricaldata_20170306-093752167_1294331273_6902.log
 mxtiming_commoditynearbyonindextofutures_20170306-093752167_1718781102_6902.log
 mxtiming_datamartdatasetlabelstransfer_20170306-093752167_1714912538_6902.log
 mxtiming_fillcommoditytimeunits_20170306-093752167_1066971732_6902.log
 mxtiming_obsolete_typology_cleanup_in_stp_rights_20170306-093752167_1206801397_6902.log
 mxtiming_new_stp_rights_migration_20170306-093752167_252516786_6902.log
 mxtiming_adapt_warehouse_rebuild_20170306-093752167_1385637444_6902.log
 mxtiming_updatepricingbookingpretraderouters_20170306-093752167_904493553_6902.log
 mxtiming_collateralinterestopsobjectupgrade_20170306-093752167_1527129704_6902.log
 mxtiming_refreshaccountssidata_20170306-093752167_421251909_6902.log

 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_adaptposnbstoredpltables_20170306-093752167_44364646_6902.log
 Or 
 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/Z/A/mxtiming_removecommodityfuturesindexplinstruments_20170306-093752167_222279393_6902.log
 OR .

Re: Why is the regex in inputs.conf not working for monitoring my log files?

robertlynch2020 — Tue, 29 Sep 2020 13:34:55 GMT

So it looks like i get this to work, with the regex - thanks.
whitelist = mxtiming_(?\d*[^]_\d).log

Re: Why is the regex in inputs.conf not working for monitoring my log files?

robertlynch2020 — Fri, 07 Apr 2017 11:23:29 GMT

Hi - This worked thanks 🙂

Re: Why is the regex in inputs.conf not working for monitoring my log files?

robertlynch2020 — Fri, 07 Apr 2017 11:27:12 GMT

Hi - I tried this, but it only give me the subdirectories and not the main directory.

So it looks like i need to lines in my inputs.conf to get the main directory and all its subdirectories

[monitor:///net/dell425srv/dell425srv/apps/SPLUNK_BACK_UP_LIVE/MXTIMING_MEDIUM5/*.log]
disabled = false
recursive = true
host = MXTIMING_LIVE_TEST5
index = mlc_live
whitelist = mxtiming_(?<ppp>\d*_[^_]*_\d*)\.log
crcSalt = <SOURCE>
sourcetype = MX_TIMING

[monitor:///net/dell425srv/dell425srv/apps/SPLUNK_BACK_UP_LIVE/MXTIMING_MEDIUM5/.../*.log]
disabled = false
recursive = true
host = MXTIMING_LIVE_TEST5
index = mlc_live
whitelist = mxtiming_(?<ppp>\d*_[^_]*_\d*)\.log
crcSalt = <SOURCE>
sourcetype = MX_TIMING