Splunk Search

Why is the regex in inputs.conf not working for monitoring my log files?

robertlynch2020
Motivator

Hi

I have the following file in multiple sub directories. I am trying to pick them up but the below is not working and i can't crack it.
The regex is good, but it just won't take them it... any help would be super... I am thinking something very small is wrong here.

-rw-rw-r-- 1 autoengine murex 4772 Apr 6 17:24 mxtiming_730010_dell427srv_121.log
-rw-rw-r-- 1 autoengine murex 4772 Apr 6 17:26 mxtiming_730018_dell427srv_504.log
-rw-rw-r-- 1 autoengine murex 4772 Apr 6 17:27 mxtiming_730022_dell427srv_531.log

[monitor:///net/dell427srv//data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/.../*.log]
disabled = false
host = RSAT_Campaign
index = mlc_live
whitelist = mxtiming_\d+_\w+_\d+.*\.log$
sourcetype = MX_TIMING
0 Karma
1 Solution

gcusello
Esteemed Legend

Hi,
Can you share sn example of not wanted files?
At a first sight you could insert part of filename in monitor
[monitor://net/dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/.../mxtiming_*.log]
Beware that there is a double slash in tour path.
Every way you could change your regex in this way
mxtiming_[^]+[^]+[^.]+.log
Bye.
Giuseppe

View solution in original post

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Can you please give the full path to a few of these? Feel free to alter directory names as needed for confidentiality, but we need to see the way the full path looks in order to check some things.

0 Karma

robertlynch2020
Motivator

Thanks for your help on this , I think we are close.
To explain the issues more.

Wanted Files
Any sub directory of the main

[dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART]

 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_730010_dell427srv_121.log

 Or 
 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/X/mxtiming_730018_dell427srv_504.log    
 Or.
 Etc..

 mxtiming_730010_dell427srv_121.log
 mxtiming_730018_dell427srv_504.log
 mxtiming_730022_dell427srv_531.log

Unwanted Files
The files I don’t want are below and again they can also come into any subdirectory

 mxtiming_commoditynearbyonindexhistoricaldata_20170306-093752167_1294331273_6902.log
 mxtiming_commoditynearbyonindextofutures_20170306-093752167_1718781102_6902.log
 mxtiming_datamartdatasetlabelstransfer_20170306-093752167_1714912538_6902.log
 mxtiming_fillcommoditytimeunits_20170306-093752167_1066971732_6902.log
 mxtiming_obsolete_typology_cleanup_in_stp_rights_20170306-093752167_1206801397_6902.log
 mxtiming_new_stp_rights_migration_20170306-093752167_252516786_6902.log
 mxtiming_adapt_warehouse_rebuild_20170306-093752167_1385637444_6902.log
 mxtiming_updatepricingbookingpretraderouters_20170306-093752167_904493553_6902.log
 mxtiming_collateralinterestopsobjectupgrade_20170306-093752167_1527129704_6902.log
 mxtiming_refreshaccountssidata_20170306-093752167_421251909_6902.log

 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_adaptposnbstoredpltables_20170306-093752167_44364646_6902.log
 Or 
 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/Z/A/mxtiming_removecommodityfuturesindexplinstruments_20170306-093752167_222279393_6902.log
 OR .
0 Karma

gcusello
Esteemed Legend

Hi,
Can you share sn example of not wanted files?
At a first sight you could insert part of filename in monitor
[monitor://net/dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/.../mxtiming_*.log]
Beware that there is a double slash in tour path.
Every way you could change your regex in this way
mxtiming_[^]+[^]+[^.]+.log
Bye.
Giuseppe

0 Karma

robertlynch2020
Motivator

Hi

I have files like below that i don't want to take in.
mxtiming_adaptposnbstoredpltables_20170306-093752167_44364646_6902.log
mxtiming_removecommodityfuturesindexplinstruments_20170306-093752167_222279393_6902.log

Cheers for you help

0 Karma

robertlynch2020
Motivator

Hi No Luck, i tried this but nothing come out

[monitor://net/dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/.../*.log]
disabled = false
host = RSAT_Campaign
index = mlc_live
whitelist = mxtiming_[^]+[^]+[^.]+.log$  
sourcetype = MX_TIMING
0 Karma

gcusello
Esteemed Legend

Hi robertlynch2020,
sorry but I answered using my smartphone that has limited function keyboards!
regex isn't correct, try:

mxtiming_[^_]+_[^_]+_[^\.]+\.log$

Bye.
Giuseppe

robertlynch2020
Motivator

Hi - This worked thanks 🙂

0 Karma

robertlynch2020
Motivator

Thanks for your help on this , I think we are close.
To explain the issues more.

Wanted Files
Any sub directory of the main

[dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART]

dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_730010_dell427srv_121.log

Or 
dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/X/mxtiming_730018_dell427srv_504.log  
Or.
Etc..

mxtiming_730010_dell427srv_121.log
mxtiming_730018_dell427srv_504.log
mxtiming_730022_dell427srv_531.log

Unwanted Files
The files I don’t want are below and again they can also come into any subdirectory

mxtiming_commoditynearbyonindexhistoricaldata_20170306-093752167_1294331273_6902.log
mxtiming_commoditynearbyonindextofutures_20170306-093752167_1718781102_6902.log
mxtiming_datamartdatasetlabelstransfer_20170306-093752167_1714912538_6902.log
mxtiming_fillcommoditytimeunits_20170306-093752167_1066971732_6902.log
mxtiming_obsolete_typology_cleanup_in_stp_rights_20170306-093752167_1206801397_6902.log
mxtiming_new_stp_rights_migration_20170306-093752167_252516786_6902.log
mxtiming_adapt_warehouse_rebuild_20170306-093752167_1385637444_6902.log
mxtiming_updatepricingbookingpretraderouters_20170306-093752167_904493553_6902.log
mxtiming_collateralinterestopsobjectupgrade_20170306-093752167_1527129704_6902.log
mxtiming_refreshaccountssidata_20170306-093752167_421251909_6902.log

dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_adaptposnbstoredpltables_20170306-093752167_44364646_6902.log
Or 
dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/Z/A/mxtiming_removecommodityfuturesindexplinstruments_20170306-093752167_222279393_6902.log
OR .

…etc..

0 Karma

gcusello
Esteemed Legend

Hi robertlynch2020,,
try this one, it should be correct (see https://regex101.com/r/8Mzm3g/1)

mxtiming_(?<ppp>\d*_[^_]*_\d*)\.log

Bye.
Giuseppe

robertlynch2020
Motivator

Hi

So it looks like i get this to work, with the regex - thanks.
whitelist = mxtiming_(?\d*[^]_\d).log

0 Karma

woodcock
Esteemed Legend

Try this change:

 [monitor:///net/dell427srv//data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/*/]

This assumes only a single directory layer between QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART and your files. Leave everything else the same. Restart your forwarder's splunk instance.

0 Karma

robertlynch2020
Motivator

Hi, Thanks, but these files could be multiple levels down from the main directory

0 Karma

robertlynch2020
Motivator

Hi - I tried this, but it only give me the subdirectories and not the main directory.

So it looks like i need to lines in my inputs.conf to get the main directory and all its subdirectories

[monitor:///net/dell425srv/dell425srv/apps/SPLUNK_BACK_UP_LIVE/MXTIMING_MEDIUM5/*.log]
disabled = false
recursive = true
host = MXTIMING_LIVE_TEST5
index = mlc_live
whitelist = mxtiming_(?<ppp>\d*_[^_]*_\d*)\.log
crcSalt = <SOURCE>
sourcetype = MX_TIMING

[monitor:///net/dell425srv/dell425srv/apps/SPLUNK_BACK_UP_LIVE/MXTIMING_MEDIUM5/.../*.log]
disabled = false
recursive = true
host = MXTIMING_LIVE_TEST5
index = mlc_live
whitelist = mxtiming_(?<ppp>\d*_[^_]*_\d*)\.log
crcSalt = <SOURCE>
sourcetype = MX_TIMING
0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...