topic Re: regex to avoid the 2016-MM-DD events in a splunk search? in Splunk Search

regex to avoid the 2016-MM-DD events in a splunk search?

pavanae — Wed, 31 Aug 2016 14:04:08 GMT

The following were some of the events

html tags 2016-04-21 09:42:38,574 DEBUG lksjfd laskdfj lskfj alsdkfj

htmltags2016-05-31T13:50:41.883450Z jhgsd kkjahdf klasjh

htmltags2016-06-11T13:50:41.883450Z kdf ouier lsijcf lkhefr

[ 2016-04-21 09:42:38,574]

abc.def.net 2016-05-31T13

Now, I am trying to write a query which can display the events which shouldn't contain 2016-MM-DD format which is YYYY-MM-DD format

For example it shouldn't display any one of the above mentioned events since they all contains the YYYY-MM-DD format

Note :-
YY- YEAR
MM-MONTH
DD-DATE

Any ideas

Re: regex to avoid the 2016-MM-DD events in a splunk search?

inventsekar — Wed, 31 Aug 2016 14:13:07 GMT

your search | regex _raw!="\d{4}-\d{2}-\d{2}"

will "retain" all YYYY-MM-DD format events and results will be all events other than YYYY-MM-DD format events.

Re: regex to avoid the 2016-MM-DD events in a splunk search?

somesoni2 — Wed, 31 Aug 2016 14:14:50 GMT

Give this a try
Updated to show host/source

your base search | regex _raw!=".*2016-\d{2}-\d{2}.*" | stats count by host, source

your base search "*2016-*" | stats count by host, source

Re: regex to avoid the 2016-MM-DD events in a splunk search?

sundareshr — Wed, 31 Aug 2016 14:18:37 GMT

Try this

your base search | regex _raw!="\d{4}-\d\d-\d\d"

Re: regex to avoid the 2016-MM-DD events in a splunk search?

pavanae — Wed, 31 Aug 2016 14:20:13 GMT

and how to list out the hosts and sources that satisfies my search?

Re: regex to avoid the 2016-MM-DD events in a splunk search?

inventsekar — Wed, 31 Aug 2016 14:21:57 GMT

to list out the hosts and sources that satisfies my search -

your search | regex _raw!="\d{4}-\d{2}-\d{2}" | table host source

Re: regex to avoid the 2016-MM-DD events in a splunk search?

tin_fish — Wed, 31 Aug 2016 14:41:35 GMT

Hi Pavanae. I agree with the other answers here, although it's not been made clear whether or not the specific format of your timestamp needs to be part of the exclusion - i.e. should the exclusion encapsulate both YYYY-MM-DD and YYYY-DD-MM?

Also as a general rule - and if possible - it's better to know what you're looking for - rather than what you're not. 🙂

Re: regex to avoid the 2016-MM-DD events in a splunk search?

pavanae — Wed, 31 Aug 2016 14:44:58 GMT

what if we want to display only the windows events. is there any unique search stanza to display only the windows hosts and filtr out the other os's?

Re: regex to avoid the 2016-MM-DD events in a splunk search?

somesoni2 — Wed, 31 Aug 2016 14:57:10 GMT

Since both the timestamp and OS info for the hosts are written in totally different logs, you can't achieve this using same base search. My suggestion would be to create a lookup table with all host and their corresponding OS and use that lookup to filter out hosts from above query.

Query to generate lookup

index=_internal source=*metrics.log os=* earliest=-1h@h  | stats latest(os) as os by host | outputlookup host_os.csv

Use lookup to exclude hosts

your base search [| inputlookup host_os.csv | where os="Windows" | table host ] | regex _raw!=".*2016-\d{2}-\d{2}.*" | stats count by host, source

Re: regex to avoid the 2016-MM-DD events in a splunk search?

pavanae — Wed, 31 Aug 2016 19:07:27 GMT

So without using the lookup's can I below the below search result was accurate?

index=* [ search index=_internal os=Windows sourcetype=splunkd | stats count by hostname | rename hostname as host | fields host ] | regex _raw!=".2016-\d{2}-\d{2}." | regex _raw!=".2016/\d{2}/\d{2}." | stats values(source) as sources values(sourcetype) as sourcetypes by host

Re: regex to avoid the 2016-MM-DD events in a splunk search?

somesoni2 — Wed, 31 Aug 2016 21:21:06 GMT

Yes it will be. Lookups will be good for performance as you don't have to go through internal logs every time.