How to change a search into a marco filter?

[cpeteman]

This is a follow up to Background exclusion The question I have now is no longer on topic with the original posted hence the following.

The title is fairly descriptive I have a search that gives logs which are regularly occuring events. In order to be able to better look and the data logs that indicat problems I want to throw out these results in future searches. In my last question it seems best to use a marco to exclude the results that the original search (below) currently gives.

search terms | eval TimeInHour=_time%3600 
    | rex mode=sed "s/ \d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}//g" 
    | stats first(_raw) by punct,TimeInHour,_raw,_time 
    |  stats count by _raw,TimeInHour,punct 
    |  addinfo| eval hours = round((info_max_time - info_min_time)/3600,0) 
    | where count > hours-1

How do I turn this in to one big exclusion/negation statement that I can then make into a macro without any subsearch garbage?

TimeInHour  punct       count hours _raw
    0   <>__[]___--_::_.    4   4   <ConMan> Console [ansel0] log at PDT.
    0   <>__[]___--_::_.    4   4   <ConMan> Console [ansel100] log at PDT.
    0   <>__[]___--_::_.    4   4   <ConMan> Console [ansel101] log at PDT.
    0   <>__[]___--_::_.    4   4   <ConMan> Console [ansel102] log at PDT.
    0   <>__[]___--_::_.    4   4   <ConMan> Console [ansel103] log at PDT.

Above are sample logs my search gives me. Over a search of 4 hours each event occured at the time time in each hour in this case on the hour hence TimeInHour is zero.

[lguinn2]

You are asking Splunk to do a lot of work, just so you know. I think you should ask Splunk to work hard for you, but I want you to realize that this search may be quite slow.

I suggest an approach similar to what some security apps use: tag the "normal" recurring events. Once tagged, they can be excluded from searches quite easily. (Easily for you, lots of work for Splunk.)

Here's how: First, for each routine event, create an eventtype. This may not be simple, because eventtypes cannot have commands (ie, no | ). You may need to create some field extractions in order to create the eventtypes you want.

Second, tag all the eventtypes with a common tag, like "routine".

Third, here is your basic search

NOT tag=routine

Caveat: If you create a large number of eventtypes, you may cause an impact on other searches. How? Because eventtypes are automatically evaluated as part of the search when you are in verbose mode. Basically, if you see the fields sidebar - Splunk has to populate it with the eventtypes. Normally, this isn't an issue. If this becomes a problem, there is a way to deal with it, though: create an app, and make all the field extractions and eventtypes private to that app. Then go to your app when you want to run this particular search.

Even if this approach doesn't work perfectly, you can probably get rid of a lot of the routine eventtypes this way. Then it might be realistic to use a subsearch to finish the job.

[rsennett_splunk]

I wonder if, since you are selecting specific events and then wanting to throw them out before you search... if you are really looking for event type or a series or set of event types that you negate prior to your search. Without an example of events, as Sean said, it's hard to see what you're trying to accomplish exactly. If you haven't used eventtypes - make a simple one with the gui and negate it in a search as a filter... I think it might spark some inspiration because you'll need to re-think your approach. You can then do a more complex structure in the eventtypes.conf file.

[sdaniels]

Hard to understand what you are looking for here. In your original question I thought you were just looking to simplify your search and abstract away all of the complexity. It might help to give an example with the data/events that you are using.

[cpeteman]

I'm afraid it's not that simple. I want to turn it into a negating statement if the conditions in the search are meet then those events are excluded from the search. Sorry for the lack of clarity I'll add more.

[linu1988]

Do you mean that you will pass the whole search into macro as an argument?

If yes, just create a macro and see if it works. It will only replace the whole thing in the original search.Where do you face the problem?