Large eventtypes reccomendations

[sonicZ]

I am in the process of making individual event types for about 175 types of log events from routers/firewall devices.

can we include regex's in the event type search syntax? Some sample log events that would each be their own eventType

.*list [0-9]+ denied
SEC-6-IPACCESSLOG
ras\d+.*Connected to \S+ port \d+
ras\d+.*Closed connection to \S+ port \d+
\d+\.\d+\.\d+\.\d+\s+.*?(?:start|stop).*?service=shell

Finally when i get all the 170 event types created i'll include them into a single event type which will include all of them in the search

sourcetype=syslog index=net eventtype=someEventType1 AND eventtype=someEventType2 AND eventtype=someEventType3 

etc etc...Will this be pretty inefficient if i go this route?
Any alternate ways i should do this would be appreciated.

[sonicZ]

Hey Gerald, how ya been??
I'll give macro's a try, I had to read up on them as not really used them before.
It seems like this will work, after testing the search with the following syntax, using a macro definition as so

regex _raw!="(\[sysd\.NOTICE\]\: WDOG\: Strobing the watchdog)|(\[sysd\.NOTICE\]\: WDOG\: Uptime" "System memory)|(ACE-4-313004\: Denied ICMP)|(.*list [a-zA-Z0-9\-]+ denied)|(ras\d+.*Connected to \S+ port \d+)" 

and using a search string to call the macro should work

index=net sourcetype=syslog source="/logs/syslog/central_netlog_repository.log" | `net_macro`

I plan to maintain the regex's to filter out certain events going forward.

Basically i am trying to replace a python script that filtered out log events with all these matching regex's, from our "net" index. This was a previously asked question.
http://splunk-base.splunk.com/answers/49995/looking-to-replace-a-custom-python-syslog-filter

[gkanapathy · Accepted solution]

No you can't use regexes like that, only base searches.

It's a bad idea to use eventtypes if you're just using them to search for things. Eventtypes are what you'd use to classify events, and are good at doing that. But every eventtype you have causes search to run slower, even if you never reference or use the eventtype in your data that you're querying. If you're just using eventtypes for searching, i.e., as an abbreviation or abstraction, you should instead use macros.

Macros can include regexes via regex commands (or any search commands), but you can not combine macros containing such commands easily with other terms.

[sonicZ]

Also i read Lowell's answer and specified sourcetype and index in the log as well as putting all new eventtypes created in a specific application, hopefully this helps performance.

http://splunk-base.splunk.com/answers/4217/performance-considerations-for-eventtypes