topic Re: How to use regex in eventtypes.conf in Splunk Search

How to use regex in eventtypes.conf

lakromani — Mon, 09 Nov 2015 12:51:31 GMT

I have data in following formats:

Nov 04 21:47:59 server1 gtu[22038]: 2833CA0D c   (master)   1A 0B 81 2D 5F 66 36 A7 DC F3 60 B0 
Nov 04 21:47:59 server1 gtu[22038]: 2833CA0D c   (master)   02 6D A0 3C B1 B3 59 CD EC BC CB 7B 55 65 85 CA 
Nov 04 21:47:59 server1 gtu[22038]: 2833CA0D c   (master)   82 70 29 01 02 06 02 BE 04 A5 FB 6C 1F 90 1D 40 
Nov 04 21:47:58 server1 gtu[22038]: 2833CA0D c   (master)   7E A0 51 E5 B2 CA

I need to set this as one eventtype.
Number of data field can go from 2 to 16.
With normal search, I can use this format:

* | regex _raw="gtu.* \(master\)\s+\w\w\s+\w\w"

But in eventypes.conf this does not work.

[gtu-master-data]
search = regex _raw="gtu.* \(master\)\s+\w\w\s+\w\w"

Does regex not work in *eventypes.conf

Re: How to use regex in eventtypes.conf

clorne — Mon, 09 Nov 2015 13:51:13 GMT

Hello,
There is some examples in this post that may help you:
https://answers.splunk.com/answers/293531/how-to-write-the-regex-for-transformsconf-to-extra.html

Regards

Re: How to use regex in eventtypes.conf

alacercogitatus — Mon, 09 Nov 2015 14:59:20 GMT

Pipes, and other non-streaming commands are not allowed in event types. The search definition must contain only the basics of a simple search - no pipes, no transactions, not regex, nothing other than the "base search" that will match your event type.

So your eventtype in this case will be the following, since this will only include things that match the search definition.

eventtypes.conf

[gtu-master-data]
 search = gtu master master_hex=*

props.conf

 [your_sourcetype]
 EXTRACT-hex_values = gtu\[(?<process_pid>[^\]]+)\]:[^\(]+\(master\)\s+(?<master_hex>(?:[a-fA-F0-9\s]{2})+)

UPDATE: so this should work, but I haven't tested it. Basically, in the props.conf you specify the Extraction for "Master Hex" values, (call it what you want), that matches the formation of hex values. Then you use the event type to limit those events that have any value of master hex.

http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Abouteventtypes
Here's the restriction documentation (from below comment) http://docs.splunk.com/Documentation/Splunk/6.3.1/Knowledge/defineeventtypes#Important_event_type_definition_restrictions

Re: How to use regex in eventtypes.conf

woodcock — Mon, 09 Nov 2015 15:39:01 GMT

@alacercogitatus; I did read that link before I posted my comment and you will note that the documentation does not mention this limitation (which I was pretty sure existed, since that was always how I wrote them). The documentation definitely needs an update to call this out.

Re: How to use regex in eventtypes.conf

lakromani — Mon, 09 Nov 2015 16:36:54 GMT

Problem is that doing this will hit lots of other stuff, that is already to tagged.
So I need to differentiate this data from other stuff.

Re: How to use regex in eventtypes.conf

alacercogitatus — Mon, 09 Nov 2015 16:44:06 GMT

The documentation in the Knowledge Manager Manual does have the restrictions in place. I agree, the spec should be updated to include the specific line.

 You cannot base an event type on a search that includes a pipe operator or a subsearch .

  In addition, you cannot base an event type on a search that references a report. For example, if you have a report with the name failed_login_search, you can't create an event type that is defined by savedsearch=failed_login_search. In a case like this you should always give the event type the same search string as the report.

http://docs.splunk.com/Documentation/Splunk/6.3.1/Knowledge/defineeventtypes#Important_event_type_definition_restrictions

Re: How to use regex in eventtypes.conf

woodcock — Mon, 09 Nov 2015 16:46:21 GMT

You need to create a field-extraction that qualifies/classifies the events (like number_of_bytes) and then create an eventtype based on that (like number_of_bytes = 2).

Re: How to use regex in eventtypes.conf

aljohnson_splun — Mon, 09 Nov 2015 19:44:50 GMT

For any search, you can look at the job inspector (Job > Inspect Job) and find the row "canBeEventType" which will be set to either 0 (cannot) or 1 (can).