topic Option to extract one specific field from different patterns in one go ? in Splunk Search

Option to extract one specific field from different patterns in one go ?

pari04home — Wed, 08 Nov 2017 03:21:08 GMT

For the same sourcetype, I have a lot many different patterns from which I want to extract one specific field. Is there an option to handle all these extractions (same sourcetype different patterns) in one go instead of going via multiple regex field extractions ?

Re: Option to extract one specific field from different patterns in one go ?

MonkeyK — Tue, 29 Sep 2020 16:38:54 GMT

regex supports an "OR" operator by using a pipe character "|" between matching expressions.

For example (comes with Splunk)
(?:objectguid|guid)\s*=\s*(?[\w-]+)

Re: Option to extract one specific field from different patterns in one go ?

yuanliu — Wed, 08 Nov 2017 05:48:38 GMT

Without clarity in data constraints, like what does it mean by "different patterns" and "one specific field", it is hard to give any definitive answer. Are you looking for the same type of data embedded in different contexts, or are you looking for a potentially different data type when context is different? Regex is extremely versatile when matching patterns if you are willing to invest in it. It all depends on the actual data, is there an order of appearance for patterns, how regular the rest of the patterns are, and so on. In my experience, there are advantages and disadvantages for one-go vs multiple rex commands. If you know your data well, it is nearly always possible to construct a single rex command to extract every pattern if you work hard enough. But if you want to capture just those combinations that you haven't thought of, multiple runs are easier.

One whacky example is this pair of constructs:

William is my nephew
My auntie is Betty
How to extract the relationship in one go? You can use ^((?<name>.+) (is|are) (my|our|your|his|her|their) (?<relation>.+)|(My|Our|Your|His|Her|Their) (?<relation>.+) (is|are) (my|our|your|his|her|their) (?<name>.+)). Is this doable? Perhaps yes. Is this practical? Probably not.

Machine data are probably better than natural language. But if you are parsing application logs, more often than not, different pieces of a pattern may appear in some context and not in others. Nowadays I just use multiple passes unless I have confidence that the data space is limited.

Re: Option to extract one specific field from different patterns in one go ?

niketn — Wed, 08 Nov 2017 06:45:15 GMT

@pari04home, as you would understand, if you have multiple patterns within your sourcetype you would need to provide us with mock/sample data for all possible patterns for us to assist you with your regular expression.