topic Re: Index time field extraction: regexp issue in Splunk Search

Index time field extraction: regexp issue

Super_Knulps — Mon, 04 May 2015 20:34:11 GMT

Hello,

Since I often search a specific expression in a large set of events, I would like to index it.

Every single instance that I am running has the following format:
instance-name.generic-name.subdomaine.domain.com

In this expression, only domain.com is static and will never change.
I would like to extract generic-name for all of my events.

props.conf

[generic-name]
TRANSFORMS-generic-name = generic-name

transforms.conf

[generic-name]

REGEX = (?&lt;instancename&gt;[^\.]+)\.(?&lt;gname&gt;[^\.]+)\.(?&lt;subdomain&gt;[^\.]+)\.(?&lt;domain&gt;[^\.]+)\.

fields.conf

[gname]
INDEXED = True

I am wondering if the fact that I am not receiving anything in the Splunk dashboard is coming from my configuration file or my regular expression ?
Thank you in advance for your help

Update: I have tried all the following regexp and there is still no result. I don't receive any data in my sourcetype.

Re: Index time field extraction: regexp issue

martin_mueller — Mon, 04 May 2015 21:03:47 GMT

Are you sure you need indexed extractions here?

What happens when you run this search:

index=foo sourcetype=generic-name gname=some-gname

Is the scanCount in the job inspector higher than the resultCount?

Re: Index time field extraction: regexp issue

woodcock — Mon, 04 May 2015 21:20:41 GMT

It is your REGEX; try this one:
(?<instancename>[^/.]+)/.(?<gname>[^/.]+)/.(?<subdomain>[^/.]+)/.(?<domain>[^/.]+)

Re: Index time field extraction: regexp issue

Super_Knulps — Mon, 04 May 2015 21:36:04 GMT

Thank you for your answer. Yes I am pretty sure that I need indexed extractions here since I am running the equivalence of gname=foo on every single search I do. Anyway, I will compare the performance before and after my change.

When I run this:
index=foo sourcetype=generic-name gname=some-gname
I got: No Results Found. Even with sourcetype=generic-name only and gname=some-gname only.

scanCount=0 resultCount=0.

I am wondering if the host is part of the data. Is the host part of the data that I can extract ? Or maybe it is just my regexp.

Re: Index time field extraction: regexp issue

rsennett_splunk — Mon, 04 May 2015 22:13:24 GMT

You are missing some parts in your regex:

YOURS: (capturing group not capturing anything, just naming the field):

[^\.]+\.(?<gname>)[^\.]\.domain\.com

MINE: (capturing group now contains the generic segment):

[^\.]+\.(?<gname>[^\.]+)\.[^\.]+\..+

in case it's not clear... here is the segment zoomed in - note the closing paren, and without the + you get the directive once... not one or more:

yours: (?<gname>)[^\.]

mine:  (?<gname>[^\.]+)

Re: Index time field extraction: regexp issue

martin_mueller — Mon, 04 May 2015 22:16:03 GMT

If you're trying to search on a part of the host you could do this:

index=foo sourcetype=generic-name host=*.some-gname.*

Should be pretty quick in terms of identifying the right events because host already is indexed. Loading the events is a different matter of course, so look at scanCount vs eventCount to check if your search is well-targeted or not.

Re: Index time field extraction: regexp issue

martin_mueller — Mon, 04 May 2015 22:19:20 GMT

If that's good in terms of scanCount vs resultCount and you want to get rid of the ugly host=*.some-gname.* you can do this field extraction:

&lt;some regex&gt; in host

That'll extract your gname from the host field to let you search using gname=some-gname backed by the host field.

Re: Index time field extraction: regexp issue

Super_Knulps — Mon, 04 May 2015 23:20:42 GMT

You are talking about search time extraction while I am asking for index time.

Re: Index time field extraction: regexp issue

Super_Knulps — Mon, 04 May 2015 23:23:35 GMT

Thank you for your answer!
I am still not receiving any result from your search.
Actually I have also tried it on regexpr.com and you are matching everything with your regexp.
Maybe I am missing something but it does not seem to work.

Re: Index time field extraction: regexp issue

Super_Knulps — Tue, 05 May 2015 00:20:06 GMT

Thank you for your answer.
Your regexp looks good and easy to understand but maybe slower due to multiple extraction.
Anyway, I still receive no data when I am trying to use yours. Am I missing something else somewhere ?

Re: Index time field extraction: regexp issue

rsennett_splunk — Tue, 05 May 2015 03:09:19 GMT

See my answer. You were missing an actual extraction. Your capturing group surrounded only the field name... so nothing was being captured. You're also representing only one iteration of "anything that is not a dot" because you were missing the + which says "Everything that is not a dot, until you hit the dot". Whether you grab all the fields, or put literals in the domain and sub domain it doesn't matter as long as you are actually capturing something. As for "Slower" as long as you are moving forward (and not doing lookbacks) speed isn't an issue.

Re: Index time field extraction: regexp issue

rsennett_splunk — Tue, 05 May 2015 03:12:10 GMT

try regex101.com that will show you what you are capturing and what you are not. It also will walk you through the regex. you can see it working click here:

https://regex101.com/r/zH0tS1/1

Re: Index time field extraction: regexp issue

martin_mueller — Tue, 05 May 2015 16:42:57 GMT

I know. I'm questioning whether indexed extractions are the right tool for the job.

Set this in props.conf:

[your_sourcetype]
...
EXTRACT-gname = ^[^.]+\.(?&lt;gname&gt;[^.]+) in host`

See if that works, and see if that selects the correct events (scanCount vs resultCount).

Re: Index time field extraction: regexp issue

woodcock — Tue, 05 May 2015 16:54:11 GMT

You need to swap the frontslashes for backslashes (stinking broken markdown). It does work; I tested it. It is important to include the other portions (but you don't necessarily have to capture them into fields) because otherwise your single capture will be capturing things you do not intend.

Re: Index time field extraction: regexp issue

Super_Knulps — Tue, 05 May 2015 22:10:17 GMT

Okay thank you, both of your regexp woocock and rsenett_splunk are matching what I want, which is perfect.
However, I still don't receive anything in the dashboard. The sourcetype is fine in the license. I have updated my first post with your regex: it is all up to date.

Re: Index time field extraction: regexp issue

woodcock — Tue, 05 May 2015 23:22:13 GMT

Post your dashboard xml.

Re: Index time field extraction: regexp issue

Super_Knulps — Wed, 06 May 2015 16:29:24 GMT

I am just using the search: "sourcetype=generic-name gname=foo", in my Splunk App.

Re: Index time field extraction: regexp issue

woodcock — Wed, 06 May 2015 16:39:45 GMT

This is probably the problem:
http://blogs.splunk.com/2011/10/07/cannot-search-based-on-an-extracted-field/

Try this search instead:
sourcetype=generic-name gname=* | search gname="foo"

Re: Index time field extraction: regexp issue

Super_Knulps — Wed, 06 May 2015 17:18:03 GMT

My problem is different: in the link you gave me, sourcetype=generic-name gname=* should give results which is not my case. I litteraly get nothing.

Re: Index time field extraction: regexp issue

rsennett_splunk — Wed, 06 May 2015 18:32:29 GMT

I think there is some confusion about exactly what your problem is.
Your question says... you often search for an expression like:
instance-name.generic-name.subdomain.domain.com
I think some folks here have assumed that this is found the host. I didn't get that from what you've said.

Also, you're giving us the text and we're giving you legitimate working regexes and still you're getting nothing. So it would be a good idea if you posted a couple of events that contain the values you're looking for so we can see what might be going wrong.

Also... see my edited answer that addresses your transforms.conf syntax