topic Re: Creating Indexed Field Extraction from Source -- FORMAT in Splunk Search

Creating Indexed Field Extraction from Source -- FORMAT

Dave98 — Mon, 28 Sep 2020 16:11:03 GMT

I have been trying to extract an indexed field by using the transforms.conf file. Here's a sample:

[serviceName]

SOURCE_KEY = MetaData:Source

REGEX = ^.{7}\(?P\w+)

FORMAT = source::ServiceName

WRITE_META = true

I believe the problem is in the FORMAT = source::ServiceName, but am not sure how to fix it.

I have tried variants like the following, but none of them work

FORMAT = source::$1

FORMAT = ServiceName::$1

Can some one help me out with the syntax to extract an indexed field from a SourceKey of source?

Re: Creating Indexed Field Extraction from Source -- FORMAT

lguinn2 — Wed, 19 Mar 2014 20:14:59 GMT

Why do you want an indexed field? A search time field is almost always a better choice...

Re: Creating Indexed Field Extraction from Source -- FORMAT

Dave98 — Wed, 19 Mar 2014 21:00:49 GMT

We have many different services (approaching 100) and the first item our developers would search on is ServiceName. Nearly every event in this type would have this attribute.

I have a search based extraction working, but have seen performance issue with broad based searches such as, "ServiceName=TestService"

I'd like to at least try out the indexed field and test it to see the performance difference.

Re: Creating Indexed Field Extraction from Source -- FORMAT

lguinn2 — Wed, 19 Mar 2014 21:07:23 GMT

Splunk already indexes every keyword in the data, plus the source sourcetype and host - so I greatly doubt that you will improve performance in this way.

Re: Creating Indexed Field Extraction from Source -- FORMAT

lguinn2 — Wed, 19 Mar 2014 21:25:31 GMT

If you want the indexed field, you should do this:

[serviceName]
SOURCE_KEY = MetaData:Source
REGEX = ^.{7}\(w+)
FORMAT = ServiceName::$1
WRITE_META = true

The resulting field will be named ServiceName.

[UPDATE] You also need to update (or create) fields.conf and add the following stanza:

[ServiceName]
INDEXED = true

Splunk will need to be restarted for these configuration changes to take effect. The indexed field will only appear in new events; if you want the field to appear in existing events, the data will need to be re-indexed.

Here is a link to the manual topic Create custom fields at index time, complete with a recommendation not to index fields! 🙂

Re: Creating Indexed Field Extraction from Source -- FORMAT

Dave98 — Wed, 19 Mar 2014 21:28:18 GMT

Well, maybe, but I'd still like to be able to try it out and see for myself.

Re: Creating Indexed Field Extraction from Source -- FORMAT

Dave98 — Wed, 19 Mar 2014 21:32:35 GMT

Ok, I'm going to try this out. But what happened to the group in the REGEX? Help me understand why you removed the
"?" syntax. How will the regex know where to extract the data?

My path is d:\logs\ServiceName\logfile.txt

Re: Creating Indexed Field Extraction from Source -- FORMAT

lguinn2 — Wed, 19 Mar 2014 22:53:23 GMT

I changed the REGEX so that still has a capture group, but now it is an unnamed capture group. $1 refers to the "first capture group", so that's how Splunk knows where to get the info to put into serviceName.

Based on your comment, here is a better REGEX; it is more general and looks for the name immediately following the \logs\ in the path:

REGEX = \\logs\\(\S+?)\\

and yes, you need the double backslashes, because a single backslash has a special meaning in regular expressions.

Re: Creating Indexed Field Extraction from Source -- FORMAT

Dave98 — Fri, 21 Mar 2014 13:17:05 GMT

I tried your solution, which makes perfect sense. Unfortunately, it still didn't index.

I tried to replace the extraction with one coming from _raw, and it worked, so the plumbing is there.

Any ideas on how to debug or other things to try?

Re: Creating Indexed Field Extraction from Source -- FORMAT

lguinn2 — Sat, 22 Mar 2014 16:46:36 GMT

I did a little research and realized that I forgot to mention fields.conf - maybe this will work?