Creating Indexed Field Extraction from Source -- FORMAT

[Dave98]

I have been trying to extract an indexed field by using the transforms.conf file. Here's a sample:

[serviceName]

SOURCE_KEY = MetaData:Source

REGEX = ^.{7}\(?P\w+)

FORMAT = source::ServiceName

WRITE_META = true

I believe the problem is in the FORMAT = source::ServiceName, but am not sure how to fix it.

I have tried variants like the following, but none of them work

FORMAT = source::$1

FORMAT = ServiceName::$1

Can some one help me out with the syntax to extract an indexed field from a SourceKey of source?

[Dave98]

Well, maybe, but I'd still like to be able to try it out and see for myself.

[lguinn2]

If you want the indexed field, you should do this:

[serviceName]
SOURCE_KEY = MetaData:Source
REGEX = ^.{7}\(w+)
FORMAT = ServiceName::$1
WRITE_META = true

The resulting field will be named ServiceName.

[UPDATE] You also need to update (or create) fields.conf and add the following stanza:

[ServiceName]
INDEXED = true

Splunk will need to be restarted for these configuration changes to take effect. The indexed field will only appear in new events; if you want the field to appear in existing events, the data will need to be re-indexed.

Here is a link to the manual topic Create custom fields at index time, complete with a recommendation not to index fields! 🙂

[lguinn2]

I did a little research and realized that I forgot to mention fields.conf - maybe this will work?

[Dave98]

I tried your solution, which makes perfect sense. Unfortunately, it still didn't index.

I tried to replace the extraction with one coming from _raw, and it worked, so the plumbing is there.

Any ideas on how to debug or other things to try?

[lguinn2]

I changed the REGEX so that still has a capture group, but now it is an unnamed capture group. $1 refers to the "first capture group", so that's how Splunk knows where to get the info to put into serviceName.

Based on your comment, here is a better REGEX; it is more general and looks for the name immediately following the \logs\ in the path:

REGEX = \\logs\\(\S+?)\\

and yes, you need the double backslashes, because a single backslash has a special meaning in regular expressions.

[Dave98]

Ok, I'm going to try this out. But what happened to the group in the REGEX? Help me understand why you removed the
"?" syntax. How will the regex know where to extract the data?

My path is d:\logs\ServiceName\logfile.txt

[lguinn2]

Splunk already indexes every keyword in the data, plus the source sourcetype and host - so I greatly doubt that you will improve performance in this way.

[Dave98]

We have many different services (approaching 100) and the first item our developers would search on is ServiceName. Nearly every event in this type would have this attribute.

I have a search based extraction working, but have seen performance issue with broad based searches such as, "ServiceName=TestService"

I'd like to at least try out the indexed field and test it to see the performance difference.

[lguinn2]

Why do you want an indexed field? A search time field is almost always a better choice...