topic Best practice for field extraction in Splunk Search

Best practice for field extraction

jangid — Fri, 06 Jul 2012 16:18:40 GMT

What is the best option for field extraction?

my log file contain some data separated with # and I want to convert these data in tabular format.

e.g.

ID#Data-1#Value-1#TRANS ID#.... so on

due to performance reason in our system I can't use key-pair in log so I want to extract this in Splunk.

Re: Best practice for field extraction

lguinn2 — Sat, 07 Jul 2012 02:07:09 GMT

There are two major types of field extraction: index time and search time. Search time is the best practice, especially for the example you have given. There are many ways to implement search time field extraction. In general, they all do the same thing with similar performance. So it is mostly a matter of "how is it easiest to specify what I want?"

In your case, I recommend that you use the REPORT extraction. It works like this. On the indexer(s) - or search head if you are using one - edit these configuration files:

props.conf

[yoursourcetypehere]
REPORT-extract1=extract-my-fields

transforms.conf

[extract-my-fields]
REGEX = ([^#]+)#([^#]+)#([^#]+)#([^#]+)# and so on
FORMAT = id::$1  Data-1::$2  Value-1::$3  transId::$4  and so on to match the previous line

HTH!

Re: Best practice for field extraction

rturk — Sat, 07 Jul 2012 02:26:21 GMT

The other way to do this is through the use of delimiters and specifying your fields like this:

props.conf

[yoursourcetypehere]
REPORT-fields = extract-my-fields

transforms.conf

[extract-my-fields]
DELIMS = "#"
FIELDS = "id", "Data-1", "Value-1", "transId", ...

I'm not sure which of the two is better (ie. less computationally expensive), but I usually use the way I have listed (so long as you have a common delimiter).

Re: Best practice for field extraction

rturk — Sat, 07 Jul 2012 02:29:24 GMT

Hi Lisa, is there any benefit using either of our methods over the other? Seems like they are both a means to the same end, except when you're extracting fields from a event with inconsistent delimiters (or delimiters in the values).

Re: Best practice for field extraction

lguinn2 — Sun, 08 Jul 2012 00:51:24 GMT

I like this answer too. It is easier when the delimiter is consistent.

Re: Best practice for field extraction

lguinn2 — Sun, 08 Jul 2012 00:56:28 GMT

Except for rare corner cases, index-time field extraction does not perform better than search-time field extraction. In addition, index-time field extraction has other issues that make it less desirable overall.

I don't think there is any performance difference between the various methods of specifying search-time field extraction -- it is just a matter of which way is easiest to specify. Your (R. Turk's) answer is excellent for this particular case. At other times, you may not be able to use DELIMS and will need to go with this solution.

Re: Best practice for field extraction

jangid — Wed, 11 Jul 2012 12:38:09 GMT

Thanks its help me.

Re: Best practice for field extraction

jangid — Mon, 28 Sep 2020 12:03:44 GMT

How do I create multiple fields in log?

e.g.
my log contains following information
[Counter-1] id#data1#data2#data2.. and so on ..
[Counter-2] id#data1#data2#data2.. and so on ..
[Counter-3] id#data1#data2#data2.. and so on ..

Now I want to create field based on the [Counter-n] keyword

[Counter-1] id#data1#data2#data2.. and so on ..
FIELDS - Counter_1_id, Counter_1_data1, Counter_1_data2, ... so on

[Counter-2] id#data1#data2#data2.. and so on ..
FIELDS - Counter_2_id, Counter_2_data1, Counter_2_data2, ... so on

[Counter-3] id#data1#data2#data2.. and so on ..
FIELDS - Counter_3_id, Counter_3_data1, Counter_3_data2, ... so on

Note - I want to create these fields only if [Counter-n] exist

Thanks
Manoj

Re: Best practice for field extraction

jangid — Mon, 28 Sep 2020 12:03:47 GMT

I tried with above given method I am able to get the fields but some unnecessary fields also creating.

props.conf

REPORT-jt_fields = extract_jt_field

transform.conf

[extract_jt_field]
REGEX = "Counter-1"
DELIMS = "#"
FIELDS = "Counter_1_id","Counter_1_data1","Counter_1_data2"

am I doing anything wrong?