Best practice for field extraction

[jangid]

What is the best option for field extraction?

my log file contain some data separated with # and I want to convert these data in tabular format.

e.g.

ID#Data-1#Value-1#TRANS ID#.... so on

due to performance reason in our system I can't use key-pair in log so I want to extract this in Splunk.

[jangid]

How do I create multiple fields in log?

e.g.
my log contains following information
[Counter-1] id#data1#data2#data2.. and so on ..
[Counter-2] id#data1#data2#data2.. and so on ..
[Counter-3] id#data1#data2#data2.. and so on ..

Now I want to create field based on the [Counter-n] keyword

[Counter-1] id#data1#data2#data2.. and so on ..
FIELDS - Counter_1_id, Counter_1_data1, Counter_1_data2, ... so on

[Counter-2] id#data1#data2#data2.. and so on ..
FIELDS - Counter_2_id, Counter_2_data1, Counter_2_data2, ... so on

[Counter-3] id#data1#data2#data2.. and so on ..
FIELDS - Counter_3_id, Counter_3_data1, Counter_3_data2, ... so on

Note - I want to create these fields only if [Counter-n] exist

Thanks
Manoj

[jangid]

I tried with above given method I am able to get the fields but some unnecessary fields also creating.

props.conf

REPORT-jt_fields = extract_jt_field

transform.conf

[extract_jt_field]
REGEX = "Counter-1"
DELIMS = "#"
FIELDS = "Counter_1_id","Counter_1_data1","Counter_1_data2"

am I doing anything wrong?

[rturk · Accepted solution]

The other way to do this is through the use of delimiters and specifying your fields like this:

props.conf

[yoursourcetypehere]
REPORT-fields = extract-my-fields

transforms.conf

[extract-my-fields]
DELIMS = "#"
FIELDS = "id", "Data-1", "Value-1", "transId", ...

I'm not sure which of the two is better (ie. less computationally expensive), but I usually use the way I have listed (so long as you have a common delimiter).

[jangid]

Thanks its help me.

[lguinn2]

I like this answer too. It is easier when the delimiter is consistent.

[lguinn2]

There are two major types of field extraction: index time and search time. Search time is the best practice, especially for the example you have given. There are many ways to implement search time field extraction. In general, they all do the same thing with similar performance. So it is mostly a matter of "how is it easiest to specify what I want?"

In your case, I recommend that you use the REPORT extraction. It works like this. On the indexer(s) - or search head if you are using one - edit these configuration files:

props.conf

[yoursourcetypehere]
REPORT-extract1=extract-my-fields

transforms.conf

[extract-my-fields]
REGEX = ([^#]+)#([^#]+)#([^#]+)#([^#]+)# and so on
FORMAT = id::$1  Data-1::$2  Value-1::$3  transId::$4  and so on to match the previous line

HTH!

[lguinn2]

Except for rare corner cases, index-time field extraction does not perform better than search-time field extraction. In addition, index-time field extraction has other issues that make it less desirable overall.

I don't think there is any performance difference between the various methods of specifying search-time field extraction -- it is just a matter of which way is easiest to specify. Your (R. Turk's) answer is excellent for this particular case. At other times, you may not be able to use DELIMS and will need to go with this solution.

[rturk]

Hi Lisa, is there any benefit using either of our methods over the other? Seems like they are both a means to the same end, except when you're extracting fields from a event with inconsistent delimiters (or delimiters in the values).