Splunk Search

Best practice for field extraction

jangid
Builder

What is the best option for field extraction?

my log file contain some data separated with # and I want to convert these data in tabular format.

e.g.

ID#Data-1#Value-1#TRANS ID#.... so on

due to performance reason in our system I can't use key-pair in log so I want to extract this in Splunk.

Tags (1)
0 Karma
1 Solution

rturk
Builder

The other way to do this is through the use of delimiters and specifying your fields like this:

props.conf

[yoursourcetypehere]
REPORT-fields = extract-my-fields

transforms.conf

[extract-my-fields]
DELIMS = "#"
FIELDS = "id", "Data-1", "Value-1", "transId", ...

I'm not sure which of the two is better (ie. less computationally expensive), but I usually use the way I have listed (so long as you have a common delimiter).

View solution in original post

jangid
Builder

How do I create multiple fields in log?

e.g.
my log contains following information
[Counter-1] id#data1#data2#data2.. and so on ..
[Counter-2] id#data1#data2#data2.. and so on ..
[Counter-3] id#data1#data2#data2.. and so on ..

Now I want to create field based on the [Counter-n] keyword

[Counter-1] id#data1#data2#data2.. and so on ..
FIELDS - Counter_1_id, Counter_1_data1, Counter_1_data2, ... so on

[Counter-2] id#data1#data2#data2.. and so on ..
FIELDS - Counter_2_id, Counter_2_data1, Counter_2_data2, ... so on

[Counter-3] id#data1#data2#data2.. and so on ..
FIELDS - Counter_3_id, Counter_3_data1, Counter_3_data2, ... so on

Note - I want to create these fields only if [Counter-n] exist

Thanks
Manoj

0 Karma

jangid
Builder

I tried with above given method I am able to get the fields but some unnecessary fields also creating.

props.conf

REPORT-jt_fields = extract_jt_field

transform.conf

[extract_jt_field]
REGEX = "Counter-1"
DELIMS = "#"
FIELDS = "Counter_1_id","Counter_1_data1","Counter_1_data2"

am I doing anything wrong?

0 Karma

rturk
Builder

The other way to do this is through the use of delimiters and specifying your fields like this:

props.conf

[yoursourcetypehere]
REPORT-fields = extract-my-fields

transforms.conf

[extract-my-fields]
DELIMS = "#"
FIELDS = "id", "Data-1", "Value-1", "transId", ...

I'm not sure which of the two is better (ie. less computationally expensive), but I usually use the way I have listed (so long as you have a common delimiter).

jangid
Builder

Thanks its help me.

0 Karma

lguinn2
Legend

I like this answer too. It is easier when the delimiter is consistent.

0 Karma

lguinn2
Legend

There are two major types of field extraction: index time and search time. Search time is the best practice, especially for the example you have given. There are many ways to implement search time field extraction. In general, they all do the same thing with similar performance. So it is mostly a matter of "how is it easiest to specify what I want?"

In your case, I recommend that you use the REPORT extraction. It works like this. On the indexer(s) - or search head if you are using one - edit these configuration files:

props.conf

[yoursourcetypehere]
REPORT-extract1=extract-my-fields

transforms.conf

[extract-my-fields]
REGEX = ([^#]+)#([^#]+)#([^#]+)#([^#]+)# and so on
FORMAT = id::$1  Data-1::$2  Value-1::$3  transId::$4  and so on to match the previous line

HTH!

lguinn2
Legend

Except for rare corner cases, index-time field extraction does not perform better than search-time field extraction. In addition, index-time field extraction has other issues that make it less desirable overall.

I don't think there is any performance difference between the various methods of specifying search-time field extraction -- it is just a matter of which way is easiest to specify. Your (R. Turk's) answer is excellent for this particular case. At other times, you may not be able to use DELIMS and will need to go with this solution.

0 Karma

rturk
Builder

Hi Lisa, is there any benefit using either of our methods over the other? Seems like they are both a means to the same end, except when you're extracting fields from a event with inconsistent delimiters (or delimiters in the values).

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...