topic Extract field in indexing time in Getting Data In

Extract field in indexing time

splunklearner — Wed, 19 Feb 2025 15:23:38 GMT

I want to extract value from the following field while indexing the data to use it to map it with index.

vs_name=v-jupiter-prd-cbc-us.sony-443-ipv6

I want to extract every field after v- and till sony. I.e., jupiter-prd-cbc-us.sony as fqdn so that this fqdn will check in lookup to map it to correct index.

Please help me with props and transforms to extract fqdn correctly.

Re: Extract field in indexing time

livehybrid — Wed, 19 Feb 2025 15:50:09 GMT

Hi @splunklearner

This is quite complex to achieve in props/transforms but shouldnt be impossible - lets have a go..

This is what it would look like as SPL - use this to tweak your eval to match your field names and config, then apply to the transforms as below.

| makeresults | eval _raw="something=v-jupiter-prd-cbc-us.sony-443-ipv6" | eval hostType=replace(_raw, ".*v\-(?<hostType>[^\.]+)\.sony.*", "\1") | eval yourIndex=json_extract(lookup("testlookup.csv",json_object("hostType",hostType), json_array(index)),"index") ``` as one line ``` | eval yourIndexNew=json_extract(lookup("testlookup.csv",json_object("hostType",replace(_raw, ".*v\-(?<hostType>[^\.]+)\.sony.*", "\1")), json_array(index)),"index")

You will also need a lookup in $SPLUNK_HOME/system/lookups - in this example its testlookup.csv. For the purposes of testing in SPL you can create a temporary lookup with this:

| makeresults | eval hostType="jupiter-prd-cbc-us", index="index1" | outputlookup testlookup.csv

Props/transforms.conf

== props.conf == [yourSourcetype] TRANSFORMS-defineIndex = defineIndex == transforms.conf == [defineIndex] INGEST_EVAL = index=json_extract(lookup("testlookup.csv",json_object("hostType",replace(_raw, ".*v\-(?<hostType>[^\.]+)\.sony.*", "\1")), json_array(index)),"index")

For more info on how the lookup command works, have a look at https://docs.splunk.com/Documentation/Splunk/9.4.0/SearchReference/ConditionalFunctions#lookup.28.26lt.3Blookup_table.26gt.3B.2C.26lt.3Bjson_object.26gt.3B.2C.26lt.3Bjson_array.26gt.3B.29

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

Re: Extract field in indexing time

splunklearner — Wed, 19 Feb 2025 16:08:00 GMT

I already have ingest eval in place. I only need to extract fqdn from vs_name in order to match there.

Re: Extract field in indexing time

livehybrid — Wed, 19 Feb 2025 20:59:44 GMT

You would therefore use "vs_name" in-place of "_raw" in the replace command.

You can use multiple transforms on a single sourcetype - even if you're already using an INGEST_EVAL.

For example

== props.conf == [yourSourcetype] TRANSFORMS-defineIndex =.defineIndex TRANSFORMS-extractServerId =.extractServerId ... etc ...

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will