topic Re: Help getting field extraction from information in field=source in Getting Data In

Help getting field extraction from information in field=source

harry17preet — Thu, 19 May 2022 02:19:21 GMT

Hi All,

I am ingesting some logs from Heavy Forwarder and then sending them to indexers.

*Snippet from inputs.conf on the Universal Forwarder

[monitor:///opt/splunk/etc/apps/nonprod_apicalls/local/ce_p2_srv_data2_env_getstats_port.txt]

disabled = false

sourcetype = my:api:ce2

index = internet

I would like to extract "data2" text from the filename. I did a rex field extraction on search head and it works giving me "instance" field name under interesting fields on Search UI. Below is the regex I used

| rex field=source "\/(.+\/)(?:[^_]+_[^_]+_[^_]+_)(?<instance>[^_]+)"

So next step I did is created props.conf with below configuration

[my:api:ce2]
EXTRACT-instance = \/(.+\/)(?:[^_]+_[^_]+_[^_]+_)(?<instance>[^_]+) in source

Restarted the splunk service on Heavy Forwarder, but it doesn't work.

Can someone advise me if I am doing something wrong here or what is the issue.

Thankyou

Harry

Re: Help getting field extraction from information in field=source

isoutamo — Thu, 19 May 2022 05:59:48 GMT

if you want to do that on search time you must add this extractions to SH's props.conf in some application. My proposal is to create your own app where put these, not in search app! See: https://community.splunk.com/t5/Splunk-Enterprise/What-kind-of-things-do-you-view-as-quot-bad-config-quot/m-p/594893#M12359

If you want to do this on ingest time and create a indexed field (probably not worth of it?) then you must use props.conf + transforms.conf on HF. I'm expecting that you need this a search time not on ingesting time + indexed field?

r. Ismo

Re: Help getting field extraction from information in field=source

VatsalJagani — Thu, 19 May 2022 06:29:21 GMT

@harry17preet - EXTRACT is a search-time parameter. You need to add on the Search Head and not on HF.

Re: Help getting field extraction from information in field=source

harry17preet — Thu, 19 May 2022 08:21:40 GMT

Hi @VatsalJagan I would like to do it during indexing.

How can I do that via props.conf and transforms.conf.

Are you able to show me that props and transforms configurations.

Re: Help getting field extraction from information in field=source

harry17preet — Thu, 19 May 2022 08:22:10 GMT

Hi @isoutamo I would like to do it during indexing.

How can I do that via props.conf and transforms.conf.

Are you able to show me the props and transforms configurations.

Re: Help getting field extraction from information in field=source

isoutamo — Thu, 19 May 2022 09:14:14 GMT

Here is quite similar case https://community.splunk.com/t5/Splunk-Enterprise/Index-time-field-extractions-path/m-p/241831. Probably you can do it based on that?

Re: Help getting field extraction from information in field=source

VatsalJagani — Thu, 19 May 2022 09:27:35 GMT

@harry17preet -

props.conf (HF)

[my:api:ce2] TRANSFORMS-extract_instance = extract_instance

transforms.conf (HF)

[extract_instance] SOURCE_KEY = MetaData:Source REGEX = \/(.+\/)(?:[^_]+_[^_]+_[^_]+_)(?<instance>[^_]+) WRITE_META = true

fields.conf (SH)

[instance] INDEXED = true

Please note you have to deploy props.conf and transforms.conf at HF (indexing level) and fields.conf on SH.

Also, not that at index time source value (MetaData:Source) will be prefixed by "source::", but I think your regex will still work, but please confirm on your end.

(Reference/Doc - https://docs.splunk.com/Documentation/Splunk/latest/Admin/transformsconf )

I hope this helps!!! Karma/upvote would be appreciated.

Re: Help getting field extraction from information in field=source

harry17preet — Fri, 20 May 2022 01:11:25 GMT

Hi @VatsalJagani , thanks heaps. that works.

Are you also able to provide me details on search time extraction. Same details only change change is source file name will vary like : /opt/splunk/etc/apps/ce/local/app/ce_*_data2_*_*.txt.

Cheers

Re: Help getting field extraction from information in field=source

VatsalJagani — Fri, 20 May 2022 04:05:22 GMT

@harry17preet - As you requested this is index-time.

You can search the index time field like this:

index=<your-index> instance::myinstance

For search-time extraction, you can use the EXTRACT that I mentioned in my previous answer. Search-time extracted field you can search with regular syntax (instance="myinstance").

I hope this helps!!!