Getting Data In

Help getting field extraction from information in field=source

harry17preet
Explorer

Hi All,

I am ingesting some logs from Heavy Forwarder and then sending them to indexers.

*Snippet from inputs.conf on the Universal Forwarder

[monitor:///opt/splunk/etc/apps/nonprod_apicalls/local/ce_p2_srv_data2_env_getstats_port.txt]

disabled = false

sourcetype = my:api:ce2

index = internet

I would like to extract "data2" text from the filename.  I did a rex field extraction on search head and it works giving me "instance" field name under interesting fields on Search UI. Below is the regex I used

| rex field=source "\/(.+\/)(?:[^_]+_[^_]+_[^_]+_)(?<instance>[^_]+)"

 

So next step I did is created props.conf with below configuration

[my:api:ce2]
EXTRACT-instance = \/(.+\/)(?:[^_]+_[^_]+_[^_]+_)(?<instance>[^_]+) in source

 

Restarted the splunk service on Heavy Forwarder, but it doesn't work.

Can someone advise me if I am doing something wrong here or what is the issue.

Thankyou

Harry

Labels (2)
0 Karma
1 Solution

VatsalJagani
Super Champion

@harry17preet - 

props.conf (HF)

[my:api:ce2]
TRANSFORMS-extract_instance = extract_instance

 

transforms.conf (HF)

[extract_instance]
SOURCE_KEY = MetaData:Source
REGEX = \/(.+\/)(?:[^_]+_[^_]+_[^_]+_)(?<instance>[^_]+)
WRITE_META = true

 

fields.conf (SH)

[instance]
INDEXED = true

 

Please note you have to deploy props.conf and transforms.conf at HF (indexing level) and fields.conf on SH.

Also, not that at index time source value (MetaData:Source) will be prefixed by "source::", but I think your regex will still work, but please confirm on your end.

(Reference/Doc - https://docs.splunk.com/Documentation/Splunk/latest/Admin/transformsconf )

 

I hope this helps!!! Karma/upvote would be appreciated.

View solution in original post

VatsalJagani
Super Champion

@harry17preet - EXTRACT is a search-time parameter. You need to add on the Search Head and not on HF.

0 Karma

harry17preet
Explorer

Hi @VatsalJagan  I would like to do it during indexing. 

How can I do that via props.conf and transforms.conf.

Are you able to show me that props and transforms configurations.

0 Karma

VatsalJagani
Super Champion

@harry17preet - 

props.conf (HF)

[my:api:ce2]
TRANSFORMS-extract_instance = extract_instance

 

transforms.conf (HF)

[extract_instance]
SOURCE_KEY = MetaData:Source
REGEX = \/(.+\/)(?:[^_]+_[^_]+_[^_]+_)(?<instance>[^_]+)
WRITE_META = true

 

fields.conf (SH)

[instance]
INDEXED = true

 

Please note you have to deploy props.conf and transforms.conf at HF (indexing level) and fields.conf on SH.

Also, not that at index time source value (MetaData:Source) will be prefixed by "source::", but I think your regex will still work, but please confirm on your end.

(Reference/Doc - https://docs.splunk.com/Documentation/Splunk/latest/Admin/transformsconf )

 

I hope this helps!!! Karma/upvote would be appreciated.

harry17preet
Explorer

Hi @VatsalJagani , thanks heaps. that works.

Are you also able to provide me details on search time extraction.  Same details only change change is source file name will vary like : /opt/splunk/etc/apps/ce/local/app/ce_*_data2_*_*.txt.

Cheers

0 Karma

VatsalJagani
Super Champion

@harry17preet - As you requested this is index-time.

You can search the index time field like this:

index=<your-index> instance::myinstance 

 

For search-time extraction, you can use the EXTRACT that I mentioned in my previous answer.  Search-time extracted field you can search with regular syntax (instance="myinstance").

 

I hope this helps!!!

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

if you want to do that on search time you must add this extractions to SH's props.conf in some application. My proposal is to create your own app where put these, not in search app!  See: https://community.splunk.com/t5/Splunk-Enterprise/What-kind-of-things-do-you-view-as-quot-bad-config...

If you want to do this on ingest time and create a indexed field (probably not worth of it?) then you must use props.conf + transforms.conf on HF. I'm expecting that you need this a search time not on ingesting time + indexed field?

r. Ismo

harry17preet
Explorer

Hi @isoutamo  I would like to do it during indexing. 

How can I do that via props.conf and transforms.conf.

Are you able to show me the props and transforms configurations.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Here is quite similar case https://community.splunk.com/t5/Splunk-Enterprise/Index-time-field-extractions-path/m-p/241831. Probably you can do it based on that? 

Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...