Hi All,
I am ingesting some logs from Heavy Forwarder and then sending them to indexers.
*Snippet from inputs.conf on the Universal Forwarder
[monitor:///opt/splunk/etc/apps/nonprod_apicalls/local/ce_p2_srv_data2_env_getstats_port.txt]
disabled = false
sourcetype = my:api:ce2
index = internet
I would like to extract "data2" text from the filename. I did a rex field extraction on search head and it works giving me "instance" field name under interesting fields on Search UI. Below is the regex I used
| rex field=source "\/(.+\/)(?:[^_]+_[^_]+_[^_]+_)(?<instance>[^_]+)"
So next step I did is created props.conf with below configuration
[my:api:ce2]
EXTRACT-instance = \/(.+\/)(?:[^_]+_[^_]+_[^_]+_)(?<instance>[^_]+) in source
Restarted the splunk service on Heavy Forwarder, but it doesn't work.
Can someone advise me if I am doing something wrong here or what is the issue.
Thankyou
Harry
props.conf (HF)
[my:api:ce2]
TRANSFORMS-extract_instance = extract_instance
transforms.conf (HF)
[extract_instance]
SOURCE_KEY = MetaData:Source
REGEX = \/(.+\/)(?:[^_]+_[^_]+_[^_]+_)(?<instance>[^_]+)
WRITE_META = true
fields.conf (SH)
[instance]
INDEXED = true
Please note you have to deploy props.conf and transforms.conf at HF (indexing level) and fields.conf on SH.
Also, not that at index time source value (MetaData:Source) will be prefixed by "source::", but I think your regex will still work, but please confirm on your end.
(Reference/Doc - https://docs.splunk.com/Documentation/Splunk/latest/Admin/transformsconf )
I hope this helps!!! Karma/upvote would be appreciated.
@harry17preet - EXTRACT is a search-time parameter. You need to add on the Search Head and not on HF.
Hi @VatsalJagan I would like to do it during indexing.
How can I do that via props.conf and transforms.conf.
Are you able to show me that props and transforms configurations.
props.conf (HF)
[my:api:ce2]
TRANSFORMS-extract_instance = extract_instance
transforms.conf (HF)
[extract_instance]
SOURCE_KEY = MetaData:Source
REGEX = \/(.+\/)(?:[^_]+_[^_]+_[^_]+_)(?<instance>[^_]+)
WRITE_META = true
fields.conf (SH)
[instance]
INDEXED = true
Please note you have to deploy props.conf and transforms.conf at HF (indexing level) and fields.conf on SH.
Also, not that at index time source value (MetaData:Source) will be prefixed by "source::", but I think your regex will still work, but please confirm on your end.
(Reference/Doc - https://docs.splunk.com/Documentation/Splunk/latest/Admin/transformsconf )
I hope this helps!!! Karma/upvote would be appreciated.
Hi @VatsalJagani , thanks heaps. that works.
Are you also able to provide me details on search time extraction. Same details only change change is source file name will vary like : /opt/splunk/etc/apps/ce/local/app/ce_*_data2_*_*.txt.
Cheers
@harry17preet - As you requested this is index-time.
You can search the index time field like this:
index=<your-index> instance::myinstance
For search-time extraction, you can use the EXTRACT that I mentioned in my previous answer. Search-time extracted field you can search with regular syntax (instance="myinstance").
I hope this helps!!!
Hi
if you want to do that on search time you must add this extractions to SH's props.conf in some application. My proposal is to create your own app where put these, not in search app! See: https://community.splunk.com/t5/Splunk-Enterprise/What-kind-of-things-do-you-view-as-quot-bad-config...
If you want to do this on ingest time and create a indexed field (probably not worth of it?) then you must use props.conf + transforms.conf on HF. I'm expecting that you need this a search time not on ingesting time + indexed field?
r. Ismo
Hi @isoutamo I would like to do it during indexing.
How can I do that via props.conf and transforms.conf.
Are you able to show me the props and transforms configurations.
Here is quite similar case https://community.splunk.com/t5/Splunk-Enterprise/Index-time-field-extractions-path/m-p/241831. Probably you can do it based on that?