topic How to edit my configurations to perform an index time field extraction? in Splunk Search

How to edit my configurations to perform an index time field extraction?

vr2312 — Fri, 17 Feb 2017 14:06:07 GMT

Hello All

My current environment is as follows :

Syslog/UF (Universal Forwarder) -> HF (Heavy Forwarder) -> Indexers

I am trying to perform an indexed time field extraction so that people can utilize the fields extracted across all Search Heads in our environment.

The following are what i have now after lots of trying :

transforms.conf

[ABC]
REGEX = ^.*host\s(?1[^ ]+)\sat.+by\s(?2.+)

FORMAT = $0:$1:$2:$3:$4:$5:$6

props.conf

[sourcetype::XYZ]
TRANSFORMS-ABC = a_B_C

I tried pushing this to the indexers to populate the extraction, but it is not working.

Also, the regex works in Search Time Extractions when i use it from the Search Head using a |rex "" command.

Please help.

Re: How to edit my configurations to perform an index time field extraction?

somesoni2 — Fri, 17 Feb 2017 14:30:02 GMT

Since you're using a Heavy forwarder in between, the event parsing will happen on the Heavy forwarder and these configurations should get deployed to Heavy forwarder not Indexers. Also, you would need to update fields.conf on the Search Head. For complete details on steps, see this link.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Configureindex-timefieldextraction#Where_to_put_the_configuration_changes_in_a_distributed_environment

Re: How to edit my configurations to perform an index time field extraction?

rjthibod — Fri, 17 Feb 2017 14:30:57 GMT

There are few issues with your approach that you need to address to do what you are trying to do:

The data passing through the HF is already "cooked" when it hits the Indexer. Unless you reroute the data at the Indexer to force it to do the extractions, you have to do Index-time extractions at the HF.
Your props.conf needs to say TRANSFORMS-ABC = ABC, where the right side of the equal sign needs to match the stanza in transforms.conf
Your transforms.conf is incomplete for Index-time extractions. You should have FORMAT and WRITE_META fields
You do not have anything in fields.conf which is required for Index-time extractions.

Look at this older question as a reference: https://answers.splunk.com/answers/103668/index-time-fields-with-heavy-forwarder.html

Re: How to edit my configurations to perform an index time field extraction?

vr2312 — Fri, 17 Feb 2017 15:07:30 GMT

@rjthibod

I will push the fields.conf only to Search Heads ?

Re: How to edit my configurations to perform an index time field extraction?

rjthibod — Fri, 17 Feb 2017 15:12:34 GMT

Yes, I believe so.

Re: How to edit my configurations to perform an index time field extraction?

vr2312 — Mon, 20 Feb 2017 10:45:11 GMT

It worked without the fields.conf being pushed to the Search Heads.

the following were pushed to the HF's

fields.conf
transforms.conf
props.conf