How to edit my configurations to perform an index time field extraction?

[vr2312]

Hello All

My current environment is as follows :

Syslog/UF (Universal Forwarder) -> HF (Heavy Forwarder) -> Indexers

I am trying to perform an indexed time field extraction so that people can utilize the fields extracted across all Search Heads in our environment.

The following are what i have now after lots of trying :

transforms.conf

[ABC]
REGEX = ^.*host\s(?1[^ ]+)\sat.+by\s(?2.+)

FORMAT = $0:$1:$2:$3:$4:$5:$6

props.conf

[sourcetype::XYZ]
TRANSFORMS-ABC = a_B_C

I tried pushing this to the indexers to populate the extraction, but it is not working.

Also, the regex works in Search Time Extractions when i use it from the Search Head using a |rex "" command.

Please help.

[rjthibod · Accepted solution]

There are few issues with your approach that you need to address to do what you are trying to do:

1. The data passing through the HF is already "cooked" when it hits the Indexer. Unless you reroute the data at the Indexer to force it to do the extractions, you have to do Index-time extractions at the HF.
2. Your props.conf needs to say TRANSFORMS-ABC = ABC, where the right side of the equal sign needs to match the stanza in transforms.conf
3. Your transforms.conf is incomplete for Index-time extractions. You should have FORMAT and WRITE_META fields
4. You do not have anything in fields.conf which is required for Index-time extractions.

Look at this older question as a reference: https://answers.splunk.com/answers/103668/index-time-fields-with-heavy-forwarder.html

[vr2312]

@rjthibod

I will push the fields.conf only to Search Heads ?

[rjthibod]

Yes, I believe so.

[vr2312]

It worked without the fields.conf being pushed to the Search Heads.

the following were pushed to the HF's

fields.conf
transforms.conf
props.conf

[somesoni2]

Since you're using a Heavy forwarder in between, the event parsing will happen on the Heavy forwarder and these configurations should get deployed to Heavy forwarder not Indexers. Also, you would need to update fields.conf on the Search Head. For complete details on steps, see this link.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Configureindex-timefieldextraction#Where_to_p...