Splunk Search

How to edit my configurations to perform an index time field extraction?

vr2312
Contributor

Hello All

My current environment is as follows :

Syslog/UF (Universal Forwarder) -> HF (Heavy Forwarder) -> Indexers

I am trying to perform an indexed time field extraction so that people can utilize the fields extracted across all Search Heads in our environment.

The following are what i have now after lots of trying :

transforms.conf

[ABC]
REGEX = ^.*host\s(?1[^ ]+)\sat.+by\s(?2.+)

FORMAT = $0:$1:$2:$3:$4:$5:$6

props.conf

[sourcetype::XYZ]
TRANSFORMS-ABC = a_B_C

I tried pushing this to the indexers to populate the extraction, but it is not working.

Also, the regex works in Search Time Extractions when i use it from the Search Head using a |rex "" command.

Please help.

0 Karma
1 Solution

rjthibod
Champion

There are few issues with your approach that you need to address to do what you are trying to do:

  1. The data passing through the HF is already "cooked" when it hits the Indexer. Unless you reroute the data at the Indexer to force it to do the extractions, you have to do Index-time extractions at the HF.
  2. Your props.conf needs to say TRANSFORMS-ABC = ABC, where the right side of the equal sign needs to match the stanza in transforms.conf
  3. Your transforms.conf is incomplete for Index-time extractions. You should have FORMAT and WRITE_META fields
  4. You do not have anything in fields.conf which is required for Index-time extractions.

Look at this older question as a reference: https://answers.splunk.com/answers/103668/index-time-fields-with-heavy-forwarder.html

View solution in original post

rjthibod
Champion

There are few issues with your approach that you need to address to do what you are trying to do:

  1. The data passing through the HF is already "cooked" when it hits the Indexer. Unless you reroute the data at the Indexer to force it to do the extractions, you have to do Index-time extractions at the HF.
  2. Your props.conf needs to say TRANSFORMS-ABC = ABC, where the right side of the equal sign needs to match the stanza in transforms.conf
  3. Your transforms.conf is incomplete for Index-time extractions. You should have FORMAT and WRITE_META fields
  4. You do not have anything in fields.conf which is required for Index-time extractions.

Look at this older question as a reference: https://answers.splunk.com/answers/103668/index-time-fields-with-heavy-forwarder.html

View solution in original post

vr2312
Contributor

@rjthibod

I will push the fields.conf only to Search Heads ?

0 Karma

rjthibod
Champion

Yes, I believe so.

0 Karma

vr2312
Contributor

It worked without the fields.conf being pushed to the Search Heads.

the following were pushed to the HF's

fields.conf
transforms.conf
props.conf

0 Karma

somesoni2
Revered Legend

Since you're using a Heavy forwarder in between, the event parsing will happen on the Heavy forwarder and these configurations should get deployed to Heavy forwarder not Indexers. Also, you would need to update fields.conf on the Search Head. For complete details on steps, see this link.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Configureindex-timefieldextraction#Where_to_p...

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!