Splunk Search

How to edit my configurations to perform an index time field extraction?

vr2312
Contributor

Hello All

My current environment is as follows :

Syslog/UF (Universal Forwarder) -> HF (Heavy Forwarder) -> Indexers

I am trying to perform an indexed time field extraction so that people can utilize the fields extracted across all Search Heads in our environment.

The following are what i have now after lots of trying :

transforms.conf

[ABC]
REGEX = ^.*host\s(?1[^ ]+)\sat.+by\s(?2.+)

FORMAT = $0:$1:$2:$3:$4:$5:$6

props.conf

[sourcetype::XYZ]
TRANSFORMS-ABC = a_B_C

I tried pushing this to the indexers to populate the extraction, but it is not working.

Also, the regex works in Search Time Extractions when i use it from the Search Head using a |rex "" command.

Please help.

0 Karma
1 Solution

rjthibod
Champion

There are few issues with your approach that you need to address to do what you are trying to do:

  1. The data passing through the HF is already "cooked" when it hits the Indexer. Unless you reroute the data at the Indexer to force it to do the extractions, you have to do Index-time extractions at the HF.
  2. Your props.conf needs to say TRANSFORMS-ABC = ABC, where the right side of the equal sign needs to match the stanza in transforms.conf
  3. Your transforms.conf is incomplete for Index-time extractions. You should have FORMAT and WRITE_META fields
  4. You do not have anything in fields.conf which is required for Index-time extractions.

Look at this older question as a reference: https://answers.splunk.com/answers/103668/index-time-fields-with-heavy-forwarder.html

View solution in original post

rjthibod
Champion

There are few issues with your approach that you need to address to do what you are trying to do:

  1. The data passing through the HF is already "cooked" when it hits the Indexer. Unless you reroute the data at the Indexer to force it to do the extractions, you have to do Index-time extractions at the HF.
  2. Your props.conf needs to say TRANSFORMS-ABC = ABC, where the right side of the equal sign needs to match the stanza in transforms.conf
  3. Your transforms.conf is incomplete for Index-time extractions. You should have FORMAT and WRITE_META fields
  4. You do not have anything in fields.conf which is required for Index-time extractions.

Look at this older question as a reference: https://answers.splunk.com/answers/103668/index-time-fields-with-heavy-forwarder.html

vr2312
Contributor

@rjthibod

I will push the fields.conf only to Search Heads ?

0 Karma

rjthibod
Champion

Yes, I believe so.

0 Karma

vr2312
Contributor

It worked without the fields.conf being pushed to the Search Heads.

the following were pushed to the HF's

fields.conf
transforms.conf
props.conf

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Since you're using a Heavy forwarder in between, the event parsing will happen on the Heavy forwarder and these configurations should get deployed to Heavy forwarder not Indexers. Also, you would need to update fields.conf on the Search Head. For complete details on steps, see this link.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Configureindex-timefieldextraction#Where_to_p...

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...