Change index at index time (chained UFs)

[danbrook]

I am collecting Windows Event Logs via either a Windows Event Collector or directly from each Windows server (with a Splunk UF installed). I am then using an intermediate forwarder to send to our indexers. This is due to network setup and location of the servers.

I have custom metadata that the intermediate forwarder is adding to identify where the logs are coming from and use in the index name.

The problem I am having is that the Windows Event Logs are hitting the Main index and not the one I've specified in transforms.

On the intermediate forwarder I have:

inputs.conf
[splunktcp:9997]
acceptFrom = *

On the destination Indexer I have:

props.conf
[source::WinEventLog:ForwardedEvents]
TRANSFORMS-Host = Set-Host-ComputerName
TRANSFORMS-LogName = Set-Sourcetype-LogName
TRANSFORMS-Index-Windows = Set-Index-Windows

[sourcetype::WinEventLog:*]
TRANSFORMS-Index-Windows = Set-Index-Windows

...

transforms.conf
[Set-Host-ComputerName]
REGEX = (?m)ComputerName=(.+)?\b
DEST_KEY = MetaData:Host
FORMAT = host::$1

[Set-Sourcetype-LogName]
REGEX = (?m)LogName=(.+)?\b
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::windows:event:$1

[Set-Index-Windows]
SOURCE_KEY = MetaData:customfield
REGEX = (.+)
DEST_KEY = _MetaData:Index
FORMAT = index-$1-win-event

[hardikJsheth]

If I understood your problem correctly, you want to put data into index named index-$custom$-win-event, I think you need to update your tranforms.conf stanza as follows:

[Set-Index-Windows]
 SOURCE_KEY = field:customfield
 REGEX = (.+)
 DEST_KEY = _MetaData:Index
 FORMAT = index-$1-win-event

[danbrook]

The custom field is set like this on the intermediate forwarder:

[default]
_meta = customfield::ACustomString

[micahkemp]

If the custom field is set on the intermediate forwarder, I have to assume the intermediate forwarder is a heavy forwarder. If that's the case, it's sending cooked data to your indexer. The indexer won't re-cook data (under normal configuration, and best practice), so changing sourcetype/index/etc won't have any affect.

For a potential (against best practice) workaround, see this previous answer:

Reparsing cooked data coming from a heavy forwarder. Possible?

[danbrook]

The Intermediate Forward is a Universal Forwarder. The custom metadata is being added to the logs.

[danbrook]

SOURCE_KEY = field:customfield - This didn't work I'm afraid.