Sending Single Input To Multiple Indexes: Works Sometimes

[jpass]

• I have a scripted input with events that I want to send to different indexes based on a string within the event.
• I do this for easier access control.
• I control the event format via the script and so have created an easy-to-regex string at the bottom of each event which Splunk should use to route the event.
• It works but only sometimes. I'll import 10 events and some go to the default index and some go to the routing index.
• These are multi-line events that are usually about 20 to 50 lines but sometimes up to a few hundred lines (rare)
• I'm wondering what I'm doing wrong.

Here's my .conf entries

inputs.conf (in a custom application directory /splunk_home/etc/apps/myapp/local/inputs.conf)

[script:///splunk_home/bin/scripts/get_test_multi.pl]
disabled = 0
interval = 60000
source = get_test_multi
sourcetype = test_multi
index=test_index

props.conf (in /splunk_home/etc/system/local/props.conf)

[test_multi]
TRANSFORMS-index=test_index_router
EXTRACT-test_checker = splunk_index :: (?<test_index_field>.*?)\n

transforms.conf (in /splunk_home/etc/system/local/transforms.conf)

[test_index_router]
REGEX = splunk_index :: (?<my_splunk_index>.*?)\n
FORMAT = $1
DEST_KEY = _MetaData:Index

In props.conf you'll notice I extract a field called 'test_index_field' which is the same as the regex I'm using to route the index. I do this just to confirm that I'm correctly extracting the value which works correctly 100% of the time.

• The events are sent to the correct index only some of the time
• Events are either routed correctly or sent to the value I designate in inputs.conf for index
• I've tried removing the entry for index in inputs.conf with the same results only it sends the events to index=main (default)

[jacobwilkins]

You might have some whitespace issues with your regex.

Try this:

REGEX = splunk_index :: (\w*)

The concern is that you may be including training whitespace in the capture of your index name, which would be tricksy to debug.

You might also want to try this:

[source::get_test_multi]
TRANSFORMS-index=test_index_router