Splunk Search

Sending Single Input To Multiple Indexes: Works Sometimes

jpass
Contributor
  • I have a scripted input with events that I want to send to different indexes based on a string within the event.
  • I do this for easier access control.
  • I control the event format via the script and so have created an easy-to-regex string at the bottom of each event which Splunk should use to route the event.
  • It works but only sometimes. I'll import 10 events and some go to the default index and some go to the routing index.
  • These are multi-line events that are usually about 20 to 50 lines but sometimes up to a few hundred lines (rare)
  • I'm wondering what I'm doing wrong.

Here's my .conf entries

inputs.conf (in a custom application directory /splunk_home/etc/apps/myapp/local/inputs.conf)

[script:///splunk_home/bin/scripts/get_test_multi.pl]
disabled = 0
interval = 60000
source = get_test_multi
sourcetype = test_multi
index=test_index

props.conf (in /splunk_home/etc/system/local/props.conf)

[test_multi]
TRANSFORMS-index=test_index_router
EXTRACT-test_checker = splunk_index :: (?<test_index_field>.*?)\n

transforms.conf (in /splunk_home/etc/system/local/transforms.conf)

[test_index_router]
REGEX = splunk_index :: (?<my_splunk_index>.*?)\n
FORMAT = $1
DEST_KEY = _MetaData:Index

In props.conf you'll notice I extract a field called 'test_index_field' which is the same as the regex I'm using to route the index. I do this just to confirm that I'm correctly extracting the value which works correctly 100% of the time.

  • The events are sent to the correct index only some of the time
  • Events are either routed correctly or sent to the value I designate in inputs.conf for index
  • I've tried removing the entry for index in inputs.conf with the same results only it sends the events to index=main (default)

jacobwilkins
Communicator

You might have some whitespace issues with your regex.

Try this:

REGEX = splunk_index :: (\w*)

The concern is that you may be including training whitespace in the capture of your index name, which would be tricksy to debug.

You might also want to try this:

[source::get_test_multi]
TRANSFORMS-index=test_index_router
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...