Getting Data In

Routing Data to specific index based on filename

Communicator

We need to route data to specific indexes based on the file names being monitored. We are trying to get the data from the file(s) /appqa/vodn01/logfiles/current/bipc9996.stdout to route to the index named "RATABASE_SIT2" With our current settings all of this data is going to index "main" We have several other files that need to be routed accordingly, we are just looking for a good example we can use.

inputs.conf:
[monitor:///appqa/vodn01/logfiles/current/.]
followSymlink=true
crcSalt =

props:
[source::bipc9996.stdout]
TRANSFORMS-routingSIT2bipc9996 = indexredirecttoRATABASESIT2_bipc9996

transforms:
[indexredirecttoRATABASESIT2bipc9996]
REGEX = bipc9996.stdout
sourcetype = bipc9996
SIT2
DESTKEY = _MetaData:Index
FORMAT = RATABASE
SIT2

we assume these files (props/transforms) should be in the etc/app/local folder on the deployment server/forwarder. We have also deployed these changes to the indexer. Is this setup incorrect?

0 Karma
1 Solution

Legend

Easiest thing would be to set the correct index directly in inputs.conf on the forwarder.

[monitor:///app_qa/vodn_01/logfiles/current/bipc9996.stdout]
index = RATABASE_SIT2
...

Otherwise if you don't want that for some reason, this can be done almost like what you've pasted but NOT on the forwarders - it needs to go on the full Splunk instance that performs parsing of the logs, which is often your indexers.

props.conf:

[source::/app_qa/vodn_01/logfiles/current/bipc9996.stdout]
TRANSFORMS-routing_SIT2_bipc9996 = index_redirect_to_RATABASE_SIT2_bipc9996

transforms.conf:

[index_redirect_to_RATABASE_SIT2_bipc9996]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = RATABASE_SIT2

As you can see I'm simply using "." as regex - this is because the source matching was already performed in props.conf, so no further matching should be needed.

View solution in original post

Influencer

I assume the issue is with source. The source that you have set in props.conf might not be right

You need to set sourcetype in your inputs.conf file. Something like

[monitor:///app_qa/vodn_01/logfiles/current/bipc999*]  
followSymlink=true  
crcSalt = <source>  
sourcetype = bipc999

props.conf can contain

[bipc999]  
TRANSFORMS-routing_SIT2_bipc9996 = index_redirect_to_RATABASE_SIT2_bipc9996

It depends on what kind of forwarder you are using. If you use light forwarder then props and transforms should be on indexer.

0 Karma

Influencer

Splunk doesn't recommend overriding source, hence i work with sourcetypes mostly 🙂

http://docs.splunk.com/Documentation/Splunk/6.1.2/admin/Inputsconf

0 Karma

Influencer

are you using heavy forwarder or light forwarder?

0 Karma

Legend

Easiest thing would be to set the correct index directly in inputs.conf on the forwarder.

[monitor:///app_qa/vodn_01/logfiles/current/bipc9996.stdout]
index = RATABASE_SIT2
...

Otherwise if you don't want that for some reason, this can be done almost like what you've pasted but NOT on the forwarders - it needs to go on the full Splunk instance that performs parsing of the logs, which is often your indexers.

props.conf:

[source::/app_qa/vodn_01/logfiles/current/bipc9996.stdout]
TRANSFORMS-routing_SIT2_bipc9996 = index_redirect_to_RATABASE_SIT2_bipc9996

transforms.conf:

[index_redirect_to_RATABASE_SIT2_bipc9996]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = RATABASE_SIT2

As you can see I'm simply using "." as regex - this is because the source matching was already performed in props.conf, so no further matching should be needed.

View solution in original post

Communicator

The path did not quite work for me, I ended up have to use this:

[source::/appqa/vodn01/logfiles/.../bipc9996.stdout]
I found that the current directory was a symbolic link and this was the only way to get it to index the log. If I used the below it would not work correctly:
[source::/appqa/vodn01/logfiles/*/bipc9996.stdout]

0 Karma