topic Routing Data to specific index based on filename in Getting Data In

Routing Data to specific index based on filename

mookiie2005 — Mon, 28 Sep 2020 17:04:33 GMT

We need to route data to specific indexes based on the file names being monitored. We are trying to get the data from the file(s) /app_qa/vodn_01/logfiles/current/bipc9996.stdout to route to the index named "RATABASE_SIT2" With our current settings all of this data is going to index "main" We have several other files that need to be routed accordingly, we are just looking for a good example we can use.

inputs.conf:
[monitor:///app_qa/vodn_01/logfiles/current/.]
followSymlink=true
crcSalt =

props:
[source::bipc9996.stdout]
TRANSFORMS-routing_SIT2_bipc9996 = index_redirect_to_RATABASE_SIT2_bipc9996

transforms:
[index_redirect_to_RATABASE_SIT2_bipc9996]
REGEX = bipc9996.stdout
sourcetype = bipc9996_SIT2
DEST_KEY = _MetaData:Index
FORMAT = RATABASE_SIT2

we assume these files (props/transforms) should be in the etc/app/local folder on the deployment server/forwarder. We have also deployed these changes to the indexer. Is this setup incorrect?

Re: Routing Data to specific index based on filename

Ayn — Tue, 15 Jul 2014 17:15:52 GMT

Easiest thing would be to set the correct index directly in inputs.conf on the forwarder.

[monitor:///app_qa/vodn_01/logfiles/current/bipc9996.stdout]
index = RATABASE_SIT2
...

Otherwise if you don't want that for some reason, this can be done almost like what you've pasted but NOT on the forwarders - it needs to go on the full Splunk instance that performs parsing of the logs, which is often your indexers.

props.conf:

[source::/app_qa/vodn_01/logfiles/current/bipc9996.stdout]
TRANSFORMS-routing_SIT2_bipc9996 = index_redirect_to_RATABASE_SIT2_bipc9996

transforms.conf:

[index_redirect_to_RATABASE_SIT2_bipc9996]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = RATABASE_SIT2

As you can see I'm simply using "." as regex - this is because the source matching was already performed in props.conf, so no further matching should be needed.

Re: Routing Data to specific index based on filename

strive — Tue, 15 Jul 2014 17:22:49 GMT

are you using heavy forwarder or light forwarder?

Re: Routing Data to specific index based on filename

strive — Tue, 15 Jul 2014 17:24:22 GMT

I assume the issue is with source. The source that you have set in props.conf might not be right

You need to set sourcetype in your inputs.conf file. Something like

[monitor:///app_qa/vodn_01/logfiles/current/bipc999*]  
followSymlink=true  
crcSalt = <source>  
sourcetype = bipc999

props.conf can contain

[bipc999]  
TRANSFORMS-routing_SIT2_bipc9996 = index_redirect_to_RATABASE_SIT2_bipc9996

It depends on what kind of forwarder you are using. If you use light forwarder then props and transforms should be on indexer.

Re: Routing Data to specific index based on filename

strive — Tue, 15 Jul 2014 17:25:17 GMT

Splunk doesn't recommend overriding source, hence i work with sourcetypes mostly 🙂

http://docs.splunk.com/Documentation/Splunk/6.1.2/admin/Inputsconf

Re: Routing Data to specific index based on filename

mookiie2005 — Mon, 28 Sep 2020 17:07:13 GMT

The path did not quite work for me, I ended up have to use this:

[source::/app_qa/vodn_01/logfiles/.../bipc9996.stdout]
I found that the current directory was a symbolic link and this was the only way to get it to index the log. If I used the below it would not work correctly:
[source::/app_qa/vodn_01/logfiles/*/bipc9996.stdout]