Having incoming TCP/UDP Traffic send to more than one index...

[balbano]

Is there someway to setup 1 TCP or UDP listening port and have it direct logs to more than one index depending on whitelist/blacklist?

For example, can you setup incoming logs from 1 listening port (e.g. TCP 514) to go to multiple indexes?

Let me know

Brian

[dikaye]

what is the SOURCE_KEY = MetaData:Host means?

"Host" means the logs source server's IP address or hostname?

and the DEST_KEY = _MetaData:Index means it routing to the specify index?

for example, I have a forwarder server and a receiver server, in the receiver server, I want the logs incoming from tcp 9997 route to different indexes, I did the configure like that:

props.conf

[source::tcp:9997]
TRANSFORMS-routing = custa_index

transforms.conf

[custa_index]
SOURCE_KEY = MetaData:Host
REGEX = 192.168.0.2
DEST_KEY = _MetaData:Index
FORMAT = custa

indexes.conf

[custa]
coldPath = /data/splunk/custA/colddb
homePath = /data/splunk/custA/db
maxDataSize = 10
thawedPath = /data/splunk/custA/thaweddb
maxHotBuckets = 2
maxWarmDBCount = 10
frozenTimePeriodInSecs = 188697600
maxHotIdleSecs = 86400

it is right?

[Paolo_Prigione · Accepted solution]

Hi, here's an untested example of the configurations you'll need to achieve your goal.

inputs.conf

[tcp://:514]
connection_host = dns
index = main

props.conf

[source::tcp:514]
TRANSFORMS-route = routeVC

transforms.conf

[routeVC]
SOURCE_KEY = MetaData:Host
REGEX = .+\.vc$
DEST_KEY = _MetaData:Index 
FORMAT = vc

Hope this helps, Paolo

[Simeon]

You can perform conditional routing where the destination key gets set to your specific index.

[balbano]

Hi gkanapathy,

I will research right away, but in the interests of time... can you provide a basic example of how the configuration would work? It would point me in the right direction. Basically whats happening right now is that traffic that is received on TCP 514 is sent to 'main' index. Can you give advice on the best approach on how to parse logs based on DNS Hostname? for example, if I have logs coming in via TCP 514 and I want all hosts with '*.vc ' DNS Extension to go to an index named 'vc'... how should I do it in the configs. Let me know when you can. Thanks for the help as always.

[gkanapathy]

It can be done where ever the parsing phase occurs, by setting the index key in a TRANSFORM. http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings%3F

[balbano]

How can this be done? Based on the documentation, looks like this is only possible for outgoing traffic from a forwarder to the indexer.

I'm interested in knowing if an indexer can take incoming log traffic from one port and route the log traffic to more than one index depending on hostname. An example or reference to the appropriate documentation would help if you have handy.

Let me know.

Thanks.

Brian