How do I split UDP:514 traffic for VMware and Cisco so they both have their respective indexes

[pdherna1]

Scenario: (DEV environment)

I have one indexer and one universal forwarder. I have both the indexer and the forwarder listening on 514:UPD. I would like to be able to split all UDP:514 traffic for VMware syslogs and Cisco network devices syslogs to their own respective index. For example, VMware syslogs would go to VM_index and Cisco network devices would go to net_index.

Both types of syslogs (VMware and Cisco) could be pointed to either the indexer or forwarder directly (dependendant upon location).

I'm not sure if I need to use Route, Filter, Regex in combination with Props.conf and Transforms.conf.

Any help with this scenario would greatly be appreciated. Examples welcomed 🙂

[mcronkrite]

Approach 1:
Set each system to use a different udp port, then setup your config to have an distinct index/sourcetype easily, just change inputs.conf to use the ports you are sending syslog to. This example, udp:1514, and udp:2514.

Approach 2
Set up a genric udp listener on 514 or 5514 (if 514 is reserved) and change the index and sourcetype based on the incoming ip address, for this change props and transforms to set the index/sourcetype by regex.

Both approaches are encompassed below in the following inputs,props,transforms.conf by utilizing a listener on the ports, and transforms to set the index/sourcetype based on host.

Setup inputs.conf

    # these are the default index/sourcetype that will be assigned to incoming data from 514, UNLESS there is an override specified in the props and transforms
    [udp://514]
     connection_host = ip 
    # connection_host = dns 
    # connection_host = none
      #approach 2  see props and transforms

    [udp://1514]
    # approach 1 and done
    disabled = 0
    source = udp:1514
    sourcetype = cisco
    index = my_cisco

    [udp://2514]
     # approach 1 and done
    disabled = 0
    source = udp:2514
    sourcetype = vmware
    index = my_vmare

   [udp://5514]
# 5515 can be used when splunk is not running as a sudoer since 514 is reserved port.
   connection_host = ip 
    # connection_host = dns 
    # connection_host = none
       # approach 2 see props and transforms

Set your props.conf like this

    # Props.conf
    [source::udp]
    NO_BINARY_CHECK = true
    SHOULD_LINEMERGE = false
    TRUNCATE = false
    MAX_TIMESTAMP_LOOKAHEAD = 20
    TIME_FORMAT = %b %d %H:%M:%S
    TZ = America/Chicago
   # set the host as from the incoming udp packet
    TRANSFORMS-gethostfromdata = set_host

    [source::udp:514]
    # set two transforms for index and sourcetype for each device coming in from this port
   #Approach 2, because we haven't separated the ports for cisco and vmware, we need to apply transforms to change to the right index/sourcetype
   # cisco
    TRANSFORMS-get-cisco-index = set_index_cisco
    TRANSFORMS-get-cisco-sourcetype = set_sourcetype_cisco
    # vmware
    TRANSFORMS-get-vmware-index = set_index_vmware
    TRANSFORMS-get-vmware-sourcetype = set_sourcetype_vmware

    # drop non-relavant events to nullqueue
    TRANSFORMS-drop-events-nullq = drop_events_nullq

    TRANSFORMS-get-syslog-index = set_index_generic_syslog
    TRANSFORMS-get-syslog-sourcetype = set_sourcetype_generic_syslog

    [source::udp:1514]
    # cisco
    TRANSFORMS-get-cisco-index = set_index_cisco
    TRANSFORMS-get-cisco-sourcetype = set_sourcetype_cisco

    # drop non-relavant events to nullqueue
    TRANSFORMS-drop-events-nullq = drop_events_nullq

    [source::udp:2514]
    # vmware
    TRANSFORMS-get-vmware-index = set_index_vmware
    TRANSFORMS-get-vmware-sourcetype = set_sourcetype_vmware

    # drop non-relavant events to nullqueue
    TRANSFORMS-drop-events-nullq = drop_events_nullq

    [source::udp:5514]
    # rerouted from 514 network data, because 514 is usually reserved port

    TRANSFORMS-get-cisco-index = set_index_cisco
    TRANSFORMS-get-cisco-sourcetype = set_sourcetype_cisco
    # vmware
    TRANSFORMS-get-vmware-index = set_index_vmware
    TRANSFORMS-get-vmware-sourcetype = set_sourcetype_vmware

    TRANSFORMS-get-syslog-index = set_index_generic_syslog
    TRANSFORMS-get-syslog-sourcetype = set_sourcetype_generic_syslog

    # drop non-relavant events to nullqueue
    TRANSFORMS-drop-events-nullq = drop_events_nullq

Setup transforms.conf

# Transforms.conf
[set_host]
#grab the first IP address and use that as the real host (syslog)
DEST_KEY = MetaData:Host
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s
FORMAT = host::$1

[TrashComments]
# drop lines that start with #
REGEX    = ^\s*#
DEST_KEY = queue
FORMAT   = nullQueue

# Set cisco BY incoming IP address by using REGEX
[set_sourcetype_cisco]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(10\.000\.00\.00)[\w\.\-]*\]?\s
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::cisco

[set_index_cisco]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(10\.000\.00\.00)[\w\.\-]*\]?\s
DEST_KEY = _MetaData:Index
FORMAT = index::my_cisco

[set_sourcetype_syslog]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(10\.000\.00\.00)[\w\.\-]*\]?\s
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::syslog

[set_index_syslog]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(10\.000\.00\.00)[\w\.\-]*\]?\s
DEST_KEY = _MetaData:Index
FORMAT = index::my_syslog

Summary:
Change inputs.conf, props.conf, and transforms.conf to collect UDP on multiple ports (on per device type) or on a single port that has to then split the traffic up incoming connection ip/name

[Damien_Dallimor]

Send everything directly to the Indexer.
Then set the "index" based upon some pattern in the syslog events.

Have a look at this answer as a guide :

http://splunk-base.splunk.com/answers/60972/split-syslog-input-into-multiple-indexes

[yannK]

To resume :
- if you have the choice, specify an index per input. (like a listening port specific for each index)
- if you can, use a syslog-server to receive the udp data and write to file (in one folder per sender) , then have splunk forwarder monitor those folders and apply the correct index.

If none of this is possible, you can at index time use a transforms (on the indexers or on the heavy forwarder if any) to route each event to the correct index. This has a cost and may impact your index speed.

see this guide and example http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setupmultipleindexes#Route_specific_event...

to configure the transforms, It comes back to this question : How do you distinguish which event should go in which index ?
for syslog : it only can be the host, or the content of the event

[pdherna1]

i'm using Splunk for syslog daemon.

[a212830]

What do you use for a syslog daemon? Rsyslog allows you to route messages to their own folder/file, and you can then process them in that manner.

[pdherna1]

Neither of the 1st two options is possible. We will be pointing everything to splunk...no "middle man".

I appreciate the link. Looks very helpful.

I'm trying to figure out how to distinguish between VMware and Cisco data. Using the host is an option. But I was hoping that others may already be using Splunk to gather syslogs from different vendors (e.g. VMware/Cisco) and would share their regex string. Btw, I have no experience using regex (so any help would be greatly appreciated).

[yannK]

How do you distinguish which one should go in which index ?

The easiest solution is use a different port per input (UDP 515 for vmware, UDP 516 for cisco, UDP 514 for regular syslog to main)

Or to use 2 forwarders with different IP's and a different index destination for the udp 514 input.

[pdherna1]

I want to make sure I'm understanding you correctly. Are you saying that I would have to configure the VMware host to use port 515, the cisco devices to use 516, etc., correct?

What if changing the default port, 514, is not an option? What are some other solutions?

[mcronkrite]

check out the answer i posted - hope this helps