topic Re: simple index segregation ? in Getting Data In

simple index segregation ?

Vtesse — Wed, 17 Apr 2013 13:35:02 GMT

Hi.

I'm trying to find a quick and simple way to separate my incoming cisco syslogs into different indexes. For complicated, and dull reasons we can only really use the udp:514 listener.

The best I've come up with so far is defining a number of different transforms in the props.conf, then use regex to define which index they go to. unfortunately my splunk and regex skills are currently rather weak and it's refusing to work.

So far I've got to here:
~~ props.conf ~~

[syslog]
TRANSFORMS-routing = generic-routing, MPLS-routing, Office-routing

~~ transforms.conf ~~

[MPLS-routing]
SOURCE_KEY = MetaData:Host
REGEX = 10\.10\.\d+
DEST_KEY = _MetaData:Index
FORMAT = MPLS

[Office-routing]
SOURCE_KEY = MetaData:Host
REGEX = 10\.100\.\d+
DEST_KEY = _MetaData:Index
FORMAT = office

[generic-routing]
DEST_KEY = _MetaData:Index
FORMAT = syslog

Any tips or thoughts?

Re: simple index segregation ?

kristian_kolb — Wed, 17 Apr 2013 13:54:33 GMT

One thing that could mess things up, and possibly be the source of your problems is that incoming syslog (i.e. with the sourcetype set to syslog) data goes through a TRANSFORM for setting the host value based on the content of the events. Have a look at the $SPLUNK_HOME/etc/system/default/props.conf and transforms.conf to see how this works.

If this happens before your TRANSFORMS the host would not be there, or rather may not have the values you expect.

You can skip the SOURCE_KEY setting (default is the _raw event) and create your regexes to match a timestamp followed by 10.10.* or 10.100.* etc.

Also, you may need to rename the the transforms so they use underscores instead of hyphens;

generic_routing
MPLS_routing
Office_routing

Also, you may have to specify a REGEX = . for the [generic_routing]. (see new spelling)

Hope this helps,

Kristian

Re: simple index segregation ?

Vtesse — Wed, 17 Apr 2013 14:26:44 GMT

Would it make more sense to use [source::udp] in the props.conf instead of [syslog] ?

Re: simple index segregation ?

kristian_kolb — Wed, 17 Apr 2013 16:57:06 GMT

Try the bottom two things first. (REGEX = . and underscores)

Re: simple index segregation ?

Vtesse — Mon, 28 Sep 2020 13:46:28 GMT

apologies for not getting back to you, but I eventually got this working in a similar method to above.
I ended up going for a [source::udp:514] thingy in props.conf then regex'ing in transforms as we discussed.
the trick was, as usual, getting the regex to work. ended up with the regex below to match the basic date/timestamp format of "MMM DD HH:MM:SS I{P OR Hostname}". pretty simple stuff but invaluable nonetheless.

[office_routing]
REGEX = (\w+\ [0-9]{1,2}\ \d+:\d+:\d+ 10.100.[0-9]{1,3}.[0-9]{1,3})|(\w+.domain.ltd)
DEST_KEY = _MetaData:Index
FORMAT = office

And it works perfectly
thanks.