create multiple sourcetypes from single syslog source

[Esky73]

Hi I'm looking to create events for syslog data from a wireless controller - and the syslog data also contains data from the AP's which is what i'm more interested in.

If i use the generic syslog sourcetype and fix the timestamp then i get all the required fields broken out for the controllers but not all the required ones for the AP's.

how best to create multiple sourcetypes in this instance ? or is there another way ?

thanks.

Blockquote

3/31/17
9:39:06.000 AM

2017-03-31 09:39:06 Local5.Notice X.X.X.X X.X.X.X stm[1234]: <1234> |APName@X.X.X.X stm| Deauth from sta: MACAddy: AP X.X.X.X-MACAddy-APName Reason STA has roamed to another AP

3/31/17
9:39:06.000 AM

2017-03-31 09:39:06 Local5.Notice X.X.X.X MC-Name stm[1234]: <1234> <1234> Assoc success @ 09:39:06.770727: MACAddy: AP X.X.X.X-MACAddy-APName

Blockquote

[beatus · Accepted solution]

Esky73,
You can accomplish this through the use of props / transforms. The general idea of what needs to be done is:
props.conf:
Create a TRANSFORMS- entry under the stanza "[syslog]" that calls a transforms.conf entry that contains a regex to re-sourcetype your events.
transforms.conf:
Create a stanza that's refrenced in props.conf containing a REGEX to match your events, DEST_KEY to set the destination of "FORMAT" to MetaData:Sourcetype and a FORMAT that sets the sourcetype to the desired new sourcetype. See example below.

For this data, my best guess would be something like this:
props.conf:

[syslog]
TRANSFORMS-rewrite_wireless_controller_st = rewrite_wireless_controller_st

transforms.conf:

[rewrite_wireless_controller_st]
REGEX = \S+\s+stm[\d+]:\s+
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::my_new_st

This should re-sourcetype your logs to "my_new_st". Obliviously change that to whatever makes sense for your environment. The last thing to consider is that these configurations need to exist on the first full Splunk instance that sees this data. In a lot of environments this is the indexer but if you have Heavy Forwarders, then this configuration needs to be there.

Lastly, if you need to further split this, you have the technique, just use additional transforms entries with regular expressions that fit the specific subset of data.

Hope this helps!

[StephenD1]

Sorry to resurrect this thread but I have a question about your last paragraph: when you say "...just use additional transforms entries with regular expressions that fit the specific subset of data..." does this mean that if I want to further extract fields from the new sourcetype=my_new_st, for example, I have to do that using TRANSFORMS? In other words, would I be able to put a new stanza further down in the props.conf for

```

[my_new_st]

...

```

then start using additional REPORTs or EXTRACTs that only apply to that new sourcetype?

[JohnEGones]

@StephenD1 

Start a new thread, this is a better practice than trying to resurrect a post that has already been answered. You can then reference this previous response.

[StephenD1]

will do, thanks

[Esky73]

Thanks for your help

** EDIT **

After testing this on a file snippet i took from a live syslog system in which it worked - i then applied the same config to my live environment only for it not to work and not understanding why.

Then i saw the following in the splunk docs :

http://docs.splunk.com/Documentation/Splunk/latest/Data/Bypassautomaticsourcetypeassignment

"Overrides only work on file and directory monitoring inputs or files you have uploaded. You cannot override the source type on network inputs. Additionally, overrides only affect new data that arrives after you set up the override. To correct the source types of events that have already been indexed, create a tag for the source type instead."

which seems to suggest this will not work in my instance as we are using a network input for syslog ?

[Esky73]

OK So it does work 🙂

My regex needed to be slightly different in my live env and it seems if one regex is incorrect in the transforms.conf it also stops the other sourcetype overrides from working.

hope this helps someone

thanks again.