Getting Data In

create multiple sourcetypes from single syslog source

Esky73
Builder

Hi I'm looking to create events for syslog data from a wireless controller - and the syslog data also contains data from the AP's which is what i'm more interested in.

If i use the generic syslog sourcetype and fix the timestamp then i get all the required fields broken out for the controllers but not all the required ones for the AP's.

how best to create multiple sourcetypes in this instance ? or is there another way ?

thanks.

Blockquote

3/31/17
9:39:06.000 AM

2017-03-31 09:39:06 Local5.Notice X.X.X.X X.X.X.X stm[1234]: <1234> |APName@X.X.X.X stm| Deauth from sta: MACAddy: AP X.X.X.X-MACAddy-APName Reason STA has roamed to another AP

3/31/17
9:39:06.000 AM

2017-03-31 09:39:06 Local5.Notice X.X.X.X MC-Name stm[1234]: <1234> <1234> Assoc success @ 09:39:06.770727: MACAddy: AP X.X.X.X-MACAddy-APName

Blockquote

Tags (1)
0 Karma
1 Solution

beatus
Communicator

Esky73,
You can accomplish this through the use of props / transforms. The general idea of what needs to be done is:
props.conf:
Create a TRANSFORMS- entry under the stanza "[syslog]" that calls a transforms.conf entry that contains a regex to re-sourcetype your events.
transforms.conf:
Create a stanza that's refrenced in props.conf containing a REGEX to match your events, DEST_KEY to set the destination of "FORMAT" to MetaData:Sourcetype and a FORMAT that sets the sourcetype to the desired new sourcetype. See example below.

For this data, my best guess would be something like this:
props.conf:

[syslog]
TRANSFORMS-rewrite_wireless_controller_st = rewrite_wireless_controller_st

transforms.conf:

[rewrite_wireless_controller_st]
REGEX = \S+\s+stm[\d+]:\s+
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::my_new_st

This should re-sourcetype your logs to "my_new_st". Obliviously change that to whatever makes sense for your environment. The last thing to consider is that these configurations need to exist on the first full Splunk instance that sees this data. In a lot of environments this is the indexer but if you have Heavy Forwarders, then this configuration needs to be there.

Lastly, if you need to further split this, you have the technique, just use additional transforms entries with regular expressions that fit the specific subset of data.

Hope this helps!

View solution in original post

beatus
Communicator

Esky73,
You can accomplish this through the use of props / transforms. The general idea of what needs to be done is:
props.conf:
Create a TRANSFORMS- entry under the stanza "[syslog]" that calls a transforms.conf entry that contains a regex to re-sourcetype your events.
transforms.conf:
Create a stanza that's refrenced in props.conf containing a REGEX to match your events, DEST_KEY to set the destination of "FORMAT" to MetaData:Sourcetype and a FORMAT that sets the sourcetype to the desired new sourcetype. See example below.

For this data, my best guess would be something like this:
props.conf:

[syslog]
TRANSFORMS-rewrite_wireless_controller_st = rewrite_wireless_controller_st

transforms.conf:

[rewrite_wireless_controller_st]
REGEX = \S+\s+stm[\d+]:\s+
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::my_new_st

This should re-sourcetype your logs to "my_new_st". Obliviously change that to whatever makes sense for your environment. The last thing to consider is that these configurations need to exist on the first full Splunk instance that sees this data. In a lot of environments this is the indexer but if you have Heavy Forwarders, then this configuration needs to be there.

Lastly, if you need to further split this, you have the technique, just use additional transforms entries with regular expressions that fit the specific subset of data.

Hope this helps!

Esky73
Builder

Thanks for your help

** EDIT **

After testing this on a file snippet i took from a live syslog system in which it worked - i then applied the same config to my live environment only for it not to work and not understanding why.

Then i saw the following in the splunk docs :

http://docs.splunk.com/Documentation/Splunk/latest/Data/Bypassautomaticsourcetypeassignment

"Overrides only work on file and directory monitoring inputs or files you have uploaded. You cannot override the source type on network inputs. Additionally, overrides only affect new data that arrives after you set up the override. To correct the source types of events that have already been indexed, create a tag for the source type instead."

which seems to suggest this will not work in my instance as we are using a network input for syslog ?

0 Karma

Esky73
Builder

OK So it does work 🙂

My regex needed to be slightly different in my live env and it seems if one regex is incorrect in the transforms.conf it also stops the other sourcetype overrides from working.

hope this helps someone

thanks again.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...