Getting Data In

How to edit my transforms.conf in order to set the sourcetype for each syslog being forwarded?

Explorer

Hello,

I have a customer sending three different kind of logs via syslog. I am pulling the logs off of a network feed where I had him point the syslogs to. It's listening on port xx514. The logs look like this:

SplunkSystem: 2017-03-20 15:00:06 servername NTP: Synchronized clock via NTP: Successfully slewed time
SplunkActivity: 2017-03-20 15:00:06 servername NTP: Synchronized clock via NTP: Successfully slewed time
SplunkAudit: 2017-03-20 15:00:06 servername NTP: Synchronized clock via NTP: Successfully slewed time

I have the syslogs going to a temp sourcetype then I am grabbing that and using a transforms to set the sourcetype for each syslog.

My inputs:

[udp://xx514]   
index=ABC
sourcetype=abc_temp_syslog
connection_host = dns
disabled = 0

My Props:

[abc_temp_syslog]
KV_MODE = auto
ANNOTATE_PUNCT = false
TIME_PREFIX = ^
TIME_FORMAT = %m $d %H:%M:%S
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false
TRANSFORMS-set1= changesourcetypetosystem
TRANSFORMS-set2= changesourcetypetoaudit
TRANSFORMS-set3= changesourcetypetoactivity

[abc_system_syslog]
KV_MODE = auto
ANNOTATE_PUNCT = false
TIME_PREFIX = ^
TIME_FORMAT = %m $d %H:%M:%S
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false

[abc_audit_syslog]
KV_MODE = auto
ANNOTATE_PUNCT = false
TIME_PREFIX = ^
TIME_FORMAT = %m $d %H:%M:%S
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false

[abc_activity_syslog]
KV_MODE = auto
ANNOTATE_PUNCT = false
TIME_PREFIX = ^
TIME_FORMAT = %m $d %H:%M:%S
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false

My Transforms.conf

[changesourcetypetoactivity]
REGEX = SplunkActivity
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::abc_activity_syslog

[changesourcetypetosystem]
REGEX = SplunkSystem
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::abc_system_syslog

[changesourcetypetoaudit]
REGEX = SplunkAudit
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::abc_audit_syslog

So what Splunk is doing is putting all three syslogs into the temp_syslog and is ignoring the transforms ... which leads me to believe that it is my transforms.conf that has some kind of error.

Thanks for the help!!

0 Karma
1 Solution

Ultra Champion

I'd try debugging to see which transform is not working by removing all and adding back one at a time. Easier to debug one than all three. Also, I think you can collapse them:

 TRANSFORMS-setST= changesourcetypetosystem, changesourcetypetoaudit, changesourcetypetoactivity

For reference, we're talking about this topic: http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

I'd also try regex like:

REGEX = ^SplunkActivity\:.*

That should look for it at the start of the event only.

View solution in original post

0 Karma

Ultra Champion

I'd try debugging to see which transform is not working by removing all and adding back one at a time. Easier to debug one than all three. Also, I think you can collapse them:

 TRANSFORMS-setST= changesourcetypetosystem, changesourcetypetoaudit, changesourcetypetoactivity

For reference, we're talking about this topic: http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

I'd also try regex like:

REGEX = ^SplunkActivity\:.*

That should look for it at the start of the event only.

View solution in original post

0 Karma

Explorer

Thanks to you both! It's working as intended.

Just updated the props to the following:

[abctempsyslog]
KVMODE = auto
ANNOTATE
PUNCT = false
TIMEPREFIX = ^
TIME
FORMAT = %m $d %H:%M:%S
LINEBREAKER = ([\r\n]+)
MAX
TIMESTAMPLOOKAHEAD = 25
SHOULD
LINEMERGE = false
TRANSFORMS-set1= changesourcetypetosystem, changesourcetypetoaudit, changesourcetypetoactivity

...and used this REGEX in the transforms for the corresponding messages:
REGEX = SplunkSystem
REGEX = SplunkActivity
REGEX = SplunkAudit

Thanks Again Burch and lguinn.

0 Karma

Ultra Champion

I would recommend reviewing which of the props you are using default values for AND perhaps remove those and let it infer from the default. It makes it easier to maintain later so you know you didn't actually care that KV_MODE = auto (for example) and that you were just using the defaults. (Check the props.conf.spec file to see that many of them are using the defaults)

Did you really want ANNOTATE_PUNCT turned off? With what I know of your environment, I'm surprised to see such an advanced setting used.

0 Karma

Legend

You don't need the .* at the end of the regex, but I like the rest of it. Should match faster and more accurately.

0 Karma

Ultra Champion

I was thinking the same.

0 Karma

Legend

This probably won't solve the problem, but your timeformat is completely off. It is shown as

 TIME_FORMAT = %m $d %H:%M:%S

should be

 TIME_FORMAT = %Y-%m-%d %H:%M:%S

Plus, you should eliminate the TIME_PREFIX, as it is doing nothing. The LINEBREAKER should also be removed.

All the parsing happens before the transforms. So the settings for the initial sourcetype will be the only setting in effect for parsing: the TIME_FORMAT, etc. does not need to be set on every sourcetype.

Finally, why not set the "default" sourcetype to one of the three, and then you would only need to transform the other two. For example, if you defaulted the sourcetype to SplunkSystem, then you would only need to transform the SplunkActivity and SplunkAudit events. This could be significantly more efficient.

Explorer

Lguinn,

The syslog has this in front of what i posted above:

Mar 21 02:00:07 Mar 21 02:00:07 SplunkSystem: 2017-03-20 15:00:06 servername NTP: Synchronized clock via NTP: Successfully slewed time

That is why I have the time format like it is. I did have it like you mentioned, but had no luck getting the transforms to work.

I'll look into setting up to one default like you mentioned.

Thanks

0 Karma

Legend

Then your time settings should be

TIME_FORMAT = %m %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD=19

You might actually get away with eliminating the TIMEFORMAT altogether. But you really do need the MAXTIMESTAMP_LOOKAHEAD because your events appear to have multiple timestamps.

0 Karma