Getting Data In

rsyslog forwarding without header

Communicator

hi ninjas

im currently dealing with some logs beeing forwarded over syslog to a 3rd party system. The question here is wheter there is an option to prevent splunk adding an additonal header to each message before its getting forwarded. So there should be a way to disable the additonal syslog header when using forwarding so the 3rd party system reveives the original raw message.

Any ideas?

0 Karma
1 Solution

Communicator

OK thanks to a collegue I just found the answer for my own question:
Apparantely, splunk adds a syslog header for every sourcetype not specified in "syslogSourceType" within outptus.conf. So referring to the docs of outputs.conf:

syslogSourceType =

  • Specifies an additional rule for handling data, in addition to that provided by the 'syslog' source type.
  • This string is used as a substring match against the sourcetype key. For example, if the string is set to 'syslog', then all source types containing the string 'syslog' will receive this special treatment.
  • To match a source type explicitly, use the pattern "sourcetype::sourcetype_name".
    • Example: syslogSourceType = sourcetype::apache_common
  • Data which is 'syslog' or matches this setting is assumed to already be in syslog format.
  • Data which does not match the rules has a header, optionally a timestamp (if defined in 'timestampformat'), and a hostname added to the front of the event. This is how Splunk causes arbitrary log data to match syslog expectations.
  • Defaults to unset.

So the solution in my case was adding the common part (as it is used as a pattern) of the forwarderd sourcetypes to the syslogSourceType stanza and restart splunkd.

View solution in original post

0 Karma

Communicator

OK thanks to a collegue I just found the answer for my own question:
Apparantely, splunk adds a syslog header for every sourcetype not specified in "syslogSourceType" within outptus.conf. So referring to the docs of outputs.conf:

syslogSourceType =

  • Specifies an additional rule for handling data, in addition to that provided by the 'syslog' source type.
  • This string is used as a substring match against the sourcetype key. For example, if the string is set to 'syslog', then all source types containing the string 'syslog' will receive this special treatment.
  • To match a source type explicitly, use the pattern "sourcetype::sourcetype_name".
    • Example: syslogSourceType = sourcetype::apache_common
  • Data which is 'syslog' or matches this setting is assumed to already be in syslog format.
  • Data which does not match the rules has a header, optionally a timestamp (if defined in 'timestampformat'), and a hostname added to the front of the event. This is how Splunk causes arbitrary log data to match syslog expectations.
  • Defaults to unset.

So the solution in my case was adding the common part (as it is used as a pattern) of the forwarderd sourcetypes to the syslogSourceType stanza and restart splunkd.

View solution in original post

0 Karma

Super Champion

Assuming you are forwarding from a Splunk HF/Indexers directly to a 3rd party system (not using a syslog server)?
Ensure timestampformat variables in outputs.conf are unset. Splunk won't put any other extra headers other timestampformat

The reason you may be experiencing headers is, these headers will be already in the event payload. Then you need to customize event itself .If you want to customize the events (eg strip first 3 words), use props & transforms

# props.conf
[my_strippable_data]
TRANSFORMS-strip_me=my_stripped_sourcetype

# transforms.conf
[my_stripped_sourcetype]
REGEX=<your_regex>
DEST_KEY=_SYSLOG_ROUTING
FORMAT=strippedGroup


# outputs.conf
[syslog]
defaultGroup=everythingElseGroup
[syslog:strippedGroup]
server = 10.1.1.197:9997
disabled = false
priority = <34>

http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Outputsconf
http://docs.splunk.com/Documentation/SplunkCloud/6.6.1/Forwarding/Forwarddatatothird-partysystemsd

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!