Getting Data In

rsyslog forwarding without header

claudio_manig
Communicator

hi ninjas

im currently dealing with some logs beeing forwarded over syslog to a 3rd party system. The question here is wheter there is an option to prevent splunk adding an additonal header to each message before its getting forwarded. So there should be a way to disable the additonal syslog header when using forwarding so the 3rd party system reveives the original raw message.

Any ideas?

0 Karma
1 Solution

claudio_manig
Communicator

OK thanks to a collegue I just found the answer for my own question:
Apparantely, splunk adds a syslog header for every sourcetype not specified in "syslogSourceType" within outptus.conf. So referring to the docs of outputs.conf:

syslogSourceType =

  • Specifies an additional rule for handling data, in addition to that provided by the 'syslog' source type.
  • This string is used as a substring match against the sourcetype key. For example, if the string is set to 'syslog', then all source types containing the string 'syslog' will receive this special treatment.
  • To match a source type explicitly, use the pattern "sourcetype::sourcetype_name".
    • Example: syslogSourceType = sourcetype::apache_common
  • Data which is 'syslog' or matches this setting is assumed to already be in syslog format.
  • Data which does not match the rules has a header, optionally a timestamp (if defined in 'timestampformat'), and a hostname added to the front of the event. This is how Splunk causes arbitrary log data to match syslog expectations.
  • Defaults to unset.

So the solution in my case was adding the common part (as it is used as a pattern) of the forwarderd sourcetypes to the syslogSourceType stanza and restart splunkd.

View solution in original post

claudio_manig
Communicator

OK thanks to a collegue I just found the answer for my own question:
Apparantely, splunk adds a syslog header for every sourcetype not specified in "syslogSourceType" within outptus.conf. So referring to the docs of outputs.conf:

syslogSourceType =

  • Specifies an additional rule for handling data, in addition to that provided by the 'syslog' source type.
  • This string is used as a substring match against the sourcetype key. For example, if the string is set to 'syslog', then all source types containing the string 'syslog' will receive this special treatment.
  • To match a source type explicitly, use the pattern "sourcetype::sourcetype_name".
    • Example: syslogSourceType = sourcetype::apache_common
  • Data which is 'syslog' or matches this setting is assumed to already be in syslog format.
  • Data which does not match the rules has a header, optionally a timestamp (if defined in 'timestampformat'), and a hostname added to the front of the event. This is how Splunk causes arbitrary log data to match syslog expectations.
  • Defaults to unset.

So the solution in my case was adding the common part (as it is used as a pattern) of the forwarderd sourcetypes to the syslogSourceType stanza and restart splunkd.

giulia_casaldi
Explorer

Hi @claudio_manig , I am trying to do as you wrote on the outputs.conf but it still has header problems.
Can you provide me a practical example please?

Thank you so much for your kindness and helpfulness,

Giulia

0 Karma

koshyk
Super Champion

Assuming you are forwarding from a Splunk HF/Indexers directly to a 3rd party system (not using a syslog server)?
Ensure timestampformat variables in outputs.conf are unset. Splunk won't put any other extra headers other timestampformat

The reason you may be experiencing headers is, these headers will be already in the event payload. Then you need to customize event itself .If you want to customize the events (eg strip first 3 words), use props & transforms

# props.conf
[my_strippable_data]
TRANSFORMS-strip_me=my_stripped_sourcetype

# transforms.conf
[my_stripped_sourcetype]
REGEX=<your_regex>
DEST_KEY=_SYSLOG_ROUTING
FORMAT=strippedGroup


# outputs.conf
[syslog]
defaultGroup=everythingElseGroup
[syslog:strippedGroup]
server = 10.1.1.197:9997
disabled = false
priority = <34>

http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Outputsconf
http://docs.splunk.com/Documentation/SplunkCloud/6.6.1/Forwarding/Forwarddatatothird-partysystemsd

0 Karma
Get Updates on the Splunk Community!

New Case Study Shows the Value of Partnering with Splunk Academic Alliance

The University of Nevada, Las Vegas (UNLV) is another premier research institution helping to shape the next ...

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...