Splunk Search

Data appended to start of each event - ....Why?

joshd
Builder

Hello,

I have a syslog-ng server forwarding logs to my Splunk server. I have just reconfigured my data input from UDP to TCP and it seems that Splunk is appending data to the start of each event for some reason (well at least it shows this way in the Search for sure). The data is a number enclosed by angled brackets and it seems the number in brackets that is being appended to the start of each logged event is a unique identifier for the process that is actually generating the event. Here is an example:

<14>Aug 12 14:07:14 server1 audispd: node=server1 type=USER_AUTH msg=audit(1281636741.920:2243091): user pid=5777 uid=0 auid=4294967295 msg='PAM: authentication acct="user" : exe="/bin/su" (hostname=?, addr=?, terminal=? res=success)' <78>Aug 12 14:07:07 server2 crond[25368]: (root) CMD (/usr/lib64/sa/sa1 1 1) <166>Aug 12 14:07:05 server3 Oracle: Audit[11084]: LENGTH: "227" SESSIONID:[8] "19110341" ENTRYID:[1] "1" USERID:[9] "USER1" ACTION:[3] "101" RETURNCODE:[1] "0" LOGOFF$PREAD:[1] "0" LOGOFF$LREAD:[3] "297" LOGOFF$LWRITE:[1] "7" LOGOFF$DEAD:[1] "0" DBID:[9] "612345678" SESSIONCPU:[1] "6" <166>Aug 12 14:07:05 server3 Oracle: Audit[11084]: LENGTH: "315" SESSIONID:[8] "19110341" ENTRYID:[1] "1" STATEMENT:[1] "1" USERID:[9] "USER1" USERHOST:[21] "desktop1" ACTION:[3] "100" RETURNCODE:[1] "0" COMMENT$TEXT:[98] "Authenticated by: DATABASE; Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=1.1.1.1)(PORT=42061))" DBID:[9] "12345678" PRIV$USED:[1] "5" <86>Aug 12 14:06:43 server1 sshd[5486]: Connection closed by 2.2.2.2 <86>Aug 12 14:06:29 server2 sshd[21659]: Connection closed by 3.3.3.3

So as you can see 14 maps to the process audispd, 78 maps to crond, 166 maps to Oracle and 86 maps to sshd.

Now can anyone tell me why splunk would append this process identifier to the event? Is this something that has been seen before?

Thanks, Josh

Tags (3)
0 Karma
1 Solution

joshd
Builder

Answering my own question here since Support got back to me a lot faster than expected...

To quote them:

" The <#> is a priority attribute syslog field. Splunk does not append this to the events, actually, splunk removes it from the events: Note the UDP additional attributes from the inputs.conf file:

no_priority_stripping = true * If this attribute is set to true, then Splunk does NOT strip the syslog field from received events. * NOTE: Do NOT include this key if you want to strip .

By default then, this is set to false.

What I assume is happening is that when you switched from UDP to TCP, either TCP doesnot have this option, or it is set to TRUE by default. "

View solution in original post

wrangler2x
Motivator

I had this same problem. Originally I had syslog-ng receiving syslogs from multiple systems which then wrote them into a series of directories which splunk monitored. This worked fine. But later I decided to simply forward them to another port that I told splunk to monitor, using TCP. This worked fine except for the code being added to the data. What I had to do to correct this was to train syslog-ng not to add this. Here is what I had originally:

destination splunk_local { tcp("localhost" port(5014)); };

I simply changed this to specify what the message should look like to match what I had been getting out of the files previously, and it worked great:

destination splunk_local { tcp("localhost" port(5014) template("$DATE $HOST $MSG\n")); };

0 Karma

dwaddle
SplunkTrust
SplunkTrust

As I recall from discussing with Splunk some time ago, the TCP input code does not have priority stripping built into it. You could probably accomplish the same thing though with a SEDCMD in props.conf, something like

[host::my_syslog_host.mynetwork.net]
SEDCMD-pristrip=s/^<[0-9]+>//

joshd
Builder

Answering my own question here since Support got back to me a lot faster than expected...

To quote them:

" The <#> is a priority attribute syslog field. Splunk does not append this to the events, actually, splunk removes it from the events: Note the UDP additional attributes from the inputs.conf file:

no_priority_stripping = true * If this attribute is set to true, then Splunk does NOT strip the syslog field from received events. * NOTE: Do NOT include this key if you want to strip .

By default then, this is set to false.

What I assume is happening is that when you switched from UDP to TCP, either TCP doesnot have this option, or it is set to TRUE by default. "

Get Updates on the Splunk Community!

Happy CX Day, Splunk Community!

Happy CX Day, Splunk Community! CX stands for Customer Experience, and today, October 3rd, is CX Day — a ...

Splunk Observability Cloud | Customer Survey!

If you use Splunk Observability Cloud, we invite you to share your valuable insights with us through a brief ...

.conf23 | Get Your Cybersecurity Defense Analyst Certification in Vegas

We’re excited to announce a new Splunk certification exam being released at .conf23! If you’re going to Las ...