In our situation we were using:
TCP protocol to send messages to the rsyslog on Ubuntu server which was writing messages to the separate file
Splunk Heavy Forwarder was reading and parsing that file and tranferring it to the Splunk indexer
The problem is if we are using TCP to establish connection, then rsyslog requires some delimiter to understand where message frame ends. Otherwise it will just buffer all messages until it fill buffer and then release part of the buffer to the log file, without really taking care about message start and end, etc...
That is why first thing we need to do is to put delimiter to the SplunkCIM.xsl file. Do this by inserting next line between line with </xsl:for-each> and line with </xsl:template> in my case it was between lines 129 and 130:
<xsl:text>
</xsl:text>
Where 
 - is a new line character in xml format it is the same as #015 in hex representation:
It is the same as: 0x0A (10 decimal) or - LF = Line Feed. Industry-strandard plain text tcp syslog uses the LF to delimit syslog frames.
From
After you did it you will need to restart Password Vault server (Do not forget to start Event Notification Engine after this, because it will be stopped after server stop, but it will not be started automatically).
After this we saw that syslog started to write message to the file each time message was recived via network.
But at this moment with default rsyslog configuration it was translating that LF control character that we defined as frame delimiter to the 3-digit octal number: #015. It means that in splunk and log file at the end of each message we saw that "#015".
To solve this we put this option to /etc/rsyslog.conf on splunk server:
$EscapeControlCharactersOnReceive off
From http://www.rsyslog.com/doc/v7-stable/configuration/input_directives/rsconf1_escapecontrolcharactersonreceive.html
Probably alternative solution is to put to the .../splunk/etc/apps/Splunk_TA_cyberark/default/props.conf LINE_BREAKER = (#015) , but we did not test this. Pay attention that you may need to escape # - special character.
Additional troubleshooting tips:
1. You can use tcpdump to check if the message from CyberArk server really arrives to the server:
tcpdump -nnvvXS -i any 'host your.vault.ip.address'
2. You can check if this message is written to the syslog using this command:
tail -f /var/log/.../your.syslog.file.name.log
Probably we can just switch to UDP protocol, but it is less reliable, that is why we stick to TCP.
Best regards,
Dima
... View more