Splunk Add-on for CyberArk: I made changes in props.conf for proper multiline event breaking, but was there a better way? (thanks for this add-on!)

[jonxilinx]

Less of a question, but just wanted to say Many thanks for this, works like a treat.

I found I had to set UseLegacySyslogFormat=No in the dbparm.ini to send the priority of the SYSLOG to Splunk, and then add this in the props.conf for line breaking for multiline events that are sent from the vault.

[cyberark:epv:cef]
LINE_BREAKER = (<5>1 )

There may have been a better way (I'm a bit of Splunk newbie), but thanks for this TA which is most useful.

[CHgeek]

We are now in 2023 and we still hit the same issue as in 2016. Is there a new xsl file for this plugin?

[nlembrechts]

In our situation we were using:
TCP protocol to send messages to the rsyslog on Ubuntu server which was writing messages to the separate file
Splunk Heavy Forwarder was reading and parsing that file and tranferring it to the Splunk indexer

The problem is if we are using TCP to establish connection, then rsyslog requires some delimiter to understand where message frame ends. Otherwise it will just buffer all messages until it fill buffer and then release part of the buffer to the log file, without really taking care about message start and end, etc...
That is why first thing we need to do is to put delimiter to the SplunkCIM.xsl file. Do this by inserting next line between line with </xsl:for-each> and line with </xsl:template> in my case it was between lines 129 and 130:
<xsl:text>&#xa;</xsl:text>

Where 
 - is a new line character in xml format it is the same as #015 in hex representation:
It is the same as: 0x0A (10 decimal) or 
 - LF = Line Feed. Industry-strandard plain text tcp syslog uses the LF to delimit syslog frames.
From

After you did it you will need to restart Password Vault server (Do not forget to start Event Notification Engine after this, because it will be stopped after server stop, but it will not be started automatically).

After this we saw that syslog started to write message to the file each time message was recived via network.
But at this moment with default rsyslog configuration it was translating that LF control character that we defined as frame delimiter to the 3-digit octal number: #015. It means that in splunk and log file at the end of each message we saw that "#015".
To solve this we put this option to /etc/rsyslog.conf on splunk server:

$EscapeControlCharactersOnReceive off

From http://www.rsyslog.com/doc/v7-stable/configuration/input_directives/rsconf1_escapecontrolcharacterso...

Probably alternative solution is to put to the .../splunk/etc/apps/Splunk_TA_cyberark/default/props.conf LINE_BREAKER = (#015), but we did not test this. Pay attention that you may need to escape # - special character.

Additional troubleshooting tips:
1. You can use tcpdump to check if the message from CyberArk server really arrives to the server:
tcpdump -nnvvXS -i any 'host your.vault.ip.address'
2. You can check if this message is written to the syslog using this command:
tail -f /var/log/.../your.syslog.file.name.log

Probably we can just switch to UDP protocol, but it is less reliable, that is why we stick to TCP.

Best regards,
Dima

[fdsc_2]

@nlembrechts  Good Explaination, Thanks for that. 

I've faced with the same issue, and captured the incoming packets to rsyslog collector and the rsyslog daemon was not really able to write all the incoming message to the target directory. when I've changed the protocol to UDP all went fine.

I'd rather like to mention about the case on CyberArk customer portal: Cyberark link

[pokng]

I am new to Splunk. I encounter the same multi-line event issue. I want to know if this an issue with the CyberArk add-on not able to recognize the multi-line event? if yes, why there is no official fix/update to the Add-on?

[jcoates_splunk]

sounds like a good approach -- did you find this helpful, or is there more help we could provide? http://docs.splunk.com/Documentation/AddOns/latest/CyberArk/Setup

[pokng]

This is not helpful. Can Splunk fix this and release an update?

The translator file SplunkCIM.xsl is provided by Splunk. Instead of changing props.conf, Can Splunk add a line break somewhere in SplunkCIM.xsl - like adding ?

BTW, what is "<5>1 "?