Cyberark Logs - Line Breaking doesn't work

[HIBE151]

Hello together,

today we tried to fix the line breaking and timestamp problems we have with logs from cyberark.:

<5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-TEST_MAIL_DualControl.ini dvc= shost=192.169.122.122 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg=<5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-SWIFT_InvestigationCenter-TEST.ini dvc= shost=192.168.122.222 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg=<5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-BLUB_InvestigationCenter.ini dvc= shost=192.168.122.221 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg=<5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-SWIFT_GTX_Bottomline.ini dvc= shost=192.168.178.212 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg=<5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-SWIFT_GTX_Bottomline-TEST.ini dvc= shost=192.122.122.122 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg=

Timestamp problem fixed with following configuration:

MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = \<\d\>\d

But for any reason I can't change Line Breaks for this text. Actually I want to break the event everytime when one of those <5>1 appears. Does anyone has an idea why?

[pellegrini]

If syslog is sent with TCP or TLS then this is an known issue in CyberArk. If you are using SC4S (Splunk Connect for Syslog) the solution is to add a line feed https://cyberark-customers.force.com/s/article/00004289

[Chawicha]

Thanks for your response!

It seems that workaround proposed in the link is for the file provided by CyberArk because it is not matching the content of SplunkCIM.xsl file provided by Splunk TA. 

Do you know how to apply it to Splunk application?

[PickleRick]

1. The thread is relatively old so you might not get many responses. It's usually better to post a new question (linking to old thread for reference) than to dig up a several-years-old thread.

2. The .xsl file is meant to be applied on the Vault side - installed to ENE, not used on Splunk's side.

https://docs.splunk.com/Documentation/AddOns/released/CyberArk/Setup

[Esky73]

did you ever fix this?

[chrisyounger]

Hi @HIBE151

This should work LINE_BREAKER = ([\r\n]*)<\d+>1\s

One important thing to remember is that the LINE_BREAKER must specify a capturing group - even if it captures nothing (like it seems to in your case). Also don't forget the LINE_BREAKER happens on the first heavy forwarder or indexer that the data goes through.

I hope this help you

[whrg]

Hello @HIBE151,

You need to use LINE_BREAKER. It specifies how the log text is broken into individual events. By default, LINE_BREAKER is set to ([\r\n]+). That is, each line break starts a new event. In your case, there is no line break, so there is only one long event.

I suggest you set LINE_BREAKER to:

LINE_BREAKER = (<5>1\s*)

Have a look at this screenshot:

Now you don't need TIME_PREFIX because now the time stamp is at the beginning of each event.

What does the "Z" in "2019-02-02T10:14:37Z" stand for? If that is the time zone, then add %Z to your TIME_FORMAT.

Overall, props.conf should look something like this:

[name]
category = Custom
pulldown_type = true
NO_BINARY_CHECK = true
LINE_BREAKER = (<5>1\s*)
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%Z

[HIBE151]

thanks for your answer. Finally the breaking is working. Anyway there is a second problem with the log.
<5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-TEST_MAIL_DualControl.ini dvc= shost=192.169.122.122 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg=CEF:0|Cyber-Ark|VaultMonitor|10.1.0000|45|1420|3|40|1|20|162|23|20|222|133|0

This Event is getting breaked into following events:
EVENT1:
<5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-TEST_MAIL_DualControl.ini dvc= shost=192.169.122.122 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg=

EVENT2:
CEF:0|Cyber-Ark|VaultMonitor|10.1.0000|45|1420|3|40|1|20|162|23|20|222|133|0

Why is Splunk splitting this event in two events? Or is Splunk doing everything correct and this are two different events? I actually thought that EVENT2 should be part of the key msg. Anyone here who has a deep understanding of cyberark logs?

Thanks

[woodcock]

Try this in props.conf:

LINE_BREAKER = ([\r\n]+)<\d+>1
SHOULD_LINEMERGE = false

Apply this to the first full version of Splunk that handles the data (HFs or Indexers) and restart all Splunk instances there. When testing, use _index_earliest=-5m with All time timepicker to make sure that you really are looking at newly indexed events that should be effected (older events will stay wrong). Also, if you have done a sourctype-override, you must apply this to the original sourcetype or it will not work.