CyberArk logs to Splunk via Syslog-ng

[bharathkumarnec]

Hello Everyone,

We have configured CyberArk logs to index into splunk based on the instructions provided in the splunk documentation. We configured to receive logs via syslog-ng through port 514, we are receiving logs but these logs are not getting processed properly by syslog-ng and we see below error:

Error processing log message: <5>1 and continued by the logs from the CyberArk.

We are using 3.5.6 version of syslog-ng, anyone faced this kind of error? this error is because of the structure of the data?

Your inputs are of great help!

Thanks in advance!

Regards,

BK

[rnowitzki]

Hi @bharathkumarnec ,

That's weird.
What makes you think, that these messages reach syslog-ng at all? 
Where do you see the error message you mentioned?

Maybe you see a more detailed error message, when you run syslog-ng in foreground.
Stop the daemon and then run:

/opt/syslog-ng/sbin/syslog-ng -Fedv

This will run Syslog-ng in foreground, so everything will go to stdout.
If you get a lot of messages, you maybe want to pipe that to a file and run it for a short period  only.

To see the messages regardless of what syslog-ng does to them, you can try:

tcpdump -i eth0 port 514 -v 

you maybe have to change the interface or port to meet your environment.
(same here: if your screen explodes due to too many  messages, pipe it to a file and run it just shortly)

You can also run a grep against tcpdump. Grep for something unique to the cyberark Logs, if you get more via the same port.

tcpdump -i eth0 port 514 -v  |  grep -C2 <cyber ark unique string>

Maybe one of the options gets you closer to find the rootcause.

Cheers
Ralph

[bharathkumarnec]

@rnowitzki , Thanks for taking time in replying.

Before writing the question here in the group below are the things that I did:

* We are receiving data from three sources on the same port, so that way only one set of data which is not seen extracting from the syslog-ng.

* So then, I tried to dump everything irrespective of the host into a different location and i observed that the cyberark logs are stored with the receiver hostname and logs are with the error "error processing log file".

* Then i did tcpdump on the host that is receiving the logs and observed that the logs are seen without any error message.

* After i realized that this is something related to the syslog-ng configuration then i used lot of rules and templates along with no parse flag, still no luck.

Now, again i realized that the no parse flag is not used\set in the way it is supposed to use, then again when i corrected the configuration, i can see the logs are receiving.

Thanks a lot for your help, so i would say no parse flag helped me in this case.

Regards,

BK

[rnowitzki · Accepted solution]

Hi @bharathkumarnec ,

You could try the no parse flag (flags(no-parse)) for the source defintion in the syslog-ng config. 
If the error relates to the format, this could at least help to get the data coming in.

It puts everything in the $message macro if you do that. You might end up with duplicate timestamps or stuff like that.  You can work around that with templates on the destinations and/or rewrite rules.

It also helps to see what the messages look like when they come in, with tcpdump e.g. Maybe it's something weird syslog-ng can not work with at all. 

Hope it helps.
BR
Ralph

[bharathkumarnec]

@rnowitzki , Thanks for your inputs, i tried with no parse flag but still no luck..