topic Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge in All Apps and Add-ons

Cisco ASA 5520 -- Problem -- no IOS - linemerge

JimDeich — Fri, 20 Mar 2015 04:28:10 GMT

Hi new to Cisco for Splunk.

We have Cisco ASA 5520 Fire Wall logs flowing to the app. They are not being converted to converted to sourcetype=cisco:ios, but are remaining syslog AND are being left with long multi-line events that should not be multi-line .

Is this just another log format or could I make some adjustment?

I see there is an ASA app but that is not for version 6.
http://answers.splunk.com/answers/124715/splunk-6-with-cisco-asa.html

Should I just set these up as sourcetype syslog with line-merge turned off?

==> Also some logs handled by the app and change to sourcetype cisco:ios look like they may have excessive line-merging. Does line breaking sometimes need adjusted for the app?

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

mikaelbje — Fri, 20 Mar 2015 05:38:08 GMT

Hi,

It sounds like you are confusing the apps and technologies.

Cisco Networks app and Cisco Networks Add-on:
Covers Cisco IOS, NX-OS, WLC devices such as Catalyst, Nexus etc

Cisco ASA Add-on and Cisco Security Suite:
Covers Cisco ASA devices

The Add-ons transform the syslog sourcetype to cisco:ios or cisco:asa respectively depending on the event contents.

SHOULD_LINEMERGE is set to true by default, so leaving it on is generally ok with these apps. I've never set a specific line breaker using these apps, but have seen rare occurences where it fails because of a timestamp in the event or in very rare cases if the logs are multi-line.

You may also try adding the following to your UDP input:

no_appending_timestamp=true

Your Splunk topology, example events and information about how you are receiving your logs is also useful, otherwise I have to make guesses to be able to help you.

Mikael,
Author of the Cisco Networks App and Add-on

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

JimDeich — Mon, 28 Sep 2020 19:18:47 GMT

So I have installed the: Splunk_TA_cisco-asa app and the app is correctly identifying logs that are cisco-ass as sourcetype cisco:asa .

The defualt props.conf for these has SHOULD_LINEMERGE = false , but lines are being merged irregularly .

The data is come in from file monitor reads but they are identified as sourcetype=syslog .

Lines end with 0A aka \n , but I have tried multiple ways to get it to stop the inappropriate line merges, but
it's not working .

Thanks!

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

mikaelbje — Mon, 30 Mar 2015 07:02:02 GMT

Could you please share your monitor stanza from inputs.conf as well as paste a few lines from your ASA log file. Anonymize any sensitive data.

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

JimDeich — Mon, 28 Sep 2020 19:19:01 GMT

I appreciate the help ..
Monitor Stanza:
[monitor:///data/newlogs/cisco/.log]
disabled = false
host_regex=([^\/]).\d{4}-\d{2}-\d{2}.log
index = ios

sourcetype = syslog

local props.conf for cisco:asa :
[cisco:asa]
CHARSET = utf-8
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S

MAX_EVENTS = 1

Sample data, each line end with 0A \n
Mar 29 00:00:01 10.232.1.103/10.232.1.103 %ASA-7-713236: IP = 169.132.18.1, IKE_DECODE RECEIVED Message (msgid=7b412844) with payloads : HDR + HASH (8) + NOTIFY (11) + NONE (0) total length : 80
Mar 29 00:00:01 10.232.1.103/10.232.1.103 %ASA-7-715047: Group = N2Pprod2, Username = cconti, IP = 169.132.18.1, processing hash payload
Mar 29 00:00:01 10.232.1.103/10.232.1.103 %ASA-7-715047: Group = N2Pprod2, Username = cconti, IP = 169.132.18.1, processing notify payload
Mar 29 00:00:01 10.232.1.103/10.232.1.103 %ASA-7-715075: Group = N2Pprod2, Username = cconti, IP = 169.132.18.1, Received keep-alive of type DPD R-U-THERE (seq number 0xced72762)
Mar 29 00:00:01 10.232.1.103/10.232.1.103 %ASA-7-715036: Group = N2Pprod2, Username = cconti, IP = 169.132.18.1, Sending keep-alive of type DPD R-U-THERE-ACK (seq number 0xced72762)
Mar 29 00:00:01 10.232.1.103/10.232.1.103 %ASA-7-715046: Group = N2Pprod2, Username = cconti, IP = 169.132.18.1, constructing blank hash payload
Mar 29 00:00:01 10.232.1.103/10.232.1.103 %ASA-7-715046: Group = N2Pprod2, Username = cconti, IP = 169.132.18.1, constructing qm hash payload
Mar 29 00:00:01 10.232.1.103/10.232.1.103 %ASA-7-713236: IP = 169.132.18.1, IKE_DECODE SENDING Message (msgid=4d9922d2) with payloads : HDR + HASH (8) + NOTIFY (11) + NONE (0) total length : 80
Mar 29 00:00:03 10.232.1.103/10.232.1.103 %ASA-7-713236: IP = 47.16.183.44, IKE_DECODE RECEIVED Message (msgid=2460272) with payloads : HDR + HASH (8) + NOTIFY (11) + NONE (0) total length : 80
Mar 29 00:00:03 10.232.1.103/10.232.1.103 %ASA-7-715047: Group = idtremote, Username = christina.rudczynski, IP = 47.16.183.44, processing hash payload
Mar 29 00:00:03 10.232.1.103/10.232.1.103 %ASA-7-715047: Group = idtremote, Username = christina.rudczynski, IP = 47.16.183.44, processing notify payload
Mar 29 00:00:03 10.232.1.103/10.232.1.103 %ASA-7-715075: Group = idtremote, Username = christina.rudczynski, IP = 47.16.183.44, Received keep-alive of type DPD R-U-THERE (seq number 0x27b70150)
Mar 29 00:00:03 10.232.1.103/10.232.1.103 %ASA-7-715036: Group = idtremote, Username = christina.rudczynski, IP = 47.16.183.44, Sending keep-alive of type DPD R-U-THERE-ACK (seq number 0x27b70150)
Mar 29 00:00:03 10.232.1.103/10.232.1.103 %ASA-7-715046: Group = idtremote, Username = christina.rudczynski, IP = 47.16.183.44, constructing blank hash payload
Mar 29 00:00:03 10.232.1.103/10.232.1.103 %ASA-7-715046: Group = idtremote, Username = christina.rudczynski, IP = 47.16.183.44, constructing qm hash payload
Mar 29 00:00:03 10.232.1.103/10.232.1.103 %ASA-7-713236: IP = 47.16.183.44, IKE_DECODE SENDING Message (msgid=e16bf0d4) with payloads : HDR + HASH (8) + NOTIFY (11) + NONE (0) total length : 80
Mar 29 00:00:04 10.232.1.103/10.232.1.103 %ASA-7-713236: IP = 169.132.18.1, IKE_DECODE RECEIVED Message (msgid=6654134b) with payloads : HDR + HASH (8) + NOTIFY (11) + NONE (0) total length : 80

Nothing more sensetive than name

The data is being handled by the: Splunk_TA_cisco-asa app.

I really cannot see why he lines a re not being split to single lines .

You help is much appreciated!

Jim

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

mikaelbje — Mon, 30 Mar 2015 16:03:44 GMT

Is the inputs.conf on a server running a Splunk forwarder sending data to an indexer? In that case your props.conf needs to reside on the indexer. Can you confirm that this is your topology? I also find it useful to set up the syslog daemon to add the local system time to the event (usually the default behaviour). That way you end up with two timestamps. The first one being the syslog server's and the next the timestamp from the sending device. I trust the syslog admin to have the right time more than I trust all the endpoint admins 🙂

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

JimDeich — Tue, 31 Mar 2015 05:12:54 GMT

The props.conf is on the indexer(s) not the light forwader.

The time stamps don't have the year. With you asking about them, and from some reading, I am thinking maybe failur to extract them is casusing the multi-line log.

I'll check with the guy that set ups the syslog piece about adding a second date / time stamp.

Thanks for helping ! more guidance appreciated.

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

mikaelbje — Mon, 28 Sep 2020 19:21:39 GMT

Your config however does look OK. Could you paste some of the events that have been incorrectly identified as one event?

You can also fiddle with MAX_TIMESTAMP_LOOKAHEAD to only look N characters into the event for a timestamp. Try setting that to 16

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

JimDeich — Mon, 28 Sep 2020 19:21:42 GMT

Here is an example of a set of lines wrongly being collapsed into a single event .
3/31/15
4:52:40.000 AM

Mar 31 04:52:40 10.232.1.103/10.232.1.103 %ASA-7-713236: IP = 200.89.4.66, IKE_DECODE RECEIVED Message (msgid=4993fc3e) with payloads : HDR + HASH (8) + NOTIFY (11) + NONE (0) total length : 80
Mar 31 04:52:40 10.232.1.103/10.232.1.103 %ASA-7-715047: Group = 200.89.4.66, IP = 200.89.4.66, processing hash payload
Mar 31 04:52:40 10.232.1.103/10.232.1.103 %ASA-7-715047: Group = 200.89.4.66, IP = 200.89.4.66, processing notify payload
Mar 31 04:52:40 10.232.1.103/10.232.1.103 %ASA-7-715075: Group = 200.89.4.66, IP = 200.89.4.66, Received keep-alive of type DPD R-U-THERE (seq number 0x5c59444a)

Mar 31 04:52:40 10.232.1.103/10.232.1.103 %ASA-7-715036: Group = 200.89.4.66, IP = 200.89.4.66, Sending keep-alive of type DPD R-U-THERE-ACK (seq number 0x5c59444a)

While in this case the lines grouped all have the same time stamp, that is not always the case.

I think it' s likely that the absence of a year in the time stamps is part of the issue.
I'll try your suggestion for MAX_TIMESTAMP_LOOKAHEAD .
Thanks!

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

JimDeich — Mon, 28 Sep 2020 19:21:45 GMT

Here is an full set of 8 lines grouped together :
3/31/15
4:52:24.000 AM

Mar 31 04:52:24 10.232.1.103/10.232.1.103 %ASA-7-713236: IP = 68.172.254.106, IKE_DECODE RECEIVED Message (msgid=108dc641) with payloads : HDR + HASH (8) + NOTIFY (11) + NONE (0) total length : 80
Mar 31 04:52:24 10.232.1.103/10.232.1.103 %ASA-7-715047: Group = idtuser, Username = olanger, IP = 68.172.254.106, processing hash payload
Mar 31 04:52:24 10.232.1.103/10.232.1.103 %ASA-7-715047: Group = idtuser, Username = olanger, IP = 68.172.254.106, processing notify payload
Mar 31 04:52:24 10.232.1.103/10.232.1.103 %ASA-7-715075: Group = idtuser, Username = olanger, IP = 68.172.254.106, Received keep-alive of type DPD R-U-THERE (seq number 0x4f9430dd)
Mar 31 04:52:24 10.232.1.103/10.232.1.103 %ASA-7-715036: Group = idtuser, Username = olanger, IP = 68.172.254.106, Sending keep-alive of type DPD R-U-THERE-ACK (seq number 0x4f9430dd)
Mar 31 04:52:24 10.232.1.103/10.232.1.103 %ASA-7-715046: Group = idtuser, Username = olanger, IP = 68.172.254.106, constructing blank hash payload
Mar 31 04:52:24 10.232.1.103/10.232.1.103 %ASA-7-715046: Group = idtuser, Username = olanger, IP = 68.172.254.106, constructing qm hash payload
Mar 31 04:52:24 10.232.1.103/10.232.1.103 %ASA-7-713236: IP = 68.172.254.106, IKE_DECODE SENDING Message (msgid=bba9f381) with payloads : HDR + HASH (8) + NOTIFY (11) + NONE (0) total length : 80

The earlier example had more lines yet.

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

JimDeich — Mon, 28 Sep 2020 19:21:51 GMT

The MAX_TIMESTAMP_LOOKAHEAD did not seem to help . Thanks again .

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

mikaelbje — Tue, 31 Mar 2015 13:59:38 GMT

Can you confirm that this is your config?

LINE_BREAKER = ([\r\n]+)

The example you provided earlier lacks the extra backslash before the r and n. Maybe it's just a paste error

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

JimDeich — Mon, 28 Sep 2020 19:23:31 GMT

Line breaker is definitely in like that.

LINE_BREAKER=([\r\n]+)

What does this parm do? I don't know it ?
LINE_BREAKER_LOOKBEHIND=100

I am really wonder if the absence of year in the date is causing an issue.

Could :
MAX_DAYS_AGO=2000
affect this?

I'll ask the guy doing the syslog to add the date business.

This is a tough one, I think I'll try support too.

Thanks Again!

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

mikaelbje — Wed, 01 Apr 2015 08:06:51 GMT

Have you tried disabling the Cisco ASA Add-on on your indexer? Doing that will effectively disable the sourcetype rewriting from syslog to cisco:asa. Do you still see multiple lines as one event? In that case I think there's something fishy with your syslog setup OR that the syslog sourcetyoe rules have been modified.
Paste the output of the following command:

cd /opt/splunk/bin
./splunk cmd btool props list syslog --debug

I want to compare that output to the default output

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

JimDeich — Mon, 28 Sep 2020 19:22:50 GMT

Here is the btool output for syslog on one of our indexers.

./splunk cmd btool props list syslog --debug
/opt/splunk/etc/apps/Splunk_TA_cisco-asa/default/props.conf [syslog]
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf EVAL-action = if(app="su" AND isnull(action),"success",action)
/opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf EVAL-signature = if(isnotnull(inbound_interface),"firewall",null())
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/apps/TA-FireEye_v3/default/props.conf LINE_BREAKER = ((?!))
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf LOOKUP-action_for_syslog = nix_action_lookup vendor_action OUTPUTNEW action
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 32
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf REPORT-0authentication_for_syslog = ssh-login-events, ssh-session-close, ssh-disconnect, sshd_authentication_kerberos_success, sshd_authentication_refused, sshd_authentication_tried, sshd_login_restricted, pam_unix_authentication_success, pam_unix_authentication_failure, sudo_cannot_identify, ksu_authentication, ksu_authorization, su_simple, su_authentication, su_successful, wksh_authentication, login_authentication
/opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf REPORT-account_management_for_syslog = useradd, userdel
/opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf REPORT-dest_for_syslog = host_as_dest
/opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf REPORT-firewall = ipfw, ipfw-stealth, ipfw-icmp, pf
/opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf REPORT-routing = iptables
/opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf REPORT-signature_for_syslog_timesync = signature_for_nix_timesync
/opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf REPORT-src_for_syslog = src_dns_as_src, src_ip_as_src, host_as_src
/opt/splunk/etc/system/default/props.conf REPORT-syslog = syslog-extractions
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/apps/TA-FireEye_v3/default/props.conf SHOULD_LINEMERGE = false
/opt/splunk/etc/system/default/props.conf TIME_FORMAT = %b %d %H:%M:%S
/opt/splunk/etc/system/default/props.conf TRANSFORMS = syslog-host
/opt/splunk/etc/apps/Splunk_TA_cisco-asa/default/props.conf TRANSFORMS-force_sourcetype_for_cisco = force_sourcetype_for_cisco_asa,force_sourcetype_for_cisco_pix,force_sourcetype_for_cisco_fwsm
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf TRANSFORMS-force_sourcetype_for_cisco_ios = force_sourcetype_for_cisco_ios, force_sourcetype_for_cisco_ios-xr, force_sourcetype_for_cisco_ios-xe
/opt/splunk/etc/apps/TA-FireEye_v3/default/props.conf TRANSFORMS-updateFireEyeSourcetypes = fix_FireEye_CEF_st, fix_FireEye_CSV_st, fix_FireEye_XML_st, fix_HX_CEF_st
/opt/splunk/etc/apps/TA-FireEye_v3/default/props.conf TRUNCATE = 0
/opt/splunk/etc/system/default/props.conf category = Operating System
/opt/splunk/etc/system/default/props.conf description = Output produced by many syslog daemons, as described in RFC3164 by the IETF
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 3
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/system/default/props.conf pulldown_type = true
/opt/splunk/etc/system/default/props.conf sourcetype =

I'm also going to set up a test splunk just to look at these logs.

I greatly appreciate your sticking with this.
JIm

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

mikaelbje — Thu, 02 Apr 2015 04:58:22 GMT

Try to use the comment button on an answer instead of adding comments as answers. Please also use the code icon (101010) when pasting output because that will preserve the original output 🙂

Have a look at this:

/opt/splunk/etc/apps/TA-FireEye_v3/default/props.conf LINE_BREAKER = ((?!))

Try disabling TA-FireEye_v3. It sets a different line breaker for the syslog sourcetype that overwrites the default. That's a bad practice in that app IMHO.

Splunk cmd btool is invaluable as it shows you what config is being used after everything is merged together.

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

JimDeich — Thu, 02 Apr 2015 05:22:48 GMT

This is a promising tip. I can't even see what they are trying to do with that line break . The regex is
(?!pattern)
a perl negative look-ahead assertion, but it's not followed by a pattern, so I don't think it would ever catch a line break.
A little to late here (NJ, USA) to try tonight.

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

mikaelbje — Thu, 02 Apr 2015 08:02:38 GMT

I believe that line breaker setting is used to prevent line breaking as it never matches anything 🙂

Happy Easter!

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

JimDeich — Fri, 03 Apr 2015 02:50:39 GMT

Mikael -- This was it! I looked at our Fireeye logs and it looks like they could tolerate normal linebreaking. So I added a local dir override to the linebreaking under props.conf in the fireeye app.

[syslog]
LINE_BREAKER = ([\r\n]+)

Thanks!!! Thanks for hanging in! Happy Easter !

Jim
...

Re: Cisco ASA 5520 -- Problem -- no IOS - linemerge

mikaelbje — Fri, 03 Apr 2015 06:02:23 GMT

You're welcome 🙂 We checked a lot of stuff before trying btool. I've made a note to myself to always do that first. Things aren't always the way they seem, so btool is really the way to go 🙂

I've converted my comment to an answer so you can accept it 🙂