topic Re: CheckPoint FW Logs from Manager with Two Timestamps in Getting Data In

CheckPoint FW Logs from Manager with Two Timestamps

wbfoxii — Thu, 17 Jan 2013 16:01:37 GMT

We are splunking logs from our CheckPoint FW. The logs are delivered from the CheckPoint manager stations, not directly from the firewalls, so there are two timestamps - first from the management station and next from the FW. We want to index based on the FW time. Here's a typical record:

Jan 17 16:47:31 aaa.bbb.178.200 fw1log: 17Jan2013 17:39:49 accept aaa.bbb.161.11 >eth2c13 rule: nnn; rule_ uid: {93622F88-3071-4FE4-BC7B-9B232AE482E6}; src: aaa.bbb.112.163; dst: aaa.bbb.4.35; proto: udp; product: VPN-1 & FireWall-1; service: 389; s_port: 3490;

The time we want is 17Jan2013 17:39:49.

I went into /etc/system/local/props.conf and added:

TIME_PREFIX = fw1log:\s

TIME_FORMAT = %d%b%Y %H:%M:%S

No luck getting this to use the second (correct) timestamp.

Re: CheckPoint FW Logs from Manager with Two Timestamps

sbrant_splunk — Thu, 17 Jan 2013 17:02:54 GMT

Try this as your time prefix:

TIME_PREFIX = ^(?:(?:[^\s]+)\s){5}

Re: CheckPoint FW Logs from Manager with Two Timestamps

wbfoxii — Thu, 17 Jan 2013 17:11:36 GMT

That didn't solve it. I assume I can run a query with | extract reload=t at the end to get the new copy of props.conf active. Checked the props list with btool and verified that this is the TIME_PREFIX being used.

Time still matches the first one found.

Re: CheckPoint FW Logs from Manager with Two Timestamps

Ayn — Thu, 17 Jan 2013 17:17:17 GMT

extract reload=t was previously a way of reloading SEARCH-TIME properties from props.conf/transforms.conf. Nowadays you don't need that because each search is run in its own process which will read the current props/transforms settings when it starts.

Index-time settings can however NOT be reloaded without restarting Splunk.

Re: CheckPoint FW Logs from Manager with Two Timestamps

sbrant_splunk — Thu, 17 Jan 2013 17:20:37 GMT

I made a few assumptions about the props.conf. Just to be sure, is the props.conf on the indexer? Does the stanza for the checkpoint data have the correct name to match the sourcetype? Can you post the entire stanza for checkpoint? It's always best to be explicit. Maybe something like this:

[checkpoint]
TIME_PREFIX = ^(?:(?:[^\s]+)\s){5}
TIME_FORMAT = %d%b%Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 25
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false

Re: CheckPoint FW Logs from Manager with Two Timestamps

wbfoxii — Mon, 28 Sep 2020 13:07:59 GMT

Props.conf is on the indexer, which is also the search head.

This is probably more than you want, but here's btool props list --debug



search     [checkpt_log]

system     ANNOTATE_PUNCT = True

system     BREAK_ONLY_BEFORE =

system     BREAK_ONLY_BEFORE_DATE = True

system     CHARSET = UTF-8

system     DATETIME_CONFIG = /etc/datetime.xml

system     HEADER_MODE =

system     LEARN_SOURCETYPE = true

system     LINE_BREAKER_LOOKBEHIND = 100

system     MAX_DAYS_AGO = 2000

system     MAX_DAYS_HENCE = 2

system     MAX_DIFF_SECS_AGO = 3600

system     MAX_DIFF_SECS_HENCE = 604800

system     MAX_EVENTS = 256

system     MAX_TIMESTAMP_LOOKAHEAD = 128

system     MUST_BREAK_AFTER =

system     MUST_NOT_BREAK_AFTER =

system     MUST_NOT_BREAK_BEFORE =

system     SEGMENTATION = indexing

system     SEGMENTATION-all = full

system     SEGMENTATION-inner = inner

system     SEGMENTATION-outer = outer

system     SEGMENTATION-raw = none

system     SEGMENTATION-standard = standard

system     SHOULD_LINEMERGE = True

system     TIME_FORMAT = %d%b%Y %H:%M:%S

system     TIME_PREFIX = ^(?:(?:[^\s]+)\s){5}

system     TRANSFORMS =

Splunk_Cis TRANSFORMS-force-sourcetype_for_cisco_devices = force_sourcetype_for_cisco_pix, force_sourcetype_for_cisco_asa, force_sourcetype_for_cisco_wap, fo

rce_sourcetype_for_cisco_fwsm, force_sourcetype_for_cisco_acs, force_sourcetype_for_cisco_ios, force_sourcetype_for_cisco_catchall

system     TRUNCATE = 10000

system     maxDist = 100

Re: CheckPoint FW Logs from Manager with Two Timestamps

wbfoxii — Thu, 17 Jan 2013 18:45:01 GMT

Couldn't put the answers to your questions in a comment.

Re: CheckPoint FW Logs from Manager with Two Timestamps

wbfoxii — Fri, 18 Jan 2013 13:12:57 GMT

TIME_PREFIX = ^(?:(?:[^\s]+)\s){5} is what is actually there. I don't know why the formatter removed the backslash in front of the s.

Re: CheckPoint FW Logs from Manager with Two Timestamps

wbfoxii — Fri, 18 Jan 2013 13:14:00 GMT

And even in this comment the backslash was removed. I guess I don't know how to markup with Markdown

Re: CheckPoint FW Logs from Manager with Two Timestamps

wbfoxii — Fri, 18 Jan 2013 13:32:57 GMT

I changed the time prefix setting to:

TIME_PREFIX = fw1log:\s

(s has a backslash in front of it)

I think there was a global setting from some other application for max timestamp lookahead that needed to be changed from the [default] stanza to a more specific stanza

Anyway, now it is working.

Many thanks to sbrant for keeping me on the right track.