Getting Data In

CheckPoint FW Logs from Manager with Two Timestamps

Communicator

We are splunking logs from our CheckPoint FW. The logs are delivered from the CheckPoint manager stations, not directly from the firewalls, so there are two timestamps - first from the management station and next from the FW. We want to index based on the FW time. Here's a typical record:

Jan 17 16:47:31 aaa.bbb.178.200 fw1log: 17Jan2013 17:39:49 accept aaa.bbb.161.11 >eth2c13 rule: nnn; rule_ uid: {93622F88-3071-4FE4-BC7B-9B232AE482E6}; src: aaa.bbb.112.163; dst: aaa.bbb.4.35; proto: udp; product: VPN-1 & FireWall-1; service: 389; s_port: 3490;

The time we want is 17Jan2013 17:39:49.

I went into /etc/system/local/props.conf and added:

TIME_PREFIX = fw1log:\s

TIME_FORMAT = %d%b%Y %H:%M:%S

No luck getting this to use the second (correct) timestamp.

Tags (1)
0 Karma

Communicator

I changed the time prefix setting to:

TIME_PREFIX = fw1log:\s


(s has a backslash in front of it)

I think there was a global setting from some other application for max timestamp lookahead that needed to be changed from the [default] stanza to a more specific stanza

Anyway, now it is working.

Many thanks to sbrant for keeping me on the right track.

0 Karma

Communicator

Props.conf is on the indexer, which is also the search head.

This is probably more than you want, but here's btool props list --debug


search [checkpt_log]
system ANNOTATE_PUNCT = True
system BREAK_ONLY_BEFORE =
system BREAK_ONLY_BEFORE_DATE = True
system CHARSET = UTF-8
system DATETIME_CONFIG = /etc/datetime.xml
system HEADER_MODE =
system LEARN_SOURCETYPE = true
system LINE_BREAKER_LOOKBEHIND = 100
system MAX_DAYS_AGO = 2000
system MAX_DAYS_HENCE = 2
system MAX_DIFF_SECS_AGO = 3600
system MAX_DIFF_SECS_HENCE = 604800
system MAX_EVENTS = 256
system MAX_TIMESTAMP_LOOKAHEAD = 128
system MUST_BREAK_AFTER =
system MUST_NOT_BREAK_AFTER =
system MUST_NOT_BREAK_BEFORE =
system SEGMENTATION = indexing
system SEGMENTATION-all = full
system SEGMENTATION-inner = inner
system SEGMENTATION-outer = outer
system SEGMENTATION-raw = none
system SEGMENTATION-standard = standard
system SHOULD_LINEMERGE = True
system TIME_FORMAT = %d%b%Y %H:%M:%S
system TIME_PREFIX = ^(?:(?:[^\s]+)\s){5}
system TRANSFORMS =
Splunk_Cis TRANSFORMS-force-sourcetype_for_cisco_devices = force_sourcetype_for_cisco_pix, force_sourcetype_for_cisco_asa, force_sourcetype_for_cisco_wap, fo
rce_sourcetype_for_cisco_fwsm, force_sourcetype_for_cisco_acs, force_sourcetype_for_cisco_ios, force_sourcetype_for_cisco_catchall
system TRUNCATE = 10000
system maxDist = 100

0 Karma

Communicator

And even in this comment the backslash was removed. I guess I don't know how to markup with Markdown

0 Karma

Communicator

TIME_PREFIX = ^(?:(?:[^\s]+)\s){5} is what is actually there. I don't know why the formatter removed the backslash in front of the s.

0 Karma

Splunk Employee
Splunk Employee

I made a few assumptions about the props.conf. Just to be sure, is the props.conf on the indexer? Does the stanza for the checkpoint data have the correct name to match the sourcetype? Can you post the entire stanza for checkpoint? It's always best to be explicit. Maybe something like this:

[checkpoint]
TIME_PREFIX = ^(?:(?:[^\s]+)\s){5}
TIME_FORMAT = %d%b%Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 25
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
0 Karma

Communicator

Couldn't put the answers to your questions in a comment.

0 Karma

Splunk Employee
Splunk Employee

Try this as your time prefix:

TIME_PREFIX = ^(?:(?:[^\s]+)\s){5}
0 Karma

Legend

extract reload=t was previously a way of reloading SEARCH-TIME properties from props.conf/transforms.conf. Nowadays you don't need that because each search is run in its own process which will read the current props/transforms settings when it starts.

Index-time settings can however NOT be reloaded without restarting Splunk.

0 Karma

Communicator

That didn't solve it. I assume I can run a query with | extract reload=t at the end to get the new copy of props.conf active. Checked the props list with btool and verified that this is the TIME_PREFIX being used.

Time still matches the first one found.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!