Splunk only partially recognizes date from OPSEC logs

[hcpr]

Hi there.
While adding Checkpoint logs to a new Splunk installation (6.1.1) with the OPSEC addon (version 2.1.0) I noticed that Splunk seems to ignore the date from the logs, and only use the time. The current date is used even when indexing old logs.

So if I have the following raw event:

loc=143934|time=2014-05-22 23:59:57|action=allow|src=132.150.36.243|s_port=63882|dst=46.137.165.40|service=80|proto=tcp|appi_name=c.richmetrics.com|matched_category=Computers / Internet|app_risk=0

(Sorry for the line breaks, the fields are separated with | )

Splunk actually indexes this with

_time=2014-05-27T23:59:57.000+02:00 

which is the tame the event was indexed. This is also the time/date shown in searches and on the graph.

Does anyone have any suggestions on how to fix this?

[araitz]

Can you confirm that you have the following in default/props.conf?

[opsec]
SHOULD_LINEMERGE = false
TIME_PREFIX      = time=
TIME_FORMAT      = %d%b%Y %H:%M:%S
KV_MODE          = none

Can you confirm that you have deployed the add-on on a Heavy Forwarder and/or have the add-on installed on your indexer(s)?

The above lines will handle time parsing, either on a HF or on your indexers. I suspect something is wrong with your configuration - maybe you manually altered the sourcetype, or the props.conf entry?

[araitz]

hcpr - you should open a support case. We can't recreate that behavior, and haven't seen that with any of the other customers using the add-on. My guess is that something somewhere else on the system is clobbering your configuration.

[hcpr]

Hi, of course 🙂 Just missed that last time 😞

The opsec app is installed on the indexers and search head plus on a heavy forwarder that is doing the actual collection from the Checkpoint system.

Also, the config is not changed apart form the testing with different data formats in fw1-loggrabber.conf that I mentioned above.

[araitz]

Can you answer my other question regarding the nature of your deployment (HF or UF, TA on indexers or not)?

[hcpr]

Yes the the default/props.conf file contains what you mention.
But still
loc=4292529|time=22May2014 0:26:54|action=accept|src=132.150.245.122|s_port=58730|dst=132.150.7.52|service=53|proto=udp
is indexed as occuring on todays date, but with correct time.

(I've tried using both DATEFORMAT="cp" (the default) and DATEFORMAT="std" in fw1-loggrabber.conf. In both cases the time is indexed properly, but the date is ignored and set to the date at indexing time)

[sroback_splunk]

Hi. You might need to edit the timestamp properties in your props.conf file for Splunk to correctly parse the original timestamp. See these docs on how Splunk reads timestamps and how to configure timestamp recognition:

http://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps#How_Splunk_assig...

http://docs.splunk.com/Documentation/Splunk/6.1.1/Data/Configuretimestamprecognition