topic Re: Splunk only partially recognizes date from OPSEC logs in Getting Data In

Splunk only partially recognizes date from OPSEC logs

hcpr — Tue, 27 May 2014 13:31:39 GMT

Hi there.
While adding Checkpoint logs to a new Splunk installation (6.1.1) with the OPSEC addon (version 2.1.0) I noticed that Splunk seems to ignore the date from the logs, and only use the time. The current date is used even when indexing old logs.

So if I have the following raw event:

loc=143934|time=2014-05-22 23:59:57|action=allow|src=132.150.36.243|s_port=63882|dst=46.137.165.40|service=80|proto=tcp|appi_name=c.richmetrics.com|matched_category=Computers / Internet|app_risk=0

(Sorry for the line breaks, the fields are separated with | )

Splunk actually indexes this with

_time=2014-05-27T23:59:57.000+02:00

which is the tame the event was indexed. This is also the time/date shown in searches and on the graph.

Does anyone have any suggestions on how to fix this?

Re: Splunk only partially recognizes date from OPSEC logs

sroback_splunk — Tue, 27 May 2014 16:08:24 GMT

Hi. You might need to edit the timestamp properties in your props.conf file for Splunk to correctly parse the original timestamp. See these docs on how Splunk reads timestamps and how to configure timestamp recognition:

http://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps#How_Splunk_assigns_timestampshere

http://docs.splunk.com/Documentation/Splunk/6.1.1/Data/Configuretimestamprecognition

Re: Splunk only partially recognizes date from OPSEC logs

araitz — Tue, 27 May 2014 16:27:57 GMT

Can you confirm that you have the following in default/props.conf?

[opsec]
SHOULD_LINEMERGE = false
TIME_PREFIX      = time=
TIME_FORMAT      = %d%b%Y %H:%M:%S
KV_MODE          = none

Can you confirm that you have deployed the add-on on a Heavy Forwarder and/or have the add-on installed on your indexer(s)?

The above lines will handle time parsing, either on a HF or on your indexers. I suspect something is wrong with your configuration - maybe you manually altered the sourcetype, or the props.conf entry?

Re: Splunk only partially recognizes date from OPSEC logs

hcpr — Wed, 28 May 2014 11:23:11 GMT

(I've tried using both DATEFORMAT="cp" (the default) and DATEFORMAT="std" in fw1-loggrabber.conf. In both cases the time is indexed properly, but the date is ignored and set to the date at indexing time)

Re: Splunk only partially recognizes date from OPSEC logs

araitz — Wed, 28 May 2014 17:12:42 GMT

Can you answer my other question regarding the nature of your deployment (HF or UF, TA on indexers or not)?

Re: Splunk only partially recognizes date from OPSEC logs

hcpr — Mon, 02 Jun 2014 11:01:57 GMT

Hi, of course 🙂 Just missed that last time 😞

The opsec app is installed on the indexers and search head plus on a heavy forwarder that is doing the actual collection from the Checkpoint system.

Also, the config is not changed apart form the testing with different data formats in fw1-loggrabber.conf that I mentioned above.

Re: Splunk only partially recognizes date from OPSEC logs

araitz — Wed, 04 Jun 2014 20:58:51 GMT

hcpr - you should open a support case. We can't recreate that behavior, and haven't seen that with any of the other customers using the add-on. My guess is that something somewhere else on the system is clobbering your configuration.