Splunk Dev

How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath
Motivator

Hi All, Currently we are facing an problem in time stamp for a Symantec log data.
Problem: When we search with the below query, we could see that the splunk _time field is different from the event's "time" field.

Query details:

index=sem sourcetype="symantec:tap:incidents" time="2017-08-11T05:01:38.134Z"

Event Details:

Time
8/24/17
3:45:33.000 PM

Event

{ [-]
tap_host: 10.x.x.x

tap_incident_id: xxxxx

deviceUid: [ [+]
]

device_time: 2017-08-11T05:01:38.134Z

domainId: [ [+]
]

event_count: 3

filehash: [ [+]
]

first_event_seen: 2017-08-11T04:41:36.000Z

last_event_seen: 2017-08-11T07:18:37.211Z

log_name: exxx_incident-2017-08-11/incident

priority_level: 2

recommended_action: You can isolate the endpoint(s), remove the file(s) and/or clean the system(s).

state: 1

summary: xxxxxxxx.

time: 2017-08-11T05:01:38.134Z

updated: 2017-08-12T12:52:06.766Z

uuid: 27fc1760-7e52-xxxxxx-0000000001eb

From the Event Action, I could see that in the event time field "2017-08-11T05:01:38.134Z" and in the _time field as "2017-08-24T15:45:33.000-04:00" for the same event, "_time" is not equal to "time".

_time is being calculated based on when it was indexed instead of when it was an event.

Question :

How to make the _time field be the same as the time field ?

Kindly guide me on this.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Try changing your MAX_TIMESTAMP_LOOKAHEAD to something crazy like 4096

Do you still have this "DATETIME_CONFIG = CURRENT"

If so, try changing it to DATETIME_CONFIG = NONE

0 Karma

Hemnaath
Motivator

Hi Jkat54, thanks for your much need support on this issue, hey we have removed this stanza from props.conf "DATETIME_CONFIG=CURRENT" and current we have the below stanza configured.

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=^time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=1024
TZ=EDT

Search details:
index=sem sourcetype="symantec:tap:incidents"

Event Details :

12/6/17
9:41:31.000 PM

{"summary": "Daily unresolved SEP detection(s)", "uuid": "f4f4e6b0-daf6-11e7-f496-000000000065", "recommended_action": "Review the SEP settings, isolate the endpoint(s), remove the file(s), and/or clean the system(s).", "filehash": ["d4d44b71598a87447b9517d0945909152d3f441132bb4867991cb6705b0d03d1", "0860cb4c27223edca4ebdc8c9d4f8043700b156b62366a56b77f3023ec3d18bf"], "time": "2017-12-07T02:33:06.459Z", "tap_incident_id": 104660, "updated": "2017-12-07T02:35:06.478Z", "domainId": ["", "www.ursoft.cn"], "tap_host": "10.X.X.X", "last_event_seen": "2017-12-07T02:28:59.000Z", "priority_level": 2, "event_count": 1, "log_name": "epmp_incident-2017-12-07/incident", "deviceUid": ["dec0cb0c-9ec9-44c6-a45c-0a00be34a303", "7bc90427-1f4e-4460-a698-a864a1cdd503"], "state": 1, "first_event_seen": "2017-12-07T02:27:42.000Z", "device_time": "2017-12-07T02:33:06.459Z"}

The "time" field in the log is 12/7/2017 at 02:33:06, but the _time field in Splunk comes through as 12/06/17 at 9:41:31

Issue still there, so kindly let me know how to calculate the MAX_TIMESTAMP_LOOKAHEAD Value.

0 Karma

Hemnaath
Motivator

Hi jkat54, I have tried that too but it did not work. I have doubt whether the changes will be applicable only to the latest events right not too the old events, is that right ?

Currently this is my props.conf details:

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=device_time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

Search details:
index=sem sourcetype="symantec:tap:incidents"

Event Details : Taken before setting the TIME_PREFIX=device_time:\s in props.conf

12/7/17
6:30:07.000 PM

{"time": "2017-12-07T23:27:17.962Z", "log_name": "epmp_incident-2017-12-07/incident", "scanners": ["stap01"], "first_event_seen": "2017-12-07T23:27:18.018Z", "filehash": ["95912bff1d9ba853b5adbafd8aa297ad0e87a0c3d1f5f378b1f37a438c7e91c5"], "priority_level": 1, "device_time": "2017-12-07T23:27:17.962Z", "deviceUid": ["0e76bdc7-f7b8-404a-9e93-9caa224c5b99"], "uuid": "2a5902a0-dba6-11e7-e6ac-000000000067", "summary": "Malicious domain www.driverwizard.org detected", "state": 1, "domainId": ["www.driverwizard.org"], "recommended_action": "Consider blacklisting the site. In addition, you may need to investigate the source of the exposure to see if further action is required.", "tap_incident_id": 104662, "tap_host": "10.X.X.X", "event_count": 1, "last_event_seen": "2017-12-07T23:27:18.018Z", "updated": "2017-12-07T23:27:17.962Z"}

The "time" field in the log is 12/7/2017 at 23:27:17, but the _time field in Splunk comes through as 12/07/17 at 6:30:07

0 Karma

jkat54
SplunkTrust
SplunkTrust

Yes it will only apply to new data coming in... not the old data.

0 Karma

Hemnaath
Motivator

Hi jkat54, good morning, hey still facing the same issue unable to make the _time value same as the time field in the events.

Props.conf details:

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = atp_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=device_time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

12/10/17
9:24:06.000 AM

{"event_count": 1, "recommended_action": "Review the SEP settings, isolate the endpoint(s), remove the file(s), and/or clean the system(s).", "priority_level": 2, "tap_host": "10.X.X.X", "device_time": "2017-12-10T14:21:06.651Z", "summary": "Daily unresolved SEP detection(s)", "domainId": [""], "state": 1, "tap_incident_id": 104669, "log_name": "epmp_incident-2017-12-10/incident", "filehash": ["104eabf550b6fe22039a3170d5262a58342afd25395bcb181bf4b407dd4a0a7d"], "first_event_seen": "2017-12-10T14:16:07.000Z", "uuid": "5c5d0ab0-ddb5-11e7-eced-00000000006e", "time": "2017-12-10T14:21:06.651Z", "updated": "2017-12-10T14:21:06.651Z", "deviceUid": ["fe66a353-28d9-47e8-9dcf-83fbfe62dbd3"], "last_event_seen": "2017-12-10T14:16:07.000Z"}

The "time" field in the log is 12/10/2017 at 14:21:06, but the _time field in Splunk comes through as 12/10/17 at 9:24:06

Kindly guide me where I am making the mistake ?

thanks in advance

0 Karma

jkat54
SplunkTrust
SplunkTrust

Do you want device_time? If so, make that your TIME_PREFIX

0 Karma

jkat54
SplunkTrust
SplunkTrust

Try this in props.conf on the forwarder or indexer that is first seeing the data:

[symantec:tap:incidents]
TIME_PREFIX=time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT
0 Karma

Hemnaath
Motivator

Hi Jkat54, thanks for your effort on this, i had run the above query provide by Dal Jeanis and got the above output. Based on the out put can I try this above stanza will that work. Kindly guide on this, as I need to get this fixed.

index=sem sourcetype="symantec:tap:incidents" | head 5| eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")| table time indextime

time

2017-08-29T16:22:01.597Z
2017-08-29T15:59:59.853Z
2017-08-29T15:59:59.852Z
2017-08-29T15:59:59.852Z
2017-08-29T15:42:05.321Z

indextime
2017-08-29T12:25:44
2017-08-29T12:05:44
2017-08-29T12:05:44
2017-08-29T12:05:44
2017-08-29T11:45:44

Also we have another sourcetype=symantec:tap:incidentevents with same issue in this indextime should match the log_time field value in the events.

index=sem sourcetype=symantec:tap:incidentevents | head 5
| eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")
| table log_time indextime

log_time indextime
2017-08-29T16:40:58.819Z 2017-08-29T12:45:43
2017-08-29T16:40:58.821Z 2017-08-29T12:45:43
2017-08-29T16:40:58.800Z 2017-08-29T12:45:43
2017-08-29T16:40:58.798Z 2017-08-29T12:45:43
2017-08-29T16:40:58.778Z 2017-08-29T12:45:43

Please guide me how to make the _time field be the same as the log_time field ?
thanks in advance.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Both are the same time_format but the time prefix is probably different. I can't tell without sample _raw data but I assume time prefix would be log_time:\s for symantec:tap:incidentevents

0 Karma

Hemnaath
Motivator

Hi Jkat, thanks for your effort on this, after executing the below query i am got this event details for a duration of 7 days

index=sem sourcetype=symantec:tap:incidentevents log_time="2017-08-22T13:38:04.899Z"

8/24/17
3:46:06.000 PM

{ [-]
actual_action: Left alone
actual_action_idx: 4

agent_version: 12.1.7004.6500
tap_host: 10.x.x.x

data_source_url_domain: xxx-my.sharepoint.com

device_ip: 10.x.x.x

device_name: xxxxxx
device_time: 2017-08-22T13:32:17.000Z

device_uid: d424209e-a2fc-4956-8355-0657630e9f13

domain_name: test.com

external_ip:

file: { [+]
}

host_name: xxxxx

incident: 20147330-873f-11ex-e5x8-00000000024b

internal_ip: 10.x.x.x

local_host_mac: xx-8b-xx-xx-0d-xx

log_name: xxx_incident-2017-08-22/event

log_time: 2017-08-22T13:38:04.899Z

no_of_viruses: 1

sep_installed: true

source: Real Time Scan
threat: { [+]
}

type_id: 4123

user_name: xxxx

uuid: 204dxxx-873f-11e7-fff5-000000005819

virus_def: 2017-08-22 rev. 001

virus_name: xx.Reputation.1

}
Kindly help me on how to make the _time field be the same as the log_time field ?

thanks in advance

0 Karma

Hemnaath
Motivator

Hi Jkat, thanks for your effort on this, can I configure the props.conf stanza for the sourcetype=symantec:tap:incidentevents. In order to make _time field be the same as the log_time field.

[symantec:tap:incidentevents]
TIME_PREFIX=log_time:\s
TIMESTAMP_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

thanks in advance.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Yes but the props must be on the first Splunk that sees the data . such as the forwarder.

0 Karma

Hemnaath
Motivator

thanks Jkat.

0 Karma

Hemnaath
Motivator

Jkat, when checked for the props stanza in the Heavy forwarder, I could see the below stanza details for this app and I have added the your stanza along with this stanza. So that fine to push it and restart the service to match the _time field be the same as the time field.

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
FIELDALIAS-dest = domainId{} as dest
FIELDALIAS-file_hash = filehash{} as file_hash
FIELDALIAS-severity_id = priority_level as severity_id
DATETIME_CONFIG = CURRENT
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=time:\s
TIMESTAMP_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

thanks in advances.

0 Karma

DalJeanis
Legend

Basically, that indicates that the indexer was not able to find the time in the event.

To confirm that that is what is happening, rather than a few similar things that might look like this, please run this and compare the times..

  index=sem sourcetype="symantec:tap:incidents"
  | head 5
  | eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")
  | table time indextime

If the two fields match each other for all 5 records, then please post your props.conf stanza for the sourcetype, and we'll help you get it figured out.

0 Karma

Hemnaath
Motivator

Hi Dal Jeanis, thanks for guiding me, I had executed the query and got this output and as you said the two fields are matching all the five records.

index=sem sourcetype="symantec:tap:incidents" | head 5 | eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S") | table time indextime

time                                        indextime

2017-08-29T16:22:01.597Z 2017-08-29T12:25:44
2017-08-29T15:59:59.853Z 2017-08-29T12:05:44
2017-08-29T15:59:59.852Z 2017-08-29T12:05:44
2017-08-29T15:59:59.852Z 2017-08-29T12:05:44
2017-08-29T15:42:05.321Z 2017-08-29T11:45:44

Kindly let me know what I have right in props stanza to match the _time field to be the same as the time field in the event.

Similarly i have another sourcetype=symantec:tap:incidentevents which is also having the same issue. in this indextime is not matching the log_time field value in the events.

index=sem sourcetype=symantec:tap:incidentevents | head 5
| eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")
| table log_time indextime

log_time indextime
2017-08-29T16:40:58.819Z 2017-08-29T12:45:43
2017-08-29T16:40:58.821Z 2017-08-29T12:45:43
2017-08-29T16:40:58.800Z 2017-08-29T12:45:43
2017-08-29T16:40:58.798Z 2017-08-29T12:45:43
2017-08-29T16:40:58.778Z 2017-08-29T12:45:43

Kindly guide me on this, on how to fix this issue.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...