Hi All, Currently we are facing an problem in time stamp for a Symantec log data.
Problem: When we search with the below query, we could see that the splunk _time field is different from the event's "time" field.
Query details:
index=sem sourcetype="symantec:tap:incidents" time="2017-08-11T05:01:38.134Z"
Event Details:
Time
8/24/17
3:45:33.000 PM
Event
{ [-]
tap_host: 10.x.x.x
tap_incident_id: xxxxx
deviceUid: [ [+]
]
device_time: 2017-08-11T05:01:38.134Z
domainId: [ [+]
]
event_count: 3
filehash: [ [+]
]
first_event_seen: 2017-08-11T04:41:36.000Z
last_event_seen: 2017-08-11T07:18:37.211Z
log_name: exxx_incident-2017-08-11/incident
priority_level: 2
recommended_action: You can isolate the endpoint(s), remove the file(s) and/or clean the system(s).
state: 1
summary: xxxxxxxx.
time: 2017-08-11T05:01:38.134Z
updated: 2017-08-12T12:52:06.766Z
uuid: 27fc1760-7e52-xxxxxx-0000000001eb
From the Event Action, I could see that in the event time field "2017-08-11T05:01:38.134Z" and in the _time field as "2017-08-24T15:45:33.000-04:00" for the same event, "_time" is not equal to "time".
_time is being calculated based on when it was indexed instead of when it was an event.
Question :
How to make the _time field be the same as the time field ?
Kindly guide me on this.
Try changing your MAX_TIMESTAMP_LOOKAHEAD to something crazy like 4096
Do you still have this "DATETIME_CONFIG = CURRENT"
If so, try changing it to DATETIME_CONFIG = NONE
Hi Jkat54, thanks for your much need support on this issue, hey we have removed this stanza from props.conf "DATETIME_CONFIG=CURRENT" and current we have the below stanza configured.
[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=^time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=1024
TZ=EDT
Search details:
index=sem sourcetype="symantec:tap:incidents"
Event Details :
12/6/17
9:41:31.000 PM
{"summary": "Daily unresolved SEP detection(s)", "uuid": "f4f4e6b0-daf6-11e7-f496-000000000065", "recommended_action": "Review the SEP settings, isolate the endpoint(s), remove the file(s), and/or clean the system(s).", "filehash": ["d4d44b71598a87447b9517d0945909152d3f441132bb4867991cb6705b0d03d1", "0860cb4c27223edca4ebdc8c9d4f8043700b156b62366a56b77f3023ec3d18bf"], "time": "2017-12-07T02:33:06.459Z", "tap_incident_id": 104660, "updated": "2017-12-07T02:35:06.478Z", "domainId": ["", "www.ursoft.cn"], "tap_host": "10.X.X.X", "last_event_seen": "2017-12-07T02:28:59.000Z", "priority_level": 2, "event_count": 1, "log_name": "epmp_incident-2017-12-07/incident", "deviceUid": ["dec0cb0c-9ec9-44c6-a45c-0a00be34a303", "7bc90427-1f4e-4460-a698-a864a1cdd503"], "state": 1, "first_event_seen": "2017-12-07T02:27:42.000Z", "device_time": "2017-12-07T02:33:06.459Z"}
The "time" field in the log is 12/7/2017 at 02:33:06, but the _time field in Splunk comes through as 12/06/17 at 9:41:31
Issue still there, so kindly let me know how to calculate the MAX_TIMESTAMP_LOOKAHEAD Value.
Hi jkat54, I have tried that too but it did not work. I have doubt whether the changes will be applicable only to the latest events right not too the old events, is that right ?
Currently this is my props.conf details:
[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=device_time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT
Search details:
index=sem sourcetype="symantec:tap:incidents"
Event Details : Taken before setting the TIME_PREFIX=device_time:\s in props.conf
12/7/17
6:30:07.000 PM
{"time": "2017-12-07T23:27:17.962Z", "log_name": "epmp_incident-2017-12-07/incident", "scanners": ["stap01"], "first_event_seen": "2017-12-07T23:27:18.018Z", "filehash": ["95912bff1d9ba853b5adbafd8aa297ad0e87a0c3d1f5f378b1f37a438c7e91c5"], "priority_level": 1, "device_time": "2017-12-07T23:27:17.962Z", "deviceUid": ["0e76bdc7-f7b8-404a-9e93-9caa224c5b99"], "uuid": "2a5902a0-dba6-11e7-e6ac-000000000067", "summary": "Malicious domain www.driverwizard.org detected", "state": 1, "domainId": ["www.driverwizard.org"], "recommended_action": "Consider blacklisting the site. In addition, you may need to investigate the source of the exposure to see if further action is required.", "tap_incident_id": 104662, "tap_host": "10.X.X.X", "event_count": 1, "last_event_seen": "2017-12-07T23:27:18.018Z", "updated": "2017-12-07T23:27:17.962Z"}
The "time" field in the log is 12/7/2017 at 23:27:17, but the _time field in Splunk comes through as 12/07/17 at 6:30:07
Yes it will only apply to new data coming in... not the old data.
Hi jkat54, good morning, hey still facing the same issue unable to make the _time value same as the time field in the events.
Props.conf details:
[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = atp_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=device_time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT
12/10/17
9:24:06.000 AM
{"event_count": 1, "recommended_action": "Review the SEP settings, isolate the endpoint(s), remove the file(s), and/or clean the system(s).", "priority_level": 2, "tap_host": "10.X.X.X", "device_time": "2017-12-10T14:21:06.651Z", "summary": "Daily unresolved SEP detection(s)", "domainId": [""], "state": 1, "tap_incident_id": 104669, "log_name": "epmp_incident-2017-12-10/incident", "filehash": ["104eabf550b6fe22039a3170d5262a58342afd25395bcb181bf4b407dd4a0a7d"], "first_event_seen": "2017-12-10T14:16:07.000Z", "uuid": "5c5d0ab0-ddb5-11e7-eced-00000000006e", "time": "2017-12-10T14:21:06.651Z", "updated": "2017-12-10T14:21:06.651Z", "deviceUid": ["fe66a353-28d9-47e8-9dcf-83fbfe62dbd3"], "last_event_seen": "2017-12-10T14:16:07.000Z"}
The "time" field in the log is 12/10/2017 at 14:21:06, but the _time field in Splunk comes through as 12/10/17 at 9:24:06
Kindly guide me where I am making the mistake ?
thanks in advance
Do you want device_time? If so, make that your TIME_PREFIX
Try this in props.conf on the forwarder or indexer that is first seeing the data:
[symantec:tap:incidents]
TIME_PREFIX=time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT
Hi Jkat54, thanks for your effort on this, i had run the above query provide by Dal Jeanis and got the above output. Based on the out put can I try this above stanza will that work. Kindly guide on this, as I need to get this fixed.
index=sem sourcetype="symantec:tap:incidents" | head 5| eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")| table time indextime
time
2017-08-29T16:22:01.597Z
2017-08-29T15:59:59.853Z
2017-08-29T15:59:59.852Z
2017-08-29T15:59:59.852Z
2017-08-29T15:42:05.321Z
indextime
2017-08-29T12:25:44
2017-08-29T12:05:44
2017-08-29T12:05:44
2017-08-29T12:05:44
2017-08-29T11:45:44
Also we have another sourcetype=symantec:tap:incidentevents with same issue in this indextime should match the log_time field value in the events.
index=sem sourcetype=symantec:tap:incidentevents | head 5
| eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")
| table log_time indextime
log_time indextime
2017-08-29T16:40:58.819Z 2017-08-29T12:45:43
2017-08-29T16:40:58.821Z 2017-08-29T12:45:43
2017-08-29T16:40:58.800Z 2017-08-29T12:45:43
2017-08-29T16:40:58.798Z 2017-08-29T12:45:43
2017-08-29T16:40:58.778Z 2017-08-29T12:45:43
Please guide me how to make the _time field be the same as the log_time field ?
thanks in advance.
Both are the same time_format but the time prefix is probably different. I can't tell without sample _raw data but I assume time prefix would be log_time:\s for symantec:tap:incidentevents
Hi Jkat, thanks for your effort on this, after executing the below query i am got this event details for a duration of 7 days
index=sem sourcetype=symantec:tap:incidentevents log_time="2017-08-22T13:38:04.899Z"
8/24/17
3:46:06.000 PM
{ [-]
actual_action: Left alone
actual_action_idx: 4
agent_version: 12.1.7004.6500
tap_host: 10.x.x.x
data_source_url_domain: xxx-my.sharepoint.com
device_ip: 10.x.x.x
device_name: xxxxxx
device_time: 2017-08-22T13:32:17.000Z
device_uid: d424209e-a2fc-4956-8355-0657630e9f13
domain_name: test.com
external_ip:
file: { [+]
}
host_name: xxxxx
incident: 20147330-873f-11ex-e5x8-00000000024b
internal_ip: 10.x.x.x
local_host_mac: xx-8b-xx-xx-0d-xx
log_name: xxx_incident-2017-08-22/event
log_time: 2017-08-22T13:38:04.899Z
no_of_viruses: 1
sep_installed: true
source: Real Time Scan
threat: { [+]
}
type_id: 4123
user_name: xxxx
uuid: 204dxxx-873f-11e7-fff5-000000005819
virus_def: 2017-08-22 rev. 001
virus_name: xx.Reputation.1
}
Kindly help me on how to make the _time field be the same as the log_time field ?
thanks in advance
Hi Jkat, thanks for your effort on this, can I configure the props.conf stanza for the sourcetype=symantec:tap:incidentevents. In order to make _time field be the same as the log_time field.
[symantec:tap:incidentevents]
TIME_PREFIX=log_time:\s
TIMESTAMP_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT
thanks in advance.
Yes but the props must be on the first Splunk that sees the data . such as the forwarder.
thanks Jkat.
Jkat, when checked for the props stanza in the Heavy forwarder, I could see the below stanza details for this app and I have added the your stanza along with this stanza. So that fine to push it and restart the service to match the _time field be the same as the time field.
[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
FIELDALIAS-dest = domainId{} as dest
FIELDALIAS-file_hash = filehash{} as file_hash
FIELDALIAS-severity_id = priority_level as severity_id
DATETIME_CONFIG = CURRENT
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=time:\s
TIMESTAMP_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT
thanks in advances.
Basically, that indicates that the indexer was not able to find the time in the event.
To confirm that that is what is happening, rather than a few similar things that might look like this, please run this and compare the times..
index=sem sourcetype="symantec:tap:incidents"
| head 5
| eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")
| table time indextime
If the two fields match each other for all 5 records, then please post your props.conf stanza for the sourcetype, and we'll help you get it figured out.
Hi Dal Jeanis, thanks for guiding me, I had executed the query and got this output and as you said the two fields are matching all the five records.
index=sem sourcetype="symantec:tap:incidents" | head 5 | eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S") | table time indextime
time indextime
2017-08-29T16:22:01.597Z 2017-08-29T12:25:44
2017-08-29T15:59:59.853Z 2017-08-29T12:05:44
2017-08-29T15:59:59.852Z 2017-08-29T12:05:44
2017-08-29T15:59:59.852Z 2017-08-29T12:05:44
2017-08-29T15:42:05.321Z 2017-08-29T11:45:44
Kindly let me know what I have right in props stanza to match the _time field to be the same as the time field in the event.
Similarly i have another sourcetype=symantec:tap:incidentevents which is also having the same issue. in this indextime is not matching the log_time field value in the events.
index=sem sourcetype=symantec:tap:incidentevents | head 5
| eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")
| table log_time indextime
log_time indextime
2017-08-29T16:40:58.819Z 2017-08-29T12:45:43
2017-08-29T16:40:58.821Z 2017-08-29T12:45:43
2017-08-29T16:40:58.800Z 2017-08-29T12:45:43
2017-08-29T16:40:58.798Z 2017-08-29T12:45:43
2017-08-29T16:40:58.778Z 2017-08-29T12:45:43
Kindly guide me on this, on how to fix this issue.