topic Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ? in Splunk Dev

How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Tue, 29 Sep 2020 15:31:32 GMT

Hi All, Currently we are facing an problem in time stamp for a Symantec log data.
Problem: When we search with the below query, we could see that the splunk _time field is different from the event's "time" field.

Query details:

index=sem sourcetype="symantec:tap:incidents" time="2017-08-11T05:01:38.134Z"

Event Details:

Time
8/24/17
3:45:33.000 PM

Event

{ [-]
tap_host: 10.x.x.x

tap_incident_id: xxxxx

deviceUid: [ [+]
]

device_time: 2017-08-11T05:01:38.134Z

domainId: [ [+]
]

event_count: 3

filehash: [ [+]
]

first_event_seen: 2017-08-11T04:41:36.000Z

last_event_seen: 2017-08-11T07:18:37.211Z

log_name: exxx_incident-2017-08-11/incident

priority_level: 2

recommended_action: You can isolate the endpoint(s), remove the file(s) and/or clean the system(s).

state: 1

summary: xxxxxxxx.

time: 2017-08-11T05:01:38.134Z

updated: 2017-08-12T12:52:06.766Z

uuid: 27fc1760-7e52-xxxxxx-0000000001eb

From the Event Action, I could see that in the event time field "2017-08-11T05:01:38.134Z" and in the _time field as "2017-08-24T15:45:33.000-04:00" for the same event, "_time" is not equal to "time".

_time is being calculated based on when it was indexed instead of when it was an event.

Question :

How to make the _time field be the same as the time field ?

Kindly guide me on this.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

DalJeanis — Tue, 29 Aug 2017 23:22:29 GMT

Basically, that indicates that the indexer was not able to find the time in the event.

To confirm that that is what is happening, rather than a few similar things that might look like this, please run this and compare the times..

  index=sem sourcetype="symantec:tap:incidents"
  | head 5
  | eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")
  | table time indextime

If the two fields match each other for all 5 records, then please post your props.conf stanza for the sourcetype, and we'll help you get it figured out.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

jkat54 — Tue, 29 Aug 2017 23:37:12 GMT

Try this in props.conf on the forwarder or indexer that is first seeing the data:

[symantec:tap:incidents]
TIME_PREFIX=time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Tue, 29 Sep 2020 15:35:15 GMT

Hi Dal Jeanis, thanks for guiding me, I had executed the query and got this output and as you said the two fields are matching all the five records.

index=sem sourcetype="symantec:tap:incidents" | head 5 | eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S") | table time indextime

time                                        indextime

2017-08-29T16:22:01.597Z 2017-08-29T12:25:44
2017-08-29T15:59:59.853Z 2017-08-29T12:05:44
2017-08-29T15:59:59.852Z 2017-08-29T12:05:44
2017-08-29T15:59:59.852Z 2017-08-29T12:05:44
2017-08-29T15:42:05.321Z 2017-08-29T11:45:44

Kindly let me know what I have right in props stanza to match the _time field to be the same as the time field in the event.

Similarly i have another sourcetype=symantec:tap:incidentevents which is also having the same issue. in this indextime is not matching the log_time field value in the events.

index=sem sourcetype=symantec:tap:incidentevents | head 5
| eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")
| table log_time indextime

log_time indextime
2017-08-29T16:40:58.819Z 2017-08-29T12:45:43
2017-08-29T16:40:58.821Z 2017-08-29T12:45:43
2017-08-29T16:40:58.800Z 2017-08-29T12:45:43
2017-08-29T16:40:58.798Z 2017-08-29T12:45:43
2017-08-29T16:40:58.778Z 2017-08-29T12:45:43

Kindly guide me on this, on how to fix this issue.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Tue, 29 Sep 2020 15:35:18 GMT

Hi Jkat54, thanks for your effort on this, i had run the above query provide by Dal Jeanis and got the above output. Based on the out put can I try this above stanza will that work. Kindly guide on this, as I need to get this fixed.

index=sem sourcetype="symantec:tap:incidents" | head 5| eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")| table time indextime

time

2017-08-29T16:22:01.597Z
2017-08-29T15:59:59.853Z
2017-08-29T15:59:59.852Z
2017-08-29T15:59:59.852Z
2017-08-29T15:42:05.321Z

indextime
2017-08-29T12:25:44
2017-08-29T12:05:44
2017-08-29T12:05:44
2017-08-29T12:05:44
2017-08-29T11:45:44

Also we have another sourcetype=symantec:tap:incidentevents with same issue in this indextime should match the log_time field value in the events.

index=sem sourcetype=symantec:tap:incidentevents | head 5
| eval indextime = strftime(_indextime, "%Y-%m-%dT%H:%M:%S")
| table log_time indextime

log_time indextime
2017-08-29T16:40:58.819Z 2017-08-29T12:45:43
2017-08-29T16:40:58.821Z 2017-08-29T12:45:43
2017-08-29T16:40:58.800Z 2017-08-29T12:45:43
2017-08-29T16:40:58.798Z 2017-08-29T12:45:43
2017-08-29T16:40:58.778Z 2017-08-29T12:45:43

Please guide me how to make the _time field be the same as the log_time field ?
thanks in advance.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

jkat54 — Tue, 29 Sep 2020 15:35:24 GMT

Both are the same time_format but the time prefix is probably different. I can't tell without sample _raw data but I assume time prefix would be log_time:\s for symantec:tap:incidentevents

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Tue, 29 Sep 2020 15:35:28 GMT

Hi Jkat, thanks for your effort on this, after executing the below query i am got this event details for a duration of 7 days

index=sem sourcetype=symantec:tap:incidentevents log_time="2017-08-22T13:38:04.899Z"

8/24/17
3:46:06.000 PM

{ [-]
actual_action: Left alone
actual_action_idx: 4

agent_version: 12.1.7004.6500
tap_host: 10.x.x.x

data_source_url_domain: xxx-my.sharepoint.com

device_ip: 10.x.x.x

device_name: xxxxxx
device_time: 2017-08-22T13:32:17.000Z

device_uid: d424209e-a2fc-4956-8355-0657630e9f13

domain_name: test.com

external_ip:

file: { [+]
}

host_name: xxxxx

incident: 20147330-873f-11ex-e5x8-00000000024b

internal_ip: 10.x.x.x

local_host_mac: xx-8b-xx-xx-0d-xx

log_name: xxx_incident-2017-08-22/event

log_time: 2017-08-22T13:38:04.899Z

no_of_viruses: 1

sep_installed: true

source: Real Time Scan
threat: { [+]
}

type_id: 4123

user_name: xxxx

uuid: 204dxxx-873f-11e7-fff5-000000005819

virus_def: 2017-08-22 rev. 001

virus_name: xx.Reputation.1

}
Kindly help me on how to make the _time field be the same as the log_time field ?

thanks in advance

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Tue, 29 Sep 2020 15:35:36 GMT

Hi Jkat, thanks for your effort on this, can I configure the props.conf stanza for the sourcetype=symantec:tap:incidentevents. In order to make _time field be the same as the log_time field.

[symantec:tap:incidentevents]
TIME_PREFIX=log_time:\s
TIMESTAMP_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

thanks in advance.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

jkat54 — Thu, 31 Aug 2017 20:08:37 GMT

Yes but the props must be on the first Splunk that sees the data . such as the forwarder.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Thu, 31 Aug 2017 20:36:17 GMT

thanks Jkat.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Tue, 29 Sep 2020 15:35:57 GMT

Jkat, when checked for the props stanza in the Heavy forwarder, I could see the below stanza details for this app and I have added the your stanza along with this stanza. So that fine to push it and restart the service to match the _time field be the same as the time field.

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
FIELDALIAS-dest = domainId{} as dest
FIELDALIAS-file_hash = filehash{} as file_hash
FIELDALIAS-severity_id = priority_level as severity_id
DATETIME_CONFIG = CURRENT
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=time:\s
TIMESTAMP_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

thanks in advances.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

jkat54 — Tue, 29 Sep 2020 15:36:00 GMT

Remove the datetime config like this if you want all data to be EDT:

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
FIELDALIAS-dest = domainId{} as dest
FIELDALIAS-file_hash = filehash{} as file_hash
FIELDALIAS-severity_id = priority_level as severity_id
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Wed, 06 Sep 2017 13:13:51 GMT

Hi Jkat54, After updating the above stanza i am getting this error after updating the above stanza.

Checking conf files for problems...
            Invalid key in stanza [symantec:tap:incidents] in /opt/splunk/etc/apps/TA-symantec_tap/default/props.conf, line 83: TIMESTAMP_FORMAT (value: %FT%T.%3N).
            Invalid key in stanza [symantec:tap:incidentevents] in /opt/splunk/etc/apps/TA-symantec_tap/default/props.conf, line 100: TIMESTAMP_FORMAT (value: %FT%T.%3N).
            Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
                    Bad strptime format value: '%Y-%m-%dT%H:%M:%S.%L%z', of param: props.conf / [oracle:auth:ovd] / TIME_FORMAT
            One or more time-format strings in your configuration are not valid. For details, please see btool.log or directly above.
    Done
    Checking default conf files for edits...
    Validating installed files against hashes from '/opt/splunk/splunk-6.6.1-aeae3fe0c5af-linux-2.6-x86_64-manifest'
    All installed files intact.
    Done

Kindly guide me on this please.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

jkat54 — Tue, 29 Sep 2020 15:39:30 GMT

Sorry, but it is TIME_FORMAT not TIMESTAMP_FORMAT. See props.conf (<- link here) for a list of valid keys.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Tue, 29 Sep 2020 15:39:44 GMT

thanks jkart, after changing the TIMESTAMP_FORMAT to TIME_FORMAT stanza in our Heavy forwarder where the splunk sees the event first .

Invalid Key Stanza got corrected.

But need to validate whether the _time field be the same as the log_time field, as we have some issue in the symantec node current we are not getting the data in splunk from this node.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

DalJeanis — Fri, 06 Oct 2017 20:19:14 GMT

@Hemnaath - Did you ever get your question answered, or do you still need help on this?

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Mon, 09 Oct 2017 14:03:04 GMT

Hi Dal Jeanis, thanks for asking, hey currently the tap application having some issue, so working with the application vendor.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Tue, 29 Sep 2020 17:08:17 GMT

Hi Jkat54, after making the above changes we are still facing the same issue unable to fix the time stamp issue.

Problem: When we search with the below query, we could see that the splunk _time field is different from the event's "time" field.

Query details:

index=sem sourcetype="symantec:tap:incidents" time="2017-12-05T01:37:08.048Z"

Event details:

12/4/17
8:38:18.000 PM

{ [-]
tap_host: 10.X.X.X

tap_incident_id: 104651
deviceUid: [ [+]
]

device_time: 2017-12-05T01:37:08.048Z

domainId: [ [+]
]

event_count: 1

filehash: [ [+]
]

first_event_seen: 2017-12-05T01:31:24.000Z

last_event_seen: 2017-12-05T01:33:12.000Z

log_name: epmp_incident-2017-12-05/incident

priority_level: 2

recommended_action: Review the SEP settings, isolate the endpoint(s), remove the file(s), and/or clean the system(s).

state: 1

summary: Daily unresolved SEP detection(s)

time: 2017-12-05T01:37:08.048Z

updated: 2017-12-05T01:37:08.441Z

uuid: ce5c8d00-d95c-11e7-d251-00000000005c

}
Show as raw text

From the Event Action, I could see that in the event time field "2017-12-05T01:37:08.048Z" and in the _time field as "2017-12-04 20:38:18" for the same event, "_time" is not equal to "time".

_time is being calculated based on when it was indexed instead of when it was an event.

Props.conf details: We have placed this configuration in Heavy forwarder where the data first reaches the splunk then gets ingested into indexer.

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

Question :

How to make the _time field be the same as the time field ?

Kindly guide me on this.

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

jkat54 — Tue, 05 Dec 2017 14:13:43 GMT

Try changing TIME_PREFIX to ^time:\s

Re: How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath — Tue, 29 Sep 2020 17:08:35 GMT

Hi jkat54, thanks for supporting me again, I will be updating the below stanza in the HF instance and restart the splunk services.

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=^time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT