Hi All, Currently we are facing an problem in time stamp for a Symantec log data.
Problem: When we search with the below query, we could see that the splunk _time field is different from the event's "time" field.
Query details:
index=sem sourcetype="symantec:tap:incidents" time="2017-08-11T05:01:38.134Z"
Event Details:
Time
8/24/17
3:45:33.000 PM
Event
{ [-]
tap_host: 10.x.x.x
tap_incident_id: xxxxx
deviceUid: [ [+]
]
device_time: 2017-08-11T05:01:38.134Z
domainId: [ [+]
]
event_count: 3
filehash: [ [+]
]
first_event_seen: 2017-08-11T04:41:36.000Z
last_event_seen: 2017-08-11T07:18:37.211Z
log_name: exxx_incident-2017-08-11/incident
priority_level: 2
recommended_action: You can isolate the endpoint(s), remove the file(s) and/or clean the system(s).
state: 1
summary: xxxxxxxx.
time: 2017-08-11T05:01:38.134Z
updated: 2017-08-12T12:52:06.766Z
uuid: 27fc1760-7e52-xxxxxx-0000000001eb
From the Event Action, I could see that in the event time field "2017-08-11T05:01:38.134Z" and in the _time field as "2017-08-24T15:45:33.000-04:00" for the same event, "_time" is not equal to "time".
_time is being calculated based on when it was indexed instead of when it was an event.
Question :
How to make the _time field be the same as the time field ?
Kindly guide me on this.
What is your architecture?
Syslog Server -> Splunk Universal Forwarder -> Splunk Indexer
Please clearly describe the architecture of your integration.
Hi Jkat54,
Architecture details :
Currently we have 5 individual indexer instances, 5 individual heavy forwarder instances, 3 clustered search head, one deployment instance and one Deployer instance, they all are running with splunk 6.6.1 version.
Data flow:
All the remote machine data (UF) are forwarder to all the 5 individual Heavy forwarder instance and then its forwarded to the Indexer instances to index the data.
Syslog:
All the five heavy forwarder instances acts as the syslogs server and so that network,firewall, ESX etc data's are read and forwarded to indexer instances from all five heavy forwarder instances.
This particular Symantec logs are forwarded from third party device called atp server to the heavy forwarder. This is the inputs.conf details
[script://$SPLUNK_HOME/etc/apps/TA-symantec_atp/bin/atp_incidents_collect.py]
disabled = False
index = intrusion
Kindly guide me how to fix this issue.
thanks in advance
So are you putting these props on your heavy forwarder?
yes, the props.conf is in one of the heavy forwarder instances. Kindly guide me how to fix this issue.
Hi jkat54, can you please guide me on this to fix the issue, i am not sure where is the issue.
thanks in advance.
I don’t know what’s wrong at this point. I suggest you reach out to splunk support or post another question and put a link to this question in the new one.
@Hemnaath - Did you ever get your question answered, or do you still need help on this?
Hi Dal Jeanis, thanks for asking, hey currently the tap application having some issue, so working with the application vendor.
Remove the datetime config like this if you want all data to be EDT:
[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
FIELDALIAS-dest = domainId{} as dest
FIELDALIAS-file_hash = filehash{} as file_hash
FIELDALIAS-severity_id = priority_level as severity_id
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT
Hi Jkat54, After updating the above stanza i am getting this error after updating the above stanza.
Checking conf files for problems...
Invalid key in stanza [symantec:tap:incidents] in /opt/splunk/etc/apps/TA-symantec_tap/default/props.conf, line 83: TIMESTAMP_FORMAT (value: %FT%T.%3N).
Invalid key in stanza [symantec:tap:incidentevents] in /opt/splunk/etc/apps/TA-symantec_tap/default/props.conf, line 100: TIMESTAMP_FORMAT (value: %FT%T.%3N).
Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
Bad strptime format value: '%Y-%m-%dT%H:%M:%S.%L%z', of param: props.conf / [oracle:auth:ovd] / TIME_FORMAT
One or more time-format strings in your configuration are not valid. For details, please see btool.log or directly above.
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-6.6.1-aeae3fe0c5af-linux-2.6-x86_64-manifest'
All installed files intact.
Done
Kindly guide me on this please.
Sorry, but it is TIME_FORMAT not TIMESTAMP_FORMAT. See props.conf (<- link here) for a list of valid keys.
thanks jkart, after changing the TIMESTAMP_FORMAT to TIME_FORMAT stanza in our Heavy forwarder where the splunk sees the event first .
Invalid Key Stanza got corrected.
But need to validate whether the _time field be the same as the log_time field, as we have some issue in the symantec node current we are not getting the data in splunk from this node.
Hi Jkat54, after making the above changes we are still facing the same issue unable to fix the time stamp issue.
Problem: When we search with the below query, we could see that the splunk _time field is different from the event's "time" field.
Query details:
index=sem sourcetype="symantec:tap:incidents" time="2017-12-05T01:37:08.048Z"
Event details:
12/4/17
8:38:18.000 PM
{ [-]
tap_host: 10.X.X.X
tap_incident_id: 104651
deviceUid: [ [+]
]
device_time: 2017-12-05T01:37:08.048Z
domainId: [ [+]
]
event_count: 1
filehash: [ [+]
]
first_event_seen: 2017-12-05T01:31:24.000Z
last_event_seen: 2017-12-05T01:33:12.000Z
log_name: epmp_incident-2017-12-05/incident
priority_level: 2
recommended_action: Review the SEP settings, isolate the endpoint(s), remove the file(s), and/or clean the system(s).
state: 1
summary: Daily unresolved SEP detection(s)
time: 2017-12-05T01:37:08.048Z
updated: 2017-12-05T01:37:08.441Z
uuid: ce5c8d00-d95c-11e7-d251-00000000005c
}
Show as raw text
From the Event Action, I could see that in the event time field "2017-12-05T01:37:08.048Z" and in the _time field as "2017-12-04 20:38:18" for the same event, "_time" is not equal to "time".
_time is being calculated based on when it was indexed instead of when it was an event.
Props.conf details: We have placed this configuration in Heavy forwarder where the data first reaches the splunk then gets ingested into indexer.
[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT
Question :
How to make the _time field be the same as the time field ?
Kindly guide me on this.
Try changing TIME_PREFIX to ^time:\s
Hi jkat54, thanks for supporting me again, I will be updating the below stanza in the HF instance and restart the splunk services.
[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=^time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT
Hi jkat54, hey I had tried the above stanza what you had mentioned in your comment but it did not work. I am getting the same output, so could you please guide me to fix this issue.
index=sem sourcetype="symantec:tap:incidents"
Event details captured after changing the props.conf
12/5/17
11:05:28.000 PM
{ [-]
atp_host: 10.x.x.x
atp_incident_id: 104656
deviceUid: [ [+]
]
device_time: 2017-12-06T04:03:08.713Z
domainId: [ [+]
]
event_count: 1
first_event_seen: 2017-12-05T09:49:58.118Z
last_event_seen: 2017-12-05T09:49:58.118Z
log_name: epmp_incident-2017-12-06/incident
priority_level: 1
recommended_action: Consider blacklisting the site. In addition, you may need to investigate the source of the exposure to see if further action is required.
state: 1
summary: Malicious domain www.xxxx.com detected
time: 2017-12-06T04:03:08.713Z
updated: 2017-12-06T04:03:08.950Z
uuid: 5e89b190-da3a-11e7-d301-000000000061
thanks in advance
HI jkat54, can you please guide me on this, issue is being there for very long time, need to fix this.
thanks in advance.
Hi All, Can anyone guide me on this issue, I am unable to make the _time field be same as the time field in the event.
thanks in advance.
Hi jkat54, can you please guide me on this issue.
Hi Jkat54, can you please help me on this issue, the issue still there.