Splunk Dev

How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Hemnaath
Motivator

Hi All, Currently we are facing an problem in time stamp for a Symantec log data.
Problem: When we search with the below query, we could see that the splunk _time field is different from the event's "time" field.

Query details:

index=sem sourcetype="symantec:tap:incidents" time="2017-08-11T05:01:38.134Z"

Event Details:

Time
8/24/17
3:45:33.000 PM

Event

{ [-]
tap_host: 10.x.x.x

tap_incident_id: xxxxx

deviceUid: [ [+]
]

device_time: 2017-08-11T05:01:38.134Z

domainId: [ [+]
]

event_count: 3

filehash: [ [+]
]

first_event_seen: 2017-08-11T04:41:36.000Z

last_event_seen: 2017-08-11T07:18:37.211Z

log_name: exxx_incident-2017-08-11/incident

priority_level: 2

recommended_action: You can isolate the endpoint(s), remove the file(s) and/or clean the system(s).

state: 1

summary: xxxxxxxx.

time: 2017-08-11T05:01:38.134Z

updated: 2017-08-12T12:52:06.766Z

uuid: 27fc1760-7e52-xxxxxx-0000000001eb

From the Event Action, I could see that in the event time field "2017-08-11T05:01:38.134Z" and in the _time field as "2017-08-24T15:45:33.000-04:00" for the same event, "_time" is not equal to "time".

_time is being calculated based on when it was indexed instead of when it was an event.

Question :

How to make the _time field be the same as the time field ?

Kindly guide me on this.

0 Karma

jkat54
SplunkTrust
SplunkTrust

What is your architecture?

Syslog Server -> Splunk Universal Forwarder -> Splunk Indexer

Please clearly describe the architecture of your integration.

0 Karma

Hemnaath
Motivator

Hi Jkat54,

Architecture details :
Currently we have 5 individual indexer instances, 5 individual heavy forwarder instances, 3 clustered search head, one deployment instance and one Deployer instance, they all are running with splunk 6.6.1 version.

Data flow:

All the remote machine data (UF) are forwarder to all the 5 individual Heavy forwarder instance and then its forwarded to the Indexer instances to index the data.

Syslog:

All the five heavy forwarder instances acts as the syslogs server and so that network,firewall, ESX etc data's are read and forwarded to indexer instances from all five heavy forwarder instances.

This particular Symantec logs are forwarded from third party device called atp server to the heavy forwarder. This is the inputs.conf details

[script://$SPLUNK_HOME/etc/apps/TA-symantec_atp/bin/atp_incidents_collect.py]
disabled = False
index = intrusion

Kindly guide me how to fix this issue.

thanks in advance

0 Karma

jkat54
SplunkTrust
SplunkTrust

So are you putting these props on your heavy forwarder?

0 Karma

Hemnaath
Motivator

yes, the props.conf is in one of the heavy forwarder instances. Kindly guide me how to fix this issue.

0 Karma

Hemnaath
Motivator

Hi jkat54, can you please guide me on this to fix the issue, i am not sure where is the issue.

thanks in advance.

0 Karma

jkat54
SplunkTrust
SplunkTrust

I don’t know what’s wrong at this point. I suggest you reach out to splunk support or post another question and put a link to this question in the new one.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

@Hemnaath - Did you ever get your question answered, or do you still need help on this?

0 Karma

Hemnaath
Motivator

Hi Dal Jeanis, thanks for asking, hey currently the tap application having some issue, so working with the application vendor.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Remove the datetime config like this if you want all data to be EDT:

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
FIELDALIAS-dest = domainId{} as dest
FIELDALIAS-file_hash = filehash{} as file_hash
FIELDALIAS-severity_id = priority_level as severity_id
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

0 Karma

Hemnaath
Motivator

Hi Jkat54, After updating the above stanza i am getting this error after updating the above stanza.

Checking conf files for problems...
            Invalid key in stanza [symantec:tap:incidents] in /opt/splunk/etc/apps/TA-symantec_tap/default/props.conf, line 83: TIMESTAMP_FORMAT (value: %FT%T.%3N).
            Invalid key in stanza [symantec:tap:incidentevents] in /opt/splunk/etc/apps/TA-symantec_tap/default/props.conf, line 100: TIMESTAMP_FORMAT (value: %FT%T.%3N).
            Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
                    Bad strptime format value: '%Y-%m-%dT%H:%M:%S.%L%z', of param: props.conf / [oracle:auth:ovd] / TIME_FORMAT
            One or more time-format strings in your configuration are not valid. For details, please see btool.log or directly above.
    Done
    Checking default conf files for edits...
    Validating installed files against hashes from '/opt/splunk/splunk-6.6.1-aeae3fe0c5af-linux-2.6-x86_64-manifest'
    All installed files intact.
    Done

Kindly guide me on this please.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Sorry, but it is TIME_FORMAT not TIMESTAMP_FORMAT. See props.conf (<- link here) for a list of valid keys.

0 Karma

Hemnaath
Motivator

thanks jkart, after changing the TIMESTAMP_FORMAT to TIME_FORMAT stanza in our Heavy forwarder where the splunk sees the event first .

Invalid Key Stanza got corrected.

But need to validate whether the _time field be the same as the log_time field, as we have some issue in the symantec node current we are not getting the data in splunk from this node.

0 Karma

Hemnaath
Motivator

Hi Jkat54, after making the above changes we are still facing the same issue unable to fix the time stamp issue.

Problem: When we search with the below query, we could see that the splunk _time field is different from the event's "time" field.

Query details:

index=sem sourcetype="symantec:tap:incidents" time="2017-12-05T01:37:08.048Z"

Event details:

12/4/17
8:38:18.000 PM

{ [-]
tap_host: 10.X.X.X

tap_incident_id: 104651
deviceUid: [ [+]
]

device_time: 2017-12-05T01:37:08.048Z

domainId: [ [+]
]

event_count: 1

filehash: [ [+]
]

first_event_seen: 2017-12-05T01:31:24.000Z

last_event_seen: 2017-12-05T01:33:12.000Z

log_name: epmp_incident-2017-12-05/incident

priority_level: 2

recommended_action: Review the SEP settings, isolate the endpoint(s), remove the file(s), and/or clean the system(s).

state: 1

summary: Daily unresolved SEP detection(s)

time: 2017-12-05T01:37:08.048Z

updated: 2017-12-05T01:37:08.441Z

uuid: ce5c8d00-d95c-11e7-d251-00000000005c

}
Show as raw text

From the Event Action, I could see that in the event time field "2017-12-05T01:37:08.048Z" and in the _time field as "2017-12-04 20:38:18" for the same event, "_time" is not equal to "time".

_time is being calculated based on when it was indexed instead of when it was an event.

Props.conf details: We have placed this configuration in Heavy forwarder where the data first reaches the splunk then gets ingested into indexer.

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

Question :

How to make the _time field be the same as the time field ?

Kindly guide me on this.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Try changing TIME_PREFIX to ^time:\s

0 Karma

Hemnaath
Motivator

Hi jkat54, thanks for supporting me again, I will be updating the below stanza in the HF instance and restart the splunk services.

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
KV_MODE = json
TRUNCATE = 0
TIME_PREFIX=^time:\s
TIME_FORMAT=%FT%T.%3N
MAX_TIMESTAMP_LOOKAHEAD=32
TZ=EDT

0 Karma

Hemnaath
Motivator

Hi jkat54, hey I had tried the above stanza what you had mentioned in your comment but it did not work. I am getting the same output, so could you please guide me to fix this issue.

index=sem sourcetype="symantec:tap:incidents"

Event details captured after changing the props.conf

12/5/17
11:05:28.000 PM
{ [-]
atp_host: 10.x.x.x
atp_incident_id: 104656
deviceUid: [ [+]
]
device_time: 2017-12-06T04:03:08.713Z
domainId: [ [+]
]
event_count: 1
first_event_seen: 2017-12-05T09:49:58.118Z
last_event_seen: 2017-12-05T09:49:58.118Z
log_name: epmp_incident-2017-12-06/incident
priority_level: 1
recommended_action: Consider blacklisting the site. In addition, you may need to investigate the source of the exposure to see if further action is required.
state: 1
summary: Malicious domain www.xxxx.com detected
time: 2017-12-06T04:03:08.713Z
updated: 2017-12-06T04:03:08.950Z
uuid: 5e89b190-da3a-11e7-d301-000000000061

thanks in advance

0 Karma

Hemnaath
Motivator

HI jkat54, can you please guide me on this, issue is being there for very long time, need to fix this.

thanks in advance.

0 Karma

Hemnaath
Motivator

Hi All, Can anyone guide me on this issue, I am unable to make the _time field be same as the time field in the event.

thanks in advance.

0 Karma

Hemnaath
Motivator

Hi jkat54, can you please guide me on this issue.

0 Karma

Hemnaath
Motivator

Hi Jkat54, can you please help me on this issue, the issue still there.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...