Archive

How to configure a time stamp for Symantec logs to correlate log_time field instead of _time field ?

Motivator

Hi All, Currently we are facing an problem in time stamp for a Symantec log data.
Problem: When we search with the below query, we could see that the splunk _time field is different from the event's "time" field.

Query details:

index=sem sourcetype="symantec:tap:incidents" time="2017-08-11T05:01:38.134Z"

Event Details:

Time
8/24/17
3:45:33.000 PM

Event

{ [-]
taphost: 10.x.x.x

tap
incidentid: xxxxx

deviceUid: [ [+]
]

device
time: 2017-08-11T05:01:38.134Z

domainId: [ [+]
]

eventcount: 3

filehash: [ [+]
]

first
eventseen: 2017-08-11T04:41:36.000Z

last
eventseen: 2017-08-11T07:18:37.211Z

log
name: exxxincident-2017-08-11/incident

priority
level: 2

recommended_action: You can isolate the endpoint(s), remove the file(s) and/or clean the system(s).

state: 1

summary: xxxxxxxx.

time: 2017-08-11T05:01:38.134Z

updated: 2017-08-12T12:52:06.766Z

uuid: 27fc1760-7e52-xxxxxx-0000000001eb

From the Event Action, I could see that in the event time field "2017-08-11T05:01:38.134Z" and in the time field as "2017-08-24T15:45:33.000-04:00" for the same event, "time" is not equal to "time".

_time is being calculated based on when it was indexed instead of when it was an event.

Question :

How to make the _time field be the same as the time field ?

Kindly guide me on this.

0 Karma

SplunkTrust
SplunkTrust

What is your architecture?

Syslog Server -> Splunk Universal Forwarder -> Splunk Indexer

Please clearly describe the architecture of your integration.

0 Karma

Motivator

Hi Jkat54,

Architecture details :
Currently we have 5 individual indexer instances, 5 individual heavy forwarder instances, 3 clustered search head, one deployment instance and one Deployer instance, they all are running with splunk 6.6.1 version.

Data flow:

All the remote machine data (UF) are forwarder to all the 5 individual Heavy forwarder instance and then its forwarded to the Indexer instances to index the data.

Syslog:

All the five heavy forwarder instances acts as the syslogs server and so that network,firewall, ESX etc data's are read and forwarded to indexer instances from all five heavy forwarder instances.

This particular Symantec logs are forwarded from third party device called atp server to the heavy forwarder. This is the inputs.conf details

[script://$SPLUNKHOME/etc/apps/TA-symantecatp/bin/atpincidentscollect.py]
disabled = False
index = intrusion

Kindly guide me how to fix this issue.

thanks in advance

0 Karma

SplunkTrust
SplunkTrust

So are you putting these props on your heavy forwarder?

0 Karma

Motivator

yes, the props.conf is in one of the heavy forwarder instances. Kindly guide me how to fix this issue.

0 Karma

Motivator

Hi jkat54, can you please guide me on this to fix the issue, i am not sure where is the issue.

thanks in advance.

0 Karma

SplunkTrust
SplunkTrust

I don’t know what’s wrong at this point. I suggest you reach out to splunk support or post another question and put a link to this question in the new one.

0 Karma

SplunkTrust
SplunkTrust

@Hemnaath - Did you ever get your question answered, or do you still need help on this?

0 Karma

Motivator

Hi Dal Jeanis, thanks for asking, hey currently the tap application having some issue, so working with the application vendor.

0 Karma

SplunkTrust
SplunkTrust

Remove the datetime config like this if you want all data to be EDT:

[symantec:tap:incidents]
SHOULDLINEMERGE = false
FIELDALIAS-event
host = taphost as eventhost
FIELDALIAS-dest = domainId{} as dest
FIELDALIAS-filehash = filehash{} as filehash
FIELDALIAS-severityid = prioritylevel as severityid
KV
MODE = json
TRUNCATE = 0
TIMEPREFIX=time:\s
TIME
FORMAT=%FT%T.%3N
MAXTIMESTAMPLOOKAHEAD=32
TZ=EDT

0 Karma

Motivator

Hi Jkat54, After updating the above stanza i am getting this error after updating the above stanza.

Checking conf files for problems...
            Invalid key in stanza [symantec:tap:incidents] in /opt/splunk/etc/apps/TA-symantec_tap/default/props.conf, line 83: TIMESTAMP_FORMAT (value: %FT%T.%3N).
            Invalid key in stanza [symantec:tap:incidentevents] in /opt/splunk/etc/apps/TA-symantec_tap/default/props.conf, line 100: TIMESTAMP_FORMAT (value: %FT%T.%3N).
            Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
                    Bad strptime format value: '%Y-%m-%dT%H:%M:%S.%L%z', of param: props.conf / [oracle:auth:ovd] / TIME_FORMAT
            One or more time-format strings in your configuration are not valid. For details, please see btool.log or directly above.
    Done
    Checking default conf files for edits...
    Validating installed files against hashes from '/opt/splunk/splunk-6.6.1-aeae3fe0c5af-linux-2.6-x86_64-manifest'
    All installed files intact.
    Done

Kindly guide me on this please.

0 Karma

SplunkTrust
SplunkTrust

Sorry, but it is TIMEFORMAT not TIMESTAMPFORMAT. See props.conf (<- link here) for a list of valid keys.

0 Karma

Motivator

thanks jkart, after changing the TIMESTAMPFORMAT to TIMEFORMAT stanza in our Heavy forwarder where the splunk sees the event first .

Invalid Key Stanza got corrected.

But need to validate whether the time field be the same as the logtime field, as we have some issue in the symantec node current we are not getting the data in splunk from this node.

0 Karma

Motivator

Hi Jkat54, after making the above changes we are still facing the same issue unable to fix the time stamp issue.

Problem: When we search with the below query, we could see that the splunk _time field is different from the event's "time" field.

Query details:

index=sem sourcetype="symantec:tap:incidents" time="2017-12-05T01:37:08.048Z"

Event details:

12/4/17
8:38:18.000 PM

{ [-]
taphost: 10.X.X.X

tap
incidentid: 104651
deviceUid: [ [+]
]

device
time: 2017-12-05T01:37:08.048Z

domainId: [ [+]
]

eventcount: 1

filehash: [ [+]
]

first
eventseen: 2017-12-05T01:31:24.000Z

last
eventseen: 2017-12-05T01:33:12.000Z

log
name: epmpincident-2017-12-05/incident

priority
level: 2

recommended_action: Review the SEP settings, isolate the endpoint(s), remove the file(s), and/or clean the system(s).

state: 1

summary: Daily unresolved SEP detection(s)

time: 2017-12-05T01:37:08.048Z

updated: 2017-12-05T01:37:08.441Z

uuid: ce5c8d00-d95c-11e7-d251-00000000005c

}
Show as raw text

From the Event Action, I could see that in the event time field "2017-12-05T01:37:08.048Z" and in the time field as "2017-12-04 20:38:18" for the same event, "time" is not equal to "time".

_time is being calculated based on when it was indexed instead of when it was an event.

Props.conf details: We have placed this configuration in Heavy forwarder where the data first reaches the splunk then gets ingested into indexer.

[symantec:tap:incidents]
SHOULDLINEMERGE = false
FIELDALIAS-event
host = taphost as eventhost
KVMODE = json
TRUNCATE = 0
TIME
PREFIX=time:\s
TIMEFORMAT=%FT%T.%3N
MAX
TIMESTAMP_LOOKAHEAD=32
TZ=EDT

Question :

How to make the _time field be the same as the time field ?

Kindly guide me on this.

0 Karma

SplunkTrust
SplunkTrust

Try changing TIME_PREFIX to ^time:\s

0 Karma

Motivator

Hi jkat54, thanks for supporting me again, I will be updating the below stanza in the HF instance and restart the splunk services.

[symantec:tap:incidents]
SHOULDLINEMERGE = false
FIELDALIAS-event
host = taphost as eventhost
KVMODE = json
TRUNCATE = 0
TIME
PREFIX=^time:\s
TIMEFORMAT=%FT%T.%3N
MAX
TIMESTAMP_LOOKAHEAD=32
TZ=EDT

0 Karma

Motivator

Hi jkat54, hey I had tried the above stanza what you had mentioned in your comment but it did not work. I am getting the same output, so could you please guide me to fix this issue.

index=sem sourcetype="symantec:tap:incidents"

Event details captured after changing the props.conf

12/5/17
11:05:28.000 PM
{ [-]
atphost: 10.x.x.x
atp
incidentid: 104656
deviceUid: [ [+]
]
device
time: 2017-12-06T04:03:08.713Z
domainId: [ [+]
]
eventcount: 1
first
eventseen: 2017-12-05T09:49:58.118Z
last
eventseen: 2017-12-05T09:49:58.118Z
log
name: epmpincident-2017-12-06/incident
priority
level: 1
recommended_action: Consider blacklisting the site. In addition, you may need to investigate the source of the exposure to see if further action is required.
state: 1
summary: Malicious domain www.xxxx.com detected
time: 2017-12-06T04:03:08.713Z
updated: 2017-12-06T04:03:08.950Z
uuid: 5e89b190-da3a-11e7-d301-000000000061

thanks in advance

0 Karma

Motivator

HI jkat54, can you please guide me on this, issue is being there for very long time, need to fix this.

thanks in advance.

0 Karma

Motivator

Hi All, Can anyone guide me on this issue, I am unable to make the _time field be same as the time field in the event.

thanks in advance.

0 Karma

Motivator

Hi jkat54, can you please guide me on this issue.

0 Karma

Motivator

Hi Jkat54, can you please help me on this issue, the issue still there.

0 Karma