How to handle the log which has the "_time" field

[Iris_Pi]

Hello Splunkers!

Your help is appreciated!

I have a log source coming into Splunk via HEC. The log is in json format, it has a field called "_time"  as showed below:

and the problem comes, this "_time" field will fail the time extraction.

I defined a source type as follow:

It's not working, the _time is set the timestamp when log arrives.

I'm using splunk cloud. 

Please share your experience in case you have encountered same issue before. 

Regards,
Iris

[Iris_Pi]

Thanks you guys for the help.

I have no control on the data source.
I changed the hec endpoint from "/service/collector" to "/service/collector/raw" on the sender, and my source type works now!

[richgalloway · Accepted solution]

The solution depends on which HEC endpoint you're using.  If you're using the event endpoint then props  are ignored and the timestamp must be in epoch format.

See https://help.splunk.com/en/splunk-enterprise/get-started/get-data-in/10.0/get-data-with-http-event-c... for details.

[inventsekar]

Hi @richgalloway and @VatsalJagani , thanks for the link and good details. 

but two more curious questions:...
the user replied as --- "I changed the hec endpoint from "/service/collector" to "/service/collector/raw" on the sender, and my source type works now!"

out of these endpoints (copied the table here for easy reference)
1) can a dev guy send logs to "any" of these end points?
2) can i assume only these 2 (services/collector/raw
and services/collector/raw/1.0) are ok endpoints to receive logs with _time inside the logs?

REST API endpoint Description

data/inputs/http	Accesses or updates HTTP Event Collector global configuration tokens and application tokens.
data/inputs/http/{name}	Manages the {name} HTTP Event Collector token. HTTP, as in data/inputs/http/http, indicates global configuration.
data/inputs/http/{name}/disable	Turns off the {name} HTTP Event Collector token.
data/inputs/http/{name}/enable	Turns on the {name} HTTP Event Collector token.
services/collector	Queries events to HTTP Event Collector using the Splunk platform JavaScript Object Notation (JSON) event protocol.
services/collector/ack	Queries event indexing status.
services/collector/event	Sends timestamped events to the HTTP Event Collector using the Splunk platform JSON event protocol when you set the auto_extract_timestamp argument to true in the /event URL. An example of a timestamp is 2017-01-02 00:00:00 Timestamps in the event's JSON envelope take priority over the extracted timestamp. If there's no timestamp in the event's JSON envelope, the merging pipeline extracts the timestamp from the event. If time=xxx is used in the /event URL, then auto_extract_timestamp is turned off.
services/collector/event/1.0	Works identically to the services/collector/event endpoint but introduces a protocol version for future scalability.
services/collector/health	Checks the health of the HTTP Event Collector. This endpoint is supported in Splunk Cloud Platform and versions 6.6.0 and higher of Splunk Enterprise.
services/collector/mint	Posts data formatted for Splunk MINT to the HTTP Event Collector.
services/collector/mint/1.0	Works identically to the receivers/token/mint endpoint but introduces a protocol version for future scalability.
services/collector/raw	Sends raw data directly to the HTTP Event Collector.
services/collector/raw/1.0	Works identically to the services/collector/raw endpoint but introduces a protocol version for future scalability.
services/collector/s2s	Receives Splunk Transmission Control Protocol (TCP) data over HTTP from the Splunk Universal Forwarder. This endpoint is supported in Splunk Enterprise 8.1.0 and higher.

[richgalloway]

1) A "dev guy" should not be sending anything to a data/inputs endpoint - only the services/collector endpoints.  

2) Use the services/collector/raw/1.0 endpoint if you need props and transforms to process your data.  Use the services/collector/event/1.0?auto_extract_timestamp=true endpoint if you only need the timestamp extracted.

[VatsalJagani]

Right,

If you have full control of the sender and you can modify the sender to send the data along with the correct time value as a request parameter called "time" with services/collector/event endpoint.

OR

If you don't have any control over which parameter gets sent, you can change the endpoint to use services/collector/raw and then write props.conf to extract the timestamp correctly.