Http Event Collector ignores JSON timestamp

[apider]

Hi,

I have this json event I put in trough HEC:

{
  "time": "2019-10-01T11:29:53.817",
  "eventType": "Computer Room Temp Monitoring",
  "location": {
    "dataCenter": "PDC1",
    "hostname": "PELLE",
    "temp": {
      "dateStart": "2019-10-02T16:24:43",
      "dateEnd": "2019-10-02T16:29:53.817",
      "average": 23,
      "min": 21,
      "max": 24
    }
  }
}

But I am unable to set the "time" as the actual event time:
Have tried with both "_json" and my own sourcetype but to no avail. Have tried with both EPOCH and time format as above.
My own sourcetype looks like this in props.conf:

[crtemp]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = time
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = 1
#TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3Q
TIME_FORMAT =
TIME_PREFIX = time
MAX_TIMESTAMP_LOOKAHEAD = 30

[jamesjarrett]

So it appears you actually can, but the trick is here:
https://docs.splunk.com/Documentation/Splunk/8.0.7/AddAWSConfigRulesSingle/ConfigureHECKinesis#Confi...

Last check, AWS does not allow you to work with this switch (for Kinesis Firehose situations), but you can test it out with your own HEC:  /services/collector/event?auto_extract_timestamp=true <etc> 

It should be noted that this enters at the merging pipeline, and not right into the typing. https://www.aplura.com/assets/pdf/hec_pipelines.pdf  

[apider]

Ah, thank you guys!
Of course it's only the RAW endpoint that enables me to do this.

Have learned something today. Did not get up from bed in vain this morning 🙂

Cheers
/Filip

{
  "time": "2019-10-02T16:29:53.817",
  "event": {
    "eventType": "Computer Room Temp Monitoring",
    "location": {
      "dataCenter": "PDC1",
      "hostname": "PELLE",
      "dateStart": "2019-10-02T16:24:43",
      "dateEnd": "2019-10-02T16:29:53.817",
      "temp": {
        "average": 23,
        "min": 22,
        "max": 24
      }
    }
  }
}

[kundeng]

Hi,
I'm assuming you used raw+ props to get what you want.

Are you able to do event protocol+ NO props to get proper time-stamping? I thought that is what HEC was designed for among other things, i.e. to simplify and speed-up data landing without data having to go through the parsing pipeline etc.

If you CAN control the formatting from the sender side, you need to change time to epoch time, and then you can just use HEC event protocol.

[starcher · Accepted solution]

If you are hitting the event endpoint instead of raw it will not process through the pipeline like that. Change to raw endpoint if you are unable to control the JSON payload to meet the HEC requirements.

[ridwanahmed]

if I must hit the event endpoint, what is the "time" field it expects?  I am currently sending something called "time" and getting ignored, similar to OP. 

[PickleRick]

A golden shovel for you for digging up such an old thread 😉

But seriously, here's the info you need.

https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/FormateventsforHTTPEventCollector

[Azeemering]

This works for me:

[ crtemp]
CHARSET=AUTO
INDEXED_EXTRACTIONS=json
KV_MODE=none
SHOULD_LINEMERGE=true
disabled=false
pulldown_type=true
LINE_BREAKER=([\r\n]+)

The events get the timestamp of _time 10/1/1911:29:53.817 AM

Where is HEC configured in your architecture and where do you apply the props? 1 system or a distributed system with heavy forwarders, indexers, search heads etc?

[apider]

Question:
Did you use the "Exact" or "Explicit" JSON fields from this example?
https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/HECExamples

[prakash007]

@apider Can you give this a try along with other configs in your props.conf

TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX=\"time\"\:\s\"
MAX_TIMESTAMP_LOOKAHEAD=30

[apider]

Actuallt posting to the RAW endpoint solved the problem and the "time" field gets set as "_time"

[apider]

Tried your conf in props, but it is not working for me. Still get the indexing time only.
It is a singe instance installation (test).
the props,conf is in $SPLUNK_HOME/etc/system/local