topic Re: Http Event Collector ignores JSON timestamp in Getting Data In

Http Event Collector ignores JSON timestamp

apider — Thu, 03 Oct 2019 13:46:31 GMT

Hi,

I have this json event I put in trough HEC:

{
  "time": "2019-10-01T11:29:53.817",
  "eventType": "Computer Room Temp Monitoring",
  "location": {
    "dataCenter": "PDC1",
    "hostname": "PELLE",
    "temp": {
      "dateStart": "2019-10-02T16:24:43",
      "dateEnd": "2019-10-02T16:29:53.817",
      "average": 23,
      "min": 21,
      "max": 24
    }
  }
}

But I am unable to set the "time" as the actual event time:
Have tried with both "_json" and my own sourcetype but to no avail. Have tried with both EPOCH and time format as above.
My own sourcetype looks like this in props.conf:

[crtemp]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = time
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = 1
#TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3Q
TIME_FORMAT =
TIME_PREFIX = time
MAX_TIMESTAMP_LOOKAHEAD = 30

Re: Http Event Collector ignores JSON timestamp

Azeemering — Wed, 30 Sep 2020 02:22:59 GMT

This works for me:

[ crtemp]
CHARSET=AUTO
INDEXED_EXTRACTIONS=json
KV_MODE=none
SHOULD_LINEMERGE=true
disabled=false
pulldown_type=true
LINE_BREAKER=([\r\n]+)

The events get the timestamp of _time 10/1/1911:29:53.817 AM

Where is HEC configured in your architecture and where do you apply the props? 1 system or a distributed system with heavy forwarders, indexers, search heads etc?

Re: Http Event Collector ignores JSON timestamp

apider — Fri, 04 Oct 2019 06:48:57 GMT

Tried your conf in props, but it is not working for me. Still get the indexing time only.
It is a singe instance installation (test).
the props,conf is in $SPLUNK_HOME/etc/system/local

Re: Http Event Collector ignores JSON timestamp

apider — Fri, 04 Oct 2019 12:07:00 GMT

Question:
Did you use the "Exact" or "Explicit" JSON fields from this example?
https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/HECExamples

Re: Http Event Collector ignores JSON timestamp

prakash007 — Fri, 04 Oct 2019 16:12:16 GMT

@apider Can you give this a try along with other configs in your props.conf

TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX=\"time\"\:\s\"
MAX_TIMESTAMP_LOOKAHEAD=30

Re: Http Event Collector ignores JSON timestamp

starcher — Fri, 04 Oct 2019 17:24:09 GMT

If you are hitting the event endpoint instead of raw it will not process through the pipeline like that. Change to raw endpoint if you are unable to control the JSON payload to meet the HEC requirements.

Re: Http Event Collector ignores JSON timestamp

apider — Mon, 07 Oct 2019 11:30:47 GMT

Ah, thank you guys!
Of course it's only the RAW endpoint that enables me to do this.

Have learned something today. Did not get up from bed in vain this morning 🙂

Cheers
/Filip

{
  "time": "2019-10-02T16:29:53.817",
  "event": {
    "eventType": "Computer Room Temp Monitoring",
    "location": {
      "dataCenter": "PDC1",
      "hostname": "PELLE",
      "dateStart": "2019-10-02T16:24:43",
      "dateEnd": "2019-10-02T16:29:53.817",
      "temp": {
        "average": 23,
        "min": 22,
        "max": 24
      }
    }
  }
}

Re: Http Event Collector ignores JSON timestamp

apider — Mon, 07 Oct 2019 11:33:04 GMT

Actuallt posting to the RAW endpoint solved the problem and the "time" field gets set as "_time"

Re: Http Event Collector ignores JSON timestamp

kundeng — Sun, 26 Apr 2020 21:02:46 GMT

Hi,
I'm assuming you used raw+ props to get what you want.

Are you able to do event protocol+ NO props to get proper time-stamping? I thought that is what HEC was designed for among other things, i.e. to simplify and speed-up data landing without data having to go through the parsing pipeline etc.

If you CAN control the formatting from the sender side, you need to change time to epoch time, and then you can just use HEC event protocol.

Re: Http Event Collector ignores JSON timestamp

jamesjarrett — Fri, 30 Oct 2020 16:31:01 GMT

So it appears you actually can, but the trick is here:
https://docs.splunk.com/Documentation/Splunk/8.0.7/AddAWSConfigRulesSingle/ConfigureHECKinesis#Configure_timestamp_extraction

Last check, AWS does not allow you to work with this switch (for Kinesis Firehose situations), but you can test it out with your own HEC: /services/collector/event?auto_extract_timestamp=true <etc>

It should be noted that this enters at the merging pipeline, and not right into the typing. https://www.aplura.com/assets/pdf/hec_pipelines.pdf

Re: Http Event Collector ignores JSON timestamp

ridwanahmed — Tue, 12 Oct 2021 19:22:28 GMT

if I must hit the event endpoint, what is the "time" field it expects? I am currently sending something called "time" and getting ignored, similar to OP.

Re: Http Event Collector ignores JSON timestamp

PickleRick — Tue, 12 Oct 2021 20:31:06 GMT

A golden shovel for you for digging up such an old thread 😉

But seriously, here's the info you need.

https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/FormateventsforHTTPEventCollector