Getting Data In

Http Event Collector ignores JSON timestamp

apider
Engager

Hi,

I have this json event I put in trough HEC:

{
  "time": "2019-10-01T11:29:53.817",
  "eventType": "Computer Room Temp Monitoring",
  "location": {
    "dataCenter": "PDC1",
    "hostname": "PELLE",
    "temp": {
      "dateStart": "2019-10-02T16:24:43",
      "dateEnd": "2019-10-02T16:29:53.817",
      "average": 23,
      "min": 21,
      "max": 24
    }
  }
}

But I am unable to set the "time" as the actual event time:
Have tried with both "_json" and my own sourcetype but to no avail. Have tried with both EPOCH and time format as above.
My own sourcetype looks like this in props.conf:

[crtemp]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = time
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = 1
#TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3Q
TIME_FORMAT =
TIME_PREFIX = time
MAX_TIMESTAMP_LOOKAHEAD = 30
0 Karma
1 Solution

starcher
SplunkTrust
SplunkTrust

If you are hitting the event endpoint instead of raw it will not process through the pipeline like that. Change to raw endpoint if you are unable to control the JSON payload to meet the HEC requirements.

View solution in original post

jamesjarrett
Path Finder

So it appears you actually can, but the trick is here:
https://docs.splunk.com/Documentation/Splunk/8.0.7/AddAWSConfigRulesSingle/ConfigureHECKinesis#Confi...

Last check, AWS does not allow you to work with this switch (for Kinesis Firehose situations), but you can test it out with your own HEC:  /services/collector/event?auto_extract_timestamp=true <etc> 

It should be noted that this enters at the merging pipeline, and not right into the typing. https://www.aplura.com/assets/pdf/hec_pipelines.pdf  

0 Karma

apider
Engager

Ah, thank you guys!
Of course it's only the RAW endpoint that enables me to do this.

Have learned something today. Did not get up from bed in vain this morning 🙂

Cheers
/Filip

{
  "time": "2019-10-02T16:29:53.817",
  "event": {
    "eventType": "Computer Room Temp Monitoring",
    "location": {
      "dataCenter": "PDC1",
      "hostname": "PELLE",
      "dateStart": "2019-10-02T16:24:43",
      "dateEnd": "2019-10-02T16:29:53.817",
      "temp": {
        "average": 23,
        "min": 22,
        "max": 24
      }
    }
  }
}
0 Karma

kundeng
Path Finder

Hi,
I'm assuming you used raw+ props to get what you want.

Are you able to do event protocol+ NO props to get proper time-stamping? I thought that is what HEC was designed for among other things, i.e. to simplify and speed-up data landing without data having to go through the parsing pipeline etc.

If you CAN control the formatting from the sender side, you need to change time to epoch time, and then you can just use HEC event protocol.

0 Karma

starcher
SplunkTrust
SplunkTrust

If you are hitting the event endpoint instead of raw it will not process through the pipeline like that. Change to raw endpoint if you are unable to control the JSON payload to meet the HEC requirements.

ridwanahmed
Path Finder

if I must hit the event endpoint, what is the "time" field it expects?  I am currently sending something called "time" and getting ignored, similar to OP. 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

A golden shovel for you for digging up such an old thread 😉

But seriously, here's the info you need.

https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/FormateventsforHTTPEventCollector

0 Karma

Azeemering
Builder

This works for me:

[ crtemp]
CHARSET=AUTO
INDEXED_EXTRACTIONS=json
KV_MODE=none
SHOULD_LINEMERGE=true
disabled=false
pulldown_type=true
LINE_BREAKER=([\r\n]+)

The events get the timestamp of _time 10/1/1911:29:53.817 AM

Where is HEC configured in your architecture and where do you apply the props? 1 system or a distributed system with heavy forwarders, indexers, search heads etc?

0 Karma

apider
Engager

Question:
Did you use the "Exact" or "Explicit" JSON fields from this example?
https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/HECExamples

0 Karma

prakash007
Builder

@apider Can you give this a try along with other configs in your props.conf

TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX=\"time\"\:\s\"
MAX_TIMESTAMP_LOOKAHEAD=30
0 Karma

apider
Engager

Actuallt posting to the RAW endpoint solved the problem and the "time" field gets set as "_time"

0 Karma

apider
Engager

Tried your conf in props, but it is not working for me. Still get the indexing time only.
It is a singe instance installation (test).
the props,conf is in $SPLUNK_HOME/etc/system/local

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...