Getting Data In

Why is _time in my JSON payload parsed correctly with milliseconds via manual upload but not via HEC?

andrewtrobec
Motivator

Hello!

I have a JSON payload whose _time field gets parsed no issue when I perform a manual upload, but when that same payload comes in through a HEC with the same sourcetype then it doesn't parse the milliseconds.

Sample payload:

{"flowid":"dc59cf7376370faadfb89764e1896a1b","id":23431,"action":"upload","request":"","response":"{\"success\":false,\"correlation_id\":\"00-dc59cf7376370faadfb89764e1896a1b-d23cff8d675709d1-01\",\"status_code\":\"401\",\"message\":\"Request unsuccessful. The following errors were found.\",\"errors\":[{\"code\":\"E_TECHNICAL\",\"value\":\"A technical error prevented the success of the request.\"}]}","midid":"","dest":"","type":"GET","requesttime":"2023-07-12T10:17:32.4327504Z","externaltime": null,"externalresponsetime": null,"middlewaretime":"2023-07-12T10:17:32.4327504Z","logtime":"2023-07-12T10:17:32.6039773","globaltime":"2023-07-12T10:17:32.4333085","responsetime":"2023-07-12T10:17:32.4364843"}

Sourcetype:

[json_sourcetype]
SHOULD_LINEMERGE = false
TIME_PREFIX = \"logtime\"\:\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N
TZ = UTC
TRUNCATE = 0
MAX_TIMESTAMP_LOOKAHEAD = 0
KV_MODE = json

Has anybody faced this issue before? What could the problem be?

Thank you and best regards,

Andrew

Labels (4)
Tags (3)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

Not necessarily. The /raw endpoint works as if you sent the events over plain TCP connection with the difference that you can send events to specific channels tied to different sourcetypes and destination indexes.

The /event endpoint skips some parts of event processing (which are assumed to have been done earlier on the sending side) and thus is lighter on the receiver's performance.

So there are several possible approaches:

1. Send to the /raw endpoint and do all the parsing yourself in Splunk

2. Send to the /event?auto_extract_timestamp=true and do the timestamp parsing yourself in Splunk

3. Parse the time on your source before sending to HEC and include the properly formatted time field along with your event contents.

I'd opt for the last option but your circumstances might - for example - not allow reformatting the event on send.

View solution in original post

PickleRick
SplunkTrust
SplunkTrust

Splunk skips several stages of event processing for HEC inputs. Most notably, line breaking but also (by default) timestamp recognition.

https://docs.splunk.com/Documentation/Splunk/9.1.0/RESTREF/RESTinput#services.2Fcollector.2Fevent

0 Karma

andrewtrobec
Motivator

Thank you for the response!  I have this same setup with a very similar payload and it works no issues.  Also, when the data comes into the HEC it parses the whole timestamp with the exception of the milliseconds for 

the timestamp.  So for example the _time for the example above is:

manual upload: 2023-07-12 10:17:32.432
HEC: 2023-07-12 10:17:32.000

One thing about the timestamp format is that it has 7 decimal places and not 6, but I am parsing %6N.  I'm not sure if this would have any bearing.

I hope this helps!

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. Which endpoint are you sending your data to:

/services/collector/raw

/services/collector/event

/services/collector/event?auto_extract_timestamp=true

0 Karma

andrewtrobec
Motivator

I need to find that out, but thank you for bringing this up!  Am I to assume that it would be best to send to /services/collector/raw?  Thanks!

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Not necessarily. The /raw endpoint works as if you sent the events over plain TCP connection with the difference that you can send events to specific channels tied to different sourcetypes and destination indexes.

The /event endpoint skips some parts of event processing (which are assumed to have been done earlier on the sending side) and thus is lighter on the receiver's performance.

So there are several possible approaches:

1. Send to the /raw endpoint and do all the parsing yourself in Splunk

2. Send to the /event?auto_extract_timestamp=true and do the timestamp parsing yourself in Splunk

3. Parse the time on your source before sending to HEC and include the properly formatted time field along with your event contents.

I'd opt for the last option but your circumstances might - for example - not allow reformatting the event on send.

andrewtrobec
Motivator

Apologies for the delayed response, I needed time to explore and test.  In the end the issue was that the source was sending to /events endpoint and not /raw.  After updating everything starting working correctly!  Thank you so much for the support!

I am accepting this as the answer because it provides multiple approaches, of which redirecting to /raw worked.

0 Karma
Get Updates on the Splunk Community!

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...