Getting Data In

How to do time extraction for HTTP Event Collector JSON logs?

scottsavareseat
Path Finder

I have an HTTP Event Collector input collecting JSON data via syslog forwarder. The syslog-ng message looks like:

 

body("{ \"source\": \"${.splunk.source}\",
        \"event\":  ${MSG}
}")

 

I can see the message and the proper source in my indexer. But time extraction is the problem. Because there is often a delay between the log and the time syslog receives it, I want to use a field in the message to grab the timestamp... A message looks like this:

 

{"data":"stuff","time_stamp":"2022-05-10 17:14:23Z","value1":"more_stuff"}

 

So, I create a props.conf on my indexer cluster that looks like:

 

[my_sourcetype]
DATETIME_CONFIG =
MAX_TIMESTAMP_LOOKAHEAD = 30
TIME_PREFIX = time_stamp\":\"
TIME_FORMAT = %Y-%m-%d %h:%M:%S%Z
TZ = UTC

 

I've confirmed the sourcetype is correct as I define it in the inputs for HEC and they match. But for the life of me, I can't seem to get Splunk to find the time. I've tried looking for errors using this search:

 

index = _internal log_level = WARN OR log_level =ERROR  "timestamp"

 

(found in an older community post, so thanks to the author) but I find nothing. I tried playing in the UI and creating a new data type (the add new data widget). In that UI, my props.conf should work. But for some reason on the cluster it doesnt.

Are there any other troubleshooting steps I can follow? Am I missing something that might help this work better?

 

Labels (3)
Tags (2)
0 Karma

somesoni2
Revered Legend

Is your syslog forwarder a universal forwarder or a heavy forwarder (Splunk Enterprise instance acting as forwarder)?

Also, make sure you're using this as TIME_FORMAT (%H instead of %h for hour).

TIME_FORMAT = %Y-%m-%d %H:%M:%S%Z
0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...