Getting Data In

Timestamp extraction with 2 timestamp fields

Radcriminal
Explorer

I have a HEC output coming to my hec receiver services/collector/event?auto_extract_timestamp=true

I want to extract time from field named "time".

The format of the event is like 

{
	"event": 
		{
			"@timestamp": "2022-05-05T10:22:44.965Z"
			"time": 1651746176018,
			"my_text": "Pony 1 has left the barn"
		}
}

 

I also have a prop.conf that have following configuration:

CHARSET=UTF-8
KV_MODE=json
LINE_BREAKER=([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD=13
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIME_FORMAT=%s%3N
TIME_PREFIX=\"time\":

In result, my timestamp is extracted from field "@timestamp" and I was experimenting a lot with TIME_PREFIX field.

But when I manually upload the json with a file, the field I need is parsed ok and "@timestamp" is ignored. 

Labels (1)
Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

What is the source of those events? Can't you simply provide a time field along with the event contents? That would be the simplest solution - no parsing whatsoever, better performance on the input...

0 Karma

Radcriminal
Explorer

I found out that debug/refresh url does not refresh configuration for my sourcetype. After complete restart splunk indexer - time became parsing.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Radcriminal,

did you tried to add a space between the prefix and the value?

TIME_PREFIX=\"time\":\s+

Ciao.

Giuseppe

Radcriminal
Explorer

@gcusello 

I have tried right now. Did not work, still extracting time from @timestamp

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Radcriminal,

did you tried %f instead %s?

TIME_PREFIX = \"time\":\s+
TIME_FORMAT = %f

Ciao.

Giuseppe

Radcriminal
Explorer

Nothing changed. I also tried to fall back TIME_PREFIX to my initial value. Result is the same.

Why does it parse correctly when I try to upload .txt or .json and choose my sourcetype, but when it comes from hec - everything works different?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Radcriminal,

in my Splunk it's running adding MAX_TIMESTAMP_LOOKAHEAD = 10:

TIME_FORMAT = %s
TIME_PREFIX = \"time\":\s
MAX_TIMESTAMP_LOOKAHEAD = 10

Ciao.

Giuseppe

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...