topic How do I reconfigure _time field? in Splunk Enterprise

How do I reconfigure _time field?

Hod152 — Fri, 30 Jun 2023 15:33:10 GMT

Hey,

It has been several days that I'm trying to solve the following issue.

I'm sending JSON data over tcp (9997), the data arrives completly but, the "_time" field that is saved is the actual indexing time instead of "time" field, that I've added before.
I tried to configure props.conf as with JSON and with regex but both hasn't work.

What makes things more wired is that some of the logs that contains another timestamp field are indexed with this field. I was also trying to play arround this but nothing helped.

I'm running an all-in-one deployment from a container.

Thanks,

------------

props.conf

[evtx] MAX_DAYS_AGO = 10951 #INDEXED_EXTRACTIONS = JSON #KV_MODE = none #TRANSFORMS-jsonextraction = json_extract #TIME_PREFIX = time.:. #TIME_FORMAT = %Y-%m-%dT%H:%M:%S #TIMESTAMP_FIELDS = time CHARSET = UTF-8 INDEXED_EXTRACTIONS = json KV_MODE = none SHOULD_LINEMERGE = false disabled = false pulldown_type = 1 TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6NZ TIMESTAMP_FIELDS=time LINE_BREAKER = ([\r\n]+) category = Structured description = ODATA SCPI MPL JSON

log sample - time is indexed based on "Event.NewTime":

{ "cribl_breaker":"fallback", "collector":"in_evtx_to_json", "time":"2023-05-18T11:41:20.687637Z", "EventData":{"NewTime":"2023-05-18T11:41:20.697000Z", "PreviousTime":"2023-05-18T11:41:20.707011Z", "ProcessId":"0x46c", "ProcessName":"C:\\Windows\\System32\\svchost.exe", "SubjectDomainName":"NT AUTHORITY", "SubjectLogonId":"0x3e5", "SubjectUserName":"LOCAL SERVICE", "SubjectUserSid":"S-1-5-19"}, "System":{"Channel":"Security", "Computer":"test123", "Correlation":null, "EventID":4616, "EventRecordID":108506811, "Execution":{"#attributes":{"ProcessID":4, "ThreadID":1752}}, "Keywords":"0x8020000000000000", "Level":0, "Opcode":0, "Provider":{"#attributes":{"Guid":"54849625-5478-4994-A5BA-3E3B0328C30D", "Name":"Microsoft-Windows-Security-Auditing"}}, "Security":null, "Task":12288, "TimeCreated":{"#attributes":{"SystemTime":"2023-05-18T11:41:20.687637Z"}}, "Version":1}}

log samle - time is indexed based on arrival time:

{"cribl_breaker":"fallback", "collector":"in_evtx_to_json", "time":"2023-05-06T20:01:58.205343Z", "EventData":{"HandleId":"0x6a30", "ObjectServer":"Security", "ProcessId":"0x4f4", "ProcessName":"C:\\Windows\\System32\\svchost.exe", "SubjectDomainName":"test.local", "SubjectLogonId":"0x3e4", "SubjectUserName":"test", "SubjectUserSid":"S-1-5-20"}, "System":{"Channel":"Security", "Computer":"test123", "Correlation":null, "EventID":4658, "EventRecordID":107343779, "Execution":{"#attributes":{"ProcessID":4, "ThreadID":109608}}, "Keywords":"0x8020000000000000", "Level":0, "Opcode":0, "Provider":{"#attributes":{"Guid":"54849625-5478-4994-A5BA-3E3B0328C30D", "Name":"Microsoft-Windows-Security-Auditing"}}, "Security":null, "Task":12800, "TimeCreated":{"#attributes":{"SystemTime":"2023-05-06T20:01:58.205343Z"}}, "Version":0}}

Re: reconfigure _time field

isoutamo — Thu, 29 Jun 2023 11:21:06 GMT

If I check correctly then these props.conf works for me.

[ <FIRST> ] SHOULD_LINEMERGE=true LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true CHARSET=UTF-8 INDEXED_EXTRACTIONS=json KV_MODE=none description=First category=Structured disabled=false pulldown_type=true TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6Q%Z TIMESTAMP_FIELDS=time

[ <SECOND> ] SHOULD_LINEMERGE=true LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true CHARSET=UTF-8 INDEXED_EXTRACTIONS=json KV_MODE=none description=second category=Structured disabled=false pulldown_type=true TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6Q%Z TIMESTAMP_FIELDS=time

I check these with 9.0.5 and another version and both works as expected.

r. Ismo

Re: reconfigure _time field

Hod152 — Sun, 02 Jul 2023 11:48:33 GMT

Thanks for the quick response!

I've tried your solution but, it still not working. My splunk version is 9.0.5

Re: How do I reconfigure _time field?

PickleRick — Sun, 02 Jul 2023 12:21:36 GMT

First things first - which endpoint are you sending your data to?

By default, unless you specify ?auto_extract_timestamp=true in your HEC endpoint URL, HEC bypasses timestamp recognition from the event itself regardless of whatever you have in it and what props you set.

https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fevent

Re: How do I reconfigure _time field?

Hod152 — Sun, 02 Jul 2023 13:45:02 GMT

I'm sending it using cribl over the default port directly to splunk. It's an all in one deployment.

Re: How do I reconfigure _time field?

PickleRick — Sun, 02 Jul 2023 15:25:22 GMT

Have you read the Cribl HEC docs?

https://docs.cribl.io/stream/destinations-splunk-hec

Re: How do I reconfigure _time field?

Hod152 — Sun, 02 Jul 2023 16:07:05 GMT

I read the Splunk single instance doc which is the how I recive the events, the events are arriving to splunk but not indexed as I need.

Re: How do I reconfigure _time field?

isoutamo — Sun, 02 Jul 2023 16:30:31 GMT

Based on port this is not an HEC endpoint. I assume that as used port 9997 it’s using S2S protocol.

Re: How do I reconfigure _time field?

PickleRick — Mon, 03 Jul 2023 07:22:37 GMT

True. I have no idea where I got this HEC from. Might have mixed it with some other thread.

Anyway, the cribl docs say that data for splunk destination is sent as "cooked and parsed" which means that if those docs are correct and the data is indeed sent parsed, the only thing that can happen in index-time on the receiving end is ingest actions.

(A side note - it's also relatively ineffective way to send events bandwidth-wise)

Re: How do I reconfigure _time field?

Hod152 — Sun, 02 Jul 2023 19:07:25 GMT

Thank you, I'll work around 🙂