Splunk Search

Why is Splunk ignoring timestamp in message?

metylkinandrey
Communicator

Good afternoon, I have already raised a similar topic. The last time I was cleared up the situation, but the problem has not been resolved.
I send messages to /services/collector/raw
And splunk takes the value from my "eventTime" field: "1985-04-12T23:21:15Z", and substitutes _time, which breaks my normal search and the normal operation of alerts. sourcetype in my case substitutes splunk itself when checking the token. I tried adjusting the sourcetype but that didn't help. In the screenshot, I displayed what I tried to do.

Unfortunately, we do not have the ability to always send messages to the Event ((

Message example:

curl --location --request POST 'http://10.10.10.10:8088/services/collector/raw' --header 'Authorization: Splunk a2-b5-48-9ff7-95r 'Content-Type: text/plain' --data-raw '{
"messageType": "RABIS",
"eventTime": "1985-04-12T23:21:15Z",
"messageId": "ED280816-E404-444A-A2D9-FFD2D171F447"
}'

Labels (1)
Tags (1)
0 Karma

yuanliu
SplunkTrust
SplunkTrust

metylkinandrey
Communicator

And I also noticed a strange thing, the problem is only with short messages.
These messages work fine:

curl --location --request POST 'http://mo.pr.dev.org:8088/services/collector/raw' --header 'Authorization: Splunk 02-93-48-96-27' --header 'Content-Type: text/plain' --data-raw '{
"messageId": "ED280816-E404-444A-A2D9-FFD2D171F8978",
"srcMsgId": "rwfsdfsfqwe121432gsgsfgd5000",
"correlationMsgId": "",
"baseSystemId": "abc1",
"routeInstanceId": "abc2",
"routepointID": "1.SA-GI.TO.KB.SEND",
"eventTime": "1985-04-12T23:20:00",
"GISGMPRequestID": "PS000BA780816-E404-444A-A2D9-FFD2D1712345",
"GISGMPResponseID": "PS000BA780816-E404-444B-A2D9-FFD2D1712345",
"messageType": "ED661",
"tid": "4234rfre09-hi4334-34lk-3j09k-fc4353j9494",
"PacketGISGMPId":"7642341379_20220512_123456789",
"result.code": "",
"resultdesc": "abc8"
}'

 

0 Karma

metylkinandrey
Communicator

Please tell me, did you find out that there is a similar problem with the splunk. But is it possible to disable checking messages for a timestamp in splunk altogether?

0 Karma

yuanliu
SplunkTrust
SplunkTrust

As I diagnosed in https://community.splunk.com/t5/Splunk-Search/A-field-is-lost-in-a-message-sent-in-raw/m-p/620555#M2... the normal behavior of Timestamp settings is completely thrown out by this bug. So no, when this bug is triggered, it is not possible to prevent Splunk from chopping off part of event and append it to the system time.  This is not "checking messages for a timestamp" at all.  You really need to file the bug and seek support.

The workaround, as I mentioned there, is to force Splunk to use _indextime for _time.  You can only do this at search time.

| rename _indextime as _time

 

metylkinandrey
Communicator

Thanks a lot!

0 Karma

metylkinandrey
Communicator

I forgot to add a screenshot. I'll add it here.
I also want to ask, is there really no other option to bypass this problem, except to edit the file: props.conf ?

Can you tell me how can I edit this file? If this is the only way out: a way and where to find it, do I need access to the file system?

0 Karma

metylkinandrey
Communicator

I'm experimenting, so far in the file: /opt/splunk/etc/system/local/props.conf set the parameter: DATETIME_CONFIG = NONE
Can you tell me how to apply the new settings in splunk?

0 Karma

metylkinandrey
Communicator

I did everything by acting on this recommendation:
Set DATETIME_CONFIG = NONE to prevent the timestamp processor from running. When timestamp processing is off, Splunk Enterprise does not look at the text of the event for the timestamp and instead uses the event time of receipt, the time the event arrives through its input. For file-based inputs, the event timestamp is taken from from the modification time of the input file.
Let me not like this option very much (
After editing the file /opt/splunk/etc/system/local/props.conf
I completely restarted the docker container so that splunk picks up the settings.
And it broke everything for me! Now I can't see my messages even if I set the filter: for the last year (((
While the messages are being sent, I get the code: "Success","code":0

0 Karma

metylkinandrey
Communicator

My current configuration:

[generic_single_line]
DATETIME_CONFIG = NONE
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TIME_FORMAT =
disabled = false
SHOULD_LINEMERGE = true

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...