How to build props.conf?

[Roy_9]

Hello,

I m trying to build the props.conf for the below log but when i am getting "failed to parse timestamp" and "defaulting to file modtime" error.

[test]
TIME_PREFIX = \["BS":
TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N
LINE_BREAKER = \}
SHOULD_LINEMERGE = true
TRUNCATE = 100000

below is the log sample
 

["BS":"2023-04-04 20-10-45.013", MessageID:"test-7657-99", UID="xy123-bc22"]
 {
 xyz
 }
["BS":"2023-04-04 20-10-46.013", MessageID:"test-7687-99", UID="xy123-bc22"]
 {
 abc
 }

              

Please help me with it


Thanks

[PickleRick]

OK, first and foremost - what is your architecture? Are you ingesting data with UF? HF? Sending directly to Cloud HEC input?

Because _what_ is in your props.conf is one thing but _where_ it should be placed is another.

[woodcock]

Your LINE_BREAKER is invalid and SHOULD_LINEMERGE is wrong.  Also, be sure that you are deploying to the first full-instance of Splunk that handles the events (i.e. are you using HF?) Are you sure that the "sourcetype" value is "rest"?  If so:

[test]
TIME_PREFIX = \["BS":
TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N
LINE_BREAKER = }([\r\n\s]+)\["BS":
SHOULD_LINEMERGE = false
TRUNCATE = 100000

[Roy_9]

@woodcock I tried your config by creating the sourcetype directly on splunk cloud but it isn’t working, 

timestamp is still getting rounded in the event instead of accurately displaying the timestamp with milli seconds.

[isoutamo]

Hi

have you try %3Q instead of %3N? Probably doesn’t help, but maybe worth for try?

r. Ismo

[Roy_9]

I am just keeping the sourcetype to test for testing purposes on my stand alone test server, I have applied the below props and i am still getting the error:

couldnot use strptime to parse timestamp from "2023-04-07 11-37-39.354"

Failed to parse timestamp, defaulting to file modtime.

these logs are already being indexed and sent to splunk cloud, we have an on prem HF in place.

is it possible to extract the timestamp from the event directly and apply it to timestamp field at search time?

I used the below search but still my timestamp is getting rounded 

index="abc" sourcetype="test"
| rex "\["BS"\":\"(?<event_time>\d{4}-\d{2}-\d{2} \d{2}-\d{2}-\d{2}\.\d{3})\""
| eval timestamp=strptime(event_time,"%Y-%m-%d %H-%M-%S.%3N")

for ex: "2023-04-07 11-37-39.354" is rounded to 4/7/23 11:37:40.000 AM under Time column in search.

[gcusello]

Hi @Roy_9,

this seems to be a json format, please try with INDEXD_EXTRACTIONS, something like this:

[test]
TIME_PREFIX = \[\"BS\":
TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N
SHOULD_LINEMERGE = true
INDEXED_EXTRACTIONS = json

in addition the regex for TIME_PREFIX is wrong because also " is a special char to escape.

Ciao.

Giuseppe

[Roy_9]

@gcusello It didn’t worked, timestamp is getting rounded to nearest second

[gcusello]

Hi @Roy_9,

it's really strange: because timestamp extraction doesn't round, it reads the time and assign it to the timestamp; then, if you used %3N, it should take the milliseconds.

Could you share your props.conf?

Ciao.

Giuseppe

[yeahnah]

The TIME_PREFIX handles regular expressions so the double quote should not need escaping.  Having said that there is no harm in escaping the double quote.

Also, it does not look like standard JSON format to me so I doubt Splunk would handle it, as it would fail to parse it as JSON.  

[yeahnah]

try this

[test]
TIME_PREFIX = \["BS":"
TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N
LINE_BREAKER = \}([\n\r]+)\[
SHOULD_LINEMERGE = false
TRUNCATE = 100000

[Roy_9]

@yeahnah I tried your config by creating the sourcetype directly on splunk cloud but it isn’t working, 

timestamp is still getting rounded in the event instead of accurately displaying the timestamp with milli seconds.