topic Re: Help needed with building props.conf in Getting Data In

How to build props.conf?

Roy_9 — Mon, 10 Apr 2023 16:13:23 GMT

Hello,

I m trying to build the props.conf for the below log but when i am getting "failed to parse timestamp" and "defaulting to file modtime" error.

[test] TIME_PREFIX = \["BS": TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N LINE_BREAKER = \} SHOULD_LINEMERGE = true TRUNCATE = 100000

below is the log sample

["BS":"2023-04-04 20-10-45.013", MessageID:"test-7657-99", UID="xy123-bc22"] { xyz } ["BS":"2023-04-04 20-10-46.013", MessageID:"test-7687-99", UID="xy123-bc22"] { abc }

Please help me with it

Thanks

Re: Help needed with building props.conf

yeahnah — Wed, 05 Apr 2023 07:08:03 GMT

try this

[test] TIME_PREFIX = \["BS":" TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N LINE_BREAKER = \}([\n\r]+)\[ SHOULD_LINEMERGE = false TRUNCATE = 100000

Re: Help needed with building props.conf

gcusello — Wed, 05 Apr 2023 06:35:49 GMT

Hi @Roy_9,

this seems to be a json format, please try with INDEXD_EXTRACTIONS, something like this:

[test] TIME_PREFIX = \[\"BS\": TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N SHOULD_LINEMERGE = true INDEXED_EXTRACTIONS = json

in addition the regex for TIME_PREFIX is wrong because also " is a special char to escape.

Ciao.

Giuseppe

Re: Help needed with building props.conf

yeahnah — Wed, 05 Apr 2023 07:13:45 GMT

The TIME_PREFIX handles regular expressions so the double quote should not need escaping. Having said that there is no harm in escaping the double quote.

Also, it does not look like standard JSON format to me so I doubt Splunk would handle it, as it would fail to parse it as JSON.

Re: Help needed with building props.conf

woodcock — Fri, 07 Apr 2023 00:52:06 GMT

Your LINE_BREAKER is invalid and SHOULD_LINEMERGE is wrong. Also, be sure that you are deploying to the first full-instance of Splunk that handles the events (i.e. are you using HF?) Are you sure that the "sourcetype" value is "rest"? If so:

[test]
TIME_PREFIX = \["BS":
TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N
LINE_BREAKER = }([\r\n\s]+)\["BS":
SHOULD_LINEMERGE = false
TRUNCATE = 100000

Re: Help needed with building props.conf

Roy_9 — Fri, 07 Apr 2023 17:42:02 GMT

I am just keeping the sourcetype to test for testing purposes on my stand alone test server, I have applied the below props and i am still getting the error:

couldnot use strptime to parse timestamp from "2023-04-07 11-37-39.354"

Failed to parse timestamp, defaulting to file modtime.

these logs are already being indexed and sent to splunk cloud, we have an on prem HF in place.

is it possible to extract the timestamp from the event directly and apply it to timestamp field at search time?

I used the below search but still my timestamp is getting rounded

index="abc" sourcetype="test"
| rex "\["BS"\":\"(?<event_time>\d{4}-\d{2}-\d{2} \d{2}-\d{2}-\d{2}\.\d{3})\""
| eval timestamp=strptime(event_time,"%Y-%m-%d %H-%M-%S.%3N")

for ex: "2023-04-07 11-37-39.354" is rounded to 4/7/23 11:37:40.000 AM under Time column in search.

Re: Help needed with building props.conf

Roy_9 — Mon, 24 Apr 2023 18:05:04 GMT

@woodcock I tried your config by creating the sourcetype directly on splunk cloud but it isn’t working,

timestamp is still getting rounded in the event instead of accurately displaying the timestamp with milli seconds.

Re: Help needed with building props.conf

Roy_9 — Mon, 24 Apr 2023 18:08:26 GMT

@yeahnah I tried your config by creating the sourcetype directly on splunk cloud but it isn’t working,

timestamp is still getting rounded in the event instead of accurately displaying the timestamp with milli seconds.

Re: Help needed with building props.conf

Roy_9 — Mon, 24 Apr 2023 18:16:11 GMT

@gcusello It didn’t worked, timestamp is getting rounded to nearest second

Re: Help needed with building props.conf

isoutamo — Mon, 24 Apr 2023 19:32:02 GMT

have you try %3Q instead of %3N? Probably doesn’t help, but maybe worth for try?

r. Ismo

Re: Help needed with building props.conf

gcusello — Tue, 25 Apr 2023 09:05:50 GMT

Hi @Roy_9,

it's really strange: because timestamp extraction doesn't round, it reads the time and assign it to the timestamp; then, if you used %3N, it should take the milliseconds.

Could you share your props.conf?

Ciao.

Giuseppe

Re: How to build props.conf?

PickleRick — Tue, 25 Apr 2023 09:11:19 GMT

OK, first and foremost - what is your architecture? Are you ingesting data with UF? HF? Sending directly to Cloud HEC input?

Because _what_ is in your props.conf is one thing but _where_ it should be placed is another.