Getting Data In

How to build props.conf?

Roy_9
Motivator

Hello,

I m trying to build the props.conf for the below log but when i am getting "failed to parse timestamp" and "defaulting to file modtime" error.

 

[test]
TIME_PREFIX = \["BS":
TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N
LINE_BREAKER = \}
SHOULD_LINEMERGE = true
TRUNCATE = 100000

 


below is the log sample
 

 

["BS":"2023-04-04 20-10-45.013", MessageID:"test-7657-99", UID="xy123-bc22"]
 {
 xyz
 }
["BS":"2023-04-04 20-10-46.013", MessageID:"test-7687-99", UID="xy123-bc22"]
 {
 abc
 }

 


              

Please help me with it


Thanks

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK, first and foremost - what is your architecture? Are you ingesting data with UF? HF? Sending directly to Cloud HEC input?

Because _what_ is in your props.conf is one thing but _where_ it should be placed is another.

0 Karma

woodcock
Esteemed Legend

Your LINE_BREAKER is invalid and SHOULD_LINEMERGE is wrong.  Also, be sure that you are deploying to the first full-instance of Splunk that handles the events (i.e. are you using HF?) Are you sure that the "sourcetype" value is "rest"?  If so:

[test]
TIME_PREFIX = \["BS":
TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N
LINE_BREAKER = }([\r\n\s]+)\["BS":
SHOULD_LINEMERGE = false
TRUNCATE = 100000

Roy_9
Motivator

@woodcock I tried your config by creating the sourcetype directly on splunk cloud but it isn’t working, 

 

timestamp is still getting rounded in the event instead of accurately displaying the timestamp with milli seconds.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

have you try %3Q instead of %3N? Probably doesn’t help, but maybe worth for try?

r. Ismo

0 Karma

Roy_9
Motivator

I am just keeping the sourcetype to test for testing purposes on my stand alone test server, I have applied the below props and i am still getting the error:

couldnot use strptime to parse timestamp from "2023-04-07 11-37-39.354"

Failed to parse timestamp, defaulting to file modtime.

these logs are already being indexed and sent to splunk cloud, we have an on prem HF in place.

is it possible to extract the timestamp from the event directly and apply it to timestamp field at search time?

 

I used the below search but still my timestamp is getting rounded 

index="abc" sourcetype="test"
| rex "\["BS"\":\"(?<event_time>\d{4}-\d{2}-\d{2} \d{2}-\d{2}-\d{2}\.\d{3})\""
| eval timestamp=strptime(event_time,"%Y-%m-%d %H-%M-%S.%3N")

for ex: "2023-04-07 11-37-39.354" is rounded to 4/7/23 11:37:40.000 AM under Time column in search.

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Roy_9,

this seems to be a json format, please try with INDEXD_EXTRACTIONS, something like this:

[test]
TIME_PREFIX = \[\"BS\":
TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N
SHOULD_LINEMERGE = true
INDEXED_EXTRACTIONS = json

in addition the regex for TIME_PREFIX is wrong because also " is a special char to escape.

Ciao.

Giuseppe

Roy_9
Motivator

@gcusello It didn’t worked, timestamp is getting rounded to nearest second

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Roy_9,

it's really strange: because timestamp extraction doesn't round, it reads the time and assign it to the timestamp; then, if you used %3N, it should take the milliseconds.

Could you share your props.conf?

Ciao.

Giuseppe

0 Karma

yeahnah
Motivator

The TIME_PREFIX handles regular expressions so the double quote should not need escaping.  Having said that there is no harm in escaping the double quote.

Also, it does not look like standard JSON format to me so I doubt Splunk would handle it, as it would fail to parse it as JSON.  

yeahnah
Motivator

try this

 

[test]
TIME_PREFIX = \["BS":"
TIME_FORMAT = %Y-%m-%d %H-%M-%S.%3N
LINE_BREAKER = \}([\n\r]+)\[
SHOULD_LINEMERGE = false
TRUNCATE = 100000

 

Roy_9
Motivator

@yeahnah I tried your config by creating the sourcetype directly on splunk cloud but it isn’t working, 

 

timestamp is still getting rounded in the event instead of accurately displaying the timestamp with milli seconds.

0 Karma
Get Updates on the Splunk Community!

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...