topic Re: Help with props.conf for timestamp in Getting Data In

Help with props.conf for timestamp

vrmandadi — Wed, 25 Mar 2020 19:48:33 GMT

Hello All ,

I have a json data format , which I am trying to import into splunk .I want to extract the timestamp from the last field value a multivalue field .For instance there is a field called appid which is a multivalue field with values 1573503539877 , 1573503539875,1573503539878,1573503539873 .I want to make the last value as the timestamp .

The last timestamp for the multivalue field appid has the following format with closed flower brackets and a square bracket but the others have just a flower bracket

MULTIVALUE FIELD "APPID" -first event
apps: [ [-]
{ [-]
addedById: 5d013c468
appId: 5d0d1fc13d418bdf5
dateAdded: /Date(1573503009489)/

MULTIVALUE FIELD APPID-last value which needs to be extracted
addedById: 398
appId:ccaaadb
dateAdded: /Date(1584128055615)/
}
]

Re: Help with props.conf for timestamp

darrenfuller — Wed, 25 Mar 2020 20:59:45 GMT

I am making a guess at how the raw json looks... but since the raw JSON will be all on one line with no carriage returns. try something like so in your props.conf:

TIME_PREFIX = (dateAdded:\s\/Date\()\d+\)\/\}\]
TIME_FORMAT = %s%3N
MAX_TIMESTAMP_LOOKAHEAD = 25

The regex is looking for a dateAdded: /Date(1234567891234) followed by the end strings of your event : }]

https://regex101.com/r/918sTd/1

Hope this helps...

.//D

Re: Help with props.conf for timestamp

vrmandadi — Wed, 30 Sep 2020 04:42:31 GMT

Thank You @darrenfuller for your reply . I tried the props you told me but that did not work .

[alt ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=JSON
KV_MODE=JSON
LINE_BREAKER=([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD=25
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TRUNCATE=10000
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
TIME_PREFIX=(dateAdded:\s\/Date()\d+)\/}]
TIME_FORMAT=%s%3N

I am pasting the raw format of data how it looks like.the one in bold before collection id is what I am looking at

"dateAdded":"\/Date(1576263356219)\/"},{"addedById":"5d013cd01758d3c468","appId":"5d013d418c2cf","dateAdded":"\/Date(1576263482497)\/"},{"addedById":"5d013cd013c468","appId":"5d35d43d17588644c6c25","dateAdded":"\/Date(1576263489027)\/"},{"addedById":"5d013cd084d3c468","appId":"5e5dc7827acaa","dateAdded":"\/Date(1583177463548)\/"},{"addedById":"5d013cd01d3c468","appId":"5e5d5c7827af0c","dateAdded":"\/Date(1583177467959)\/"}],"collectionId"

Re: Help with props.conf for timestamp

to4kawa — Wed, 30 Sep 2020 04:42:38 GMT

[alt ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=none
KV_MODE=JSON
LINE_BREAKER=([\r\n]+){
MAX_TIMESTAMP_LOOKAHEAD=1000
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TRUNCATE=0
category=Structured
description=JSON
disabled=false
pulldown_type=true
TIME_PREFIX=.*Date\(
TIME_FORMAT=%s%3N

INDEXED_EXTRACTIONS OR KV_MODE should be set only one.
If LINE_BREAKER is good, TIME_PREFIX 's REGEX match greedy.
so, latest Date match _time

Re: Help with props.conf for timestamp

vrmandadi — Wed, 30 Sep 2020 04:42:41 GMT

This worked.

[ alt ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=JSON
KV_MODE=JSON
LINE_BREAKER=([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD=60
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TRUNCATE=10000
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
TIME_PREFIX=(Date()\d+)\\/\"}]