Getting Data In

Help with props.conf for timestamp

vrmandadi
Builder

Hello All ,

I have a json data format , which I am trying to import into splunk .I want to extract the timestamp from the last field value a multivalue field .For instance there is a field called appid which is a multivalue field with values 1573503539877 , 1573503539875,1573503539878,1573503539873 .I want to make the last value as the timestamp .

The last timestamp for the multivalue field appid has the following format with closed flower brackets and a square bracket but the others have just a flower bracket

MULTIVALUE FIELD "APPID" -first event
apps: [ [-]
{ [-]
addedById: 5d013c468
appId: 5d0d1fc13d418bdf5
dateAdded: /Date(1573503009489)/

MULTIVALUE FIELD APPID-last value which needs to be extracted
addedById: 398
appId:ccaaadb
dateAdded: /Date(1584128055615)/
}
]

alt text

0 Karma
1 Solution

vrmandadi
Builder

This worked.

[ alt ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=JSON
KV_MODE=JSON
LINE_BREAKER=([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD=60
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TRUNCATE=10000
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
TIME_PREFIX=(Date()\d+)\\/\"}]

View solution in original post

0 Karma

vrmandadi
Builder

This worked.

[ alt ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=JSON
KV_MODE=JSON
LINE_BREAKER=([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD=60
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TRUNCATE=10000
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
TIME_PREFIX=(Date()\d+)\\/\"}]

0 Karma

to4kawa
Ultra Champion
[alt ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=none
KV_MODE=JSON
LINE_BREAKER=([\r\n]+){
MAX_TIMESTAMP_LOOKAHEAD=1000
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TRUNCATE=0
category=Structured
description=JSON
disabled=false
pulldown_type=true
TIME_PREFIX=.*Date\(
TIME_FORMAT=%s%3N

INDEXED_EXTRACTIONS OR KV_MODE should be set only one.
If LINE_BREAKER is good, TIME_PREFIX 's REGEX match greedy.
so, latest Date match _time

0 Karma

darrenfuller
Contributor

I am making a guess at how the raw json looks... but since the raw JSON will be all on one line with no carriage returns. try something like so in your props.conf:

TIME_PREFIX = (dateAdded:\s\/Date\()\d+\)\/\}\]
TIME_FORMAT = %s%3N
MAX_TIMESTAMP_LOOKAHEAD = 25

The regex is looking for a dateAdded: /Date(1234567891234) followed by the end strings of your event : }]

https://regex101.com/r/918sTd/1

Hope this helps...

.//D

0 Karma

vrmandadi
Builder

Thank You @darrenfuller for your reply . I tried the props you told me but that did not work .

[alt ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=JSON
KV_MODE=JSON
LINE_BREAKER=([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD=25
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TRUNCATE=10000
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
TIME_PREFIX=(dateAdded:\s\/Date()\d+)\/}]
TIME_FORMAT=%s%3N

I am pasting the raw format of data how it looks like.the one in bold before collection id is what I am looking at

"dateAdded":"\/Date(1576263356219)\/"},{"addedById":"5d013cd01758d3c468","appId":"5d013d418c2cf","dateAdded":"\/Date(1576263482497)\/"},{"addedById":"5d013cd013c468","appId":"5d35d43d17588644c6c25","dateAdded":"\/Date(1576263489027)\/"},{"addedById":"5d013cd084d3c468","appId":"5e5dc7827acaa","dateAdded":"\/Date(1583177463548)\/"},{"addedById":"5d013cd01d3c468","appId":"5e5d5c7827af0c","dateAdded":"\/Date(1583177467959)\/"}],"collectionId"

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...