Modify the format of events in splunk UI

[sarnagar]

Hi All,
I have JSON Logs like below:

SAMPLE EVENT:

"line":" 2016-10-21 19:16:00 INFO [CollectorAccess] Updating peer collector list: [172.17.0.6:9998, 172.18.0.3:9998|dynatrace_collector_dev-wlp_wlp.8.9of6d5ylz2g8yc22wioksjhek.ingress:9998|10.255.0.13:9998|10.255.0.15:9998, 10.255.0.20:9998|172.18.0.4:9998|dynatrace_collector_dev-wlp_wlp.5.04w8s09obp9pwe2mu1s9elq7i.ingress:9998|10.255.0.13:9998, 172.17.0.9:9998, 172.17.0.3:9998, 172.18.0.3:9998|dynatrace_collector_dev-wlp_wlp.1.0bs4sokp2fpkkz0nfh4mo4xsc.ingress:9998|10.255.0.13:9998|10.255.0.16:9998, 172.17.0.8:9998, dynatrace_collector_dev-wlp_wlp.7.3hvzd4e5b5zdby4blgu1v8rm8.ingress:9998|172.18.0.4:9998|10.255.0.13:9998|10.255.0.22:9998, dynatrace_collector_dev-wlp_wlp.6.5huzo8ot2fnzavm9ji5it4p9d.ingress:9998|172.18.0.4:9998|10.255.0.13:9998|10.255.0.21:9998, 172.17.0.5:9998, 172.17.0.7:9998, 172.18.0.3:9998|10.255.0.13:9998|10.255.0.17:9998|dynatrace_collector_dev-wlp_wlp.2.5a4bbjtne3hxm3i3f0a0xj7lv.ingress:9998, 10.255.0.18:9998|172.18.0.3:9998|dynatrace_collector_dev-wlp_wlp.3.4tr7zjjhgqqtn0fh9hd0l7qoo.ingress:9998|10.255.0.13:9998, 172.17.0.4:9998, 10.255.0.19:9998|172.18.0.3:9998|dynatrace_collector_dev-wlp_wlp.4.ep9fshp8i94rdz7p7cpwi9v77.ingress:9998|10.255.0.13:9998",
"source":" stdout",
"tag":" itec-artifactory.hec.com:6555/com.hec.pl000123.ezpaas.ezpaas-dynatrace-collector:6.3-11/Dynatrace_Collector_DEV-WLP_WLP.7.3hvzd4e5b5zdby4blgu1v8rm8/5125046f7489 "

I WANT TO MODIFY THE LAST ITALICISED LINE IN ABOVE EVENT TO SOMETHING LIKE BELOW : ( CHANGES HIGHLIGHTED AS BOLD)
source=stdout,
container-image=itec-artifactory.hec.com:6555/com.hec.pl000123.ezpaas.ezpaas-dynatrace-collector:6.3-11,container-service=Dynatrace_Collector_DEV-WLP_WLP.7.3hvzd4e5b5zdby4blgu1v8rm8,container-id=5125046f7489

Basically I want to create field value pairs from the events and also modify its appearance in the UI.

How can I achieve this??

[lukejadamec]

You can extract the fields so that they show up in Interesting Fields, and use those fields directly in searches, but you cannot change how they display in the UI with config changes without using SEDCMD.

[somesoni2 · Accepted solution]

Give this a try.

your base search | rex mode=sed "s/\"tag\":\s\"([^\/]+\/[^\/]+)\/([^\/]+)\/([^\"]+)/\"container-image\":\"\1\",\"container-service\":\"\2\",\"container-id\":\"\3/" | spath

The regex was created assuming your raw event, in text format looks like this. You might have to adjust the regex if there is a diff

{  
"line": "2016-10-21 19:16:00 INFO [CollectorAccess] Updating peer collector", "list": "[172.17.0.6:9998, 172.18.0.3:9998|dynatrace_collector_dev-wlp_wlp.8.9of6d5ylz2g8yc22wioksjhek.ingress:9998|10.255.0.13:9998|10.255.0.15:9998, 10.255.0.20:9998|172.18.0.4:9998|dynatrace_collector_dev-wlp_wlp.5.04w8s09obp9pwe2mu1s9elq7i.ingress:9998|10.255.0.13:9998, 172.17.0.9:9998, 172.17.0.3:9998, 172.18.0.3:9998|dynatrace_collector_dev-wlp_wlp.1.0bs4sokp2fpkkz0nfh4mo4xsc.ingress:9998|10.255.0.13:9998|10.255.0.16:9998, 172.17.0.8:9998, dynatrace_collector_dev-wlp_wlp.7.3hvzd4e5b5zdby4blgu1v8rm8.ingress:9998|172.18.0.4:9998|10.255.0.13:9998|10.255.0.22:9998, dynatrace_collector_dev-wlp_wlp.6.5huzo8ot2fnzavm9ji5it4p9d.ingress:9998|172.18.0.4:9998|10.255.0.13:9998|10.255.0.21:9998, 172.17.0.5:9998, 172.17.0.7:9998, 172.18.0.3:9998|10.255.0.13:9998|10.255.0.17:9998|dynatrace_collector_dev-wlp_wlp.2.5a4bbjtne3hxm3i3f0a0xj7lv.ingress:9998, 10.255.0.18:9998|172.18.0.3:9998|dynatrace_collector_dev-wlp_wlp.3.4tr7zjjhgqqtn0fh9hd0l7qoo.ingress:9998|10.255.0.13:9998, 172.17.0.4:9998, 10.255.0.19:9998|172.18.0.3:9998|dynatrace_collector_dev-wlp_wlp.4.ep9fshp8i94rdz7p7cpwi9v77.ingress:9998|10.255.0.13:9998]", 
"source": "stdout", 
"tag": "itec-artifactory.fmr.com:6555/com.fmr.pl000123.ezpaas.ezpaas-dynatrace-collector:6.3-11/Dynatrace_Collector_DEV-WLP_WLP.7.3hvzd4e5b5zdby4blgu1v8rm8/5125046f7489" 
}

[sarnagar]

Hi @somesoni2 ,
I want to thank you so much for your efforts and time trying to help me out with my problem.
This is exactly what I was looking for.

As @lukejadamec said , Is there a way to get this done via configs without running the special search everytime?

Highly appreciate your help on this.

[lukejadamec]

@somesoni2 how can this be adapted to happen behind the scenes? We've already figured out how to extract the fields in configs without a special search. How do you change the UI without a special search?

[lukejadamec]

Can you post your current inputs, props, and transforms configs for this input?

[sarnagar]

Hi @lukejadamec

I tried these two configs and none worked.

PROPS

[ _json ]
CHARSET=UTF-8

INDEXED_EXTRACTIONS=json

KV_MODE=json
SHOULD_LINEMERGE=true
category=Structured
disabled=false
pulldown_type=true
TIMESTAMP_FIELDS=line
EXTRACT-fields = (\"source\":\")(?\w+)\",\"tag\":\"(?.\/{1}.)\/(?.)\/(?.)\"

2.

PROPS

[ _json ]
CHARSET=UTF-8

INDEXED_EXTRACTIONS=json

KV_MODE=json
SHOULD_LINEMERGE=true
category=Structured
disabled=false
pulldown_type=true
TIMESTAMP_FIELDS=line
REPORT-transform = fields

TRANSFORMS

[fields]
REGEX = (\"source\":\")(?\w+)\",\"tag\":\"(?.\/{1}.)\/(?.)\/(?.)\"

[sarnagar]

Adding the screenshot since the REGEX ain't captured correctly in comment section:

[lukejadamec]

I'm trying some things to restructure/pull this data out at index time.

[sarnagar]

Hi @lukejadamec ,

Thankyou so much for your time and efforts for helping me with this problem.
The Splunk support advised us to use search-time rather than index-time which would affect splunk performance although I'm not aware of the problems caused by index-time extractions.
It would be great if you could help me for a solution on this..

[lukejadamec]

I have created a SEDCMD string that when added to props.conf (and a restart) for the sourcetype will change all new indexed events:
"source to LogSource=
"tag to ContainerImage=
the second / from the end to ,ContainerService=
the first / from the end to ,ContainerID=

It works fine on all of the example data you've posted so far. Splunk automatically recognizes new field= names and sets the values accordingly.
However, the SEDCMD literally changes the event data before it is indexed and the change is permanent (no going back without re-indexing the events). Also, this will not affect any data that has already been indexed, so a separate search like those posted by @somesoni2 would be required for searching old data (or you could re-index the entire data set with the new config).
I can post the string, but I highly recommend that you create a test index, test sourcetype, and test input to test it before you implement it in production.

[sarnagar]

Wow..Thats great..!! Please post the configs and I will try it DEV first ..
Thankyou soo much for your help 🙂
1) Also after I use these configs , I will be unable to use the query that @somesoni2 provided since the events format gets changed before its indexed right??
2) Infact I modified query to view it in a format I wanted. Below is my query:
my base query | rex mode=sed "s/\"tag\":\"([^\/]+\/[^\/]+)\/([^\/]+)\/([^\"]+)/\n\"container-image\":\"\1\",\n\"container-service\":\"\2\",\n\"container-id\":\"\3/
s/\"line\":\"/\"/
s/{//
s/}//
s/\",\"s/\"\n\"s/"

[lukejadamec]

The current query you're using will work on events before this SEDCMD implementation but not after.
Good luck.

SEDCMD-modifySource = s/\"source\":/LogSource=/g s/\"tag\":/ContainerImage=/g s/\/(?=[^\/]+$)/,ContainerID=/1 s/\/(?=[^\/]+[^\/]+$)/,ContainerService=/1

SEDCMD is space delimited, so you'll actually find 4 replacement commands in the string.

[lukejadamec]

I should also mention that SEDCMD cannot be used post parsing, so you can't use automated Splunk sourcetypes that include extractions, and it is almost always required that the config is on the forwarder (not the indexer).

[sarnagar]

Hi @lukejadamec @somesoni2 ,

Thankyou for all your inputs and help. I really appreciate all your support and help and it was really helpfull.
I used the SEDCMD and modified the format as required. But the team doesn't want to get this done during index-time due to the complications involved. Is there a possibility of getting it done during search-time??