Does Splunk have an option to only index part of a JSON file?

[tamduong16]

My json file is very long but most of the information in there is redundant. I just want to get all the segments that start with the line callIdentifier and end with the line endTime. The number of segment like this is in the json file is unpreditable. Is there a way I could do this with Splunk?

Here is the example of the json file:

{
"plcmCallList" : [ {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/f5a9acac-2878-460e-8744-aa64a110a128",
"rel" : "self",
"type" : "application/vnd.plcm.plcm-call",
"title" : "Self Relationship"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/f5a9acac-2878-460e-8744-aa64a110a128/call-...",
"rel" : "urn:com:polycom:api:rest:link-relations:events",
"type" : "application/vnd.plcm.plcm-audit-event-list",
"title" : "Call Events"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/f5a9acac-2878-460e-8744-aa64a110a128/subsc...",
"rel" : "urn:com:polycom:api:rest:link-relations:subscription-events",
"type" : "application/vnd.plcm.plcm-subscription-event-list",
"title" : "Subscription Events"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/f5a9acac-2878-460e-8744-aa64a110a128/prope...",
"rel" : "urn:com:polycom:api:rest:link-relations:property-changes",
"type" : "application/vnd.plcm.plcm-audit-property-change-list",
"title" : "Property Changes"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/f5a9acac-2878-460e-8744-aa64a110a128/signa...",
"rel" : "urn:com:polycom:api:rest:link-relations:signaling-diagram",
"type" : "image/png",
"title" : "Signaling Diagram"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/f5a9acac-2878-460e-8744-aa64a110a128/bandw...",
"rel" : "urn:com:polycom:api:rest:link-relations:bandwidth",
"type" : "application/vnd.plcm.plcm-bandwidth",
"title" : "Bandwidth"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/f5a9acac-2878-460e-8744-aa64a110a128/qos",
"rel" : "urn:com:polycom:api:rest:link-relations:qos",
"type" : "application/vnd.plcm.plcm-qos",
"title" : "QoS"
} ],
"destinationDetails" : {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/devices/c6acb1b6-4e7b-45da-ae74-4dd7f26de805",
"rel" : "urn:com:polycom:api:rest:link-relations:device-details",
"type" : "application/vnd.plcm.plcm-device",
"title" : "device-details"
} ],
"deviceIdentifier" : "c6acb1b6-4e7b-45da-ae74-4dd7f26de805",
"ipAddress" : "199.81.66.22",
"deviceName" : "abc70-tam",
"deviceModel" : "PolycomRealPresenceGroup310",
"deviceVersion" : "6.1.0",
"aliases" : [ "sip:abc70-tam@ute.jjjj.com" ],
"registrationStatus" : "ACTIVE",
"site" : "ABC",
"territory" : "Default DMA Territory (dma-l4)",
"authenticationStatus" : "NOT_APPLICABLE"
},
"originatorDetails" : {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/devices/642b14bb-c624-4e72-aa0e-7dba421b22ba",
"rel" : "urn:com:polycom:api:rest:link-relations:device-details",
"type" : "application/vnd.plcm.plcm-device",
"title" : "device-details"
} ],
"deviceIdentifier" : "642b14bb-c624-4e72-aa0e-7dba421b22ba",
"ipAddress" : "199.81.66.17",
"deviceName" : "abc70-oled",
"deviceModel" : "PolycomRealPresenceGroup310",
"deviceVersion" : "6.1.0",
"aliases" : [ "sip:abc70-oled@ute.jjjj.com" ],
"registrationStatus" : "ACTIVE",
"site" : "ABC",
"territory" : "Default DMA Territory (dma-l4)",
"authenticationStatus" : "NOT_APPLICABLE"
},
"callIdentifier" : "f5a9acac-2878-460e-8744-aa64a110a128",
"originator" : "sip:abc70-oled@ute.jjjj.com",
"dialString" : "sip:abc70-tam@ute.jjjj.com",
"destination" : "abc70-tam",
"nodeId" : "fc4d797f-e368-485e-94b6-58fb8c13f683",
"callStatus" : "Ended",
"callDuration" : "0:1:70",
"callSignaling" : "SIP",
"cluster" : "vrh12345.ute.jjjj.com",
"entityTag" : "ecb9cf92882c65d6e1dce00f759e515c",
"startTime" : "2017-10-27T14:06:55.912-0500",
"endTime" : "2017-10-27T14:08:05.980-0500"
}, {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/64d5b15a-84aa-419c-b861-ce632311fb55",
"rel" : "self",
"type" : "application/vnd.plcm.plcm-call",
"title" : "Self Relationship"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/64d5b15a-84aa-419c-b861-ce632311fb55/call-...",
"rel" : "urn:com:polycom:api:rest:link-relations:events",
"type" : "application/vnd.plcm.plcm-audit-event-list",
"title" : "Call Events"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/64d5b15a-84aa-419c-b861-ce632311fb55/subsc...",
"rel" : "urn:com:polycom:api:rest:link-relations:subscription-events",
"type" : "application/vnd.plcm.plcm-subscription-event-list",
"title" : "Subscription Events"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/64d5b15a-84aa-419c-b861-ce632311fb55/prope...",
"rel" : "urn:com:polycom:api:rest:link-relations:property-changes",
"type" : "application/vnd.plcm.plcm-audit-property-change-list",
"title" : "Property Changes"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/64d5b15a-84aa-419c-b861-ce632311fb55/signa...",
"rel" : "urn:com:polycom:api:rest:link-relations:signaling-diagram",
"type" : "image/png",
"title" : "Signaling Diagram"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/64d5b15a-84aa-419c-b861-ce632311fb55/bandw...",
"rel" : "urn:com:polycom:api:rest:link-relations:bandwidth",
"type" : "application/vnd.plcm.plcm-bandwidth",
"title" : "Bandwidth"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/64d5b15a-84aa-419c-b861-ce632311fb55/qos",
"rel" : "urn:com:polycom:api:rest:link-relations:qos",
"type" : "application/vnd.plcm.plcm-qos",
"title" : "QoS"
} ],
"destinationDetails" : {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/devices/c6acb1b6-4e7b-45da-ae74-4dd7f26de805",
"rel" : "urn:com:polycom:api:rest:link-relations:device-details",
"type" : "application/vnd.plcm.plcm-device",
"title" : "device-details"
} ],
"deviceIdentifier" : "c6acb1b6-4e7b-45da-ae74-4dd7f26de805",
"ipAddress" : "199.81.66.22",
"deviceName" : "abc70-tam",
"deviceModel" : "PolycomRealPresenceGroup310",
"deviceVersion" : "6.1.0",
"aliases" : [ "sip:abc70-tam@ute.jjjj.com" ],
"registrationStatus" : "ACTIVE",
"site" : "ABC",
"territory" : "Default DMA Territory (dma-l4)",
"authenticationStatus" : "NOT_APPLICABLE"
},
"originatorDetails" : {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/devices/642b14bb-c624-4e72-aa0e-7dba421b22ba",
"rel" : "urn:com:polycom:api:rest:link-relations:device-details",
"type" : "application/vnd.plcm.plcm-device",
"title" : "device-details"
} ],
"deviceIdentifier" : "642b14bb-c624-4e72-aa0e-7dba421b22ba",
"ipAddress" : "199.81.66.17",
"deviceName" : "abc70-oled",
"deviceModel" : "PolycomRealPresenceGroup310",
"deviceVersion" : "6.1.0",
"aliases" : [ "sip:abc70-oled@ute.jjjj.com" ],
"registrationStatus" : "ACTIVE",
"site" : "ABC",
"territory" : "Default DMA Territory (dma-l4)",
"authenticationStatus" : "NOT_APPLICABLE"
},
"callIdentifier" : "64d5b15a-84aa-419c-b861-ce632311fb55",
"originator" : "sip:abc70-oled@ute.jjjj.com",
"dialString" : "sip:abc70-tam@ute.jjjj.com",
"destination" : "abc70-tam",
"nodeId" : "fc4d797f-e368-485e-94b6-58fb8c13f683",
"callStatus" : "Ended",
"callDuration" : "0:0:22",
"callSignaling" : "SIP",
"cluster" : "vrh12345.ute.jjjj.com",
"entityTag" : "115072029b78ce0d56d958e1d8abb9e6",
"startTime" : "2017-10-27T14:06:25.459-0500",
"endTime" : "2017-10-27T14:06:47.467-0500"
}, {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/b08725c4-ad38-4d48-9d32-faf5bf8134d8",
"rel" : "self",
"type" : "application/vnd.plcm.plcm-call",
"title" : "Self Relationship"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/b08725c4-ad38-4d48-9d32-faf5bf8134d8/call-...",
"rel" : "urn:com:polycom:api:rest:link-relations:events",
"type" : "application/vnd.plcm.plcm-audit-event-list",
"title" : "Call Events"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/b08725c4-ad38-4d48-9d32-faf5bf8134d8/subsc...",
"rel" : "urn:com:polycom:api:rest:link-relations:subscription-events",
"type" : "application/vnd.plcm.plcm-subscription-event-list",
"title" : "Subscription Events"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/b08725c4-ad38-4d48-9d32-faf5bf8134d8/prope...",
"rel" : "urn:com:polycom:api:rest:link-relations:property-changes",
"type" : "application/vnd.plcm.plcm-audit-property-change-list",
"title" : "Property Changes"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/b08725c4-ad38-4d48-9d32-faf5bf8134d8/signa...",
"rel" : "urn:com:polycom:api:rest:link-relations:signaling-diagram",
"type" : "image/png",
"title" : "Signaling Diagram"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/b08725c4-ad38-4d48-9d32-faf5bf8134d8/bandw...",
"rel" : "urn:com:polycom:api:rest:link-relations:bandwidth",
"type" : "application/vnd.plcm.plcm-bandwidth",
"title" : "Bandwidth"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/b08725c4-ad38-4d48-9d32-faf5bf8134d8/qos",
"rel" : "urn:com:polycom:api:rest:link-relations:qos",
"type" : "application/vnd.plcm.plcm-qos",
"title" : "QoS"
} ],
"destinationDetails" : {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/devices/c6acb1b6-4e7b-45da-ae74-4dd7f26de805",
"rel" : "urn:com:polycom:api:rest:link-relations:device-details",
"type" : "application/vnd.plcm.plcm-device",
"title" : "device-details"
} ],
"deviceIdentifier" : "c6acb1b6-4e7b-45da-ae74-4dd7f26de805",
"ipAddress" : "199.81.66.22",
"deviceName" : "abc70-tam",
"deviceModel" : "PolycomRealPresenceGroup310",
"deviceVersion" : "6.1.0",
"aliases" : [ "sip:abc70-tam@ute.jjjj.com" ],
"registrationStatus" : "ACTIVE",
"site" : "ABC",
"territory" : "Default DMA Territory (dma-l4)",
"authenticationStatus" : "NOT_APPLICABLE"
},
"originatorDetails" : {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/devices/642b14bb-c624-4e72-aa0e-7dba421b22ba",
"rel" : "urn:com:polycom:api:rest:link-relations:device-details",
"type" : "application/vnd.plcm.plcm-device",
"title" : "device-details"
} ],
"deviceIdentifier" : "642b14bb-c624-4e72-aa0e-7dba421b22ba",
"ipAddress" : "199.81.66.17",
"deviceName" : "abc70-oled",
"deviceModel" : "PolycomRealPresenceGroup310",
"deviceVersion" : "6.1.0",
"aliases" : [ "sip:abc70-oled@ute.jjjj.com" ],
"registrationStatus" : "ACTIVE",
"site" : "ABC",
"territory" : "Default DMA Territory (dma-l4)",
"authenticationStatus" : "NOT_APPLICABLE"
},
"callIdentifier" : "b08725c4-ad38-4d48-9d32-faf5bf8134d8",
"originator" : "sip:abc70-oled@ute.jjjj.com",
"dialString" : "sip:abc70-tam@ute.jjjj.com",
"destination" : "abc70-tam",
"nodeId" : "fc4d797f-e368-485e-94b6-58fb8c13f683",
"callStatus" : "Ended",
"callDuration" : "0:0:30",
"callSignaling" : "SIP",
"cluster" : "vrh12345.ute.jjjj.com",
"entityTag" : "8bda37701498b0b2f6a5a5aaec15cb74",
"startTime" : "2017-10-27T14:05:45.421-0500",
"endTime" : "2017-10-27T14:06:16.282-0500"
}, {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/71bc1d27-d101-469d-9519-a7639410c0a7",
"rel" : "self",
"type" : "application/vnd.plcm.plcm-call",
"title" : "Self Relationship"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/71bc1d27-d101-469d-9519-a7639410c0a7/call-...",
"rel" : "urn:com:polycom:api:rest:link-relations:events",
"type" : "application/vnd.plcm.plcm-audit-event-list",
"title" : "Call Events"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/71bc1d27-d101-469d-9519-a7639410c0a7/subsc...",
"rel" : "urn:com:polycom:api:rest:link-relations:subscription-events",
"type" : "application/vnd.plcm.plcm-subscription-event-list",
"title" : "Subscription Events"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/71bc1d27-d101-469d-9519-a7639410c0a7/prope...",
"rel" : "urn:com:polycom:api:rest:link-relations:property-changes",
"type" : "application/vnd.plcm.plcm-audit-property-change-list",
"title" : "Property Changes"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/71bc1d27-d101-469d-9519-a7639410c0a7/signa...",
"rel" : "urn:com:polycom:api:rest:link-relations:signaling-diagram",
"type" : "image/png",
"title" : "Signaling Diagram"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/71bc1d27-d101-469d-9519-a7639410c0a7/bandw...",
"rel" : "urn:com:polycom:api:rest:link-relations:bandwidth",
"type" : "application/vnd.plcm.plcm-bandwidth",
"title" : "Bandwidth"
}, {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/reports/calls/71bc1d27-d101-469d-9519-a7639410c0a7/qos",
"rel" : "urn:com:polycom:api:rest:link-relations:qos",
"type" : "application/vnd.plcm.plcm-qos",
"title" : "QoS"
} ],
"destinationDetails" : {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/devices/642b14bb-c624-4e72-aa0e-7dba421b22ba",
"rel" : "urn:com:polycom:api:rest:link-relations:device-details",
"type" : "application/vnd.plcm.plcm-device",
"title" : "device-details"
} ],
"deviceIdentifier" : "642b14bb-c624-4e72-aa0e-7dba421b22ba",
"ipAddress" : "199.81.66.17",
"deviceName" : "abc70-oled",
"deviceModel" : "PolycomRealPresenceGroup310",
"deviceVersion" : "6.1.0",
"aliases" : [ "sip:abc70-oled@ute.jjjj.com" ],
"registrationStatus" : "ACTIVE",
"site" : "ABC",
"territory" : "Default DMA Territory (dma-l4)",
"authenticationStatus" : "NOT_APPLICABLE"
},
"originatorDetails" : {
"atomLinkList" : [ {
"href" : "https://vrh12345.ute.jjjj.com:8443/api/rest/devices/c6acb1b6-4e7b-45da-ae74-4dd7f26de805",
"rel" : "urn:com:polycom:api:rest:link-relations:device-details",
"type" : "application/vnd.plcm.plcm-device",
"title" : "device-details"
} ],
"deviceIdentifier" : "c6acb1b6-4e7b-45da-ae74-4dd7f26de805",
"ipAddress" : "199.81.66.22",
"deviceName" : "abc70-tam",
"deviceModel" : "PolycomRealPresenceGroup310",
"deviceVersion" : "6.1.0",
"aliases" : [ "sip:abc70-tam@ute.jjjj.com" ],
"registrationStatus" : "ACTIVE",
"site" : "ABC",
"territory" : "Default DMA Territory (dma-l4)",
"authenticationStatus" : "NOT_APPLICABLE"
},
"callIdentifier" : "71bc1d27-d101-469d-9519-a7639410c0a7",
"originator" : "sip:abc70-tam@ute.jjjj.com",
"dialString" : "sip:abc70-oled@ute.jjjj.com",
"destination" : "abc70-oled",
"nodeId" : "fc4d797f-e368-485e-94b6-58fb8c13f683",
"callStatus" : "Ended",
"callDuration" : "0:2:157",
"callSignaling" : "SIP",
"cluster" : "vrh12345.ute.jjjj.com",
"entityTag" : "758e7384f69a1b27afc26323ee014d9a",
"startTime" : "2017-10-27T13:55:00.837-0500",
"endTime" : "2017-10-27T13:57:38.545-0500"
} ]
}

[Damien_Dallimor · Accepted solution]

In that case (you are using the REST API Modular Input) , then you should use a custom response handler to pull out the "callIdentifier" to "endTime" keys and index this chunk as individual events. Very easy to do. I would not use SEDCMD.

[skoelpin]

There's a few approaches you can take here. You could either split the json message into multiple events and then route all unwanted data to nullQueue or you could use SEDCMD.

Either way will require writing a lot of regex, but I think using SEDCMD will be less effort

[tamduong16]

Hi, do you know where is a good start for me if I choose to go with SEDCMD? Thanks!

[skoelpin]

SEDCMD has 3 parts to it, each part is separated by a forward slash.

First part is s/ which means you want to match a pattern and replace it with something.
The second part is your regex pattern that you want to match and replace
The third part is what to replace it with. In your case, you want to remove so you should replace it with an empty string

So one part will look like this

SEDCMD-remove_line_one = s/\"plcmCallList\"\s:\s\[\s{//g

Edit your props.conf and add this in there. Fields are relative to sourcetype, so make sure your adding the correct sourcetype to the staza below.

[Yoursourcetype]
 ...Other configurations...
 SEDCMD-remove_line_one = s/\"plcmCallList\"\s:\s\[\s{//g

I would test this in a dev environment before applying to production.

[Damien_Dallimor]

How are you getting that JSON into Splunk ?

[tamduong16]

I plan to use rest api modular input for this. That is an example of the json file that I get from the api call.