https://community.splunk.com/t5/Splunk-Search/How-to-get-a-sourcetype-of-JSON-mixed-with-text-that-uses-the/m-p/141515#M39218 <P>I downvoted this post because dosnt answer the question</P> Tue, 01 Dec 2015 15:09:58 GMT nmanolak 2015-12-01T15:09:58Z https://community.splunk.com/t5/Splunk-Search/How-to-get-a-sourcetype-of-JSON-mixed-with-text-that-uses-the/m-p/141511#M39214 <P>Hi I'm trying to get JSON data from a message queue into splunk. This works very well but the imported events also contain some message header information</P> <PRE><CODE>Tue Dec 02 15:45:56 CET 2014 name="amqp_msg_received" event_id="null" msg_queue="dns-events2splunk-splunk" msg_exchange="dns-events2splunk" msg_body="{"timestamp":"2014-12-02T14:45:56.021Z","src":"192.168.16.56","src_port":"53","dest":"192.168.0.2","dest_port":7521,"dns_meta":{"status":"NOERROR","id":42347,"flags":["qr","rd","ra"],"queries":"1","answers":"1"},"dns_query":{"domain":"lopq.ch.","class":"IN","type":"A"},"dns_response":{"domain":"lopq.ch.","ttl":8477,"class":"IN","type":"A","IP":"10.14.183.106"}}" </CODE></PRE> <P>How can i now get a sourcetype that</P> <OL> <LI>using the timestamp within the json object</LI> <LI>correctly extracts the kv pairs as well as the json kv pairs</LI> </OL> <P>Alternatively only taking the msg_body and correctly extract the json object</P> Tue, 02 Dec 2014 15:04:23 GMT https://community.splunk.com/t5/Splunk-Search/How-to-get-a-sourcetype-of-JSON-mixed-with-text-that-uses-the/m-p/141511#M39214 mathiask 2014-12-02T15:04:23Z https://community.splunk.com/t5/Splunk-Search/How-to-get-a-sourcetype-of-JSON-mixed-with-text-that-uses-the/m-p/141512#M39215 <P>btw im using Splunk Enterprise and not Splunk Cloud, and there i cannot find the sourcetype json_predefined_timestamp</P> Mon, 28 Sep 2020 18:18:39 GMT https://community.splunk.com/t5/Splunk-Search/How-to-get-a-sourcetype-of-JSON-mixed-with-text-that-uses-the/m-p/141512#M39215 mathiask 2020-09-28T18:18:39Z https://community.splunk.com/t5/Splunk-Search/How-to-get-a-sourcetype-of-JSON-mixed-with-text-that-uses-the/m-p/141513#M39216 <P>I also use Websphere, and the jms connector...</P> <P>There are a lot of possible actions, but nothing that covers your needs at 100%...<BR /> I don't use json, but xml, but problems and solutions are the same...</P> <UL> <LI>I'm happy about the header, because we put a lot of good informations in it, we can search at, because there are automatically converted to fields by splunk</LI> <LI>To handle the body, I've made a field-extraction with a regex like<BR /> "body=(?P.*). After that I can user spath : spath input=xml ...</LI> </UL> <P>There two other options :<BR /> - make a custom message handler, for the jms connector, there you could put all your json data into key-value, to have splunk extract it automatically<BR /> - or with a custom message handler or changes in the jms connector, but everything in json, and use kvmode=json (I'm not sure it works, for xml you can make it... you have to check).</P> <P>.</P> Tue, 02 Dec 2014 17:43:41 GMT https://community.splunk.com/t5/Splunk-Search/How-to-get-a-sourcetype-of-JSON-mixed-with-text-that-uses-the/m-p/141513#M39216 sbsbb 2014-12-02T17:43:41Z https://community.splunk.com/t5/Splunk-Search/How-to-get-a-sourcetype-of-JSON-mixed-with-text-that-uses-the/m-p/141514#M39217 <P>I you don't mind loosing the data that doesn't fall within the msg_body, then you can use SEDCMD to eliminate all the the JSON string, something like:</P> <PRE><CODE>SEDCMD-stripnonjson1 = s/^.*msg_body="// SEDCMD-stripnonjson2 = s/}"$/}/ </CODE></PRE> <P>This will leave only the JSON string for it index, which can then be used to get the timestamp and be parsed into the kv pairs:</P> <PRE><CODE>KV_MODE = json TIME_PREFIX = "timestamp":" </CODE></PRE> <P>These would go in the props.conf file for the data coming in to the indexers. You may also want to define the linebreak to do it on each JSON string at the beginning of a line if they cross lines:</P> <PRE><CODE>BREAK_ONLY_BEFORE = ^{ NO_BINARY_CHECK = 1 SHOULD_LINEMERGE = true </CODE></PRE> <P>I have tried this on your data, and it appears to have worked.</P> Tue, 02 Dec 2014 20:44:06 GMT https://community.splunk.com/t5/Splunk-Search/How-to-get-a-sourcetype-of-JSON-mixed-with-text-that-uses-the/m-p/141514#M39217 cpetterborg 2014-12-02T20:44:06Z https://community.splunk.com/t5/Splunk-Search/How-to-get-a-sourcetype-of-JSON-mixed-with-text-that-uses-the/m-p/141515#M39218 <P>I downvoted this post because dosnt answer the question</P> Tue, 01 Dec 2015 15:09:58 GMT https://community.splunk.com/t5/Splunk-Search/How-to-get-a-sourcetype-of-JSON-mixed-with-text-that-uses-the/m-p/141515#M39218 nmanolak 2015-12-01T15:09:58Z