Web Page Input Splunk for Json feeds from Hybrid-Analysis.com

[dmenon84]

Hi,

I added the following web-page config on Search Head.

URL - https://www.hybrid-analysis.com/feed?json
Selector-Td
Index=main
sourcetype=hybrid-feeds

I am getting the feeds but the format is not what I desire to see

Logs on Splunk

---

browser="integrated_client" response_size="282029" response_code="200" request_time="520.60508728" url="https://www.hybrid-analysis.com/feed?json" content_md5="b30597afe6c188d0254022546120ad58" content_sha224="ca641b20ad4acfe098e0d78901586c098948facd65c2b49d1a487c58" encoding="ascii" content="{ \"count\": 205, \"status\": \"ok\", \"data\": [ { \"md5\": \"8488817780e9de9d705e2bee0e299e44\", \"sha1\": \"80d0a0d98b60be0b8d57c3e197dc73d80d8a936f\", \"sha256\": \"fbd35e3151052bfb33f74d9083b158929220d1b47b48944fa6d0181b30cee9f4\", \"isinteresting\": false, \"analysis_start_time\": \"2016-08-26 11:45:12\", \"threatscore\": 100, \"threatlevel\": 2, \"avdetect\": 12, \"isunknown\": false, \"vxfamily\": \"QVM20.1.0000.Malware\", \"submitname\": \"ransomware.zip\", \"isurlanalysis\": false, \"size\": 725148, \"type\": \"PE32 executable (GUI) Intel 80386, for MS Windows\", \"et_alerts_total\": 4, \"et_alerts_real_total\": 15, \"domains\": [ \"api.ipify.org\", \"rd7v7mhidgrulwqg.onion.link\" ], \"hosts\": [ \"103.198.0.2\", \"23.23.167.0\" ], \"compromised_hosts\": [ \"103.198.0.2\" ], \"et_alerts\": [ { \"destip\": \"8.8.8.8\", \"destport\": \"53\", \"protocol\": \"UDP\", \"action\": { \"signatureid\": \"2022332\", \"signaturerev\": \"3\", \"severity\": \"1\", \"category\": \"A Network Trojan was detected\", \"description\": \"ET POLICY DNS Query to .onion proxy Domain (onion.link)\" } }, { \"destip\": \"23.23.167.0\", \"destport\": \"80\", \"protocol\": \"TCP\", \"action\": { \"signatureid\": \"2021997\", \"signaturerev\": \"2\", \"severity\": \"1\", \"category\": \"Potential Corporate Privacy Violation\", \"description\": \"ET POLICY External IP Lookup api.ipify.org\" } }, { \"destip\": \"23.23.167.0\", \"destport\": \"80\", \"protocol\": \"TCP\", \"action\": { \"signatureid\": \"2021997\", \"signaturerev\": \"2\", \"severity\": \"1\", \"category\": \"Potential Corporate Privacy Violation\", \"description\": \"ET POLICY External IP Lookup api.ipify.org\" } }, { \"destip\": \"23.23.167.0\", \"destport\": \"80\", \"protocol\": \"TCP\", \"action\": { \"signatureid\": \"2021997\", \"signaturerev\": \"2\", \"severity\": \"1\", \"category\": \"Potential Corporate Privacy Violation\", \"description\": \"ET POLICY External IP Lookup api.ipify.org\" } } ], \"environmentId\": \"100\", \"environmentDescription\": \"Windows 7 32 bit\", \"sharedanalysis\": true, \"isreliable\": true, \"reporturl\": \"\/sample\/fbd35e3151052bfb33f74d9083b158929220d1b47b48944fa6d0181b30cee9f4\/?environmentId=100\", \"vt_detect\": 12, \"ms_detect\": 12 }, { \"md5\": \"5f791c9ef260305a483dd28a972c96f2\", \"sha1\": \"b01641b8e3083d44c34dfa9b57c6e04a73e9405c\", \"sha256\": \"5146d4ab415390c08f30135588e5e871e54e8c774d0dd7e8949ae010ddfd6394\", \"isinteresting\": false, \"analysis_start_time\": \"2016-08-26 11:41:05\", \"threatscore\": 8, \"threatlevel\": 0, \"avdetect\": 0, \"isunknown\": false, \"submitname\": \"Normal.dotm\", \"isurlanalysis\": false, \"size\": 20635, \"type\": \"Microsoft Word 2007+\", \"environmentId\": \"100\", \"environmentDescription\": \"Windows 7 32 bit\", \"sharedanalysis\": true, \"isreliable\": true, \"reporturl\": \"\/sample\/5146d4ab415390c08f30135588e5e871e54e8c774d0dd7e8949ae010ddfd6394\/?environmentId=100\", \"vt_detect\": 0, \"ms_detect\": 0 }, { \"md5\": \"0fdaa37867ca1a6b392ff5842b1ad167\", \"sha1\": \"1fbb0916e1efc68df54faf6f2e4f6524279058b1\", \"sha256\": \"9ac01ce2b88ce1c41a53ac967a8bbf434076f71c1d52ec54de404e2c7929d01f\", \"isinteresting\": false, \"analysis_start_time\": \"2016-08-26 11:38:42\", \"threatscore\": 87, \"threatlevel\": 2, \"avdetect\": 44, \"isunknown\": false, \"vxfamily\": \"Unwanted\", \"submitname\": \"Service_KMS.exe\", \"isurlanalysis\": false, \"size\": 974016, \"type\": \"PE32 executable (GUI) Intel 80386 Mono\/.Net assemb ...\", \"environmentId\": \"100\", \"environmentDescription\": \"Windows 7 32 bit\", \"sharedanalysis\": true, \"isreliable\": true, \"reporturl\": \"\/sample\/9ac01ce2b88ce1c41a53ac967a8bbf434076f71c1d52ec54de404e2c7929d01f\/?environmentId=100\", \"vt_detect\": 44, \"ms_detect\": 44 }, { \"md5\": \"5d2b528ecec2b102fa5e8dc94db33316\", \"sha1\": \"d0b3349e135295dea7fe64e288caf89e90368c19\", \"sha256\": \"5b52d60f833fd2c27be55ba24c02cb91773f9a7fa0261278aa783e2fb436b8b9\", \"isinteresting\": false, \"analysis_start_time\": \"2016-08-26 11:37:06\", \"threatscore\": 100, \"threatlevel\": 2, \"avdetect\": 40, \"isunknown\": false, \"vxfamily\": \"W97M.Downloader\", \"submitname\": \"guy.mackenzie.doc\", \"isurlanalysis\": false, \"size\": 42470, \"type\": \"Microsoft Word 2007+\", \"domains\": [ \"www.maxmind.com\" ], \"environmentId\": \"100\", \"environmentDescription\": \"Windows 7 32 bit\", \"sharedanalysis\": false, \"isreliable\": true, \"reporturl\": \"\/sample\/5b52d60f833fd2c27be55ba24c02cb91773f9a7fa0261278aa783e2fb436b8b9\/?environmentId=100\", \"vt_detect\": 40, \"ms_detect\": 40 }, { \"md5\": \"36280b99d6f882abbb843776a2f995ce\", \"sha1\": \"64a7bd642ecc672a9ac1420a7dd4087db31f93c4\", \"sha256\": \"c9866f3d453936bb71a84b13703dbb507f56b7b192ae2692900339295cf48f60\", \"isunknown\": true, \"isinteresting\": false, \"analysis_start_time\": \"2016-08-26 11:36:27\", \"threatscore\": 56, \"threatlevel\": 2, \"submitname\": \"eBILL_BritishGas.js\", \"isurlanalysis\": false, \"size\": 6793, \"type\": \"ASCII text\", \"et_alerts_total\": 2, \"et_alerts_real_total\": 2, \"domains\": [ \"www.numengo.com\" ], \"hosts\": [ \"217.70.180.131\" ], \"compromised_hosts\": [ \"217.70.180.131\" ], \"et_alerts\": [ { \"destip\": \"217.70.180.131\", \"destport\": \"80\", \"protocol\": \"TCP\", \"action\": { \"signatureid\": \"2021697\", \"signaturerev\": \"2\", \"severity\": \"1\", \"category\": \"A Network Trojan was detected\", \"description\": \"ET TROJAN EXE Download Request To Wordpress Folder Likely Malicious\" } }, { \"destip\": \"217.70.180.131\", \"destport\": \"80\", \"protocol\": \"TCP\", \"action\": { \"signatureid\": \"2022239\", \"signaturerev\": \"4\", \"severity\": \"1\", \"category\": \"A Network Trojan was detected\", \"description\": \"ET TROJAN EXE Download Request To Wordpress Folder Likely Malicious\" } } ], \"environmentId\": \"100\", \"environmentDescription\": \"Windows 7 32 bit\", \"sharedanalysis\": true, \"isreliable\": true, \"reporturl\": \"\/sample\/c9866f3d453936bb71a84b13703dbb507f56b7b192ae2692900339295cf48f60\/?environmentId=100\" }, { \"md5\": \"5a39973622ed4230bfcf003b4ac9f18b\", \"sha1\": \"2661f82b0ffd7ac3f274a31fa564dcb785a4fe36\", \"sha256\": \"d42135cf81df795b76fe0c6d0cac61de55966efd59cb1768c561b57e49ad7ab2\", \"isunknown\": true, \"isinteresting\": true, \"analysis_start_time\": \"2016-08-26 11:35:47\", \"threatscore\": 100, \"threatlevel\": 2, \"submitname\": \"vii_pay_commission_scales.doc\", \"isurlanalysis\": false, \"size\": 1052402, \"type\": \"Rich Text Format data, v

---

I have the following questions -
1) Is there another way to input data from https://www.hybrid-analysis.com/feed?json (public feeds)?
2) Also I am getting duplicates with the method I am using ? How to get rid of the duplicates?
3) Also should I add this in Master instead of Search Head ?

Thanks in advance for any guidance.

[neelamsantosh]

Mate have u integrated the feeds into splunk.

I have the same requirement as yours.