topic Scripted input with cURL in Getting Data In

Scripted input with cURL

jedatt01 — Fri, 02 Nov 2012 20:01:20 GMT

I would like to have splunk index the output of cURL but instead of seeing the entire html I'd like to be able to have splunk just index the data that a browser renders. Is there anyway to do this without some major scripting?

Re: Scripted input with cURL

gcoles — Fri, 02 Nov 2012 20:21:27 GMT

In the scripted input, you could pipe the output of the curl command through a command like hxnormalize, which is part of the html-xml-utils package in Linux.

Re: Scripted input with cURL

gkanapathy — Mon, 05 Nov 2012 05:49:29 GMT

You mean you want to parse the HTML and extract only selected parts of it?

Re: Scripted input with cURL

jedatt01 — Wed, 21 Nov 2012 19:08:25 GMT

gcoles, this answer didn't quite work for me, however what did work was that I found out how the page was getting the dynamic information. It was using a .cgi call. So, I just created a scripted input that calls the .cgi and formats the output to key value pairs