Scripted input with cURL

[jedatt01]

I would like to have splunk index the output of cURL but instead of seeing the entire html I'd like to be able to have splunk just index the data that a browser renders. Is there anyway to do this without some major scripting?

[gkanapathy]

You mean you want to parse the HTML and extract only selected parts of it?

[gcoles]

In the scripted input, you could pipe the output of the curl command through a command like hxnormalize, which is part of the html-xml-utils package in Linux.

[jedatt01]

gcoles, this answer didn't quite work for me, however what did work was that I found out how the page was getting the dynamic information. It was using a .cgi call. So, I just created a scripted input that calls the .cgi and formats the output to key value pairs