We have several servers that support an HTTP request that presents a page of activity in a simple HTML table. I'd like to GET the page, find the table, and use the values as Splunk fields, and then the s as the values, with one Splunk event per table row. Website input seems like the right app for this, but I can't quite get it working. I defined the input to produce a preview like this: | webscrape selector="#myTable tr" url="http://server1.ourcompany.com/cgi-bin/reportStatus.cgi" depth_limit=25 text_separator="|" empty_matches=0 And I use a sourcetype of, say, "server_status". The results include the html table such as: <table id="myTable" border="1"> <tbody> <tr> <th>Seq</th> <th>Time</th> <th>Source</th> <th>Event</th> </tr> <tr style="background: rgb(224, 224, 224);"> <td>#1</td> <td>2012_02_07 03:02:08</td> <td>some-process_log</td> <td>file received</td> </tr> <tr>...</tr> </table> I'm aiming for Splunk events that look like: Seq=#1 Time=2012_02_07 03:02:08 Source=some-process_log Event=file receive What I see is: The header row with <TH>'s isn't captured as part of the the match field, I've played with other CSS selectors and, while results differ, they're still not right In the above preview query I can sort of get close by piping to ** | fields match | mvexpand match |...** where I could then probably further split on the '|' separator, however When I do a Splunk query such as sourcetype="server_status" I get a single event in which match is single valued and I have nothing seemingly to split on The match field actually gets leading/trailing and doubled up "|" delimiters, presumably because of the start/end <> tags? A la |39||#45||2013_08_09 22:20||some processing source||Sending confirmation email| This all seems like I'm going down the wrong rabbit hole. Can someone advise the right way to get this done? Thanks! ... View more

I'm have a custom command that parses an input field in each given record and emits 0 to N records as its output. I'm doing this to avoid a bunch of mvzip/mvexp and logic in the calling SPL. It does seem to work fine, but I'd like some reasurance that this is really supported in the SDK. The SDK doc for StreamingCommand of the PythonSDK (http://docs.splunk.com/Documentation/PythonSDK) says (bold highlighting mine), Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of up to 50,000 records.... This (and the rest of the article and others like it that I've found) don't really seem to specify how many records can be returned. The above seems to suggest that it's really supposed to be 1:1 and not 1:n. Here's an edited down version of my code. (I'm also a Python newbie, so apologies for any ugliness there.) import sys from mytokeninfo import Info sys.path.append("splunk_sdk-1.6.5-py2.7.egg") from splunklib.searchcommands import dispatch, StreamingCommand, Configuration @Configuration(local=True) # Per doc on "stateful" streaming commands class ExStatefulCommand(StreamingCommand): def stream(self, records): for record in records: tokens = self.parseRecordForTokens(record) for token in tokens: info = self.processToken(token) record['newField1'] = info.field1 # Application specifics simplified here for clarity (hopefully) record['newField2'] = info.field2 # ...etc yield record So, for each record, I'm augmenting it one or more times and also yield'ing it each time. If so, I'd love to see the doc for it. If not, can I get an explanation as to why and also suggestions for how best to deal with this in a clean and proper manner? ... View more

I'm trying to optimize execution of a custom command by caching information it processes, but just for the duration of the currently execution SPL. My custom command does something like this for each input record Parse a field from an application event in Splunk into 1 or more tokens for further processing For each token, do some expensive proccessing on the token and cache the results. This cache is really only valid for the scope of the SPL query since the results of the processing can differ each time. E.g., a restful API query for information that can be updated over time The SDK doc for StreamingCommand of the PythonSDK (http://docs.splunk.com/Documentation/PythonSDK) says (bold highlighting mine), Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of up to 50,000 records. Hence, _a search command must be prepared to be invoked many times during the course of pipeline processing. _ Each invocation should produce a set of results independently usable by downstream processors. My question here is: How can I maintain my cache of expensive processing results for the full scope of the SPL query and only for the query duration? That is, maintain the cached information over multiple command invocations for a given SPL query. I do see multiple invocations causing certain tokens to be needlessly processed multiply. My current cache is simply a Python dict() but I'm not picky. I do, however, need to know SPL start/end somehow so I can init and delete the chache. Such as a pre/post query callback. Or, for that matter, some way of hooking up my stateful command data to the query so that I can fetch it again via, say, the self.service.token or some such. Here's an edited down version of my code. (I'm also a Python newbie, so apologies for any ugliness there.) import sys from mytokeninfo import Info sys.path.append("splunk_sdk-1.6.5-py2.7.egg") from splunklib.searchcommands import dispatch, StreamingCommand, Configuration # A global cache of already processed "token" results to avoid # doing more than absolutely necessary knownTokens = None @Configuration(local=True) # Per doc on "stateful" streaming commands class ExStatefulCommand(StreamingCommand): def stream(self, records): for record in records: token = self.parseRecordForToken(record) if token not in knownTokens: self.processAndCache(token) # Call one or more restful APIs for this token and save results info = knownTokens[token] record['newField'] = info.field # Application specifics simplified here for clarity (hopefully) yield record Also, in the working code, I've put in some logging and definitely see the knownTokens size go back to zero in the search log. So, to restate question, how to I keep my cache populated, but just until the end of the calling SPL? ... View more