All Apps and Add-ons
Highlighted

Automatic search time extraction using external command

Path Finder

Is it possible to use external commands in for automatic search-time field extractions ?

Reason: I've got a case where performing a search-time extraction using regex isn't an option (well.. it is, but it'll need numerous heavy regexes and the performance would be awful), so the only option is using an external command to do the extractions. That works fine, but the end users have to pipe their searches through a command and that appears to be inconvenient and presents a performance penalty on the searches that have majority of the data coming from sources that don't require to be extracted with the external command.


Data in question comes from Arcsight. The particular problem is with the "custom strings" - stuff that isn't in Arcsight's schema, so it comes as a pair csN={data} csNLabel={key name}. For example:

cs1=Example Text cs1Label=key1

The custom script extracts those pairs, so the data from the example above will be parsed as:

key1="Example Text"

It wouldn't be too hard to write a regex to extract that, if only the data would have been ordered in some way. But it isn't - csN and csNLabel can come anywhere in the sting, the data can contain csNLabel with no csN and so on... For example:

cs1=Example Text cs2Label=key2Name cs1Label=key1 cs3=Some more text eventId=12345678 end=1294637812000 art=1294648800578 cs3Label=anoterKey
0 Karma
Highlighted

Re: Automatic search time extraction using external command

Communicator

Have you considered parsing the fields out at index time rather than at search time? Any chance we could see a sample of the data so the gurus here can possibly assist further with the extractions?

0 Karma
Highlighted

Re: Automatic search time extraction using external command

Path Finder

I have considered that, but then run into the usual problem - I've run into a new type of data that wouldn't have been caught by the extraction code I had in place. So if I've parsed the data at index time I would have to reindex whatever I have acquired by then 😞

Data comes from Arcsight. I'll add an explanation to the question body - don't have enough characters left in the comment

0 Karma
Highlighted

Re: Automatic search time extraction using external command

Splunk Employee
Splunk Employee

Yes you can, there's a bit of work you need to do: http://www.splunk.com/base/Documentation/latest/SearchReference/Aboutcustomsearchcommands

A built-in example that does something similar to what you probably is the | xmlkv search command, which is implemented by $SPLUNK_HOME/etc/apps/search/bin/xmlkv.py and configured in $SPLUNK_HOME/etc/apps/search/default/commands.conf.

But as for whether you can set up automatic extractions, the answer is no, but it is a good enhancement request.

0 Karma
Highlighted

Re: Automatic search time extraction using external command

Path Finder

The search command already exists, I've even shared it as an app. The question was really about automatic extractions, sorry for not being clear on that.
I'll file an RFE with support. Any idea if it may be implemented before 4.3?

0 Karma