topic Re: Problem with using SOURCE_KEY in Splunk Search

Problem with using SOURCE_KEY

dmaislin_splunk — Fri, 23 Dec 2011 17:16:20 GMT

I have some XML data that I parse into many fields, one of which is "relativePath" why can't I get the transforms to extract a new field "fileName" from the SOURCE_KEY? The rex command works fine in the search bar:

  | rex field=relativePath "^.*[\\\/](?<fileName>.*)"

Sample Event:

<CheckEventRequest>
  <EventList count="1">
    <Event event="0x20000" path="\\cepapoc.emcsplunk.com\CHECK$\server2fs1\davidpoc2" flag="0x2" protocol="0" server="CEPAPOC" share="server2fs1" clientIP="10.0.0.2" serverIP="10.0.0.4" timeStamp="0x4EF4883C00014D1D" userSid="S-1-5-21-175151209-4036982877-1867759480-500" ownerSid="S-1-5-32-544" fileSize="0x0" newName="\\cepapoc.emcsplunk.com\CHECK$\server2fs1\SplunkEMC" desiredAccess="0x0" createDispo="0x0" ntStatus="0x0" relativePath="\\CEPAPOC\server2fs1\davidpoc2"/>
  </EventList>
</CheckEventRequest>

props.conf

[cepa]
LINE_BREAKER = ([\r\n]+20\d{2}/[01]\d/[0123]\d\s[012]\d:[0-5]\d:[0-5]\d[\r\n]+)
SHOULD_LINEMERGE = FALSE
TIME_PREFIX = timeStamp="
MAX_TIMESTAMP_LOOKAHEAD = 18
DATETIME_CONFIG = /etc/apps/vnx/default/emc-epoch.xml
REPORT-xmlkv = xmlkv-alternative
REPORT-getFileName = getFileName

transforms.conf

[xmlkv-alternative]
REGEX = <([^\s\>]*)[^\>]*\>([^<]*)\<\/\1\>
FORMAT = $1::$2
MV_ADD = True

[getFileName]
SOURCE_KEY = relativePath
REGEX = ^.*[\\\/](.*)
FORMAT = fileName::"$1"

Re: Problem with using SOURCE_KEY

_d_ — Fri, 23 Dec 2011 17:44:25 GMT

See if this works:

[cepa] LINE_BREAKER = ([\r\n]+20\d{2}/[01]\d/[0123]\d\s[012]\d:[0-5]\d:[0-5]\d[\r\n]+) SHOULD_LINEMERGE = FALSE TIME_PREFIX = timeStamp=" MAX_TIMESTAMP_LOOKAHEAD = 18 DATETIME_CONFIG = /etc/apps/vnx/default/emc-epoch.xml REPORT-my_name = xmlkv-alternative, getFileName

This particular REPORT sequence insures that the [xmlkv-alternative] transform stanza gets applied first, then [getFileName].

Hope this helps.

> please upvote and accept answer if you find it useful - thanks!

Re: Problem with using SOURCE_KEY

_d_ — Fri, 23 Dec 2011 17:55:17 GMT

Perhaps could be hitting the problem described here: http://blogs.splunk.com/2011/10/07/cannot-search-based-on-an-extracted-field/ 🙂

Re: Problem with using SOURCE_KEY

Ledion_Bitincka — Fri, 23 Dec 2011 18:09:16 GMT

This is most likely related to a bad regex. Assuming relativePath="\CEPAPOC\server2fs1\davidpoc2" and that you want extract fileName=davidpoc2 then the following should do the trick (note the updated regex in getFileName)

props.conf
[cepa] 
LINE_BREAKER = ([\r\n]+20\d{2}/[01]\d/[0123]\d\s[012]\d:[0-5]\d:[0-5]\d[\r\n]+)
SHOULD_LINEMERGE = FALSE TIME_PREFIX = timeStamp="
MAX_TIMESTAMP_LOOKAHEAD = 18 
DATETIME_CONFIG = /etc/apps/vnx/default/emc-epoch.xml 
REPORT-my_name = xmlkv-alternative, getFileName

transforms.conf
[xmlkv-alternative]
REGEX = <([^\s\>]*)[^\>]*\>([^<]*)\<\/\1\>
FORMAT = $1::$2
MV_ADD = True

[getFileName]
SOURCE_KEY = relativePath
# need to extract filenames from unix and windows paths, so use both forward/backward slashes
REGEX = (?<fileName>[^\\/]+)$

Re: Problem with using SOURCE_KEY

dmaislin_splunk — Fri, 23 Dec 2011 18:56:48 GMT

The answer is.....

The data was using autokv to extract all the delimited fields, not my xmlkv-alternative. SOURCE_KEY does not work well with the default splunk autokv. I replaced it with kv-alternative.

props.conf

[cepa]
LINE_BREAKER = ([\r\n]+20\d{2}/[01]\d/[0123]\d\s[012]\d:[0-5]\d:[0-5]\d[\r\n]+)
SHOULD_LINEMERGE = FALSE
TIME_PREFIX = timeStamp="
MAX_TIMESTAMP_LOOKAHEAD = 18
DATETIME_CONFIG = /etc/apps/vnx/default/emc-epoch.xml
REPORT-parsefields = kv-alternative,getFileName
TRANSFORMS-removehb = removehb
LOOKUP-event = eventlookup event OUTPUTNEW event_description
LOOKUP-dispo = dispolookup createDispo OUTPUTNEW createDispo_Description
KV_MODE = none

transforms.conf

[kv-alternative]
REGEX = (\w+)="([^"]+)"
FORMAT = $1::$2
MV_ADD = True

[getFileName]
SOURCE_KEY = relativePath
REGEX = (?<fileName>[^\\]+)$