topic Re: cant concatenate indexed data in Deployment Architecture

cant concatenate indexed data

cwi — Mon, 28 Sep 2020 10:10:18 GMT

concatenating fields at index time doesn't seem to work. I have the following transform:

[gztdnv]
REGEX = <td>\s+(\S+)\s+(\S+)
FORMAT = td_nv::$1.$2
MV_ADD = true

the data looks like:

"<td> BW  400"

I would expect to see a value for td_nv of BW400 after the search. However, the transform remains in its unprocessed state indicating that td_nv is simply "$1.$2". If I just use $1 or $2 it is evaluated correctly with that piece.

This is described in the splunk docs section under "create custom fields at index time". Any ideas?

Re: cant concatenate indexed data

Ayn — Thu, 01 Dec 2011 07:57:22 GMT

Is this transform referred to in props.conf using the EXTRACT directive or TRANSFORMS directive? Could you paste the props.conf section?

Re: cant concatenate indexed data

tgow — Thu, 01 Dec 2011 15:15:27 GMT

There might be a couple of issues here but try this and let me know if it works:

[gztdnv]
REGEX = \<td\>\s+(\S+)\s+(\S+)
FORMAT = $1::$2
MV_ADD = true

The "<" and ">" have to be escaped with a backslash "\". Also if you are creating a grouping in the REGEX then the FORMAT has to be "$1::$2".

Here is a link to more information:

http://docs.splunk.com/Documentation/Splunk/4.2.4/Data/Configureindex-timefieldextraction

Re: cant concatenate indexed data

cwi — Mon, 28 Sep 2020 10:10:26 GMT

Tried it with both FORMAT suggestions with no change so far. The props.conf entry is below. FYI, all the other extracts and reports pieces work.

Thx.

[MyLogs]
TRUNCATE = 0
LINE_BREAKER = (?!)
BREAK_ONLY_BEFORE = ^
TIME_PREFIX = ^\s+
EXTRACT-segName = ^ (?[^\n]+)
EXTRACT-testName = ^ (?[^\n]+)
EXTRACT-testExec = ^ (?[^\n]+)
EXTRACT-npes = ^ (?[^\n]+)
EXTRACT-JobID = ^ (?[^\n]+)
EXTRACT-compiler = ^ (?[^\n]+)
EXTRACT-params = ^ (?[^\n]+)
EXTRACT-rmgr = ^ (?[^\n]+)
EXTRACT-user = ^ (?[^\n]+)
EXTRACT-vers = ^ (?[^\n]+)
EXTRACT-start = ^ (?[^\n]+)
EXTRACT-end = ^ (?[^\n]+)
EXTRACT-results = ^\s*\s+(?[^\n]+)
EXTRACT-nodes = ^ (?[^\n]+)
EXTRACT-real = ^real\s+(?[^\n]+)
REPORT-gztd = gztd
REPORT-gztdnv = gztdnv
REPORT-gzresults = gzresults

Re: cant concatenate indexed data

Ayn — Thu, 01 Dec 2011 20:21:47 GMT

You're using REPORT instead of TRANSFORMS for the fields you are trying to concatenate. REPORT is a definition for a search-time extraction, not an index-time extraction. You need to use TRANSFORMS if you want to concatenate values, as this is not supported for search-time extractions.

Re: cant concatenate indexed data

Ayn — Thu, 01 Dec 2011 20:22:20 GMT

< and > do not need to be escaped.

Re: cant concatenate indexed data

cwi — Mon, 28 Sep 2020 10:10:43 GMT

no luck so far... here are the three key files. Restarted splunk and no td_nv field even appears....

transforms.conf :

[gztdnv]
REGEX = \s+(\S+)\s+(\S+)\s+(\S+)
FORMAT = td_nv::"$1.$2"
WRITE_META = true

props.conf :

[mylogs]
TRANSFORMS-gztdnv = gztdnv

fields.conf:

[td_nv]
INDEXED=true