Archive
Highlighted

Problem using multikv to parse tabular data

Motivator

Hello!

I'm trying to create a custom multikv.conf stanza for the first time, and am having trouble getting it working. Here is an example of the events I'm working with:

Object ID Volume                    Aggregate     Storage Server          Daily Growth Rate   Days To Full Daily Growth Rate (%)
--------- ------------------------- ------------- ----------------------  ------------------- ------------ ---------------------
36906     volume_1                  aggr1_vmw_sas nas-vmw-prod02.foo.com           1074009984            1                  10.0
25031     volume_2                  aggr2_vmw_sas nas-vmw-prod03.foo.com             29670000           75                   0.6
25033     volume_3                  aggr2_vmw_sas nas-vmw-prod03.foo.com             23464000          146                   0.4
5430      volume_4                  aggr1_vmw_sas nas-vmw-prod02.foo.com             22181400          273                   0.3
Totals                                                                             1249102095
The multikv.conf stanza I created is as follows:
[volume_growth_rates]

#ignore the first two lines, as getting proper tokens with the spaces and % will be problematic
pre.start = "Object"
pre.linecount = 2
pre.ignore = _all_

#Manually define the header tokens
header.tokens = _token_list_, ObjectId, Volume, Aggregate, StorageServer, DailyGrowthRate, DaysToFull, DailyGrowthRatePct

#tokenize with space separators
body.end = "Totals"
body.tokens = _tokenize_, 0, " "

#ignore the 'Totals' line, as it doesn't really contain useful data.
post.start = "Totals"
post.linecount = 1
post.ignore = _all_

When I try to use this in search, nothing happens. Trying both:

... | multikv

and

... | multkv conf=volume_growth_rates

Any ideas on what I'm missing here?

Thanks!

Tags (2)
Highlighted

Re: Problem using multikv to parse tabular data

Splunk Employee
Splunk Employee

Tested in v6.0.

multkv.conf is annoying.
It is acting funny. We cannot trust the spec file.... sad. Examples do not work....

Solution 1. Forget about multikv.conf and use general Field Extraction

  • props.conf

    [source::.../Log/testmultikvtable.log]
    REPORT-multikv = testmultikvtable1 testmultikvtable2

  • transforms.conf

    [testmultikvtable1]
    REGEX = ^((\d+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+))
    FORMAT = Row::$1 objectID::$2 volume::$3 aggregate::$4 storageServer::$5 dailyGrowthRate::$6 daysToFull::$7 growthRate::$8
    MV
    ADD = true

    [testmultikvtable_2]
    REGEX = ^Totals\s+(\S+)
    FORMAT = Totals::$1

Solution 2. Stick with buggy multikv.conf

# Put multikv.conf in all of Splunk instance (Search Head and Peers)
- props.conf
[source::.../Log/test_multikv_table.log]
LINE_BREAKER = ([\n\r]+)Object ID
SHOULD_LINEMERGE = false
DATETIME_CONFIG  = CURRENT
KV_MODE = multi_volumeGrowthRates
  • multikv.conf

    Bug: must be located in Search Peer's local directory, instead of bundle from SH

    [volumeGrowthRates]

    Not sure if "pre" is working

    pre.start = "Object"
    pre.end = "----.*"

    pre.linecount = 2

    pre.ignore = all

    header setting

    header.tokens = tokenlist_,"ObjectId","Volume","Aggregate","StorageServer","DailyGrowthRate","DaysToFull","DailyGrowthRatePct"

    Body needs to set ignoring pre and post lines

    body.tokens = tokenize, 0, " "
    body.ignore = regex "----|Totals"

    Post... ignore is not working???

    post.start = "Totals"
    post.linecount = 1
    post.ignore = all

0 Karma
Highlighted

Re: Problem using multikv to parse tabular data

Contributor

Maybe it is issue with identifying the header?

How does SPLUNk know when to ignore the header here?

example with the manual approach of assigning headers, we specify start of header & linecount:

# specify table header location and processing 
header.start = "Object ID" 
header.linecount = 1 
header.tokens = _tokenize_, -1," "
0 Karma