topic Re: multikv.conf for data with pipe delimeter in Getting Data In

multikv.conf for data with pipe delimeter

Parameshwara — Wed, 17 Apr 2013 07:11:51 GMT

multikv.conf

[testmultikv]
pre.linecount    = 1
header.linecount = 1
header.tokens    = _tokenize_, -1, "1"
body.tokens      = _tokenize_, 0, "1"

Sample data file:

School|Month|Subject_A_Score|Subject_B_Score
SchoolA|January|0|20
SchoolB|January|50|99
SchoolC|January|11|88
...

Search:

index=xxx | multikv conf=testmultikv | table School Month Subject...

Search results does not pick up the defined fields. What is missing in the configuration file?

Re: multikv.conf for data with pipe delimeter

andreas — Wed, 08 May 2013 18:14:14 GMT

A working configuration for multikv.conf is:

[testmultikv]
header.linecount = 1
header.tokens = _tokenize_, -1, "|"
body.tokens = _tokenize_, 0, "|"

(no pre section, and "|" (pipe) instead of "1" (one))

And you have to ensure that your whole data file will be indexed as one event, because multikv works on "table-formatted events".

The whole "table"

School|Month|Subject_A_Score|Subject_B_Score
SchoolA|January|0|20
SchoolB|January|50|99
SchoolC|January|11|88
...

has to be one event.

You can do this by setting BREAK_ONLY_BEFORE for the sourcetype to a pattern that never match like (?!) and SHOULD_LINEMERGE to true. Depending on your file size (number of lines) you probably also need to increase MAX_EVENTS (default is 500).

Sample sourcetype definition in props.conf:

[schooldata]
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=(?!)
MAX_EVENTS=10000
NO_BINARY_CHECK=1

Re: multikv.conf for data with pipe delimeter

Parameshwara — Fri, 10 May 2013 05:17:04 GMT

multikv.conf:

[testmultikv]
header.linecount = 1
header.tokens = _tokenize_, -1, "|"
body.tokens = _tokenize_, 0, "|"

props.conf

[datav1]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
pulldown_type = 1
BREAK_ONLY_BEFORE=(?!)
MAX_EVENTS=100000

my data is:

School|Month|SubjectA|SubjectB
SchoolZ|January|0|20
SchoolX|January|50|99
SchoolM|January|11|88

my search is:

index=multikv | multikv conf=testmultikv

What I get is the first data row becomes the field. In 'interesting field' one items appears as SchoolZ|January|0|20.

Re: multikv.conf for data with pipe delimeter

andreas — Fri, 10 May 2013 09:24:49 GMT

Is this line
School|Month|SubjectA|SubjectB
the first line in your data file? No empty line(s) before?

How does your inputs.conf look like? (Did you use sourcetype=datav1?)

What is the output of the search
index=multikv sourcetype=datav1
(should be one event containing all the data)

and what is the output of
index=multikv sourcetype=datav1 | multikv conf=testmultikv | table School Month SubjectA SubjectB

Which Splunk version are you using?

Re: multikv.conf for data with pipe delimeter

Parameshwara — Mon, 13 May 2013 00:15:50 GMT

School|Month|SubjectA|SubjectB is the first line in my data and there are no empty lines before.

my inputs.conf

[monitor:///opt/testdata/multikv]
disabled = false
followTail = 0
host = datav1
index = multikv
sourcetype = datav1

"index=multikv sourcetype=datav1" gives only 1 event which is the 4 lines of my data (1 header, 3 data)

"index=multikv sourcetype=datav1 | multikv conf=testmultikv | table School Month SubjectA SubjectB" gives "No results found"

I'm running Splunk 5.0.2, build 149561.