topic Re: log with key value pair or transforms.conf performance diffrence? in Getting Data In

log with key value pair or transforms.conf performance diffrence?

jazzythemartian — Mon, 28 Sep 2020 14:55:05 GMT

Hi,

to gain index size I made the log format as below. I didn't use key value pair.

20121101095842|192.168.1.2|KRQQQShcnQdRK8pLKTXC|20138494756382|I|PLAY|this the detailed info|1

And in transforms.conf I defined the fields.
DELIMS="|"
FIELDS=time,sourceip,session_id,customer_id,channel,op_type,detail,result_code

What if I made the log format like;

time=20121101095842,sourceip=192.168.1.2,sessiın_id=KRQQQShcnQdRK8pLKTXC,customer_id=20138494756382,channel=I, op_type=PLAY, detail=this the detailed info|result_code=1

Is there any performance diffrence between these two? a big diffrence in speed?

thanks,

Re: log with key value pair or transforms.conf performance diffrence?

kristian_kolb — Tue, 08 Oct 2013 08:36:27 GMT

Well, as you've probably calculated, you'll save some license space - in this case like 40%. I cannot see any immediate downside to the approach - as long as you keep the number and order of fields constant. With key=value pairs, that is not relevant, as the extraction takes place automatically.

You should probably set KV_MODE=none for this sourcetype in props.conf.

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Whether a REPORT is faster than KV_MODE=auto... I don't know - perhaps a little.

Re: log with key value pair or transforms.conf performance diffrence?

sowings — Tue, 08 Oct 2013 13:04:11 GMT

REPORT with DELIMS is definitely faster if you turn off KV_MODE=auto for that type. 🙂 I'm not sure if "properly configured" REPORT with DELIMS alone is faster than key=value pairs, however.

Re: log with key value pair or transforms.conf performance diffrence?

kristian_kolb — Mon, 28 Sep 2020 14:55:16 GMT

Naturally - having both is the worst 🙂

Gut feeling says that REPORT + KV_MODE=none should be faster than KV_MODE=auto. Should be fewer, less complicated steps. Though for some searches the difference might not be even noticeable.

Re: log with key value pair or transforms.conf performance diffrence?

sowings — Tue, 08 Oct 2013 14:00:02 GMT

I agree with your gut.