topic Re: indexed_extractions=json not working for FIFO file in Getting Data In

indexed_extractions=json not working for FIFO file

moneybox — Tue, 29 Sep 2020 23:37:52 GMT

Hi everyone,

In my inputs.conf I am monitoring a fifo file receiving json events.

Inputs.conf :
[fifo:///tmp/a.fifo]
disabled=0
index=main
sourcetype=json_test

Props.conf :
[json_test]
INDEXED_EXTRACTIONS=JSON
KV_MODE=none
AUTO_KV_JSON=false
SHOULD_LINEMERGE=false
TIME_FORMAT=%s.%6N

I see that all fields are parsed correctly, yet I cannot tstats by none metadata fields.
I then tried to add TRANSFORMS-js=test_js to props.conf and have the following stanza in transforms.conf :

[test_js]
REGEX=\"([a-zA-Z0-9_.]+)\":\"([^"]+)\"
FORMAT=$1::$2
REPEAT_MATCH=true
WRITE_META=true

In this case, I am able to tstats on some json fields. However, this regex does not cover all cases of keys and values of a json line and it seems very redundant to reparse all json fields after INDEXED_EXTRACTIONS=json.

Any ideas on how to solve this issue ?

Re: indexed_extractions=json not working for FIFO file

harsmarvania57 — Wed, 13 Mar 2019 19:24:08 GMT

Hi,

Can you please provide some sample data (Please mask sensitive data) ?

Re: indexed_extractions=json not working for FIFO file

moneybox — Thu, 14 Mar 2019 13:34:43 GMT

Hi,

Sure, here is a sample line that will go into the fifo file:
{"value": "New", "onclick": 123}

"value" and "onclick" will not be available in a tstats command such as :

| tstats count where index=main by value

Re: indexed_extractions=json not working for FIFO file

harsmarvania57 — Thu, 14 Mar 2019 13:51:39 GMT

For me it is working fine with indexing data directly not with fifo input (On Splunk 7.2.3) . Can you please let us know which version of splunk are you running?

I used only below config in props.conf while on boarding the data on my lab.

props.conf

[mysourcetype]
INDEXED_EXTRACTIONS = JSON

Re: indexed_extractions=json not working for FIFO file

moneybox — Thu, 14 Mar 2019 14:12:12 GMT

Hi,

I am using Splunk 7.2.4

Yes the above configuration works for monitor or batch.
It does not work with fifo files though

Re: indexed_extractions=json not working for FIFO file

harsmarvania57 — Fri, 15 Mar 2019 10:34:54 GMT

Yes, reproduced this issue with [fifo://...] stanza, it is not honoring INDEXED_EXTRACTIONS = JSON and due to that value and onclick are not indexing as index fields and due to that you can't use tstats (Because tstats read data from .tsidx files which contains only indexed fields name and data).

Re: indexed_extractions=json not working for FIFO file

moneybox — Sun, 17 Mar 2019 09:21:39 GMT

So is it possible to report a bug for splunk ?
Any idea how to achieve that ?
thanks

Re: indexed_extractions=json not working for FIFO file

harsmarvania57 — Sun, 17 Mar 2019 10:40:05 GMT

If you have active splunk support entitlement then you can raise case with splunk support and if they will say that this is expected behavior then I'll suggest to provide docs feedback on props.conf docs page to mention this as fifo input stanza doesn't support INDEXED_EXTRACTIONS.

Re: indexed_extractions=json not working for FIFO file

moneybox — Tue, 19 Mar 2019 15:25:50 GMT

I am afraid I use the free license for this project.
@harsmarvania57 is it possible for you to open a case ?

Thnk you very much