Getting Data In
Highlighted

indexed_extractions=json not working for FIFO file

Explorer

Hi everyone,

In my inputs.conf I am monitoring a fifo file receiving json events.

Inputs.conf :
[fifo:///tmp/a.fifo]
disabled=0
index=main
sourcetype=json_test

Props.conf :
[jsontest]
INDEXED
EXTRACTIONS=JSON
KVMODE=none
AUTO
KVJSON=false
SHOULD
LINEMERGE=false
TIME_FORMAT=%s.%6N

I see that all fields are parsed correctly, yet I cannot tstats by none metadata fields.
I then tried to add TRANSFORMS-js=test_js to props.conf and have the following stanza in transforms.conf :

[testjs]
REGEX=\"([a-zA-Z0-9
.]+)\":\"([^"]+)\"
FORMAT=$1::$2
REPEATMATCH=true
WRITE
META=true

In this case, I am able to tstats on some json fields. However, this regex does not cover all cases of keys and values of a json line and it seems very redundant to reparse all json fields after INDEXED_EXTRACTIONS=json.

Any ideas on how to solve this issue ?

0 Karma
Highlighted

Re: indexed_extractions=json not working for FIFO file

SplunkTrust
SplunkTrust

Hi,

Can you please provide some sample data (Please mask sensitive data) ?

0 Karma
Highlighted

Re: indexed_extractions=json not working for FIFO file

Explorer

Hi,

Sure, here is a sample line that will go into the fifo file:
{"value": "New", "onclick": 123}

"value" and "onclick" will not be available in a tstats command such as :

| tstats count where index=main by value

0 Karma
Highlighted

Re: indexed_extractions=json not working for FIFO file

SplunkTrust
SplunkTrust

For me it is working fine with indexing data directly not with fifo input (On Splunk 7.2.3) . Can you please let us know which version of splunk are you running?

I used only below config in props.conf while on boarding the data on my lab.

props.conf

[mysourcetype]
INDEXED_EXTRACTIONS = JSON
0 Karma
Highlighted

Re: indexed_extractions=json not working for FIFO file

Explorer

Hi,

I am using Splunk 7.2.4

Yes the above configuration works for monitor or batch.
It does not work with fifo files though

0 Karma
Highlighted

Re: indexed_extractions=json not working for FIFO file

SplunkTrust
SplunkTrust

Yes, reproduced this issue with [fifo://...] stanza, it is not honoring INDEXED_EXTRACTIONS = JSON and due to that value and onclick are not indexing as index fields and due to that you can't use tstats (Because tstats read data from .tsidx files which contains only indexed fields name and data).

0 Karma
Highlighted

Re: indexed_extractions=json not working for FIFO file

Explorer

So is it possible to report a bug for splunk ?
Any idea how to achieve that ?
thanks

0 Karma
Highlighted

Re: indexed_extractions=json not working for FIFO file

SplunkTrust
SplunkTrust

If you have active splunk support entitlement then you can raise case with splunk support and if they will say that this is expected behavior then I'll suggest to provide docs feedback on props.conf docs page to mention this as fifo input stanza doesn't support INDEXED_EXTRACTIONS.

0 Karma
Highlighted

Re: indexed_extractions=json not working for FIFO file

Explorer

I am afraid I use the free license for this project.
@harsmarvania57 is it possible for you to open a case ?

Thnk you very much

0 Karma