Getting Data In

indexed_extractions=json not working for FIFO file

moneybox
Explorer

Hi everyone,

In my inputs.conf I am monitoring a fifo file receiving json events.

Inputs.conf :
[fifo:///tmp/a.fifo]
disabled=0
index=main
sourcetype=json_test

Props.conf :
[json_test]
INDEXED_EXTRACTIONS=JSON
KV_MODE=none
AUTO_KV_JSON=false
SHOULD_LINEMERGE=false
TIME_FORMAT=%s.%6N

I see that all fields are parsed correctly, yet I cannot tstats by none metadata fields.
I then tried to add TRANSFORMS-js=test_js to props.conf and have the following stanza in transforms.conf :

[test_js]
REGEX=\"([a-zA-Z0-9_.]+)\":\"([^"]+)\"
FORMAT=$1::$2
REPEAT_MATCH=true
WRITE_META=true

In this case, I am able to tstats on some json fields. However, this regex does not cover all cases of keys and values of a json line and it seems very redundant to reparse all json fields after INDEXED_EXTRACTIONS=json.

Any ideas on how to solve this issue ?

0 Karma

harsmarvania57
Ultra Champion

Hi,

Can you please provide some sample data (Please mask sensitive data) ?

0 Karma

moneybox
Explorer

Hi,

Sure, here is a sample line that will go into the fifo file:
{"value": "New", "onclick": 123}

"value" and "onclick" will not be available in a tstats command such as :

| tstats count where index=main by value

0 Karma

harsmarvania57
Ultra Champion

For me it is working fine with indexing data directly not with fifo input (On Splunk 7.2.3) . Can you please let us know which version of splunk are you running?

I used only below config in props.conf while on boarding the data on my lab.

props.conf

[mysourcetype]
INDEXED_EXTRACTIONS = JSON
0 Karma

moneybox
Explorer

Hi,

I am using Splunk 7.2.4

Yes the above configuration works for monitor or batch.
It does not work with fifo files though

0 Karma

harsmarvania57
Ultra Champion

Yes, reproduced this issue with [fifo://...] stanza, it is not honoring INDEXED_EXTRACTIONS = JSON and due to that value and onclick are not indexing as index fields and due to that you can't use tstats (Because tstats read data from .tsidx files which contains only indexed fields name and data).

0 Karma

moneybox
Explorer

So is it possible to report a bug for splunk ?
Any idea how to achieve that ?
thanks

0 Karma

harsmarvania57
Ultra Champion

If you have active splunk support entitlement then you can raise case with splunk support and if they will say that this is expected behavior then I'll suggest to provide docs feedback on props.conf docs page to mention this as fifo input stanza doesn't support INDEXED_EXTRACTIONS.

0 Karma

moneybox
Explorer

I am afraid I use the free license for this project.
@harsmarvania57 is it possible for you to open a case ?

Thnk you very much

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...