topic Re: Issue searching data in Splunk Search

Why does Splunk return 0 results when filtering data that we know is there?

FedeCarrizo — Fri, 21 Aug 2020 21:44:35 GMT

Hi everyone!

We're sending events to Splunk using the HTTP Collector but we have an issue when we try to search for that data (using fields)

Example event data

{ "event": "Sample message", "sourcetype": "my-backend-json", "fields": { "function.name": "lambda-2", "function.version": "0.0.1", "function.env": "prod", "function.flow": "cashin", "function.country": "ARG", "request.awsRequestId": "0000001", "user.accountId": "00001", "logtype": "error" } }

We see the events in Splunk search:

But the issue is when we select any field for filtering data, Splunk returns 0 results.

Any ideas?

Thank you!

Re: Issue searching data

Nisha18789 — Fri, 21 Aug 2020 00:22:34 GMT

Hi @FedeCarrizo , it appears that the fields are having new line characters in them or something. To confirm this , can you try running query with wildcards in the field value and see if that returns the results?

”function.env”=*prod*

Also, what props.conf have you used for this ingestion?

Re: Issue searching data

FedeCarrizo — Fri, 21 Aug 2020 00:31:26 GMT

Thanks for your answer Nisha!

Now it´s working with

function.env=*prod*

Any ideas how we can fix the "new line or something" issue?

Here´s the request to the HTTP Collector:

Thank you so much!

Re: Issue searching data

Nisha18789 — Fri, 21 Aug 2020 00:41:19 GMT

Hi @FedeCarrizo , can you check in your local props.conf the stanza for the sourcetype you are getting this data into?

Re: Issue searching data

FedeCarrizo — Fri, 21 Aug 2020 00:49:29 GMT

Hi @Nisha18789 !

I´m not sure if we are using (or modifying) the props.conf file. It´s posible?

Here´s the SourceType config:

Re: Issue searching data

Nisha18789 — Fri, 21 Aug 2020 08:41:04 GMT

Hi @FedeCarrizo , can you try replacing

INDEXED_EXTRACTIONS = json

KV_MODE= json

and see it that helps?

Re: Issue searching data

FedeCarrizo — Fri, 21 Aug 2020 13:55:07 GMT

Hi @Nisha18789

Just created a new SourceType with KV_MODE=json but no luck 😞

I tried with a new event:

{ "event": "Error getting user info", "sourcetype": "new-backend-json", "fields": { "function.name": "lambda-cashin-1", "function.version": "0.0.1", "function.env": "PROD", "function.flow": "cashin", "function.country": "ARG", "request.awsRequestId": "0000001", "user.accountId": "00001", "user.username": "fgc@uala.com.ar", "test":"sample", "logtype": "error" } }

but the only "searchable" field is "logtype" (I tried with "test", which is at the same json level, but it didn't work)

Re: Issue searching data

Nisha18789 — Fri, 21 Aug 2020 22:07:34 GMT

hello @FedeCarrizo , can you try removing LINE_BREAKER from the set up and add below two if not present already.

DATETIME_CONFIG=CURRENT

SHOULD_LINEMERGE=false

Re: Issue searching data

FedeCarrizo — Mon, 24 Aug 2020 19:43:45 GMT

No luck with that configuration :(.