Why does Splunk return 0 results when filtering data that we know is there?

[FedeCarrizo]

Hi everyone!

We're sending events to Splunk using the HTTP Collector but we have an issue when we try to search for that data (using fields)

Example event data

{
	"event": "Sample message", 
	"sourcetype": "my-backend-json", 
	"fields": {
		"function.name": "lambda-2", 
		"function.version": "0.0.1",
        "function.env": "prod",
		"function.flow": "cashin",
		"function.country": "ARG",
		"request.awsRequestId": "0000001",
		"user.accountId": "00001",
		"logtype": "error"
	}
}

We see the events in Splunk search:

 

But the issue is when we select any field for filtering data, Splunk returns 0 results.

Any ideas?

Thank you!

[Nisha18789]

Hi @FedeCarrizo , it appears that the fields are having new line characters in them or something. To confirm this , can you try running query with wildcards in the field value and see if that returns the results?

”function.env”=*prod*

Also, what props.conf have you used for this ingestion?

[FedeCarrizo]

Thanks for your answer Nisha! 

Now it´s working with 

function.env=*prod* 

Any ideas how we can fix the "new line or something" issue?

Here´s the request to the HTTP Collector:

Thank you so much! 

[Nisha18789]

Hi @FedeCarrizo , can you check in your local props.conf the stanza for the sourcetype you are getting this data into?

[FedeCarrizo]

Hi @Nisha18789 !

I´m not sure if we are using (or modifying) the props.conf file. It´s posible? 

Here´s the SourceType config:

[Nisha18789]

Hi @FedeCarrizo , can you try replacing

INDEXED_EXTRACTIONS =  json 

by 

KV_MODE= json

and see it that helps?

[FedeCarrizo]

Hi @Nisha18789 

Just created a new SourceType with KV_MODE=json but no luck 😞 

I tried with a new event: 

{
	"event": "Error getting user info", 
	"sourcetype": "new-backend-json", 
	"fields": {
		"function.name": "lambda-cashin-1", 
		"function.version": "0.0.1",
        "function.env": "PROD",
		"function.flow": "cashin",
		"function.country": "ARG",
		"request.awsRequestId": "0000001",
		"user.accountId": "00001",
		"user.username": "fgc@uala.com.ar",
		"test":"sample",
		"logtype": "error"
	}
}

but the only "searchable" field is "logtype" (I tried with "test", which is at the same json level, but it didn't work)

[Nisha18789]

hello @FedeCarrizo , can you try removing  LINE_BREAKER from the set up and add below two if not present already.

DATETIME_CONFIG=CURRENT

SHOULD_LINEMERGE=false

[FedeCarrizo]

No luck with that configuration :(.