Splunk Search

Why does Splunk return 0 results when filtering data that we know is there?

FedeCarrizo
Engager

Hi everyone!

We're sending events to Splunk using the HTTP Collector but we have an issue when we try to search for that data (using fields)

Example event data

 

 

{
	"event": "Sample message", 
	"sourcetype": "my-backend-json", 
	"fields": {
		"function.name": "lambda-2", 
		"function.version": "0.0.1",
        "function.env": "prod",
		"function.flow": "cashin",
		"function.country": "ARG",
		"request.awsRequestId": "0000001",
		"user.accountId": "00001",
		"logtype": "error"
	}
}

 

 

 

We see the events in Splunk search:

 

Screen Shot 2020-08-20 at 20.43.38.png

Screen Shot 2020-08-20 at 20.50.54.png

But the issue is when we select any field for filtering data, Splunk returns 0 results.

 

1.png2.png

 

Any ideas?

Thank you!

 

 

 

 

 

 

Labels (1)
Tags (1)
0 Karma

Nisha18789
Builder

Hi @FedeCarrizo , it appears that the fields are having new line characters in them or something. To confirm this , can you try running query with wildcards in the field value and see if that returns the results?

”function.env”=*prod*

Also, what props.conf have you used for this ingestion?

 

FedeCarrizo
Engager

Thanks for your answer Nisha! 

 

Now it´s working with 

function.env=*prod* 

 

Any ideas how we can fix the "new line or something" issue?

 

Here´s the request to the HTTP Collector:

 

Screen Shot 2020-08-20 at 21.29.31.png

 

Thank you so much! 

0 Karma

Nisha18789
Builder

Hi @FedeCarrizo , can you check in your local props.conf the stanza for the sourcetype you are getting this data into?

 

0 Karma

FedeCarrizo
Engager

Hi @Nisha18789 !

I´m not sure if we are using (or modifying) the props.conf file. It´s posible? 

 

Here´s the SourceType config:

Screen Shot 2020-08-20 at 21.47.40.png

 

0 Karma

Nisha18789
Builder

Hi @FedeCarrizo , can you try replacing

INDEXED_EXTRACTIONS =  json 

by 

KV_MODE= json

and see it that helps?

0 Karma

FedeCarrizo
Engager

Hi @Nisha18789 

 

Just created a new SourceType with KV_MODE=json but no luck 😞 

 

I tried with a new event: 

{
	"event": "Error getting user info", 
	"sourcetype": "new-backend-json", 
	"fields": {
		"function.name": "lambda-cashin-1", 
		"function.version": "0.0.1",
        "function.env": "PROD",
		"function.flow": "cashin",
		"function.country": "ARG",
		"request.awsRequestId": "0000001",
		"user.accountId": "00001",
		"user.username": "fgc@uala.com.ar",
		"test":"sample",
		"logtype": "error"
	}
}

 

but the only "searchable" field is "logtype" (I tried with "test", which is at the same json level, but it didn't work)

 

Screen Shot 2020-08-21 at 10.51.27.png

0 Karma

Nisha18789
Builder

hello @FedeCarrizo , can you try removing  LINE_BREAKER from the set up and add below two if not present already.

DATETIME_CONFIG=CURRENT

SHOULD_LINEMERGE=false

 

0 Karma

FedeCarrizo
Engager

No luck with that configuration :(.

 

Screen Shot 2020-08-24 at 16.43.06.png

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!