Splunk Search

Why does Stats command ocassionally produces job search components in results?

splunkkitty
Path Finder

I have the following stats search:

 

 

index=servers1 OR index=servers2 DBNAME=DATABASENAME source="/my/log/source/*" | stats list(Tablespace), list("Total MB"), list("Used MB"), list("Free MB"), list("Percent Free") by DBNAME

 

 

Which is supposed to produce the following results:

DBNAME list(Tablespace) list(Total MB) list(Used MB) list(Free MB) list(Percent Free)
DATABASENAME

RED_DATA

165890

46350

119540

72

 

BLUE_DATA

2116

1016

1100

52

 

PINK_DATA

10

0

10

100

 

PURPLE_DATA

34

17

17

50

 

GREEN_DATA

6717

0

6717

100

 

ORANGE_DATA

51323

295

51028

99

…. Cont'd for 20+ rows          


About 25-30% of the time (as both a live search and scheduled report) I get the following results instead. The list(Used MB) column will have extra garbage data which appears to be job search components:

DBNAME list(Tablespace) list(Total MB) list(Used MB) list(Free MB) list(Percent Free)
DATABASENAME

RED_DATA

165890

46350

119540

72

 

BLUE_DATA

2116

1016

1100

52

 

PINK_DATA

10

0

10

100

 

PURPLE_DATA

34

17

17

50

 

GREEN_DATA

6717

0

6717

100

 

ORANGE_DATA

51323

295

51028

99

     

ion.command.search

   
     

4

   
     

duration.command.search.calcfields

   
     

0

   
     

duration.command.search.fieldalias

   
     

0

   
     

duration.command.search.filter

   
     

0

   
     

duration.command.search.index

   
     

2

   
     

duration.command.search.index.usec_1_8

   
     

0

   
     

duration.command.search.index.usec_8_64

   
     

0

   
     

duration.command.search.kv

   
     

0

   
     

duration.command.search.lookups

   
…. Cont'd for 20+ rows          


I've adjusted the following limits.conf Search Head settings with no luck:

 

 

[stats]
maxresultrows = 50000
maxvalues = 10000
maxvaluesize = 10000
list_maxsize = 10000

 


The search inspector also does not produce any notable error messages.

Any ideas as to what is happening here and how I can solve it?

Labels (4)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The erroneous results imply an error extracting the "Used MB" field.  To help fix it, we'll need to see sample data and the props.conf settings for the sourcetype.

---
If this reply helps you, Karma would be appreciated.
0 Karma

splunkkitty
Path Finder

The pretrained CSV sourcetype is used. No customized props is referenced in the input.

Inputs.conf:

[monitor:///my/log/source/]
index = my_index
disabled = 0
crcSalt = <SOURCE>


CSV source example:

"Tablespace","Used MB","Free MB","Total MB","Percent Free","DBNAME"
"RED_DATA",1974,6,1980,0,"DATABASENAME"
"BLUE_DATA",217532,605514,823046,74,"DATABASENAME"
"PINK_DATA",0,10,10,100,"DATABASENAME"
"PURPLE_DATA",0,100,100,100,"DATABASENAME"
"GREEN_DATA",23682,1258,24940,5,"DATABASENAME"
"ORANGE_DATA",241,996,1237,81,"DATABASENAME"

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You say the csv sourcetype is used, but the inputs.conf does not specify any sourcetype.  Let's start by adding sourcetype = csv to the inputs.conf stanza.

---
If this reply helps you, Karma would be appreciated.
0 Karma

splunkkitty
Path Finder

Updated and deployed inputs.conf with csv sourcetype (and additionally allow list for CSVs only):

[monitor:///my/log/source/]
index = my_index
whitelist = (\.csv$)
sourcetype = csv
recursive = false
disabled = 0
crcSalt = <SOURCE>


As of this time nothing has changed with the execution of the search (job search components still appear under the "Used MB" field with a small percentage of searches).

Thank you for your input!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Thanks for trying that.  Have you looked at the raw data?  It sure seems like there are some non-numerical values in the "Used MB" field that are throwing off the results.

---
If this reply helps you, Karma would be appreciated.
0 Karma

splunkkitty
Path Finder

I had a chance to sit down with the owner of the hosts. Nothing looks amiss with any of the CSV files (we have this same file ingesting from multiple hosts). Formatting is good.

When we were originally set things up the "Percentage Free" column was named "% Free" which was causing a problem. We had to update it so the files would ingest.

0 Karma

brijebi
New Member

I am running into the same problem.   I am also using stats list() and getting job inspector data in the list column.  My source type is syslog.

It only happens when I use a search head to search another indexer.  When I search on the indexer itself, the results are fine.

.

0 Karma

splunkkitty
Path Finder

I ended up reaching out to Splunk support on the issue. Per their reply there were bugs in 7.3.8 and some versions of 8.2.* that were similar. Our deployment is running on 8.2.2.1.

Unfortunately after some lab testing and back and forth we were unable to isolate and resolve it. In the meantime I've been using the table command with this particular dataset to get around the garbage output.

They recommended upgrading our Splunk deployment to the most recent release and then checking to see if it is still happening. We haven't gotten this far yet.

Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...