Display row, even when count over value is zero

[JMFrank215]

I have the following search:

index=aa sourcetype="bb" Service="/abc" OR Service="/mno" OR Service="/xyz" | chart count over Service by ZCode | addtotals

This returns the count of ZCode across /abc, /mno, and/xyz. Example would look like this:

Service	200	400	500	504	Total
/abc	2	3	1	2	8
/mno	1	4	0	1	6
/xyz	4	0	5	2	11

The problem I'm facing is if the values for 200, 400, 500, and 504 are all zeros across one service, it will no longer display that service. Very new to Splunk but did a fair amount of research but could not find a solution. Thank you!

[ITWhisperer]

The reason you couldn't find a solution is that basically there is no (simple) solution. splunk reports what is there, not what is not there. If there are no entries for a service, let's say /abc, from your example, how does splunk know to report on it? What about service /cba or /nonexistent or /completelymadeup? In order for splunk to report on things it hasn't found events for, you need to add dummy events as part of your search. You can do this with append and makeresults or inputlookup or a join from a wider search where the services you are interested in did have events.

[JMFrank215]

interesting, I would think it would still know to report on it because it is counting over all three of those services. That makes sense though. Anywhere you can point me to on how I would create dummy events for these services? Very very new to Splunk and this type of thing in general - everything I'm seeing and reading is a little overwhelming

[bowesmana · Accepted solution]

@JMFrank215 

It's not actually counting over anything, it's just looking at the data that exists as @ITWhisperer  says.

There is no what to know that your search term is supposed to be interpreted as a 'service'.

The simplest thing to do is to add this before your addtotals

| append [
  | makeresults
  | eval Service=split("/abc,/mno,/xyz",",")
]
| stats values(*) as * by Service
| fillnull value=0
| addtotals

Clearly your service names are not real, so in practice, you would most likely have a CSV file with your expected service names in there and inside the append, you would have a

| inputlookup your_service_list.csv

which would contain a column called Service

[JMFrank215]

Thank you, that explanation was really helpful. I tried adding what you suggested but even after playing around with some of the syntax, I am still getting "Error in 'eval' command: The arguments to the 'split' function are invalid." I can't seem to figure out why, maybe there's a small error in what I'm doing?

I can use my actual service names if that helps at all, was just being cautious about using actual information but can't imagine it matters at all. My search that returns the above error is below. It just has the three services called. Any input would be much appreciated

index=xx sourcetype="yy" Service="POST /loan/api/" OR Service="POST /credit/api/ OR Service="POST /transfer/api"

| chart count over Service by ZCode

| append [

            | makeresults

            | eval Service=split("POST /loan/api/", "POST /deposit/api/", "POST /transfer/api")

]

| stats values(*) as * by Service

| fillnull value=0

| addtotals

[bowesmana]

@JMFrank215 

Your split statement is wrong. It takes 2 params

1=the string to split

2=the delimiter

So, all the service names should be in a single string separated by comma and the delimiter is ","

So, that is creating a multi-value field called Service in the appended row, which is then used in the stats aggregation.

[JMFrank215]

Ah okay, I totally misinterpreted that part of your first response - still a novice. Seems to run correctly now but is there a way for it to only show those newly created services if there are no counts for the service? So would like for it to show the actual call counts for loan/api/ and /credit/api (rows 4 and 5) and then since there were no calls for /transfer/api/, display the newly created service for that (row 3). Not sure if that is too complicated or not possible? Table displayed is below, really appreciate all your help!

Row #	Service	200	400	500	504	Total
1	POST /loan/api/	0	0	0	0	0
2	POST /credit/api/	0	0	0	0	0
3	POST /transfer/api	0	0	0	0	0
4	POST /loan/api/	2	3	1	2	8
5	POST /credit/api/	1	4	0	1	6

[bowesmana]

The table is not right in that it is not treating your POST /loan/api service (row 1) the same as your row 4. Looks like that's most likely because you have a space in front of the POST in your append split clause. Without the space you would only get 3 rows as it would treat 1+4 and 2+5 as the same services.

[JMFrank215]

Yup, that did it. I missed that there couldn't be spaces between the services I wanted to split. Didn't realize Splunk was this strict, going to be tough to learn but hoping I get there eventually. Thanks so much for your help!