Why does searching fields in a summary index generate single value fields that have two identical values?

[Xisor]

Hey folks,

I'm looking at a summary index that's being generated through the Splunk Web (e.g. the source is being set to the savedsearch's name, into an index selected right at the bottom of Splunk Web for the saved search).

We're plugging in plenty of data, and it's ticking along nicely. The search command is basically this:

index=raw_index_with_stuff source=specific_alpha 
| stats avg(field_a) AS field_a avg(field_b) AS field_b avg(field_c) AS field_c by host
| rename host AS hostname

Nothing too complicated.

The events in the summary index similarly look right. E.g.:

01/09/2017 11:55:00 -0500, search_name="Summary_Stuff_5m", search_now=1483981500.000, info_min_time=1483980900.000, info_max_time=1483981200.000, info_search_time=1483981502.975, field_a=1.000 field_b=1.921 field_c=1.114 hostname=laptop1701a

Now, my problem is looking at the fields. If we do a simple query like:

index=summary_index_with_stuff source=Summary_Stuff_5m
| timechart avg(field_a) by hostname

It doesn't generate results. (It's also averaging averages, which is somewhat dubious, I know.)

Clicking on the events that feed into that (e.g. into the events tab), we see all the individual events properly summarised, as above, which leaves me scratching my head as to why that's the case.

When I click on an individual event (the > under the i column), we get closer to the heart of the issue:

A colleague of mine suggests this could be the result of the summarising search running twice - but that would surely generate two identical events, not an event with identical values in what should be single valued fields?

My suspicion is that it could be interplay with a field extraction (or similar)? (As it looks, to me, like certain fields have been extracted twice), but beyond 'a field extraction' (there are many), I'm currently stumped as to what I'm looking for.

[somesoni2 · Accepted solution]

Your summary index search and data looks good. It seems more to me like a field extraction issue where Splunk has tried to extract the field twice and causing the field to be a multivalued field. Since summary index writes in key-value fashion, you don't need any explicit field extractions. Check if anything is setup for your source.

[Xisor]

Good shout, somesoni2.

Looking in the transforms, we found:
transforms
[Summary_Stuff_5m]
DELIMS = ",", "="
CAN_OPTIMIZE = false
MV_ADD = true
CLEAN_KEYS = false

And I think it was that MV_ADD line that was the trouble. Testing it out in dev, it transpired to be the case. We think this'd been set inadvertently previously, and moving to MV_ADD = false looks to have entirely resolved the issue.

Huge thanks for everyone's input!

[DalJeanis]

I'd start with this -

index=summary_index_with_stuff source=Summary_Stuff_5m | head 5 | eval mytime = _time| table mytime field_a hostname

Are there results for that? If not, then one of the required fields is missing for "timechart avg(field_a) by hostname". The timechart command depends on the _time. Is it present in your summary index?

[Xisor]

Good shout, but it's more than that. So the search:
index=summary_index_with_stuff source=Summary_Stuff_5m
| head 5
| eval mytime = _time
| table mytime "Bytes_Received/sec" hostname

That 'properly' yields results:
mytime || field_a || hostname
1484039400 || 0.000000, 0.000000 || WS4HECC4, WS4HECC4
1484039400 || 0.000000, 0.000000 || WS4HECA2, WS4HECA2
1484039400 || 52.590353, 52.590353 || WS4ALAR8, WS4ALAR8
1484039400 || 1.093472, 1.093472 || WR8YX2KC,, WR8YX2KC
1484039400 || 2.613286, 2.613286 || WR8WX02V, WR8WX02V

Except that field_a & hostname are duplicate values.

[DalJeanis]

You can use mvdedup to kill the extra values, but that doesn't account for why they're there in the first place. The link posted by rjthibod looks like a good place to start.

[rjthibod]

It might be helpful if you shared the configuration in savedsearches.conf that is generating the summary data. Those details might be helpful.

Also, what kind of environment are you running: Clustered or standalone?

[Xisor]

Cheers, this is a clustered environment.

On one of the dedicated 'job' search heads (not used for ad-hoc searches, only scheduled ones) is where this savedsearches.conf piece lives:

[Summary_PerfmonMK:Redirector_5m]
action.email.useNSSubject = 1
action.summary_index = 1
action.summary_index._name = stuff_wks_summary
alert.digest_mode = True
alert.suppress = 0
alert.track = 0
auto_summarize.dispatch.earliest_time = -1d@h
cron_schedule = */5 * * * *
enableSched = 1
realtime_schedule = 0
schedule_window = 1
search = index=stuff_wks_perfmon sourcetype="PerfmonMK:Redirector" earliest=-10m@m latest=-5m@m | stats \
avg(Bytes_Received/sec) as Bytes_Received/sec \
avg(Bytes_Total/sec) as Bytes_Total/sec \
avg(Bytes_Transmitted/sec) as Bytes_Transmitted/sec \
avg(Current_Commands) as Current_Commands \
avg(File_Data_Operations/sec) as File_Data_Operations/sec \
avg(File_Read_Operations/sec) as File_Read_Operations/sec \
avg(File_Write_Operations/sec) as File_Write_Operations/sec \
avg(Network_Errors/sec) as Network_Errors/sec \
avg(Reads_Denied/sec) as Reads_Denied/sec \
avg(Server_Disconnects) as Server_Disconnects \
avg(Server_Sessions_Hung) as Server_Sessions_Hung \
avg(Writes_Denied/sec) as Writes_Denied/sec by host | rename host as hostname

As you see, it's perfmon and pretty straight forward.

All of those fields generated are duplicated, so too is info_max_time, info_min_time, info_search_time, search_name, search_now, and hostname (as mentioned before).

This occurs both on the regular clustered search heads (that live behind a load balancer) and also on the job search heads (breaking my own rules and doing an ad-hoc search on them gets the same results).

[rjthibod]

Since you are not using the summary indexing transforming commands (e.g., sistats), I think you might need to use addinfo and collect in your search. This is based on the guidance here http://docs.splunk.com/Documentation/Splunk/6.5.0/Knowledge/Configuresummaryindexes#Search_commands_....