Transforms in Summary Index report smart and verbose mode generating huge amount of fields

[PatrickButterly]

Im trying to make transaction more usable for the end user ans the summary index seems to be the best option available for this because of the large data.
However I put the transaction in a report and summarise it with a list of fields and it works great...
Except i end up with several thousand fields extracted when i look for index=summary sourcetype=MyTransaction. Is there anyway to get it to only take the filed I list in the report? or at least not break it all up into nonsense. Or is my best option to only run it in fast mode and accept the limitation?
Thanks

[micahkemp · Accepted solution]

You should perform some kind of reporting command prior to summary indexing. This will define the fields that get placed in your summary.

Something like:

index=_internal | stats count BY sourcetype

Will yield more compact and efficient data in your summary index, and will also result in fewer unnecessary fields when searching it. Whereas something like:

index=_internal

(even if you include | fields) sent directly to a summary will result in the full _raw value being added to your summary index.

What you may have been doing was something like this:

index=_internal | table *

prior to the summary indexing. That would result in all fields, almost all of which you don't want. If you know you want to include each event separately, but only a few fields, do something like this instead:

index=_internal | table sourcetype host source

to retain just those fields in your summary.

[PatrickButterly]

Thanks, I think that table is what I was looking for. Using the fields command is what thew me off

[nickhills]

You can use the fields command to limit what gets returned in the fields list:

index=summary sourcetype=MyTransaction|fields fieldA fieldB fieldC

https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/SearchReference/Fields

In addition you could create a macro which conatins the above search, and your users could invoke it with just

`your_macro_name`