https://community.splunk.com/t5/Monitoring-Splunk/Data-cleaning-for-better-performance-and-usage/m-p/386791#M3325 <P>Ok and thanks for the advice.<BR /> Do they icnrease the cost as the mcollect command will "convert events into metric data to be stored in a metric index on the search head" ?</P> <P>Someone as proposed me to use ES+data Model / SIEM to make the job but not sure that it will reply to my expectation. From my understanding it's more to do the metric analytics than clean and format metrics. What do you think?</P> Mon, 27 May 2019 07:22:26 GMT guillain 2019-05-27T07:22:26Z https://community.splunk.com/t5/Monitoring-Splunk/Data-cleaning-for-better-performance-and-usage/m-p/386789#M3323 <P>Hello people,</P> <P>I try to figure out a design for the metric indexing with the following constrainst:<BR />- keep the original raw data<BR />- availability of the metrics (ok for 15/30min)<BR />- high number of indexes and TB by day<BR />- lot of data manipulation for metric name and format alignement (factor the volume)<BR />- high search complexity (accross many indexes...)</P> <P>In that case, what do you suggest as what I've in mind is not really good...?<BR />- lookup: with the volume, the data manipulation and the search it's not sure to have a good performance result<BR />- kafka: add design complexity (ms, infra...) and imply to rewrite the current transformation rules<BR />- transformation during the indexing: it's not recommanded and it doesn't match with the need to keep the original raw <BR />- reindexing: data in new indexes will duplicate cost (infra but splunk lic also?) and increase the delay to have the metrics</P> <P>Thanks in advance for your help and enjoy your weekend <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span></P> Sat, 06 Jun 2020 02:07:21 GMT https://community.splunk.com/t5/Monitoring-Splunk/Data-cleaning-for-better-performance-and-usage/m-p/386789#M3323 guillain 2020-06-06T02:07:21Z https://community.splunk.com/t5/Monitoring-Splunk/Data-cleaning-for-better-performance-and-usage/m-p/386790#M3324 <P>Do a savedsearch that runs <CODE>mcollect</CODE> (<A href="https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/mcollect">doc reference</A>) into a metrics index. Give this metrics index the desired retention time.<BR /> Also, docs: <A href="https://docs.splunk.com/Documentation/Splunk/7.2.6/Indexer/Setupmultipleindexes#Create_metrics_indexes">Create metrics indexes</A></P> <P>Skalli</P> Fri, 24 May 2019 21:00:39 GMT https://community.splunk.com/t5/Monitoring-Splunk/Data-cleaning-for-better-performance-and-usage/m-p/386790#M3324 skalliger 2019-05-24T21:00:39Z https://community.splunk.com/t5/Monitoring-Splunk/Data-cleaning-for-better-performance-and-usage/m-p/386791#M3325 <P>Ok and thanks for the advice.<BR /> Do they icnrease the cost as the mcollect command will "convert events into metric data to be stored in a metric index on the search head" ?</P> <P>Someone as proposed me to use ES+data Model / SIEM to make the job but not sure that it will reply to my expectation. From my understanding it's more to do the metric analytics than clean and format metrics. What do you think?</P> Mon, 27 May 2019 07:22:26 GMT https://community.splunk.com/t5/Monitoring-Splunk/Data-cleaning-for-better-performance-and-usage/m-p/386791#M3325 guillain 2019-05-27T07:22:26Z https://community.splunk.com/t5/Monitoring-Splunk/Data-cleaning-for-better-performance-and-usage/m-p/386792#M3326 <P>ES doesn't really use metrics. You could build a Data Model and use your own accelerated one for custom dashboards but ES requires an additional license. If you don't have an ES running, you won't need it just for some metrics.<BR /> Also, Splunk Enterprise also has the option to build data models and accelerate them. Docs: <A href="https://docs.splunk.com/Documentation/Splunk/7.2.6/Knowledge/Aboutdatamodels">About data models</A></P> Tue, 28 May 2019 16:33:19 GMT https://community.splunk.com/t5/Monitoring-Splunk/Data-cleaning-for-better-performance-and-usage/m-p/386792#M3326 skalliger 2019-05-28T16:33:19Z