topic Re: How to make search efficient with spath and regex in Splunk Search https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492410#M137352 <P>Its not too bad. I was wondering if there was a better way to do this. Since it has both regex and spath and also an eval</P> Thu, 21 Nov 2019 21:03:48 GMT vjzone 2019-11-21T21:03:48Z How to make search efficient with spath and regex https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492408#M137350 <P>How can I make this search efficient?</P> <PRE><CODE>earliest=-1m source="/var/log/aws/opsworks/opsworks-agent.statistics.log" host="*prod*" Reported statistics data | dedup host | rex field=_raw "Reported statistics data: (?&lt;json&gt;.*)\N" | fields json, host | spath input=json | rename stats.memory.free as memFree, stats.memory.total as memtotal | eval memFreePer=memFree/memtotal*100 | table host, memFreePer, stats.cpu.idle </CODE></PRE> Thu, 21 Nov 2019 17:29:59 GMT https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492408#M137350 vjzone 2019-11-21T17:29:59Z Re: How to make search efficient with spath and regex https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492409#M137351 <P>What makes you think it's inefficient? What does Job Inspector say?<BR /> Have you tried replacing 'Reported statistics data' with ' "Reported statistics data:" ' in the base search?</P> Thu, 21 Nov 2019 18:15:49 GMT https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492409#M137351 richgalloway 2019-11-21T18:15:49Z Re: How to make search efficient with spath and regex https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492410#M137352 <P>Its not too bad. I was wondering if there was a better way to do this. Since it has both regex and spath and also an eval</P> Thu, 21 Nov 2019 21:03:48 GMT https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492410#M137352 vjzone 2019-11-21T21:03:48Z Re: How to make search efficient with spath and regex https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492411#M137353 <P>There is not greater efficiency to be had other than to explicitly specify an index; here is that along with some other clarification adjustments:</P> <PRE><CODE>index="YouShouldAlwaysSpecifyAnIndex" AND source="/var/log/aws/opsworks/opsworks-agent.statistics.log" AND host="prod" AND Reported AND statistics AND data | dedup host | rex "Reported statistics data: (?&lt;json&gt;.*)\N" | fields json host | spath input=json | rename stats.memory.free AS memFree, stats.memory.total AS memtotal | eval memFreePer = 100 * memFree / memtotal | table host, memFreePer, stats.cpu.idle </CODE></PRE> Thu, 21 Nov 2019 22:53:13 GMT https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492411#M137353 woodcock 2019-11-21T22:53:13Z Re: How to make search efficient with spath and regex https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492412#M137354 <P>Thanks woodcock. Every one of our logs has index=main, which is why i chose to ignore it. Apart from that, is the query alright?</P> Fri, 22 Nov 2019 01:33:08 GMT https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492412#M137354 vjzone 2019-11-22T01:33:08Z Re: How to make search efficient with spath and regex https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492413#M137355 <P>Yes, efficiency-wise, there is nothing to do, but there some best-practices that provide additional clarity as shown in my answer. Is ALL of your data in <CODE>index=main</CODE> or just the data you need for this purpose? If the former, you should REALLY fix that. In any case, do not ever run a search without specifying <CODE>index=</CODE> somewhere.</P> Fri, 22 Nov 2019 16:07:39 GMT https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492413#M137355 woodcock 2019-11-22T16:07:39Z Re: How to make search efficient with spath and regex https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492414#M137356 <P>Is it possible for stats.memory.free and stats.memory.total to be extracted as fields as the log is ingested? This would save you search-time overhead and the cost of some disk space and ingest overhead.</P> Fri, 22 Nov 2019 16:50:07 GMT https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492414#M137356 jpolvino 2019-11-22T16:50:07Z Re: How to make search efficient with spath and regex https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492415#M137357 <P>Thanks Jpolvino. Unfortunately splunk is administered by another group and I'll have to raise a request for it.</P> Fri, 22 Nov 2019 16:56:32 GMT https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492415#M137357 vjzone 2019-11-22T16:56:32Z Re: How to make search efficient with spath and regex https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492416#M137358 <P>Yes every application log is with index=main. I have asked for a change and it'll hopefully happen soon.</P> <P>Appreciate your response. </P> Fri, 22 Nov 2019 16:58:58 GMT https://community.splunk.com/t5/Splunk-Search/How-to-make-search-efficient-with-spath-and-regex/m-p/492416#M137358 vjzone 2019-11-22T16:58:58Z