topic Re: "Collect" command doesnt work with "All time (Real-Time)" in Knowledge Management

"Collect" command doesnt work with "All time (Real-Time)"

DavidGirsvaldas — Wed, 22 Aug 2018 10:13:17 GMT

Hi,
I have a use case where I need to check for incomming events with measurements, combine and modify them and save as a new event. This is different from a simple summary as I might need to apply math to its values. What Im trying to use now is "Collect" command:

index=main (host="host1" AND MeasurementChannel=1) OR (host="host2" AND MeasurementChannel=2) | stats latest(MeanValue) as sumMean latest(Timestamp) as latestTImestamps latest(_time) as _time by MeasurementChannel| stats sum(sumMean) as MeanValue latest(latestTImestamps) as Timestamp latest(_time) as _time | eval Alias="myCustomChannel" | collect index=main host=host1

This search works fine when executed as non-real time, but when I set time interval to "All Time(Real-time)" nothings gets collected. in documentation for Collect command (http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Collect) it says

The collect command also works with real-time searches that have a time range of All time.

Re: "Collect" command doesnt work with "All time (Real-Time)"

inventsekar — Wed, 22 Aug 2018 11:09:20 GMT

collect command Description....
Adds the results of a search to a summary index that you specify. You must create the summary index before you invoke the collect command.

index=main ....................| collect index=main host=host1
here you are collecting the events from main index and sending it again to main index.

probably you should re-write your query to... (beforehand, you have to create this mainCollectSummary index )
index=main ....................| collect index=mainCollectSummary

Re: "Collect" command doesnt work with "All time (Real-Time)"

DavidGirsvaldas — Wed, 22 Aug 2018 11:20:12 GMT

thank you for reply.
I also tried collecting events to "summary" index and it behaved in a same way. It worked with non-real time queries. Also my use case actually requires events to be saved in a same index.
"You must create the summary index before you invoke the collect command."- the way I read it, is that Splunk says search will not create a new index automatically by running search and so index should be created prior.
"Adds the results of a search to a summary index that you specify"- as far as I know Real-Time searches never finishes so therefore they do not produce results. So I wouldnt expect it to work, but the Collect command documentation clearly states that command works with All-time(real time) which confuses me

Re: "Collect" command doesnt work with "All time (Real-Time)"

inventsekar — Wed, 22 Aug 2018 11:52:37 GMT

maybe, update the authorize.conf file that gives a way to grant/remove this collect command from a user...

[capability::run_collect]
* Lets a user run the collect command.

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Authorizeconf

Re: "Collect" command doesnt work with "All time (Real-Time)"

DavidGirsvaldas — Wed, 22 Aug 2018 13:04:44 GMT

The documentation page points to Splunk version 7.1.2 meanwhile Im using a bit older 7.0.2. This capability doesnt appear as option in my version and gets ignored if set in config files. However I doubt this is an issue since Im able to successfully use Collect command as long as it is not real time.

Re: "Collect" command doesnt work with "All time (Real-Time)"

inventsekar — Wed, 22 Aug 2018 13:18:27 GMT

ok, are you able to run other real time searches?!?!

[capability::rtsearch]
* Lets a user run realtime searches.

Re: "Collect" command doesnt work with "All time (Real-Time)"

DavidGirsvaldas — Wed, 22 Aug 2018 13:26:03 GMT

yes, they all work as expected. Im currently running it all using Admin role. rtsearch is enabled.