https://community.splunk.com/t5/Splunk-Search/When-using-an-API-to-enrich-my-data-how-can-I-control-how-many/m-p/191406#M55108 <P>From logs created by my script, Splunk is sending the hash one at a time to the script. I have caching implemented with the script so it doesn't use the API for something its received an answer for in the last few hours.</P> <P>What I'm after is configuring Splunk to send more than one hash at a time to the script as it goes down the list.</P> Sun, 05 Jul 2015 18:41:46 GMT rharrisssi 2015-07-05T18:41:46Z https://community.splunk.com/t5/Splunk-Search/When-using-an-API-to-enrich-my-data-how-can-I-control-how-many/m-p/191404#M55106 <P>When using an API to enrich my data, for example running MD5 hashes in my logs against VirusTotal's API, how can I control how many requests Splunk sends to my external lookup script? There are often maximum limits on APIs provided, and its most efficient to take advantage and send in bulk since it will be going off-box.</P> Sun, 05 Jul 2015 04:54:44 GMT https://community.splunk.com/t5/Splunk-Search/When-using-an-API-to-enrich-my-data-how-can-I-control-how-many/m-p/191404#M55106 rharrisssi 2015-07-05T04:54:44Z https://community.splunk.com/t5/Splunk-Search/When-using-an-API-to-enrich-my-data-how-can-I-control-how-many/m-p/191405#M55107 <P>Splunk will send all the requests to your external lookup script because Splunk can't know what the script is doing.</P> <P>However, the script can be smart about what it sends off and what it answers itself from a cache.</P> Sun, 05 Jul 2015 11:18:28 GMT https://community.splunk.com/t5/Splunk-Search/When-using-an-API-to-enrich-my-data-how-can-I-control-how-many/m-p/191405#M55107 martin_mueller 2015-07-05T11:18:28Z https://community.splunk.com/t5/Splunk-Search/When-using-an-API-to-enrich-my-data-how-can-I-control-how-many/m-p/191406#M55108 <P>From logs created by my script, Splunk is sending the hash one at a time to the script. I have caching implemented with the script so it doesn't use the API for something its received an answer for in the last few hours.</P> <P>What I'm after is configuring Splunk to send more than one hash at a time to the script as it goes down the list.</P> Sun, 05 Jul 2015 18:41:46 GMT https://community.splunk.com/t5/Splunk-Search/When-using-an-API-to-enrich-my-data-how-can-I-control-how-many/m-p/191406#M55108 rharrisssi 2015-07-05T18:41:46Z https://community.splunk.com/t5/Splunk-Search/When-using-an-API-to-enrich-my-data-how-can-I-control-how-many/m-p/191407#M55109 <P>I see. While I'm not so sure about lookups, custom search commands are by default configured to receive up to 50000 rows per invocation: <A href="http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/Commandsconf">http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/Commandsconf</A></P> <P>If there's no way to tell this to a lookup then you could at least convert your lookup script.</P> Sun, 05 Jul 2015 21:56:46 GMT https://community.splunk.com/t5/Splunk-Search/When-using-an-API-to-enrich-my-data-how-can-I-control-how-many/m-p/191407#M55109 martin_mueller 2015-07-05T21:56:46Z