I receive the following error on a lot of pivots starting with the top most "Cisco IOS Event"
09-30-2015 16:42:24.542 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: invalid vector<T> subscript
Just want to let you know that I have also confirmed that searches for raw events, i.e. sourcetype=cisco:ios do not work. This is because of the Vendor Message lookup which is a large CSV file. This worked in 6.2.4. It was reported as a bug in Splunk Beta, but was apparently not fixed before official release.
Splunk bug, not an app bug.
This is a Known Issue SPL-107253 and is fixed in Splunk version 6.3.1
There is a bug filed for this issue (SPL-107253). A recommended work around is to add the following to your limits.conf configuration on both your Search Heads and Indexers.
[lookup]
max_memtable_bytes=15000000
Once that setting is in place, restart the Splunk process. Customers who have applied this work around have reported back that it resolved the issue. I hope you find it does for you as well.
More information about that setting can be found under the limits.conf.spec document.
workaround worked for me as well, although only had to change it on the index layer, not the search head(s). the lookup table in question, cisco_ios_messages.csv, for me is almost 12mb.
Confirmed to work for me. Thanks
I have edited limits.conf file in C:\Program Files\Splunk\etc\system\local
[lookup]
max_memtable_bytes=15000000
But it didn't work
You will need to configure the max_memtable_bytes to at least the size of the lookup table it is attempting to load into memory. If it is greater than 15000000 bytes, the error will persists. You will want to determine the size of the lookup table and adjust your setting appropriately. One method would be to enable debugging on search, execute the search that is affected then review the search.log for that search.
On the Search Head:
Browse to $SPLUNK_HOME/etc and edit log-searchprocess.cfg and change "rootCategory=INFO,searchprocessAppender" to "rootCategory=DEBUG,searchprocessAppender" and save the change.
Ex :
rootCategory=DEBUG,searchprocessAppender
Restart your splunk instance and run the search to reproduce the issue.
Click on Job -> Inspect Job then Search.log. Look for an entry along these lines:
10-26-2015 09:59:33.166 DEBUG LookupOperator - Found static lookup file: /opt/splunk/etc/apps/splunk_app_whatever/lookups/sys_lookup.csv
10-26-2015 09:59:33.166 DEBUG LookupOperator - Loading lookup table 'sys_lookup', file size = 219629642, modtime = 1445287388
In that example, the lookup file is almost 210 Mb. You would then need to configure your setting as follows:
lookup]
max_memtable_bytes=230000000
I would recommend using caution when configuring this setting above the default. Adjusting that setting can cause high memory pressure and if there are any adverse affects you will want to remove it. This should be addressed in a maintenance release in the very near future.
Thank you! Your post is very helful. In Cisco Networks App has new problem, when i click Inventory - > Devices then it don't show Software versions, Models, Mnemonics by model... Althought, Cisco network overview, routing.... are works.
Check the Help page in the app. There's an explanation in there 🙂
i read the help page in the app. But Perfomance and Devices field show No results found
while other field works very good.
getting the same error on splunk enterprise 6.3 cleaning the index resolves the issue
Just want to let you know that I have also confirmed that searches for raw events, i.e. sourcetype=cisco:ios do not work. This is because of the Vendor Message lookup which is a large CSV file. This worked in 6.2.4. It was reported as a bug in Splunk Beta, but was apparently not fixed before official release.
Splunk bug, not an app bug.
So the solution for this is to wait for splunk to fix the bug?
Yes, unfortunately, or downgrade your servers. I have not filed a bug report. If you can do that it would be great. Reference this thread and the fact that I pointed this out in the beta phase.
Can you provide me with some more info, please?
This may be a Splunk bug if you recently upgraded to Splunk Enterprise 6.3
Hi
Splunk Version 6.3.0
Cisco Networks Add-on TA-cisco_ios 2.3.0
Cisco Networks cisco_ios 2.3.0
And yes I upgraded to Splunk Enterprise 6.3
Could you try deleting the app and add-on, then reinstalling?
What OS are you on, by the way?
windows server 2012r2
Hmm, this thread seems related: http://answers.splunk.com/answers/312282/why-is-my-search-with-the-strcat-command-failing-t.html
Could you open a case with Splunk, please?
Case 275638 opened.