All Apps and Add-ons

Cisco Networks App for Splunk Enterprise: Why am I getting error "invalid vector T subscript" on a lot of pivots?

gallaway
Explorer

I receive the following error on a lot of pivots starting with the top most "Cisco IOS Event"

09-30-2015 16:42:24.542 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: invalid vector<T> subscript
1 Solution

mikaelbje
Motivator

Just want to let you know that I have also confirmed that searches for raw events, i.e. sourcetype=cisco:ios do not work. This is because of the Vendor Message lookup which is a large CSV file. This worked in 6.2.4. It was reported as a bug in Splunk Beta, but was apparently not fixed before official release.

Splunk bug, not an app bug.

View solution in original post

0 Karma

sgarvin55
Splunk Employee
Splunk Employee

This is a Known Issue SPL-107253 and is fixed in Splunk version 6.3.1

0 Karma

jcrabb_splunk
Splunk Employee
Splunk Employee

There is a bug filed for this issue (SPL-107253). A recommended work around is to add the following to your limits.conf configuration on both your Search Heads and Indexers.


[lookup]
max_memtable_bytes=15000000


Once that setting is in place, restart the Splunk process. Customers who have applied this work around have reported back that it resolved the issue. I hope you find it does for you as well.

More information about that setting can be found under the limits.conf.spec document.

Jacob
Sr. Technical Support Engineer

brooklynotss
Path Finder

workaround worked for me as well, although only had to change it on the index layer, not the search head(s). the lookup table in question, cisco_ios_messages.csv, for me is almost 12mb.

0 Karma

gallaway
Explorer

Confirmed to work for me. Thanks

0 Karma

leenguyen07
Explorer

I have edited limits.conf file in C:\Program Files\Splunk\etc\system\local
[lookup]
max_memtable_bytes=15000000
But it didn't work

0 Karma

jcrabb_splunk
Splunk Employee
Splunk Employee

You will need to configure the max_memtable_bytes to at least the size of the lookup table it is attempting to load into memory. If it is greater than 15000000 bytes, the error will persists. You will want to determine the size of the lookup table and adjust your setting appropriately. One method would be to enable debugging on search, execute the search that is affected then review the search.log for that search.

On the Search Head:

  1. Browse to $SPLUNK_HOME/etc and edit log-searchprocess.cfg and change "rootCategory=INFO,searchprocessAppender" to "rootCategory=DEBUG,searchprocessAppender" and save the change.
    Ex :

    search logs go a separate file

    rootCategory=DEBUG,searchprocessAppender

  2. Restart your splunk instance and run the search to reproduce the issue.

  3. Click on Job -> Inspect Job then Search.log. Look for an entry along these lines:


10-26-2015 09:59:33.166 DEBUG LookupOperator - Found static lookup file: /opt/splunk/etc/apps/splunk_app_whatever/lookups/sys_lookup.csv
10-26-2015 09:59:33.166 DEBUG LookupOperator - Loading lookup table 'sys_lookup', file size = 219629642, modtime = 1445287388


In that example, the lookup file is almost 210 Mb. You would then need to configure your setting as follows:

lookup]
max_memtable_bytes=230000000

I would recommend using caution when configuring this setting above the default. Adjusting that setting can cause high memory pressure and if there are any adverse affects you will want to remove it. This should be addressed in a maintenance release in the very near future.

Jacob
Sr. Technical Support Engineer
0 Karma

leenguyen07
Explorer

Thank you! Your post is very helful. In Cisco Networks App has new problem, when i click Inventory - > Devices then it don't show Software versions, Models, Mnemonics by model... Althought, Cisco network overview, routing.... are works.

mikaelbje
Motivator

Check the Help page in the app. There's an explanation in there 🙂

0 Karma

leenguyen07
Explorer

i read the help page in the app. But Perfomance and Devices field show No results found

0 Karma

leenguyen07
Explorer

while other field works very good.

0 Karma

ictsecman
New Member

getting the same error on splunk enterprise 6.3 cleaning the index resolves the issue

0 Karma

mikaelbje
Motivator

Just want to let you know that I have also confirmed that searches for raw events, i.e. sourcetype=cisco:ios do not work. This is because of the Vendor Message lookup which is a large CSV file. This worked in 6.2.4. It was reported as a bug in Splunk Beta, but was apparently not fixed before official release.

Splunk bug, not an app bug.

0 Karma

gallaway
Explorer

So the solution for this is to wait for splunk to fix the bug?

0 Karma

mikaelbje
Motivator

Yes, unfortunately, or downgrade your servers. I have not filed a bug report. If you can do that it would be great. Reference this thread and the fact that I pointed this out in the beta phase.

0 Karma

mikaelbje
Motivator

Can you provide me with some more info, please?

  1. What Splunk Enterprise version?
  2. What Cisco Networks App version?
  3. What Cisco Networks Add-on version?

This may be a Splunk bug if you recently upgraded to Splunk Enterprise 6.3

0 Karma

gallaway
Explorer

Hi

Splunk Version 6.3.0
Cisco Networks Add-on TA-cisco_ios 2.3.0
Cisco Networks cisco_ios 2.3.0

And yes I upgraded to Splunk Enterprise 6.3

0 Karma

mikaelbje
Motivator

Could you try deleting the app and add-on, then reinstalling?

What OS are you on, by the way?

0 Karma

gallaway
Explorer

windows server 2012r2

0 Karma

mikaelbje
Motivator

Hmm, this thread seems related: http://answers.splunk.com/answers/312282/why-is-my-search-with-the-strcat-command-failing-t.html

Could you open a case with Splunk, please?

0 Karma

gallaway
Explorer

Case 275638 opened.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...