Hi everyone! We've moved some of heavy lookups to kv store and now they work faster and more stable. But one of them barely can calculate something, being inserted in a search with lookup even on 24 hour span, while it easily loads with inputlookup . Accelerated field is a hexadecimal uid of 16 symbols. To make it easier for splunk to accelerate it, we converted it in a bigint and accelerated on this new field. Still the same slowness. It was rewritten but it didn't help. And its csv version lookups much faster, but we had some issues with updating large lookups so we moved to kv. Slow-lookuped kv has 10mln entries and around 5 fields. Fast-lookuped kv with 40mln entries and 10 fields works fine (with accelerated field also converted from hex to bigint) Server: 251.81 GB Physical Memory, 20 CPU Cores in limits.conf: max_threads_per_outputlookup = 10 in server.conf: oplogSize = 2000 What can cause such slowness? What params can we tune? ... View more

Hi splunkers! Need your advice again. We have a db table with users registrations. Millions of entries. There's an option for a user to merge with another account if for some reason he created another one. Merge happens pretty often. But there's no earliest date of registration for merged users in original table. So we address to another table where this info is stored. Make lookup of it and then make kv_store lookup of original table taking in consideration merged accounts lookup. This kvstore lookup is used to correctly segment users' purchases, logins by their registration date. Rewriting of this kv_store lookup takes hours and sometimes it skips.. Now we want to create outside splunk separate db table with correct registration information and to use it queries. And here's a question: which way is better to use in this case? Dbxlookup without need to update something in splunk kv-store lookup with its complete rewriting several times a day kv-store lookup updating it values through REST API ? ... View more

Hi! We have some searches on a dashboard that work way too long as they include several subsearches and calculate data for the latest 30 days, that lead daily scheduled pdf of that dashboard not to send pdf by email with pdf-sending error. We splitted heavy searches into several ones but from day to day it may take different time to complete each of them. Is there a way to set up saved searches to run in sequence not only by setting a sparse schedule for them? ... View more

Hi! What setting in advanced edit list of a scheduled saved search stands for cleaning search artifacts after its fulfilling? I have a scheduled saved search that runs about 30 minutes and generates a file of lets say 1Gb, after this I cannot search anything with an error message that I've exhausted space limit for my user. And I need to clean my jobs then. Thanks! ... View more

Hi everyone How do I leave just unique events by specified field in an accelerated data model? My base search looks like: index=main source=transactions tx_type=purchase | `registration_time` | `type_user` And I'm trying to add child dataset with just one constraint: dedup transaction_id But it didn't work. Duplicated transaction_id cannot be deleted from original source 'cause it's a feature of some transactions... But, in data model, we just need one occurrence of it, e.g. to sum revenue. With this query we get a greater result than we actually have because of counting several times transactions with the same transaction_id. How can we get correct result? Thanks in advance! | tstats sum(transactions.price) AS Revenue from datamodel=transact.transactions where (nodename = transactions) groupby _time span=1month | rename transactions.* as * | timechart span=1month first(Revenue) as revenue ... View more

Hi everyone! We've been randomly facing with rather annoying and critical issue while working with lookups: sometimes only several entries get lookup fields when there should be many more of them. Rewriting lookup file helps in most of cases. But it is not stable performance and means you cannot trust results especially when lookup is used in scheduled searches. Can't say for sure what conditions cause such behaviour: often it happens with large csv files (over 1mln lines) that are rewritten on a daily basis and it happens in all Splunk versions. This time I managed to save search logs to the same query when lookup worked incorrectly and when it worked ok (after its rewriting). Comparison of these two files showed that they are mainly identical except several lines. In "good" log file there is an entry, that misses in "bad" log: INFO CMBucketId CMIndexId: New indexName=main inserted, mapping to id=1 Also in good log: INFO DispatchThread SrchOptMetrics optimize_toJson=1 While in bad: INFO DispatchThread SrchOptMetrics optimize_toJson=2 Excel with comparison is attached Hope for your help, guys! ... View more

We've created a dashboard with tabs using steps from this post https://www.splunk.com/blog/2015/03/30/making-a-dashboard-with-tabs-and-searches-that-run-when-clicked.html Now we need several areas with tabs in one dashboard. E.g. first several tabs show data on sourcetypes, next block of tabs shows data on sources etc. And they do not depend on each other. What we get - just one active panel on the dashboard. We see chart either on sourcetypes or on sources when we need to see both with ability to select tabs. Why tabs counstruction is preferable and not linkswitcher - because we need to reduce number of searches that launch when opening dashboard. At the moment I know that tabs can do such trick but if splunk can do this with links as well I'll be glad to know how. Thanks in advance! .... <row id="tabs"> <panel> <html> <ul id="tabs" class="nav nav-tabs"> <li class="active"> <a href="#" class="toggle-tab" data-toggle="tab" data-elements="link1" data-token="link1">...</a> </li> <li> <a href="#" class="toggle-tab" data-toggle="tab" data-elements="link2" data-token="link2">...</a> </li> <li> <a href="#" class="toggle-tab" data-toggle="tab" data-elements="link3" data-token="link3">...</a> </li> <li> <a href="#" class="toggle-tab" data-toggle="tab" data-elements="link4" data-token="link4">...</a> </li> </ul> </html> </panel> </row> <row id="link1"> <panel> <chart> <title>All sourcetypes</title> <search id="sourcetypes_all"> <query>... $link1$... </query> </search> <option name="charting.chart">line</option> <option name="charting.chart.showDataLabels">all</option> <option name="charting.drilldown">all</option> </chart> </panel> </row> .... <row id="tabs2"> <panel> <html> <ul id="tabs2" class="nav nav-tabs"> <li class="active"> <a href="#" class="toggle-tab" data-toggle="tab" data-elements="link5" data-token="link5">...</a> </li> <li> <a href="#" class="toggle-tab" data-toggle="tab" data-elements="link6" data-token="link6">...</a> </li> <li> <a href="#" class="toggle-tab" data-toggle="tab" data-elements="link7" data-token="link7">...</a> </li> </ul> </html> </panel> </row> <row id="link5"> <panel> <chart> <title>All CR</title> <search> <query>... $link5$...</query> </search> <option name="charting.chart">line</option> <option name="charting.chart.showDataLabels">all</option> <option name="charting.drilldown">all</option> </chart> </panel> </row> ... ... View more