About Lucas_K

Lucas_K · ‎01-22-2018

Is there anyway to get the mongodb COMPACT command run by splunk? We have quite a few kvstores that have used around 100GB of local disk space per search head. In reading into how the kvstore works at some stage each of these kvstores have preallocated this large amount of space and then deleted the data. Upon checking the dmc -> kvstore -> instance and looking at collection sizes many of these are significantly smaller than the utilized disk on each host. As a test I even updated a collection so it had only a single key and the total collection size was 20gb on disk. In reading on how to remedy this outside of splunk I need to do a compact and then a resync of the kvstores. I'd already tried the resync and this doesn't recover the disk space so I need to know if there is a way to do the compact first and then another resync.

Lucas_K · ‎11-30-2017

I've actually found the root cause with is a module that is included with this app. As the source is no longer available there is noupdated method to utilise to fix this. In a nutshell there is a bug in the underlying dependencies that causes super slow performance in any ldap environment larger than a few users. I tried it against an internal group ldap server which only had a few groups and members and it was fine.

Lucas_K · ‎11-08-2017

Thanks. Looks like the only way is an outputcsv file. collect puts to many extra fields and also stores the file into the spool dir which will be picked up by the default stash inputs stanzas.

Lucas_K · ‎11-08-2017

We'd tried accelerated data models for this specific use case but it just killed our indexers (multi million user radius accounting stats) I'll probably revisit again as a couple of big splunk versions have come out so perhaps performance has improved since then. I do agree accelerate data models makes keeping those statistics up to date a much simpler process though!

Lucas_K · ‎11-08-2017

In the metrics getting started documentation ( http://docs.splunk.com/Documentation/Splunk/7.0.0/Metrics/GetStarted ) it says "Summary indexing does not work with metrics." When I read the rest of the documentation I don't see any specific reason I couldn't craft my own data to fit the metrics format. If I massage an event into having all the correct fields ( http://docs.splunk.com/Documentation/Splunk/7.0.0/Metrics/GetStarted#Metrics_data_format ) could I save that event to a metrics store? I am looking to leverage the speed increase in the metric store with data I already process and save into summaries. My only concern might be that the sourcetype would be stash so I may need custom input stash parsing to make it "fit".

Lucas_K · ‎10-31-2017

I think you're out of luck. You can't display exact "_indextime" as the output will always filter. You can fake it and put in a space though. index=_internal | rename _indextime AS " _indextime"| table host " _indextime" Notice the space in the quotes.

Lucas_K · ‎10-31-2017

Standalone splunk search head with distributed search utilising local = true on all commands.conf stanza's. This means it is only running on the search head.

Lucas_K · ‎10-30-2017

Has anyone figured out the root cause of the ridiculously poor performance of the SA-ldapsearch apps ldapsearch command? I've hunted through the code, i've wiresharked the communications and the delays are all internally within the app itself. I've tried active directory servers (millions of entries, redirections etc), openldap (~10 groups, 40 users) and both respond in the same time frame with the same splunk-side delays. I've tried ssl and non-ssl connections and the slow queries remain. There is a minimum 10 second delay before the query is sent to the ldap server. You can see it in the picture below.

Lucas_K · ‎10-22-2017

😞 yeah didn't really expect it to be added officially.

Lucas_K · ‎10-17-2017

6 years on and there is no update to having this as a standard feature or supported config option. Latest versions also break the previous work arounds above. Anyone have this working correctly in 6.6.3+?

Lucas_K · ‎10-17-2017

Deletion of the en_US dir breaks the splunk ui. There must be some other dependency that the UI requires from the en_US directory. "Error connecting: Connection refused"

Lucas_K · ‎10-01-2017

We had to uninstall the app long prior to that answer.

Lucas_K · ‎07-04-2017

so nothing? I'll have to uninstall as this app as it still throws thousands of errors making troubleshooting startup impossible.

Lucas_K · ‎06-21-2017

This app seems to add lots of invalid stanzas to all user configurations. ie. Invalid key in stanza [blah] in /opt/splunk/etc/apps/blah/local/savedsearches.conf, line 953: display.visualizations.custom.wordcloud_app.wordcloud.useColors (value: 1). Invalid key in stanza [blah] in /opt/splunk/etc/apps/blah/local/savedsearches.conf, line 967: display.visualizations.custom.wordcloud_app.wordcloud.splunkTastic (value: 0). Invalid key in stanza [blah] in /opt/splunk/etc/apps/blah/local/savedsearches.conf, line 968: display.visualizations.custom.wordcloud_app.wordcloud.useColors (value: 1). Invalid key in stanza [blah] in /opt/splunk/etc/apps/blah/local/savedsearches.conf, line 1051: display.visualizations.custom.wordcloud_app.wordcloud.splunkTastic (value: 0). Invalid key in stanza [blah] in /opt/splunk/etc/apps/blah/local/savedsearches.conf, line 1052: display.visualizations.custom.wordcloud_app.wordcloud.useColors (value: 1). There are thousands of these invalid stanzas as it infects every user and every app.

Lucas_K · ‎04-03-2017

All search heads need to be able to access the indexers not just the captain. Make sure they can all access the index cluster master also. This allows search heads to know where primary buckets are located. The captain is responsible for distributing the search head cluster bundle but individual search heads talk to indexers. Check the search head cluster status and that it is stable.

Lucas_K · ‎03-07-2017

I ran into this same missing icon today and found that can be part of a capability that has been split out. To see those icons users roles must have "rest_properties_get" somewhere.

Lucas_K · ‎02-23-2017

v6.5.2. the saga continues. Seriously splunk devs I've posted the code!

Lucas_K · ‎02-14-2017

All those will need their own ports as they will complain about it during start up. Splunkweb, splunkd, app port,kvstore ports should all be unique. Also create individual host names in server.conf and inputs.conf. This servers two purposes. You can tell what is going on when you check internals. Distsearch requires unique names to work (it's used for the public key storing directory name). Also perhaps throw in a heavy forwarder so you can try out advanced routing techniques.

Lucas_K · ‎02-13-2017

Solved. No splunk settings were changed from the above to make this work. This warning can be removed by using ellipitcal curve cert keys. Old blog post with examples here : http://blogs.splunk.com/2014/06/03/generate-elliptical-curve-certkeys-for-splunk/ The down side is that your CA needs to be signing EC certs. If they aren't then you can't use ec keys.

Lucas_K · ‎02-05-2017

Has anyone been able to get Splunk Web settings that do not trigger these browser reports? web.conf [settings] enableSplunkWebSSL = 1 cipherSuite = TLSv1.2:!eNULL:!aNULL:!SHA1 supportSSLV3Only = False sslVersions = tls1.2 I still get this Obsolete Connection Settings The connection to this site uses a strong protocol (TLS 1.2), an obsolete key exchange (RSA), and a strong cipher (AES_128_GCM).

Lucas_K · ‎01-09-2017

Here ya go example code at the bottom. Explanation. Create an eval via a progress tag. Then insert that new eval somewhere on your dashboard. Any of the job tokens can be grabbed in the same manner. <dashboard> <label>Token Example</label> <row> <panel> <search id="search_logic"> <query>index=_internal sourcetype=splunkd| top sourcetype</query> <earliest>-1m</earliest> <latest>now</latest> <progress> <eval token="scancount">$job.scanCount$</eval> <eval token="resultcount">$job.resultCount$</eval> </progress> </search> <chart> <title>Top sourcetypes for index=_internal</title> <search base="search_logic" /> <option name="charting.chart">bar</option> </chart> <html> <h3>Scancount : $scancount$</h3> <h3>resultcount : $resultcount$</h3> </html> </panel> </row> </dashboard>

Lucas_K · ‎11-29-2016

Do a splunk search 🙂 In your example you are looking for a savedsearch modification but you can do it for any object. index=_internal source=*conf.log | xmlkv | search "data.optype_desc"=WRITE_STANZA data.asset_uri{}=*savedsearches* | table _time host data.from_repo data.asset_uri{} data.task | stats values(host) AS host count by data.asset_uri{}, data.task | addcoltotals labelfield=Total count Total count should be the number of members in your search head cluster.

Lucas_K · ‎11-28-2016

I need to fully understand this pipeline as we have some high availability compliance issues around where the processing occurs. If the documentation is true and the searches run on the indexers then the search head is just a UI to the mappings fields as you said. If the app is working the way dshpritz suggests isn't the TA-cefout missing the custom command? Looking at the doco again there is this bit. When you save this search, it begins to run immediately to filter and map data in real time. The running search is appended with an additional command: | cefout routing=<name_of_your_routing_group> cefout is a custom streaming command which takes a single parameter, routing. The custom streaming command allows the search to perform summary indexing directly on indexers and then routes the output to the destination specified by the routing parameter. This would suggest that the search head is NOT just a ui to mapping the fields but integral in how the entire app needs to function in an on going basis. ie. if you shut down the search head none of the searches would run and no events would be forwarded. I am also guessing that the custom streaming command and its required scripts are accessed as part of the bundle that the search head sends to the indexer as they are not included in the created TA but are required if the indexers are to properly process the search. So a better description is : 1.Search head job scheduler triggers a normal scheduled search. 2.This search is distributed to indexers as per normal. A bundle that includes the local search heads app that includes the custom streaming command is also sent to each indexer. 3.The search is executed by each indexer however by utilising this streaming command it keeps the data local to the indexer and does not return it to the requesting search as is normal splunk behaviour. 4.custom script directs the events via local stash tcp routing markings to be delivered as per the TA's predefined destinations. If the streaming script is sent via bundle wouldn't this make the TA redundant as you could send up to date output definitions via the bundle instead? It also feels like the app has a split personality. On one hand removing search head dependency, yet on the other all scheduling is still performed there. I can see that it gives you flexibility on modifying the mappings, yet the outputs are static as they need to be deployed. If outputs are static then wouldn't it have been better to wrap the entire bundle into a TA that you pushed to the indexers? There would be a zero search head dependency then.

Lucas_K · ‎11-27-2016

I am trying to understand the data path for the latest CEF app release ( https://splunkbase.splunk.com/app/1847/ ). In the new app, on creation of a new output, you need to push out a created TA to your indexers. This app contains app/indexes/inputs/outputs and props.conf files and that's it. This is where I am a little lost if I follow the documentation to the letter. http://docs.splunk.com/Documentation/CEFapp/2.0.0/DeployCEFapp/Overview Ripping the app apart it looks like this: 1.The searches are kicked off on the search head. 2. Transformed into cef. 3. Saved as a stash file into the spool dir. 4. Local search heads inputs.conf will pick these up via a batch input. How do the indexers send this data to the output destination? The app instructions do not explicitly say you need to set the outputs on the search head to the indexer (even though this is good standard practice for _internal logs). So if we assume the search head forwards the stash to the indexer there is no props/transforms that would enable this app to actually recognise this data So in short ... The old 1.0 app processing pipeline was : search head -> dist search to indexers -> local cef event conversion -> local stash parsing -> local output group -> destination cef tcp output via tcp routing New 2.0 app processing pipeline seems to be : search head -> dist search to indexers -> local cef event conversion -> local stash parsing .... some process here ... indexer stash ingestion -> output via tcp routing to 3rd party destination. I don't understand how individual indexers are supposed to find this data to forward to the 3rd party as it doesn't have a savedsearches.conf. Is the indexer based inputs.conf a furphy and it actually uses the index based props as the basis for forwarding instead and thus searches are actually STILL run on the search head and not the indexer as documentation states. " The indexers are responsible for performing the CEF mapping searches and forwarding the results" - http://docs.splunk.com/Documentation/CEFapp/2.0.0/DeployCEFapp/Howtheappworks Anyone know the cef 2.0 processing order?

Lucas_K · ‎11-24-2016

Which part errors for your data in the macro? What parameters were you passing through to forecast5w(val,confidence,rel@me,days) ? Comparing your source and his presentation i'd guess the macro should look like this? forcast5w(OrderCount,90,1,7) With your full search looking something like this : index=summary_trend source=orders_10min earliest=-5w | `forcast5w(OrderCount,90,1,7)`

Posts	498
Solutions	46
Karma Given	157
Karma Received	263
Member Since	‎01-18-2012

Online Status	Offline
Date Last Visited	‎08-26-2024 07:26 PM

REST API Modular Input : Polling interval is not c...

User profile web UI option "Restart background job...

Multiple time frame search with one of the time fr...

Search affinity and data replication in multisite ...

KVstore / mongodb compact command

Summarizing data and storing it in a metrics index...

Poor ldapsearch performance - SA-ldapsearch app

Wordcloud Custom Visualization : Stanza errors

SSL "obsolete key" warning in chrome browser

Splunk App For CEF 2.0: How do indexers run search...

KVstore / mongodb compact command

Re: Poor ldapsearch performance - SA-ldapsearch ap...

Re: Summarizing data and storing it in a metrics i...

Re: Summarizing data and storing it in a metrics i...

Summarizing data and storing it in a metrics index...

Re: Use of _indextime field in table or stats comm...

Re: Poor ldapsearch performance - SA-ldapsearch ap...

Poor ldapsearch performance - SA-ldapsearch app

Re: Adding Melbourne and Sydney Australia as avail...

Re: Change default web UI locale?

Re: How to change the default local for the web UI

Re: Wordcloud Custom Visualization : Stanza errors

Re: Wordcloud Custom Visualization : Stanza errors

Wordcloud Custom Visualization : Stanza errors

Re: Splunk search heads "waiting for data..."

Re: How to fix broken Menu Icons in the app naviga...

Re: Adding Melbourne and Sydney Australia as avail...

Re: Splunk Lab - Minimum servers

Re: SSL "obsolete key" warning in chrome browser

SSL "obsolete key" warning in chrome browser

Re: Which search commands allow you to display sea...

Re: How do you make sure Search Head Cluster objec...

Re: Splunk App For CEF 2.0: How do indexers run se...

Splunk App For CEF 2.0: How do indexers run search...

Re: Looking for the Forecasting Code/Macro from Mi...

Join the Conversation