Update: I forgot to also delete the details for the dashboard in the metadata file on the Deployer. Once that was also removed I was able to delete the local copy. ... View more

I just tried this but it did not work for me: 1. Deleted dashbaord (xml file) on deployer 2. Applied the shcluster-bundle 3. Went to delete the dashboard on a Search Head but the delete option is still not available. 4. Verified on each SHCluster member in the app's default directory that the dashboard (xml file) is no longer present. It is now only in the local directory. 5. I restarted Splunk on one of the Search heads and still can not delete the (now local only) dashboard there. ... View more

I added the transforms to the Universal Forwarder to send the unwanted stuff to the nullQueue and it is now working as I need it to. I didn't think that would work (even on structured data) but it seems that it does. thank you! ... View more

Thankx. This is certainly a confusing topic. Before last month I would have said that this type of thing had to be done on the indexer, and we have done it that way many times in the past. I just happened to come across the Forwarder configurations. I'm not sure when this functionality got added to the forwarder, but it does help offload some of the work from me / the Splunk infrastructure team and allows server/service admins the capability of adding these types of inputs on their own. Thank you @woodcock and @richgalloway for taking the time to answer and help me with this. ... View more

I think that may have been the case in previous Forwarder versions, but you can now configure this on the Forwarder. "If you have Splunk Enterprise, you can edit the settings on indexer machines or machines where you are running the Splunk universal forwarder." http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileswithstructureddata I already the CSV files being forwarded to Splunk, with props.conf configured on the Forwarder - I tested and verified that last month. It was just today when I went to add the TSV files that I asked myself why I was using both INDEXED_EXTRACTIONS and FIELD_DELIMITER for the CSV files and whether or not I actually needed to. ... View more

I am in the process of working on a standard way to create new HEC tokens, and have them automatically configured on all Heavy Forwarders (I use a Deployment Server and, like you, my own custom app for Heavy Forwarder configs.) So if I understand you correctly, you generate new tokens (disabled) on your deployment server using the web UI, and then you are copying the new stanza from inputs.conf in the splunk_httpinput app to your custom app and then enabling them there? That is what I was thinking of doing, and was looking around to see if anyone else was doing this or had any other options when I came across this. My only other option so far is to keep using the splunk_httpinput app, have it configured and deployed via the Deployment Server, but in this case the tokens would then also be enabled on the Deployment Server - which probably doesn't matter but Id rather not have it set up this way. I already have a Deployment Server in place, so can not set it up on one of the Heavy Forwarders as Splunk documentation recommends. ... View more

That worked for me too! Thank you!! I wish I had seen this 6 hours ago!!! Any idea why this is required only for settings in alert_actions.conf? All of my other settings in the same app work fine. ... View more

Here is the Dashboard XML.... I removed all but one panel to make it easier to sort through: <form> <label>Dashboard Clone</label> <description>Dashboard for TESTING - validating token value</description> <fieldset submitButton="false" autoRun="true"> <input type="time" token="timePicker" searchWhenChanged="true"> <label>Time</label> <default> <earliest>-15m</earliest> <latest>now</latest> </default> </input> <input type="dropdown" token="selectedSite" searchWhenChanged="true"> <label>Sites</label> <selectFirstChoice>true</selectFirstChoice> <search> <query>| rest /services/authentication/current-context/context splunk_server=local | fields + username | lookup customers.csv ID as username OUTPUT Site as SitesAllowed | mvexpand SitesAllowed | fields + SitesAllowed</query> <earliest>-15m</earliest> <latest>now</latest> </search> <fieldForLabel>SitesAllowed</fieldForLabel> <fieldForValue>SitesAllowed</fieldForValue> </input> </fieldset> <search id="BaseSearch" ref="utweb_basesearch"> <earliest>$timePicker.earliest$</earliest> <latest>$timePicker.latest$</latest> </search> <row> <panel> <chart> <title>Count By Status</title> <search base="BaseSearch"> <query>| search req_host=*$selectedSite$* AND [| rest /services/authentication/current-context/context splunk_server=local | fields + username | lookup customers.csv ID as username OUTPUT Site as SitesAllowed | mvexpand SitesAllowed | fields + SitesAllowed | search SitesAllowed=*$selectedSite$* ] | stats count by status</query> </search> <option name="count">10</option> <option name="list.drilldown">full</option> <option name="list.wrap">1</option> <option name="maxLines">5</option> <option name="raw.drilldown">full</option> <option name="rowNumbers">0</option> <option name="table.drilldown">all</option> <option name="table.wrap">1</option> <option name="type">list</option> <fields>[]</fields> <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option> <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option> <option name="charting.axisTitleX.visibility">visible</option> <option name="charting.axisTitleY.visibility">visible</option> <option name="charting.axisTitleY2.visibility">visible</option> <option name="charting.axisX.scale">linear</option> <option name="charting.axisY.scale">linear</option> <option name="charting.axisY2.enabled">0</option> <option name="charting.axisY2.scale">inherit</option> <option name="charting.chart">pie</option> <option name="charting.chart.bubbleMaximumSize">50</option> <option name="charting.chart.bubbleMinimumSize">10</option> <option name="charting.chart.bubbleSizeBy">area</option> <option name="charting.chart.nullValueMode">gaps</option> <option name="charting.chart.showDataLabels">none</option> <option name="charting.chart.sliceCollapsingThreshold">0.01</option> <option name="charting.chart.stackMode">default</option> <option name="charting.chart.style">shiny</option> <option name="charting.drilldown">all</option> <option name="charting.layout.splitSeries">0</option> <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option> <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option> <option name="charting.legend.placement">right</option> </chart> </panel> </row> </form> ... View more

That is what I had tried when you first suggested the subsearch method, but it was not returning anything. I just tried again, and get No results found. I know the subsearch works, since I can run it on its own and get the desired output (a site name) when I replace the token with an actual site value. Maybe it is an issue because I am using a basesearch on the dashboard? I made a new panel that used the base search and for the query I entered the subsearch and got no results. Once I removed the base search from this new panel, things worked for this new test panel and the Selected Site was returned. ... View more

The Dropdown search is: | rest /services/authentication/current-context/context splunk_server=local | fields + username | lookup customers.csv ID as username OUTPUT Site as SitesAllowed | mvexpand SitesAllowed | fields + SitesAllowed The lookup is just a csv that indicates which site a user should be able to see logs for: user1,site1.com user1,site2.com user2,site3.com One of the panel searches is: <search base="BaseSearch"> <query>| search requested_site=*$selectedSite$* | stats count by status</query> </search> The Base search is: index=webhosting | fields * and is required because it runs as owner - users do not have access to the index. ... View more

Thankx for the suggestion, but I don't think I can get that to work in my situation. When using the inputlookup command it has to be the first command in a search. That will not work in my situation because I have a base search in use so the inputlookup would not be the first search (I am using a basesearch that runs as the owner because the users do not have access to the index themselves.) Also, the search I am using to populate the dropdown list uses the rest command, which also has to be the first command in a search (I am doing this to get the currently logged in user name.) ... View more

We had a similar issue, where ServiceNow logs were no longer getting indexed. A restart of Splunk resolved the issue for us. I am not really sure what triggered it to stop getting logs though. I noticed a new version of the TA is available (we are running v 2.7.0, and 2.9.0 is available.) Not sure if that will resolve the issue or not. ... View more

We are building a new Splunk environment (hardware refresh and migrating from search head pooling to clustering) and I am wanting to prevent clutter in the Search app (right now everyone seems to share all of their searches out and there is just a lot of reports and dashboards to sort through in the Search app.) We also want users from each department to save things in their Department's app to help organise things and also to let us know who owns something which is useful when an object's owner leaves and we are asked to change the owner or permissions on something.) Part of the problem is now anyone can save and share things in the Search and Reporting app. I investigated this and it seems that currently Everyone has write access to the Search app... if that is not (and never was) the default configuration, I am not sure why we set it up that way years ago, but that will certainly change in the future. We will be trying our best with end-user education, asking them to save to their Department's app. And the default app will be assigned by role to help by at least getting them started out in the appropriate place. With read only permissions on the search app that will help address the clutter issue as they will not be able to share anything there. ... View more

Our current Splunk infrastructure (with Search Head Pooling) has been around for several years, and we are planning to move to Search Head Clustering next spring when we stand up our new Splunk environment... but until then I still need to support our search head pooling. I ran the search you provided, but not sure that it will help me solve my problem. I need to find out why real time searches seem to not be running (due to absence of alerts from them, and not seeing them in the job queue.) I have turned debugging messages on for the scheduler, but that has not produced anything helpful. I also wanted to see if I could find a list of all real time searches configured so I could try to persuade the owners to change them to saved searches. The closest I have been able to come is the search index=_internal "SavedSplunker" "realtime=1" - where I assumed "realtime=1" would indicate the search is a real time search, but when looking at the searches returned settings, I see I was wrong because they are not all real time searches (start time/finish time is not rt, and they are configured to run on a cron schedule) Prior to this issue, I have seen the need for additional Search Heads, so am in the process of standing up two more SHs, I am just not sure this will address the real time search/alert issue since there only seems to be 10 RT searches running currently with the two SHs when my calculations show that each SH should be able to handle 10 RT searches. A search of the _internal logs does not show any reference to search limit reached though. ... View more

Yes, that looks like that is what has happened... Is there any good explanation for this... why a bucket created a year and a half ago would still be written to? I assumed a bucket was written to until it filled up and then a new bucket was created. I could maybe understand data a couple of weeks/months beyond retention depending on how much data comes in for the index.... but a bucket still being written to 20 months later?!?? startEpoch endEpoch Thu Aug 8 09:56:40 CDT 2013 Fri Feb 27 10:35:57 CST 2015 Thu Nov 6 17:36:01 CST 2014 Fri Feb 13 00:10:41 CST 2015 Wed Dec 24 09:47:11 CST 2014 Tue Jan 20 15:40:03 CST 2015 ... View more