Probably those hosts are not sending data any more. Metadata doesn't depend on time range but index=_internal does.
So those hosts might have sent data earlier but stopped now. Can you increase your timerange to last 30 days or even "all time" and look for one of the hosts from the above list? or just run the below search and see when these hosts were last updated.
| metadata type=hosts | rename firstTime as "First Event" lastTime as "Last Event" recentTime as "Last Update" | fieldformat "First Event"=strftime('First Event', "%c") | fieldformat "Last Event"=strftime('Last Event', "%c") | fieldformat "Last Update"=strftime('Last Update', "%c")|table host "First Event" "Last Event" "Last Update"
Splunk should send it's internal data if the connection is established. It's quite interesting if it's forwarding other data but not internal
... View more
Normally forwarder switches at every regular interval, say 30 mins or if the indexer goes down
The selection of receiver from the group is random
If you want to select only two out of three, you can configure that.
Detailed doc available here : http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Setuploadbalancingd
On top of this, If you have a specific issue, then somebody might be able to help you
... View more
Please make sure that your new license is enough to process the amount of data you have
Add license to the correct license master
Allocate the new license to the pool : http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Createalicensepool
Check your license quota has reflected new license
If this doesn't work, contact Splunk support and they can help get you back up and running
... View more
Try this. This should give you the difference for a week per day
...your search earliest=-7d@d
| bucket _time span=1d
| stats earliest(_time) as FirstTime,earliest(counter) as FirstCounter, latest(_time) as LastTime,latest(counter) as LastCCounter
by _time
|eval Difference=LastCCounter-FirstCounter
|table FirstTime FirstCounter LastTime LastCounter Difference
Please test it with your events
Also refer to http://docs.splunk.com/Documentation/Splunk/6.2.0/Search/Specifytimemodifiersinyoursearch
... View more
Normally it takes the scheduled time but Splunk considers different methods to run scheduled reports. http://docs.splunk.com/Documentation/Splunk/6.3.2/Report/Configurethepriorityofscheduledreports
... View more
Add a loop to your Java script to list all table elements and apply renderer to each element.
Something like below. Please test it and feel free to amend for your requirements
function addStatusIcon(id) {
mvc.Components.get(id).getVisualization(function(tableView){
tableView.table.addCellRenderer(new CustomRangeRenderer());
tableView.table.render();
});
}
//Get all table elements in the dashboard
var s = document.getElementsByClassName("dashboard-element table splunk-view");
// Loop thru the list
for (i=0;i<s.length;i++) {
id=s[i].getAttribute("id");
addStatusIcon(id);
}
... View more
Your first search gives you run time and your second search gives you the delay in execution; ie; if you have scheduled a search at 9:00 and executed at 9:05, then the delay is 5 minutes.
... View more
Try summary indexing. Run your search everyday and store only the results you want in summary index. Use the summary index in dashboard. This will speed up your dashboard especially if you have to search for one month data
http://docs.splunk.com/Documentation/Splunk/6.3.1511/Knowledge/Usesummaryindexing
... View more
are these single line or multi-line?
If its multi-line, is there a common field in those events?
If its multi-line, are they coming in the mentioned order ie; hostname 1, status 1 , hostname2,status etc?
... View more
For any manual change in configs, splunk needs a restart
Ref : http://docs.splunk.com/Documentation/Splunk/6.2.0/Troubleshooting/Enabledebuglogging
... View more
Can you share your search which is used to get data from the KV store ? For eg: if my drop down token is id_token, then the search will be
|inputlookup my_lookup where ID=$id_token$
More read : http://dev.splunk.com/view/SP-CAAAEZH
... View more
Can you just try adding a table/fields with status and ip before the chart command and run the search without chart to make sure that status and ip are listed and then add the chart command. Something like below.
sourcetype=weblogs status=200 OR status=410 [search sourcetype=weblogs status=200 earliest=-3d@d latest=now request="GET /path/*" [search sourcetype=weblogs request="GET /path/*" status=410 earliest=-3d@d latest=now [search sourcetype=firewalllogs "/blocked/" earliest=-3d@d latest=now | dedup ip | table ip] | dedup ip | table ip] | dedup ip | table ip] | fields status,ip| chart count over ip by status
... View more
There are multiple ways to achieve this.
Easiest and best method is to merge events : http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Indexmulti-lineevents
If you have common fields, join the events with one of the commands (join,transaction,stats etc) . https://answers.splunk.com/answers/335149/how-do-get-the-combined-search-result.html
... View more
Instead of hidden search, you can use search base=main search and then use base search in sub panel
See here
https://answers.splunk.com/answers/239159/multiple-base-searches-in-a-dasboard-with-post-pro.html
... View more
If you were able to see the fields before and not now, then most probably it's due to different search mode unless your raw events are changed.
See here for information : http://docs.splunk.com/Documentation/Splunk/6.2.0/Search/Changethesearchmode
... View more
Ok if you are using html, then just get the current time from java script itself instead of creating a search manager just for this.
var currentDate=new Date();
You can either use currentDate directly which has full timestamp including timezone or get date/time elements from the date object currentDate according to your requirements.
Check javascript datetime documents for further reference
... View more
Try this
|metadata type=sources|eval difference=now() - firstTime|fields source difference|where difference < 300
This will list the sources which have appeared in last 5 minutes. Schedule this for reasonable time window according to your requirement and adjust the difference accordingly. Try the above search without the where clause and test it
... View more
Did you get a chance to look at the documentation mentioned above ?: http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/MonitorWindowsdata
also
http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Setupforwardingandreceiving
and
http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Configureyourinputs
Go thru these docs and refer to the sub links if necessary. It has all details about configuring your UF
... View more
We have used more than 100 especially when splunk converts sub searches to OR conditions and even in format. So most probably there are no limits we are aware of.
If you are facing an issue in searches it might be because of other limits in http://docs.splunk.com/Documentation/Splunk/6.2.0/admin/Limitsconf
There should be better ways to write search without using a lot of OR conditions.
... View more
You can do it from your search itself using outputlookup
for eg :
|stats count|eval timestamp=now()|fields timestamp|outputlookup kvstorelookup name
Its already mentioned in the docs provided above. also refer to
http://docs.splunk.com/Documentation/Splunk/6.3.1511/SearchReference/Outputlookup
In general, you can insert a record programmatically using rest end points as well.
http://dev.splunk.com/view/webframework-tutorials/SP-CAAAEZV
... View more
Splunk monitors files added to the directory automatically if you have configured monitor for a directory
Ref : http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Monitorfilesanddirectories
Whenever a new file is added, splunk creates a new source for that file unless you override the source parameter in the inputs.conf.
In that case, you can create an alert when a new source is created by comparing the count with previous one or looking for a specific event from the new file etc.
... View more