Yes, the the authorization is done on the search head, so if there are nor roles which allow access to indexes, no search will be performed against this indexes. But you have to make certain this roles are maintained properly. In case someone is able to become admin on this search head, this person might change the roles, and in this case this person might then access anything on your indexer cluster. This makes it not feasible to give for example developers a search head with admin privilege, which is connected to the indexer cluster. ... View more

It is possible to have another search head or search head cluster accessing an indexer cluster. A search head is not able to filter out some results or block some indexes based on the search head. The indexers know which search heads are allowed to send searches, they accept any search from any attached search head. The whole authorization is performed on the search heads, so on a second search head you should possibly use the same identity provider (LDAP, SAML or whatever), otherwise you have to be very careful about your users, groups and roles. ... View more

A very lazy solution to keep a daily copy for a week and a monthly copy for a year of the whole $SPLUNK_HOME/etc needs tow simple cron jobs. 50 23 * * * umask 0007 ; tar -czf /opt/splunk/etc-`date +"\%A"`.tgz /opt/splunk/etc > /dev/null 2>&1 55 23 1 * * umask 0007 ; tar -czf /opt/splunk/etc-`date +"\%B"`.tgz /opt/splunk/etc > /dev/null 2>&1 Uses one GB of disk space and allows to recover from any mishaps in the etc directory. You can of course use a find to select just the csv files: tar -czf /opt/splunk/etc-`date +"\%A"`.tgz `find /opt/splunk/etc -name "*.csv"` No need for any (python) script at all ... View more

do you intend to run the script on the search head, will it be triggered by cron? ... View more

Depends on your intentions and environment. If you have deployed forwarders, and you want to be alerted if a host does not send any data, you could go for the events in the _internal index use a join with a query some (short) time ago to figure out which hosts came or have gone. In the internal index you have regular events from each host every second or so, so you can react quickly. If you go for certain other indexes, you are bound by the maximum interval between two events per host, if your test period is shorter than the maximum time between two events, a host will appear as having gone. ... View more

I assume you intend to create some statistics about the events from the last full hour, e.g. at 09:36, you want all events wich occurred between 08:00 and 08:59.59.999 so, earliest=-1h@h latest=@h will give you just this events. There you may just append your stats function. I don't know if you can have an eval in a count, I had to take the eval out of the count function on some occasions. If you need the count per full hour for some time, e.g the last day, then my best guess would be to use span. ... View more

index=xyz earliest=-1h@h latest=@h ... see Time Modifiers ... View more

You can use a combination: in props.conf you setup a transform based on the source type, and then have a regex on the host in transform, so the transform is only applied for some hosts. Of course you could as well do it the other way around by having a prosp.conf entry for the hosts and the regex on the sourcetype in transforms.conf props.conf [abc] TRANSFORMS-abc_conditional_routing = TRANSFORMS-abc_conditional_routing transforms.conf [TRANSFORMS-abc_conditional_routing] SOURCE_KEY = MetaData:Host REGEX = zxc DEST_KEY = _MetaData:Index FORMAT = web ... View more

Thanks for your reply. We are aware of the 1 million bucket limit, we already hit it and had to upgrade to 6.6 to overcome it ... View more

yes indeed, Oracle Linux is based on RHEL, see also this statement ... View more

We are using the current version of the universal forwarder on Oracle Linux. We installed on most system the RPM, but for Oracle Engineered Systems I would recommend to take the tarball so you have no dependencies in the local repository, otherwise an upgrade trough Oracle Platinum Services may fail. ... View more

Two more options Setup ip filters on your Splunk Indexer(s) to block the ip address of this servers. Details depend on your operating system. Configure props.conf and transforms.conf to redirect events from this systems to the nullQueue props.conf [host::YOUR_HOSTS] TRANSFORMS-DiscardHosts = DiscardHosts transforms.conf [DiscardHosts] SOURCE_KEY = _TCP_ROUTING REGEX = . DEST_KEY = queue WRITE_META = true FORMAT = nullQueue ... View more

should work, what happens? does the second search alone give some results? ... View more

We separated all our configurations into different apps: UI-Apps for the Search Heads IA-Apps for the Indexer FA-Apps for the Heavy and Universal Forwarders. and some more for special cases All this apps are maintained in a git repository and distributed from there. Theses apps are developed and maintained on separate integration servers or local test installation. The finished apps are then, as a directory tree, into the git repository. The deployment from the git repository to the deployment server, the deployer and the master node is currently a manual task. At the moment, we are working on a continuous integration / continuous delivery solution where we will control the workflows with something like Jenkins or Bamboo. The goal is to have a workflow in which a developer commits the apps to the git repo which then triggers a check with Splunk Appinspect, and a deployment to the integration servers. An accepted change could then be merged into another git branch which in turn would trigger deployment to the production environment. A deployment which results in a rolling restart will be queued to some defined 'quiet' hours. Of course, this description is a bit simplified as reality is always a bit more complicated, but it should explain the big picture. ... View more

So the script works, but timestamp and event breaking fails. As long as you don't remove the colons Splunk will use the default settings and recognise this timestamps and brake the events there. There are various ways to fix it, but the laziest one would be to look at the Splunk Add-on for Unix, or at least the definition of the scripted input and the lastlog stanza from the props.conf. ... View more

did yo check the ip tables with iptables -vL what is the error you get can you connect to port 8443 ... View more

We have implemented an architecture with heavy forwarders in front of the indexer cluster, no data source connects directly to the indexers, the parsing is done on the heavy forwarders. So for most changes only the heavy forwarders are affected and restarted one by one, which saves us a rolling restart of the cluster. ... View more

I assume you want the 3 lines starting with a timestamp, so i would use [\d{2}\/\d{2}\/\d{2}\ (.*?[\r\n]){3} ... View more

For Redhat the following commands are doing it: /sbin/iptables -t nat -I PREROUTING -p tcp --destination-port 443 -j REDIRECT --to-ports 8443 /etc/init.d/iptables save ... View more

your pattern in the props stanza name [host::template...] matches hostnames 'template' followed by exactly 3 more arbitrary chars. You should use something like [host::template.*] this matches template followed by an arbitrary number of arbitrary chars. Just be careful, this matches anything which starts with template. ... View more

gaierror(-3, 'Temporary failure in name resolution') (gaierro= Get Address Info Error) points towards a DNS issue, did you try to load one of this urls with wget, curl or plain telnet, from the Splunk server on which Website Monitoring is running? See also this answer ... View more

It gets tricky as soon as your changes are to complex to be done in an index time transform either on an indexer or an intermediate heavy forwarder. If you know your source systems well and receive the events from an input which reads a log file, you could probably change the input to a scripted input where a script reads the log file and only passes the desired parts to Splunk. Mika Borner has written a blog about using Apache NiFi to preprocess data, this gives you a lot of flexibility although it has also its complexity. ... View more