Here's how I solved this. I noticed that the savedsearch dhcpd_alert_new_mac_address_15m was configured to send an email every 15 minutes. By default, it sends email to example@example.org . That is a ton of email (96 incorrect emails per day?). This is viewable under "Splunk> Manager » Searches and reports » dhcpd_alert_new_mac_address_15m", and on the commandline at $SPLUNK_HOME/etc/apps/dhcpd/default/savedsearches.conf has this: [dhcpd_alert_new_mac_address_15m] action.email = 1 action.email.sendresults = 1 action.email.to = example@example.com counttype = number of events cron_schedule = */15 * * * * description = Alerts on mac addresses seen in the last 15 minutes that were not in the dhcpd_mac-hostname lookup table To disable this, I simply unchecked the box next to "Schedule this search". On the commandline, the following file was added to $SPLUNK_HOME/etc/apps/dhcpd/local/savedsearches.conf , and now the emails have stopped. [dhcpd_alert_new_mac_address_15m] disabled = 1 ... View more

I found out how to do this. I am following the instructions written at Documentation > Splunk > Developing Dashboards, Views, and Apps for Splunk Web > Step 6: Build navigation for your app. Navigate to the Search App, then navigate to Manager > User interface > Navigation menus . Click default . Add the following to the XML. It might need to go below the other saved searches < divider /> < collection label="DHCP App"> < view name="DHCP App View" default="true" /> < saved source="unclassified" match="dhcp" /> < /collection> Note that I had to fudge the XML code a bit to make it appear correctly inside this post. ... View more

Took a while, but we finally found the cause of my problem with the help of Splunk Technical Support and many hours on the phone. Splunk did not have permission to read /etc/httpd/mime.types , which is a special file that we created for our own local Apache installation. I don't know why Splunk is trying to read that file or why the error is not properly reported to the logfile. According to Splunk support, this is a bug and will get fixed. I changed the permissions on that file so that the user splunk could read that file. Splunk is now able to be run as user splunk . ... View more

Deep in the Knowledge Manager Manual, I found more details about punct . In the section "Use the punct field to search on similar events " says: The punct field stores the first 30 punctuation characters in the first line of the event. This field is useful for finding similar events quickly. When you use punct, keep in mind: Quotes and backslashes are escaped. Spaces are replaced with an underscore (_). Tabs are replaced with a "t". Dashes that follow alphanumeric characters are ignored. Interesting punctuation characters are: ",;-#$%&+./:=?@\'|*\n\r\"(){}<>[]^!" In addition, wildcards are supported, according to "Identify similar events with punct". However, this description is vague. You may want to consider wildcarding the punctuation to match insignificant variations (for example, "punct=::[]/"). ... View more

Well, in the end I just ended up doing a loop like: for USER in $USERLIST do $SPLUNK_HOME/bin/splunk add user ${USER}@example.org -password jibberish done Not quite a bulk import, but it gets the job done. ... View more