Not sure if this has been seen by others and it didn't turn up in my searches... I have a 7.3.3 instance where I forgot the admin password. So I created a $SPLUNK_HOME/etc/system/local/user-seed.conf, restarted, but I couldn't log in with the password. Additionally, the user-seed.conf file was still present. Turns out there was still a $SPLUNK_HOME/etc/passwd file (presumably from previous upgrades). I moved that to the $SPLUNK_HOME/etc/passwd.bak, restarted and then Splunk used the user-seed.conf file to reset the admin password. Hope this helps someone else... ... View more

I'm collecting DNS logs and I'm trying to drop all logs with sub.domain.com as the query. In my transforms.conf I have the following: [dropdomain] REGEX = sub.domain.com DEST_KEY=queue FORMAT=nullQueue But those domains still show up in my index. I have this on both the HF and the Indexer for that sourcetype. I also am collecting logs from windows DNS debug log. As you know those come across in (#)string(#)string(#)string(#) format. So when the above comes through one of those logs, I have (3)sub(6)domain(3)com(0) in my log. I'm trying to drop those also and here is my transforms.conf for that log: [dropdomain] REGEX = sub(6)domain(3)com DEST_KEY=queue FORMAT=nullQueue But that isn't working either. Is my syntax correct? Do I need to escape the period or not? Do I escape the parenthesis or not? Thanks. (I'm sure this question has been asked before, but I have not found the right google-fu to get the answer) ... View more

There is a lot of information regarding the order of the precedence of props.conf within a single Splunk server, but how about between servers? I assumed it would be the last server (i.e. the SH), but I am having a problem where the time extraction works great in one spot but not in the SH despite the props.conf being the same. Let me explain. I have this sequence: UF -> IDX cluster -> SH cluster. I used the input wizard on one of the IDX servers to help create the props.conf file. The main reason for doing it this way was to get the correct config for the timestamp recognition (https://docs.splunk.com/Documentation/SplunkCloud/7.2.7/Data/Configuretimestamprecognition ). That config works just fine for any of that data sourcetype I uploaded to that IDX server. I then copied that working props.conf to the UF AND pushed it out to the SH cluster. But when I start indexing data via the UF, the timestamp field is ignored and Splunk uses the time of index instead. So do I need to push the props.conf out to all the indexers also? Is there any possibility that something else (an old config props.conf) on the SH is overriding the one from the UF? If so, how would I find out? Can btool pull from multiple servers to determine configuration conflicts? Additional troubleshooting tips appreciated. Thanks in advance. ... View more

I have a search that starts off with | metadata type=hosts .... The problem is that the results are pulling back a host that is not sending logs or should be sending logs. A search for the host in Splunk turns up nothing. So my question is why it's showing up in metadata if no logs are being collected (or have ever been collected) from that host? ... View more

So I created a way to use Mark Baggett's freq_server script as a lookup and blogged about it here (http://shadowtrackers.net/blog/get-your-freq-on-in-splunk). But, that is not the point of this forum post. I had based my blog post on my setup at home. I was running a single Splunk server and making the URL connection to a separate Linux server running in a vm on the same box as the Splunk server. So, when I got to work and tried to implement the same lookup, I was surprised and frustrated when it didn't work right out of the box. On the small Splunk instance where I implemented the lookup, I have a single search head and two Indexers. When I ran the lookup search: index=dns | rename query as domain | lookup freqserver domain I received the following error: 2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors. [INDEX01] Script for lookup table 'freqserver' returned error code 1. Results may be incorrect. [INDEX02] Script for lookup table 'freqserver' returned error code 1. Results may be incorrect. ... View more

I was troubleshooting a weird issue in a Splunk Universal Forwarder installed on a Windows 2012R2 Print Server. I was getting the print logs but not the Application/System/Security logs. I looked at the inputs.conf for both and they were correct. I checked to see if the print server was sending data and if the indexers were receiving data and they were. I didn't see any errors in splunkd.log on the print server nor any errors in the _internal log in my Splunk index cluster. Splunk is running as local system and as I said, seems to have no problem reading the print logs Then I started going through all the logs on the UF var/log directory and found btool.log had the following after the most recent restart: ConfPathMapper - Failed to open: C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\inputs.conf: Access is denied. Well, I thought that should be simple, except I didn't know the simple answer. In *nix, the entire Splunk directory, subdirectory, and files need to owned by the user Splunk. If there is an equivalent error, I just run chown -R splunk.splunk /opt/splunk/ and everything is (usually) fixed. But Windows doesn't have the equivalent of a Splunk user. Usually installing as local admin or domain admin takes care of setting all the correct permissions. But now that I have to fix the permissions on a single file/folder, what are the permissions that should be set on the whole Splunk directory structure? Googling hasn't yet found the answer. Closest I've found is this: https://answers.splunk.com/answering/8980/view.html and this: https://answers.splunk.com/answering/9545/view.html ... View more