Awesome! it's working now. I was able to start Splunk with your commands. root@indexer:/opt/splunk/bin# ./splunk start Splunk> CSI: Logfiles. Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary Done Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunk/splunk-6.6.1-aeae3fe0c5af-linux-2.6-x86_64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... Done Waiting for web server at http://127.0.0.1:8000 to be available..... Done If you get stuck, we're here to help. Look for answers here: http://docs.splunk.com The Splunk web interface is at http://indexer:8000 root@indexer:/opt/splunk/bin# ./splunk status splunkd is running (PID: 5391). splunk helpers are running (PIDs: 5392 5401 5521 5561 9931 9932). So what does (root)# chown -R indexer:indexer /opt/splunk command do can you explain to me? ... View more

Initially when I checked in $SPLUNK_HOME/etc/splunk-launch.conf user name was csoc but then I changed it to indexer yesterday since I was having some issues. It's currently still set as the indexer. SPLUNK_OS_USER=indexer When I ran cat grep command on all 3 (csoc, indexer, splunk), it only brought results for indexer and splunk. root@indexer:~# cat /etc/passwd | grep indexer indexer:x:1000:1000:indexer,,,:/home/indexer:/bin/bash root@indexer:~# cat /etc/passwd | grep splunk splunk:x:1001:1001::/home/splunk: Version 6.5.2 Modify the following line to suit the location of your Splunk install. If unset, Splunk will use the parent of the directory containing the splunk CLI executable. SPLUNK_HOME=/home/build/build-home/ivory By default, Splunk stores its indexes under SPLUNK_HOME in the var/lib/splunk subdirectory. This can be overridden here: SPLUNK_DB=/home/build/build-home/ivory/var/lib/splunk Splunkd daemon name SPLUNK_SERVER_NAME=Splunkd Splunkweb daemon name SPLUNK_WEB_NAME=splunkweb If SPLUNK_OS_USER is set, then Splunk service will only start if the 'splunk [re]start [splunkd]' command is invoked by a user who is, or can effectively become via setuid(2), $SPLUNK_OS_USER. (This setting can be specified as username or as UID.) SPLUNK_OS_USER SPLUNK_OS_USER=indexer Ok, where to go from here? ... View more

how do you check? ... View more

I've ran the chown command, removed splunkd.pid and tried to start splunk from root. chown -R splunk:splunk /opt/splunk ... View more

I have just removed splunkd.pid file from below location and tried to start splunk but still getting permission error. root@indexer:/opt/splunk/var/run/splunk# ls -l total 68 drwx--x--- 4 splunk splunk 4096 Mar 30 14:31 appserver -rw------- 1 splunk splunk 10990 Jun 26 10:54 composite.xml drwx------ 2 splunk splunk 4096 Mar 30 14:31 csv drwx--x--x 43 splunk splunk 20480 Jun 23 18:26 dispatch drwx------ 2 splunk splunk 4096 Jun 22 18:09 merged drwx------ 4 splunk splunk 4096 Mar 30 14:32 scheduler -rw------- 1 splunk splunk 556 Jun 23 18:26 session-86d3a80d3dbec0935375ba9adeb589bfc7ef3468 -rw------- 1 splunk splunk 0 Jun 23 16:14 session-86d3a80d3dbec0935375ba9adeb589bfc7ef3468.lock -rw------- 1 splunk splunk 15 Jun 26 10:54 splunkd.pid drwx------ 2 splunk splunk 4096 Jun 22 18:09 srtemp drwx------ 2 splunk splunk 4096 May 25 18:44 typeahead drwx------ 2 splunk splunk 4096 Mar 30 14:31 upload indexer@indexer:~$ sudo -H -u splunk /opt/splunk/bin/splunk start [sudo] password for indexer: This command can only be run by bootstart user. root@indexer:/opt/splunk/bin# ./splunk start Warning: cannot create "/opt/splunk/var/log/splunk" Warning: cannot create "/opt/splunk/var/log/introspection" Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied Splunk> Be an IT superhero. Go home early. Checking prerequisites... Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Pid file "/opt/splunk/var/run/splunk/splunkweb.pid" unreadable.: Permission denied Checking http port [8000]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied open Checking mgmt port [8089]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied open Checking appserver port [127.0.0.1:8065]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied open ERROR - Failed opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Checking configuration... Done. Creating: /opt/splunk/var/lib/splunk Warning: cannot create "/opt/splunk/var/lib/splunk" ... View more

I think the original user name has been changed since after running esix's command above. I know what my original user name is which was not splunk but something else. root@indexer:/opt/splunk/etc/init.d# ls -l total 4 -r--r--r-- 1 splunk splunk 819 May 19 07:37 README So where to go from here? ... View more

I tried your commands and now stuck in below where it's prompting me for password which don't work with the one that I setup. Is there a default password for this? root@indexer:~# killall splunk & killall mongod [1] 27060 splunk: no process found mongod: no process found [1]+ Exit 1 killall splunk root@indexer:~# chown -R splunk:splunk /opt/splunk root@indexer:~# sudo su - splunk No directory, logging in with HOME=/ $ /opt/splunk/bin/splunk start This command can only be run by bootstart user. $ sudo -i [sudo] password for splunk: Sorry, try again. [sudo] password for splunk: Sorry, try again. [sudo] password for splunk: sudo: 3 incorrect password attempts Splunk instance is running on a vmware workstation. ... View more

actually I got it working, strange it didn't work but I didn't do anything more than configuring logging it again in the switch maybe I fogot to wr mem. Anyways looks good now. I will try some dodgy router next. ... View more

I forgot to mention that was already done. Thanks, logging 10.10.50.11 How do I verifiy that I'm getting the logs from HP Procurve Switch? What commands do you need to run to determine whether logs are being received or not? ... View more

"You stated you setup syslog-ng on your UF, the .conf is usually in either /etc or /use/local/etc, run a find" Are you referring to inputs.conf and outputs.conf? ... View more

This indicates udp514 is open by Splunkd isn't it? root@forwarder:~# netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.1.1:53 0.0.0.0: LISTEN 799/dnsmasq tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN 755/sshd tcp 0 0 0.0.0.0:8089 0.0.0.0: LISTEN 1338/splunkd tcp6 0 0 :::22 ::: LISTEN 755/sshd udp 0 0 0.0.0.0:5353 0.0.0.0: 544/avahi-daemon: r udp 0 0 0.0.0.0:54600 0.0.0.0: 544/avahi-daemon: r udp 0 0 127.0.1.1:53 0.0.0.0: 799/dnsmasq udp 0 0 0.0.0.0:514 0.0.0.0: 1338/splunkd <<<<<<<<< udp 0 0 0.0.0.0:631 0.0.0.0: 630/cups-browsed udp6 0 0 :::5353 ::: 544/avahi-daemon: r udp6 0 0 :::44497 :::* 544/avahi-daemon: r root@forwarder:~# ... View more

root@forwarder:/opt/splunkforwarder/bin# ./splunk set deploy-poll 10.10.50.11:8089 root@forwarder:/opt/splunkforwarder/bin# ./splunk add monitor /opt/splunk/copyright.txt -sourcetype configtest -index main Parameter name: Path does not exist. root@forwarder:/opt/splunkforwarder/bin# ./splunk add monitor /opt/splunkforwarder/copyright.txt -sourcetype configtest -index main Added monitor of '/opt/splunkforwarder/copyright.txt'. In the forwarder I configured add monitor but the Path does not exist on /opt/splunk/copyright.txt so I added on /opt/splunkforwarder/copyright.txt However, when I do a search index=main sourcetype="configtest"  there were No results found. ... View more

Thanks, I've installed Cisco ASA app, will look into that. It's not showing vpn geolocation etc but will try to dig into that as well. ... View more

Done, I couldn't accept my own but all good. I also didn't know that, thanks for the tip. ... View more

oh when did you ask me to post syslog-ng.conf? I wasn't sure how to do that. I see it's in the last line. Let me do that for you, just trying to figure out how. ... View more

Can you give me an example? When I goto Add data>Forward it says, there are currently no forwarders configured as deployment clients to this instance. I'm not sure if that's normal. Can you do it from the terminal CLI? ... View more

Hi Guys, Think I got it working! I ran a search sourcetype="cisco:asa" and it brought up 484,558 events. Now I have tons of new questions with things like how to filter syslogs and how to setup and configure NTP as my time stamps are wrong and alot more but for now I'm so happy I got it to work finally!! Thanks to all of you guys who helped me to achieve this I couldn't have done it if I hadn't come to this forum with absolutely no help from anywhere else. One more thing I'd like to ask is with the query commands, could someone tell me how I can do standard, general and more discoverable searches with filters, limitations etc with Cisco asa? All the search command I know is sourcetype="cisco:asa" Thanks heaps guys, really appreciate it. Now I'll see if I can get the logs in for the routers and switches as well. ... View more

You're right, I created UDP input in the indexer rather than in the forwarder. Maybe that's why something's been screwing up. I didn't know that. I've now configured UDP 514 input in the forwarder where the syslog-ng server is installed and removed it from the indexer. Port 9997 was already enabled in the indexer and add forward-server is already present as well since I've configured it before. See the output below: Enable listen on port 9997 on the Indexer to recieve incoming data from the UF. <<<<< This has been done ./splunk enable listen 9997 -auth username:password root@indexer:/opt/splunk/bin# ./splunk enable listen 9997 -auth admin:seeshock Failed to create. Configuration for port 9997 already exists. Configure your forwarder to send data to the indexer. <<<<<<<<<<<< This has been done ./splunk add forward-server :9997 root@forwarder:/opt/splunkforwarder/bin# ./splunk add forward-server 10.10.50.11:9997 10.10.50.11:9997 forwarded-server already present Create the UDP input on your UF to listen that syslog. <<<<<<<<<<<<<<< This has been done ./splunk add udp 514 -sourcetype cisco:asa root@forwarder:/opt/splunkforwarder/bin# ./splunk add udp 514 -sourcetype cisco:asa Listening for UDP input on port 514. Also Removed UDP 514 from indexer root@indexer:/opt/splunk/bin# ./splunk remove udp 514 -sourcetype cisco:asa Removed udp://514 Now, how do I test everything is working properly? Though, I'm still not confident syslog-ng server is configured properly with Cisco ASA. I need some clear instructions on this. Any help would be appreciated. ... View more