About jhl226116

jhl226116 · ‎06-28-2017

Awesome! it's working now. I was able to start Splunk with your commands. root@indexer:/opt/splunk/bin# ./splunk start Splunk> CSI: Logfiles. Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary Done Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunk/splunk-6.6.1-aeae3fe0c5af-linux-2.6-x86_64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... Done Waiting for web server at http://127.0.0.1:8000 to be available..... Done If you get stuck, we're here to help. Look for answers here: http://docs.splunk.com The Splunk web interface is at http://indexer:8000 root@indexer:/opt/splunk/bin# ./splunk status splunkd is running (PID: 5391). splunk helpers are running (PIDs: 5392 5401 5521 5561 9931 9932). So what does (root)# chown -R indexer:indexer /opt/splunk command do can you explain to me?

jhl226116 · ‎06-26-2017

Initially when I checked in $SPLUNK_HOME/etc/splunk-launch.conf user name was csoc but then I changed it to indexer yesterday since I was having some issues. It's currently still set as the indexer. SPLUNK_OS_USER=indexer When I ran cat grep command on all 3 (csoc, indexer, splunk), it only brought results for indexer and splunk. root@indexer:~# cat /etc/passwd | grep indexer indexer:x:1000:1000:indexer,,,:/home/indexer:/bin/bash root@indexer:~# cat /etc/passwd | grep splunk splunk:x:1001:1001::/home/splunk: Version 6.5.2 Modify the following line to suit the location of your Splunk install. If unset, Splunk will use the parent of the directory containing the splunk CLI executable. SPLUNK_HOME=/home/build/build-home/ivory By default, Splunk stores its indexes under SPLUNK_HOME in the var/lib/splunk subdirectory. This can be overridden here: SPLUNK_DB=/home/build/build-home/ivory/var/lib/splunk Splunkd daemon name SPLUNK_SERVER_NAME=Splunkd Splunkweb daemon name SPLUNK_WEB_NAME=splunkweb If SPLUNK_OS_USER is set, then Splunk service will only start if the 'splunk [re]start [splunkd]' command is invoked by a user who is, or can effectively become via setuid(2), $SPLUNK_OS_USER. (This setting can be specified as username or as UID.) SPLUNK_OS_USER SPLUNK_OS_USER=indexer Ok, where to go from here?

jhl226116 · ‎06-26-2017

how do you check?

jhl226116 · ‎06-26-2017

I've ran the chown command, removed splunkd.pid and tried to start splunk from root. chown -R splunk:splunk /opt/splunk

jhl226116 · ‎06-26-2017

I have just removed splunkd.pid file from below location and tried to start splunk but still getting permission error. root@indexer:/opt/splunk/var/run/splunk# ls -l total 68 drwx--x--- 4 splunk splunk 4096 Mar 30 14:31 appserver -rw------- 1 splunk splunk 10990 Jun 26 10:54 composite.xml drwx------ 2 splunk splunk 4096 Mar 30 14:31 csv drwx--x--x 43 splunk splunk 20480 Jun 23 18:26 dispatch drwx------ 2 splunk splunk 4096 Jun 22 18:09 merged drwx------ 4 splunk splunk 4096 Mar 30 14:32 scheduler -rw------- 1 splunk splunk 556 Jun 23 18:26 session-86d3a80d3dbec0935375ba9adeb589bfc7ef3468 -rw------- 1 splunk splunk 0 Jun 23 16:14 session-86d3a80d3dbec0935375ba9adeb589bfc7ef3468.lock -rw------- 1 splunk splunk 15 Jun 26 10:54 splunkd.pid drwx------ 2 splunk splunk 4096 Jun 22 18:09 srtemp drwx------ 2 splunk splunk 4096 May 25 18:44 typeahead drwx------ 2 splunk splunk 4096 Mar 30 14:31 upload indexer@indexer:~$ sudo -H -u splunk /opt/splunk/bin/splunk start [sudo] password for indexer: This command can only be run by bootstart user. root@indexer:/opt/splunk/bin# ./splunk start Warning: cannot create "/opt/splunk/var/log/splunk" Warning: cannot create "/opt/splunk/var/log/introspection" Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied Splunk> Be an IT superhero. Go home early. Checking prerequisites... Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Pid file "/opt/splunk/var/run/splunk/splunkweb.pid" unreadable.: Permission denied Checking http port [8000]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied open Checking mgmt port [8089]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied open Checking appserver port [127.0.0.1:8065]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied open ERROR - Failed opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Checking configuration... Done. Creating: /opt/splunk/var/lib/splunk Warning: cannot create "/opt/splunk/var/lib/splunk"

jhl226116 · ‎06-26-2017

I think the original user name has been changed since after running esix's command above. I know what my original user name is which was not splunk but something else. root@indexer:/opt/splunk/etc/init.d# ls -l total 4 -r--r--r-- 1 splunk splunk 819 May 19 07:37 README So where to go from here?

jhl226116 · ‎06-26-2017

I tried your commands and now stuck in below where it's prompting me for password which don't work with the one that I setup. Is there a default password for this? root@indexer:~# killall splunk & killall mongod [1] 27060 splunk: no process found mongod: no process found [1]+ Exit 1 killall splunk root@indexer:~# chown -R splunk:splunk /opt/splunk root@indexer:~# sudo su - splunk No directory, logging in with HOME=/ $ /opt/splunk/bin/splunk start This command can only be run by bootstart user. $ sudo -i [sudo] password for splunk: Sorry, try again. [sudo] password for splunk: Sorry, try again. [sudo] password for splunk: sudo: 3 incorrect password attempts Splunk instance is running on a vmware workstation.

jhl226116 · ‎06-25-2017

Hi Guys, I upgraded Splunk from 6.5 to 6.6 since then I'm unable to start Splunk properly, it seems there is some issue in the permission level. Could you walk me through in steps what I suppose to do to fix the permission denied error message pls? I'm just few months into Splunk so fairly new to everything. See the error message below: root@indexer:/opt/splunk/bin# ./splunk status Warning: cannot create "/opt/splunk/var/log/splunk" Warning: cannot create "/opt/splunk/var/log/introspection" Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied splunkd.pid file is unreadable. Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

jhl226116 · ‎05-18-2017

actually I got it working, strange it didn't work but I didn't do anything more than configuring logging it again in the switch maybe I fogot to wr mem. Anyways looks good now. I will try some dodgy router next.

jhl226116 · ‎05-18-2017

I forgot to mention that was already done. Thanks, logging 10.10.50.11 How do I verifiy that I'm getting the logs from HP Procurve Switch? What commands do you need to run to determine whether logs are being received or not?

jhl226116 · ‎05-18-2017

I have hard time getting logs from Procurve to the Splunk server. Any help would be greatly appreciated. I can ping between the Splunk server and HP Procurve switch vice-versa, they are in the same subnet. No firewall is blocking the connection: Indexer: 10.10.50.11 Forwarder2: 10.10.50.15 root@indexer:~# ping 10.10.50.3 PING 10.10.50.3 (10.10.50.3) 56(84) bytes of data. 64 bytes from 10.10.50.3: icmp_seq=1 ttl=255 time=0.873 ms 64 bytes from 10.10.50.3: icmp_seq=2 ttl=255 time=0.858 ms root@indexer:/opt/splunk/bin# ./splunk display listen Receiving is enabled on port 9997. root@indexer:/opt/splunk/bin# ./splunk btool inputs list splunktcp --debug | grep -v default /opt/splunk/etc/system/local/inputs.conf host = indexer /opt/splunk/etc/apps/launcher/local/inputs.conf [splunktcp://9997] /opt/splunk/etc/apps/launcher/local/inputs.conf connection_host = ip /opt/splunk/etc/system/local/inputs.conf host = indexer root@indexer:/opt/splunk/bin# ./splunk btool outputs list splunktcp --debug | grep -v default root@indexer:/opt/splunk/bin# ./splunk list inputstatus Cooked:tcp : 9997:10.10.50.12:8089 time opened = 2017-04-19T21:49:41+1000 9997:10.10.50.15:8089 time opened = 2017-04-21T19:19:01+1000 tcp_cooked:listenerports : 9997 UDP:listenerports : 514 root@indexer:/opt/splunk/bin# root@forwarder2:~# netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3824/sshd tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 4304/splunkd tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN 896/dnsmasq tcp6 0 0 :::22 :::* LISTEN 3824/sshd udp 0 0 0.0.0.0:514 0.0.0.0:* 4304/splunkd udp 0 0 0.0.0.0:631 0.0.0.0:* 8112/cups-browsed udp 0 0 0.0.0.0:57978 0.0.0.0:* 767/avahi-daemon: r udp 0 0 0.0.0.0:5353 0.0.0.0:* 767/avahi-daemon: r udp 0 0 127.0.1.1:53 0.0.0.0:* 896/dnsmasq udp 0 0 0.0.0.0:68 0.0.0.0:* 883/dhclient udp6 0 0 :::46130 :::* 767/avahi-daemon: r udp6 0 0 :::5353 :::* 767/avahi-daemon: r root@forwarder2:/opt/splunkforwarder/bin# ./splunk list forward-server Active forwards: 10.10.50.11:9997 Configured but inactive forwards: None root@forwarder2:/opt/splunkforwarder/bin# ./splunk show deploy-poll Deployment Server URI is set to "10.10.50.15:8089". root@forwarder2:/opt/splunkforwarder/bin# ./splunk add udp 514 -sourcetype hp:switch Listening for UDP input on port 514. root@forwarder:/opt/splunkforwarder/bin# ./splunk add forward-server 10.10.50.11:9997 10.10.50.11:9997 forwarded-server already present Listening port 9997 has already been enable on the indexer. What other information do you need from me and where do I go from here?

jhl226116 · ‎04-21-2017

"You stated you setup syslog-ng on your UF, the .conf is usually in either /etc or /use/local/etc, run a find" Are you referring to inputs.conf and outputs.conf?

jhl226116 · ‎04-20-2017

This indicates udp514 is open by Splunkd isn't it? root@forwarder:~# netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.1.1:53 0.0.0.0: LISTEN 799/dnsmasq tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN 755/sshd tcp 0 0 0.0.0.0:8089 0.0.0.0: LISTEN 1338/splunkd tcp6 0 0 :::22 ::: LISTEN 755/sshd udp 0 0 0.0.0.0:5353 0.0.0.0: 544/avahi-daemon: r udp 0 0 0.0.0.0:54600 0.0.0.0: 544/avahi-daemon: r udp 0 0 127.0.1.1:53 0.0.0.0: 799/dnsmasq udp 0 0 0.0.0.0:514 0.0.0.0: 1338/splunkd <<<<<<<<< udp 0 0 0.0.0.0:631 0.0.0.0: 630/cups-browsed udp6 0 0 :::5353 ::: 544/avahi-daemon: r udp6 0 0 :::44497 :::* 544/avahi-daemon: r root@forwarder:~#

jhl226116 · ‎04-20-2017

root@forwarder:/opt/splunkforwarder/bin# ./splunk set deploy-poll 10.10.50.11:8089 root@forwarder:/opt/splunkforwarder/bin# ./splunk add monitor /opt/splunk/copyright.txt -sourcetype configtest -index main Parameter name: Path does not exist. root@forwarder:/opt/splunkforwarder/bin# ./splunk add monitor /opt/splunkforwarder/copyright.txt -sourcetype configtest -index main Added monitor of '/opt/splunkforwarder/copyright.txt'. In the forwarder I configured add monitor but the Path does not exist on /opt/splunk/copyright.txt so I added on /opt/splunkforwarder/copyright.txt However, when I do a search index=main sourcetype="configtest" there were No results found.

jhl226116 · ‎04-20-2017

Thanks, I've installed Cisco ASA app, will look into that. It's not showing vpn geolocation etc but will try to dig into that as well.

jhl226116 · ‎04-20-2017

Done, I couldn't accept my own but all good. I also didn't know that, thanks for the tip.

jhl226116 · ‎04-19-2017

oh when did you ask me to post syslog-ng.conf? I wasn't sure how to do that. I see it's in the last line. Let me do that for you, just trying to figure out how.

jhl226116 · ‎04-19-2017

Can you give me an example? When I goto Add data>Forward it says, there are currently no forwarders configured as deployment clients to this instance. I'm not sure if that's normal. Can you do it from the terminal CLI?

jhl226116 · ‎04-19-2017

Hi Guys, Think I got it working! I ran a search sourcetype="cisco:asa" and it brought up 484,558 events. Now I have tons of new questions with things like how to filter syslogs and how to setup and configure NTP as my time stamps are wrong and alot more but for now I'm so happy I got it to work finally!! Thanks to all of you guys who helped me to achieve this I couldn't have done it if I hadn't come to this forum with absolutely no help from anywhere else. One more thing I'd like to ask is with the query commands, could someone tell me how I can do standard, general and more discoverable searches with filters, limitations etc with Cisco asa? All the search command I know is sourcetype="cisco:asa" Thanks heaps guys, really appreciate it. Now I'll see if I can get the logs in for the routers and switches as well.

jhl226116 · ‎04-18-2017

You're right, I created UDP input in the indexer rather than in the forwarder. Maybe that's why something's been screwing up. I didn't know that. I've now configured UDP 514 input in the forwarder where the syslog-ng server is installed and removed it from the indexer. Port 9997 was already enabled in the indexer and add forward-server is already present as well since I've configured it before. See the output below: Enable listen on port 9997 on the Indexer to recieve incoming data from the UF. <<<<< This has been done ./splunk enable listen 9997 -auth username:password root@indexer:/opt/splunk/bin# ./splunk enable listen 9997 -auth admin:seeshock Failed to create. Configuration for port 9997 already exists. Configure your forwarder to send data to the indexer. <<<<<<<<<<<< This has been done ./splunk add forward-server :9997 root@forwarder:/opt/splunkforwarder/bin# ./splunk add forward-server 10.10.50.11:9997 10.10.50.11:9997 forwarded-server already present Create the UDP input on your UF to listen that syslog. <<<<<<<<<<<<<<< This has been done ./splunk add udp 514 -sourcetype cisco:asa root@forwarder:/opt/splunkforwarder/bin# ./splunk add udp 514 -sourcetype cisco:asa Listening for UDP input on port 514. Also Removed UDP 514 from indexer root@indexer:/opt/splunk/bin# ./splunk remove udp 514 -sourcetype cisco:asa Removed udp://514 Now, how do I test everything is working properly? Though, I'm still not confident syslog-ng server is configured properly with Cisco ASA. I need some clear instructions on this. Any help would be appreciated.

jhl226116 · ‎04-18-2017

Yep, I'm just back on it now. Additionally, pls see my response to alemarzu below. I've mistakenly configured UDP 514 on the indexer rather than the forwarder earlier which could have been causing some problems but it has been fixed now as I reconfigured it on the forwarder and removed it from the indexer.

jhl226116 · ‎04-18-2017

Sorry for the delayed response, I was on a 5 day Easter long weekend break and just got back to work. Thanks for your help. I ran your command and it seems like I'm getting something from the ASA firewall. Do you see the output as data coming in from ASA firewall? If that's the case then as you already predicted it could be a misconfigured syslog-ng which I agree to as I was never confident with it because none of the Instructions really worked well and I don't know what I did really. I need some clear instructions on this. root@forwarder:~# tcpdump -vvv -i ens160 udp port 514 tcpdump: listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes 10:01:14.409934 IP (tos 0x0, ttl 255, id 4312, offset 0, flags [none], proto UDP (17), length 188) 10.10.50.1.syslog > 10.10.50.12.syslog: [udp sum ok] SYSLOG, length: 160 Facility local5 (21), Severity critical (2) Msg: Apr 19 2017 11:05:08 CSOCLABFW01 : %ASA-2-106001: Inbound TCP connection denied from 178.47.249.32/22733 to 61.68.11.85/23 flags SYN on interface outside\0x0a 0x0000: 3c31 3730 3e41 7072 2031 3920 3230 3137 0x0010: 2031 313a 3035 3a30 3820 4353 4f43 4c41 0x0020: 4246 5730 3120 3a20 2541 5341 2d32 2d31 0x0030: 3036 3030 313a 2049 6e62 6f75 6e64 2054 0x0040: 4350 2063 6f6e 6e65 6374 696f 6e20 6465 0x0050: 6e69 6564 2066 726f 6d20 3137 382e 3437 0x0060: 2e32 3439 2e33 322f 3232 3733 3320 746f 0x0070: 2036 312e 3638 2e31 312e 3835 2f32 3320 0x0080: 666c 6167 7320 5359 4e20 206f 6e20 696e 0x0090: 7465 7266 6163 6520 6f75 7473 6964 650a 10:01:19.657510 IP (tos 0x0, ttl 255, id 6363, offset 0, flags [none], proto UDP (17), length 197) 10.10.50.1.syslog > 10.10.50.12.syslog: [udp sum ok] SYSLOG, length: 169 Facility local5 (21), Severity warning (4) Msg: Apr 19 2017 11:05:14 CSOCLABFW01 : %ASA-4-106023: Deny udp src inside:10.10.50.2/1028 dst outside:172.30.234.150/5516 by access-group "inside_access_in" [0x0, 0x0]\0x0a 0x0000: 3c31 3732 3e41 7072 2031 3920 3230 3137 0x0010: 2031 313a 3035 3a31 3420 4353 4f43 4c41 0x0020: 4246 5730 3120 3a20 2541 5341 2d34 2d31 0x0030: 3036 3032 333a 2044 656e 7920 7564 7020 0x0040: 7372 6320 696e 7369 6465 3a31 302e 3130 0x0050: 2e35 302e 322f 3130 3238 2064 7374 206f 0x0060: 7574 7369 6465 3a31 3732 2e33 302e 3233 0x0070: 342e 3135 302f 3535 3136 2062 7920 6163 0x0080: 6365 7373 2d67 726f 7570 2022 696e 7369 0x0090: 6465 5f61 6363 6573 735f 696e 2220 5b30 0x00a0: 7830 2c20 3078 305d 0a 10:01:19.657586 IP (tos 0x0, ttl 255, id 25234, offset 0, flags [none], proto UDP (17), length 197) 10.10.50.1.syslog > 10.10.50.12.syslog: [udp sum ok] SYSLOG, length: 169 root@forwarder:~# tcpdump -vvv -i ens160 tcp port 514 tcpdump: listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes I've already provided IP's for the ASA, Indexer and Forwarder in my previous post (In the very beginning). Cisco ASA: 10.10.50.1 Indexer: 10.10.50.11 Forwarder: 10.10.50.12 I think I configured each of these to communicate using IP addresses, not the hostnames but how do I check? Check that you ASA is sending to an IP address, not a hostname. I think it's using an IP address, not a hostname but how do I check? Check that your syslog-NG server, which is running your UF, is sending (output.conf) to your indexer using an IP address, not a hostname. I think it's using an IP address, not a hostname but how do I check? root@forwarder:~# nano /opt/splunkforwarder/etc/system/local/outputs.conf [tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = 10.10.50.11:9997 [tcpout-server://10.10.50.11:9997]

jhl226116 · ‎04-12-2017

I'm too noob to undestand that but please see my further outputs below in response to nychawk. What do you think?

jhl226116 · ‎04-12-2017

Cisco ASA: 10.10.50.1 Indexer: 10.10.50.11 Forwarder: 10.10.50.12 In the command line, are you referring "src host" as indexer or Cisco asa? I tried to run the command from forwarder pointing src host as both indexer and cisco asa but result shows No such device. Am I running the command correctly and from the right directory location? Sorry I'm a newbie just 4mth into Splunk, trying to understand how everything works. It's quite difficult trying to learn everything by myself alone, it would be x10 times faster if someone taught me how to do it. Please see output below: From Forwarder pointing src host as Indexer 10.10.50.11 root@forwarder:~# tcpdump -vvv -i 10.10.50.11 udp port 514 tcpdump: 10.10.50.11: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device root@forwarder:~# tcpdump -vvv -i 10.10.50.11 tcp port 514 tcpdump: 10.10.50.11: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device From Forwarder pointing src host as Cisco ASA 10.10.50.1 root@forwarder:~# tcpdump -vvv -i 10.10.50.1 udp port 514 tcpdump: 10.10.50.1: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device root@forwarder:~# tcpdump -vvv -i 10.10.50.1 tcp port 514 tcpdump: 10.10.50.1: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device I'm not sure what's going on but my gut tells me that Ciasco asa is not communicating with syslog-ng which is installed on the forwarder server. I installed TCPdump and tried running it for the first time use, I'm not sure if some ports are missing or anything but I'm not sure where to look. Does everything look ok? Or anything doesn't look right? TCPDUMP output from forwarder root@forwarder:~# tcpdump -i ens160 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes 10:08:21.205181 IP 10.10.50.1.syslog > 10.10.50.12.syslog: SYSLOG local5.error, length: 141 10:08:21.205221 IP 10.10.50.12 > 10.10.50.1: ICMP 10.10.50.12 udp port syslog unreachable, length 177 10:08:21.205963 IP 10.10.50.12.28344 > dns1.tpgi.com.au.domain: 38283+ PTR? 12.50.10.10.in-addr.arpa. (42) 10:08:21.208036 IP dns1.tpgi.com.au.domain > 10.10.50.12.28344: 38283 NXDomain 0/1/0 (119) 10:08:21.208235 IP 10.10.50.12.45615 > dns1.tpgi.com.au.domain: 57567+ PTR? 1.50.10.10.in-addr.arpa. (41) 10:08:21.209522 IP dns1.tpgi.com.au.domain > 10.10.50.12.45615: 57567 NXDomain 0/1/0 (118) 10:08:21.209752 IP 10.10.50.12.38074 > dns1.tpgi.com.au.domain: 11768+ PTR? 35.160.12.203.in-addr.arpa. (44) 10:08:21.211518 IP dns1.tpgi.com.au.domain > 10.10.50.12.38074: 11768 1/2/0 PTR dns1.tpgi.com.au. (110) 10:08:21.332932 IP 10.10.50.12.35998 > 10.10.50.11.9997: Flags [P.], seq 4241108933:4241109329, ack 1934590004, win 237, options [nop,nop,TS val 128417542 ecr 285094319], length 396 10:08:21.333142 IP 10.10.50.11.9997 > 10.10.50.12.35998: Flags [.], ack 396, win 1452, options [nop,nop,TS val 285094735 ecr 128417542], length 0 [2]+ Stopped tcpdump -i ens160 TCPDUMP output from Indexer root@indexer:~# tcpdump -i ens160 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes 17:35:54.469974 IP 10.10.50.12.35998 > 10.10.50.11.9997: Flags [P.], seq 4241118889:4241119208, ack 1934590004, win 237, options [nop,nop,TS val 128428299 ecr 285104916], length 319 17:35:54.470013 IP 10.10.50.12.60808 > 10.10.50.11.9997: Flags [S], seq 604967860, win 29200, options [mss 1460,sackOK,TS val 128428299 ecr 0,nop,wscale 7], length 0 17:35:54.470038 IP 10.10.50.11.9997 > 10.10.50.12.35998: Flags [.], ack 319, win 1452, options [nop,nop,TS val 285105492 ecr 128428299], length 0 17:35:54.470052 IP 10.10.50.11.9997 > 10.10.50.12.60808: Flags [S.], seq 2536124428, ack 604967861, win 28960, options [mss 1460,sackOK,TS val 285105492 ecr 128428299,nop,wscale 7], length 0 17:35:54.470199 IP 10.10.50.12.60808 > 10.10.50.11.9997: Flags [.], ack 1, win 229, options [nop,nop,TS val 128428299 ecr 285105492], length 0 17:35:54.470254 IP 10.10.50.12.60808 > 10.10.50.11.9997: Flags [F.], seq 1, ack 1, win 229, options [nop,nop,TS val 128428299 ecr 285105492], length 0 17:35:54.470520 IP 10.10.50.11.9997 > 10.10.50.12.60808: Flags [F.], seq 1, ack 2, win 227, options [nop,nop,TS val 285105492 ecr 128428299], length 0 17:35:59.441948 IP 10.10.50.12.35998 > 10.10.50.11.9997: Flags [P.], seq 319:688, ack 1, win 237, options [nop,nop,TS val 128429542 ecr 285105492], length 369 17:35:59.442022 IP 10.10.50.11.9997 > 10.10.50.12.35998: Flags [.], ack 688, win 1452, options [nop,nop,TS val 285106735 ecr 128429542], length 0 17:35:59.442137 IP 10.10.50.12.35998 > 10.10.50.11.9997: Flags [P.], seq 688:779, ack 1, win 237, options [nop,nop,TS val 128429542 ecr 285106735], length 91 17:35:59.442169 IP 10.10.50.11.9997 > 10.10.50.12.35998: Flags [.], ack 779, win 1452, options [nop,nop,TS val 285106735 ecr 128429542], length 0 [7]+ Stopped tcpdump -i ens160

jhl226116 · ‎04-09-2017

Hi Guys, I've been trying to send Cisco ASA firewall logs to syslog-ng server where the forwarder is installed but I just can't get it working. I've setup a forwarder and installed syslog-ng in Ubuntu VM. I have tried to follow the instructions on this link and also from other various sources but I'm stressful enough to say that I just can't get it working. https://www.splunk.com/blog/2016/03/11/using-syslog-ng-with-splunk/ I really need some clear detailed step by step instructions on how to configure Cisco ASA to store syslogs into the syslog-ng server and forward the data to indexer. I'm not sure if I configured syslog-ng server properly in Ubuntu. I used following tutorial but no success. https://www.youtube.com/watch?v=glvsJJFbzZc&feature=em-share_video_user Could you check if everything looks ok below and advise the next step from here? My goal is to send Cisco ASA Firewall logs to syslog-ng server and push it out to the indexer with universal forwarder so that I'm able to see all the cisco asa logs from the search. My setup is as below: All servers have been built with Ubuntu in VM. Indexer: 10.10.50.11 Forwarder: 10.10.50.12 (Installed syslog-ng here) I can ping and SSH between Indexer and forwarder. Configured universal forwarder to send data to the receiving indexer. root@forwarder:/opt/splunkforwarder/bin# ./splunk add forward-server 10.10.50.11:9997 -auth admin:seeshock root@forwarder:/opt/splunkforwarder/bin# ./splunk list forward-server Active forwards: None Configured but inactive forwards: 10.10.50.11:9997 Configure deployment client root@forwarder:/opt/splunkforwarder/bin# ./splunk show deploy-poll Deployment Server URI is set to "10.10.50.12:8089". root@indexer:/opt/splunk/bin# ./splunk status splunkd is running (PID: 3109). splunk helpers are running (PIDs: 3110 3118 3183 3200). root@indexer:/opt/splunk/bin# ./splunk btool inputs list splunktcp --debug | grep -v default /opt/splunk/etc/system/local/inputs.conf host = indexer /opt/splunk/etc/apps/launcher/local/inputs.conf [splunktcp://9997] /opt/splunk/etc/apps/launcher/local/inputs.conf connection_host = ip /opt/splunk/etc/system/local/inputs.conf host = indexer root@indexer:/opt/splunk/bin# ./splunk list inputstatus tcp_cooked:listenerports : 9997 index=internal host="indexer" Time Event 4/2/17 10.10.50.11 - admin [02/Apr/2017:20:47:25.825 +1000] "GET /en-US/splunkd/raw/services/search/shelper?output_mode=json&snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D_internal+host%3D&useTypeahead=true&useAssistant=false&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&=1491126297345 HTTP/1.1" 200 5141 "http://10.10.50.11:8000/en-US/app/search/search?q=search%20index%3D_internal%20host%3Dforwarder&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=&latest=&sid=1491130033.479" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0" - b44fb5fd27b3677a2443d6de473370fe 8ms 8:47:25.825 PM · host = indexer · source = /opt/splunk/var/log/splunk/splunkd_ui_access.log · sourcetype = splunkd_ui_access 4/2/17 10.10.50.11 - admin [02/Apr/2017:20:47:25.681 +1000] "GET /en-US/splunkd/_raw/services/search/shelper?output_mode=json&snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D_internal+host%3Df&useTypeahead=true&useAssistant=false&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&=1491126297344 HTTP/1.1" 200 5109 "http://10.10.50.11:8000/en-US/app/search/search?q=search%20index%3D_internal%20host%3Dforwarder&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=&latest=&sid=1491130033.479" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0" - b44fb5fd27b3677a2443d6de473370fe 40ms 8:47:25.681 PM · host = indexer · source = /opt/splunk/var/log/splunk/splunkd_ui_access.log · sourcetype = splunkd_ui_access index=_internal host="forwarder" 4/3/17 04-03-2017 14:15:24.393 +1000 INFO DC:PhonehomeThread - Attempted handshake 260 times. Will try to re-subscribe to handshake reply 2:15:24.393 PM · host = forwarder · source = /opt/splunkforwarder/var/log/splunk/splunkd.log · sourcetype = splunkd 4/3/17 04-03-2017 14:15:24.393 +1000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected 2:15:24.393 PM · host = forwarder · source = /opt/splunkforwarder/var/log/splunk/splunkd.log · sourcetype = splunkd 4/3/17 04-03-2017 14:15:16.083 +1000 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0.000000, instantaneous_eps=0.000000, average_kbps=0.000000, total_k_processed=0.000000, kb=0.000000, ev=0.000000 2:15:16.083 PM · host = forwarder · source = /opt/splunkforwarder/var/log/splunk/metrics.log · sourcetype = splunkd root@indexer:/etc/syslog-ng# netstat -an | grep 9997 tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN tcp 0 0 10.10.50.11:9997 10.10.50.12:53380 ESTABLISHED root@indexer:/opt/splunk/etc/system/local# mkdir firewall_asa root@indexer:/opt/splunk/etc/system/local# cd firewall_asa root@indexer:/opt/splunk/etc/system/local/firewall_asa# cd root@indexer:~# cd /opt/splunk/bin root@indexer:/opt/splunk/bin# ./splunk add udp 514 -sourcetype cisco:asa Listening for UDP input on port 514. root@forwarder:/opt/splunkforwarder/bin# ./splunk show deploy-poll Deployment Server URI is set to "10.10.50.12:8089". root@forwarder:/opt/splunkforwarder/bin# ./splunk list forward-server Active forwards: 10.10.50.11:9997 Configured but inactive forwards: None root@forwarder:/opt/splunkforwarder/bin# ./splunk show servername Server name: forwarder root@forwarder:/opt/splunkforwarder/bin# ./splunk show default-hostname Default hostname for data inputs: forwarder. root@forwarder:/opt/splunkforwarder/bin# ./splunk show servername Server name: forwarder root@forwarder:/opt/splunkforwarder/bin# ./splunk show default-hostname Default hostname for data inputs: forwarder. root@forwarder:/opt/splunkforwarder/bin# ./splunk add forward-server 10.10.50.11:9997 -auth admin:seeshock Added forwarding to: 10.10.50.11:9997. root@forwarder:/opt/splunkforwarder/bin# ./splunk list forward-server Active forwards: None Configured but inactive forwards: 10.10.50.11:9997

Posts	46
Solutions	0
Karma Given	2
Karma Received	1
Member Since	‎03-20-2017

Online Status	Offline
Date Last Visited	‎06-05-2020 02:03 AM

After upgrading to Splunk 6.6, the splunkd.pid fil...

How to setup HP Procurve switch to send logs to Sp...

How to send Cisco ASA Firewall logs to syslog-ng s...

How to activate forwarder server?

Re: After upgrading to Splunk 6.6, the splunkd.pid...

Re: After upgrading to Splunk 6.6, the splunkd.pid...

Re: After upgrading to Splunk 6.6, the splunkd.pid...

Re: After upgrading to Splunk 6.6, the splunkd.pid...

Re: After upgrading to Splunk 6.6, the splunkd.pid...

Re: After upgrading to Splunk 6.6, the splunkd.pid...

Re: After upgrading to Splunk 6.6, the splunkd.pid...

After upgrading to Splunk 6.6, the splunkd.pid fil...

Re: How to setup HP Procurve switch to send logs t...

Re: How to setup HP Procurve switch to send logs t...

How to setup HP Procurve switch to send logs to Sp...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

Re: How to send Cisco ASA Firewall logs to syslog-...

How to send Cisco ASA Firewall logs to syslog-ng s...

Are you a member of the Splunk Community?