topic Re: modsecurity / Source doesn't show up in Getting Data In

modsecurity / Source doesn't show up

thierryit — Mon, 28 Sep 2020 16:16:05 GMT

Hi,

Running both Splunk server and Splunkforwarder on V6.0.2.
Both machine (web server and Splunk server) have their FW off.
After an "netstat -a" on both machine, I can see that there is a TCP connection established between my web server (port TCP 56xxx) and my Splunk server (port TCP 9997).
My inputs.conf is:

[monitor:///var/log/apache2/modsec_audit.log]
disabled = false
host = name_of_my_server
index = main
sourcetype = modsec_audit

On my Splunk server when going to: Search & Reporting/Search/Data Summary I only see one source (udp:514 -> my firewall) nothing else.
On hosts, I can see only my firewall .....

If I add in my inputs.conf one of my apache2 log, as example access.log, it will work like a charm ...
But not for my modsecurity log file .....

Any ideas ?

Thx

Re: modsecurity / Source doesn't show up

martin_mueller — Fri, 28 Mar 2014 17:36:02 GMT

Check the input's status on the forwarder: http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

Re: modsecurity / Source doesn't show up

thierryit — Mon, 28 Sep 2020 16:16:08 GMT

For full status, visit:
https://127.0.0.1:8089/services/admin/inputstatus/TailingProcessor:FileStatus

Updated: Fri Mar 28 19:42:32 2014 (took 0.0 sec)
Have seen 2 dirs. (+0)
Finished with 19 tracked files. (+0)

Currently reading 4 files.
some open files (showing up to 5):
/opt/splunk/var/log/splunk/audit.log (100%)
/opt/splunk/var/log/splunk/web_access.log (100%)
/opt/splunk/var/log/splunk/metrics.log (100%)
/opt/splunk/var/log/splunk/splunkd_access.log (100%)

Ignoring 0 items.

Hum ....

Re: modsecurity / Source doesn't show up

martin_mueller — Fri, 28 Mar 2014 17:44:56 GMT

Is that on the forwarder?

Re: modsecurity / Source doesn't show up

thierryit — Fri, 28 Mar 2014 17:52:13 GMT

No on the server .... I have understood between line to do it on the forwarder 🙂
I do it now.

Re: modsecurity / Source doesn't show up

thierryit — Fri, 28 Mar 2014 18:43:14 GMT

There is no Python interpreter included with Splunkforwarder ... And I cannot use the one provided with the system.

Re: modsecurity / Source doesn't show up

martin_mueller — Fri, 28 Mar 2014 18:51:19 GMT

You can call that REST endpoint manually.

Re: modsecurity / Source doesn't show up

thierryit — Fri, 28 Mar 2014 18:52:57 GMT

I am not a developer ... I do not understand your answer ... Sorry.

Re: modsecurity / Source doesn't show up

martin_mueller — Fri, 28 Mar 2014 18:54:23 GMT

As suggested by the script run on the server, go to https://127.0.0.1:8089/services/admin/inputstatus/TailingProcessor:FileStatus (replace 127.0.0.1 with the forwarder's host).

Re: modsecurity / Source doesn't show up

thierryit — Fri, 28 Mar 2014 19:14:42 GMT

splunk ~ # https://192.168.1.xx:8089/services/admin/inputstatus/TailingProcessor:FileStatus
-bash: https://192.168.1.xx:8089/services/admin/inputstatus/TailingProcessor:FileStatus: Aucun fichier ou dossier de ce type (No file or folder of this type)

Re: modsecurity / Source doesn't show up

martin_mueller — Fri, 28 Mar 2014 19:17:02 GMT

...in an https-capable client, such as your browser.

Re: modsecurity / Source doesn't show up

thierryit — Sat, 29 Mar 2014 03:42:44 GMT

A part of the answer, seems to be too big:
app canlist 1 canwrite 1 modifiable 0 owner system perms
read
* write
removable 0 sharing system eai:attributes
optionalFields
requiredFields
wildcardFields
inputs
/opt/splunkforwarder/var/log/splunk/audit.log
file position 50835 file size 50835 parent $SPLUNKHOME/var/log/splunk/splunkd.log percent 100.00 type finished reading /opt/splunkforwarder/var/log/splunk/btool.log
file position 0 file size 0 parent $SPLUNKHOME/var/log/splunk/splunkd.log percent 100 type finished reading /opt/splunkforwarder/var/log/splunk/licenseaudit.log

Re: modsecurity / Source doesn't show up

thierryit — Sat, 29 Mar 2014 03:43:58 GMT

file position 0 file size 0 parent $SPLUNKHOME/var/log/splunk/splunkd.log percent 100 type finished reading /opt/splunkforwarder/var/log/splunk/licenseusage.log
file position 0 file size 0 parent $SPLUNKHOME/var/log/splunk/splunkd.log percent 100 type finished reading /opt/splunkforwarder/var/log/splunk/metrics.log file position 1144937 file size 1144937 parent $SPLUNKHOME/var/log/splunk/splunkd.log percent 100.00 type finished
reading /opt/splunkforwarder/var/log/splunk/scheduler.log
file position 0 file size 0 parent $SPLUNKHOME/var/log/splunk/splunkd.log percent 100 type finished

Re: modsecurity / Source doesn't show up

martin_mueller — Sat, 29 Mar 2014 10:10:04 GMT

That's utterly unreadable, but it seems to me as if it only lists Splunk's own internal log files - so it's not even trying to read your log.

Run this from the CLI of the forwarder:

/opt/splunkforwarder/bin/splunk cmd btool inputs list monitor

Re: modsecurity / Source doesn't show up

thierryit — Mon, 28 Sep 2020 16:16:16 GMT

[monitor:///opt/splunkforwarder/etc/splunk.version]
_TCP_ROUTING = *
_rcvbuf = 1572864
host = server_hostname
index = _internal
sourcetype = splunk_version
[monitor:///opt/splunkforwarder/var/log/splunk]
_rcvbuf = 1572864
host = server_hostname
index = _internal
[monitor:///opt/splunkforwarder/var/log/splunk/splunkd.log]
_TCP_ROUTING = *
_rcvbuf = 1572864
host = server_hostname
index = _internal
[monitor:///var/log/apache2/modsec_audit.log]
_rcvbuf = 1572864
crcSalt =
disabled = false
host = toto.domain.org
index = main
sourcetype = modsec_audit

With or without crcSalt, same pb.

Re: modsecurity / Source doesn't show up

martin_mueller — Sat, 29 Mar 2014 13:23:51 GMT

Okay, is that file readable by the user running the forwarder?

Re: modsecurity / Source doesn't show up

thierryit — Sat, 29 Mar 2014 13:31:58 GMT

sudo ps auxxx |grep splunk*
root 1247 0.8 0.7 161860 32520 ? Sl 15:18 0:06 splunkd -p 8089 start
root 1251 0.0 0.0 49116 2884 ? Ss 15:18 0:00 [splunkd pid=1247] splunkd -p 8089 start [process-runner]

-rw-r--r-- 1 root adm 8528077 Mar 29 14:52 modsec_audit.log

Thx

Re: modsecurity / Source doesn't show up

martin_mueller — Sat, 29 Mar 2014 16:15:32 GMT

No entries in the _internal log files from that host?

Re: modsecurity / Source doesn't show up

thierryit — Sat, 29 Mar 2014 16:18:42 GMT

Can you be more precise ? What entry ? Where ?
Thx for your help even during the week end 🙂

Re: modsecurity / Source doesn't show up

martin_mueller — Sat, 29 Mar 2014 16:22:59 GMT

Run a search on the indexer, something like this:

index=_internal host=yourforwarderhost modsec_audit.log