Hey splunksters , -Just curious if anyone has had success getting secure syslog over tcp -port 6514 . The safenet applicance is supposed to send data to the indexer which is being treated like the "syslog" server. I have tried using my own certificates and carefully pointing the various inputs, web, and server.conf files LIKE THIS: https://wiki.splunk.com/Community:SplunkWeb_SSL_SelfSignedCert_NewRootCA AND LIKE THIS: https://community.splunk.com/t5/Getting-Data-In/How-to-configure-my-splunk-app-to-get-data-over-SSL/td-p/85793 -Through playing with the configuration stanzas, I am no longer getting any splunkd errors. -However, the INFO field (in splunkd ) provides these msg: IPv4 port 6514 is reserved for raw input (SSL) IPv4 port 6514 is reserved for splunk 2 splunk IPv4 port 6514 will negotiate s2s protocol level 4 creating raw acceptor for IPv4 port 6514 with SSL the server IS listening for port 6514, but wireshark does not show anything coming in or any flags for that port -So, I'm wondering if I need to allow client authentication?? - Do I have to use the Certificates from the safenet side instead? They have sent over 3 certificates ( KeySecure client certificate and PKI CA certificate/certificate chain ) If so, How do I do I import/install their certificates and apply them in the .confs Thanks!
... View more
If your deployment server is forwarding its internal logs to your indexing layer, you should be able to use a query like:
index=_internal host="Your_deployment_server_hostname" "/services/broker/phonehome/"
| stats max(_time) AS last_checkin_epoch by clientip
| eval now_epoch=now()
| eval time_since_last_checkin=now_epoch-last_checkin_epoch
| sort - time_since_last_checkin
... View more
This has been solved many times including:
Meta Woot!: https://splunkbase.splunk.com/app/2949/
Broken Hosts App for Splunk: https://splunkbase.splunk.com/app/3247/
Alerts for Splunk Admins ("ForwarderLevel" alerts): https://splunkbase.splunk.com/app/3796/
Splunk Security Essentials(https://docs.splunksecurityessentials.com/features/sse_data_availability/): https://splunkbase.splunk.com/app/3435/
Monitoring Console: https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
Deployment Server: https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarder_warnings
... View more
This is what i ended up doing. (obviously, you will have to create your own lookup like the one below this paragraph) And you may or may not be referencing a host and sourcetype blacklist like mine.....if not, just remove those lines...As you can see I'm filtering on percent changes, which is a threshold you can change or remove the where command altogether..I'm still working on the math, but for the most part i think its right.
| tstats latest(_indextime) as Latest where index=* by host sourcetype index
| search NOT
[ inputlookup sourcetype_blacklist.csv
| table sourcetype]
| lookup sourcetype_interval.csv sourcetype OUTPUT interval as intervals
| eval intervals=round(intervals/60/60,2)
| eval intervals=coalesce(intervals,0)
| eval current=now()
| eval Minimum_Age=round(((current-Latest)/60)/60,2)
| eval perc_change=((Minimum_Age-intervals)/Minimum_Age*100)
| where perc_change > 90
| rangemap field=Minimum_Age default=Critical Normal=0-0.5 Elevated=0.5-2 Warning=2-3
| eval stIDX=tostring(index) + " -- " + tostring(sourcetype)
| eval stINT=tostring(sourcetype) + " -- " + tostring(intervals)
| eval stLast=tostring(sourcetype) + " -- " + tostring(Minimum_Age)
| eval pcChange=tostring(sourcetype) + " -- " + tostring(perc_change)
| stats values(stIDX) as Index--Sourcetype list(Latest) as "Latest Event" list(Minimum_Age) as Minimum_Age list(range) as Threshold list(stINT) as Sourcetype--Interval list(stLast) as Sourcetype--HoursSinceLast list(pcChange) as Sourcetype--PercChange by host
| convert ctime("Latest Event") timeformat="%Y/%m/%d %H:%M"
| eventstats avg(Minimum_Age) as average by host
| eval average=round(average,2)
| rename Minimum_Age as "Hours Since Last Seen" average as "Avg Hours Since Last Seen" lintervals as ST_Interval
| sort "Latest Event"
| fields - "Avg Hours Since Last Seen"
| table host "Latest Event" Threshold Sourcetype--Interval Sourcetype--HoursSinceLast Sourcetype--PercChange
... View more