All, Thought there was a one stop shop command for this, but I can't find it. Basically I just want an alert when I see a drop in count of events. Say I have 1stddev of change in 15 minutes? index=* app=clamav | stats count ... View more

All, I have installed Splunk Enterprise Security (ES) and the Clam AV apps. Searching tag=malware tag=attack works, bringing up the expected event. I can even see it in the visualizations. But I am not seeing it the host itself showing as infected. Any idea why? Very new to ES so I am not sure where to start on troubleshooting this. ... View more

All, Playing a little game with security right now. The ask is that we have a robots.txt at https://mysplunk.mydomain.local:8443/robots.txt Is there anyway I can place a text file at the root of Splunk to present? ... View more

All, I am trying to tune performance on a set of data. Basically I have narrowed it down to search time extractions being the issue but I really don’t see any resource limits on the search heads that indicate that it’s working all that hard on the data set. That is, low CPU ~2% and yet I am waiting minutes for fields to extract. Any recommendation on how I might get more performance from this search? Here are my notes - My search index=akamai over 1 hour Fast Mode I get - slow but workable --9,525,499 events in 39.804 seconds However, this set of data normally needs the fields. So in smart mode --9,571,210 events in 243.244 seconds This means we're talking 4 minutes an hour, which not an acceptable performance for our user. As it's "smart mode" this implies it's a search tier issue. Correct? I went ahead and setup batch mode searh parallelization and here are my new results. Fast mode --9,440,668 events in 17.909 seconds Smart Mode --9,453,800 events in 134.911 second While this improvement is great. We're still looking at Smart mode being over 2 minutes per hour of data. We continue to need raw data searches to be more performant. Search time extractions are just taking too long. I was wondering if there are any ways to tell what extraction is taking too long? The data is well formed, cooked JSON data coming from a heavy forwarder, which pull data down from Akamai. Perhaps I need to convert some fields to index time? There are common ones like "site" which are usually used. So I went one step farther end upped it to 3 pipelines. Fast Mode --9,510,314 events in 15.77 seconds Smart Mode --9,498,542 events in 130.875 seconds Over all, we're still exceeding 2 minutes per hour of data search time for Akamai with extractions. I'd like to get that down closer to 1 minute, per hour of data. That is asking our user to wait 24 minutes for just a day of data. So some reading shows I might want to try INDEXED_EXTRACTIONS = json, so I applied that to my heavy forwarder. I went ahead and applied that to one of my two heavy forwarders that process the akamai data and let it bake for an hour. Fast Mode time went up to 17 seconds and Search Mode time went up to 160 seconds With the clear decrease in performance over all from indexed time extractions I went ahead and disabled that immediately. I am using the Splunk Add-on for Akamai from Splunkbase here for props.conf. https://splunkbase.splunk.com/app/3030/ Overall I am not seeing much in terms of CPU usage on the Search head. 1.9% - 3% CPU during the search. So I am not sure how to get the field extraction process on the search head to use all the idle resources. ... View more

All, Is there a way in Splunk to measure my total IO/thoughout on my indexers? Looking for a search or an app? thanks -Daniel ... View more

All, I have 20+ 8 CPU search heads. Going to be installing 24 CPUs search heads and retiring these old ones that are currently configured in a search head cluster. Basic plan is to add them to the pool, wait a day, then remove the old ones. Anything in there I am forgetting? Any special tuning to make Splunk realize the new search heads have more cores? ... View more

All, A bit concern for us lately is Splunk downtime. Search head clustering has been helpful, so now we're looking at the indexing tier.. Per my reading here: http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Takeapeeroffline Seems I can upgrade node by node without downtime as along as my search factor is high enough? Am I reading this correctly? ... View more

All, Not really a SunOS guy, so might be missing something fundamental. I wrote the script I need, and it runs fine as the user I expect it do. Splunk is root and running as root. Here I have a simple script that echos out some system information for our Splunk ES demo. #/usr/bin/bash ip=`/usr/sbin/ifconfig -a | grep -i inet | grep -i 10 | awk '{print $2}'` mac=`/usr/sbin/ifconfig -a | grep -i ether | head -n 1 | awk '{print $2}'` nt_host=`hostname` dns=`nslookup $ip | grep -i name | awk '{print $4}'` owner="user" owner_email="[email protected]" owner_DL="[email protected]" # Lat and long work - default lat=76.057837 long=-115.214195 # if sjr override the above with sjr lat and long datacenter=${nt_host:0:3} if [ $datacenter == "sjr" ] then lat=140.5622 long=121.9297 fi echo "ip=$ip, mac=$mac, nt_host=$nt_host, dns=$dns, owner=$owner, owner_email=$owner_email, owner_DL=$owner_DL, priority=high,lat=$lat,long=$long,city=\"Las Vegas\", country=US, bunit=operations, category=cardholder, pci_domain=\"trust|cardholder\", is_expected=true, should_timesync=true, should_update=true, requires_av=false" and here is the inputs.conf: [script://./bin/getinfo.sh] index=asset_discovery sourcetype=assets:SunOS interval=60 This script executes when run locally. I also verified the app runs fine on a LInux machine. I am getting other data from this forwarder. At this point I am out of ideas as to why this scripted input is failing. ... View more

All, I have the following props.conf / transforms.conf setup. Trying to set aside PCI into it's own set of indexers. However, redirection only works with a fully qualified hostname. Wildcards don't seem to work. Is there something I am missing? transforms.conf REGEX=. DEST_KEY=_TCP_ROUTING FORMAT=pci-group props.conf doesn't work [host::myvery.*] TRANSFORMS-routing=pciRoutingbeta doesn't work [(?::){0}myvery*] TRANSFORMS-routing=pciRoutingbeta doesn't work [(?::){0}myvery.*] TRANSFORMS-routing=pciRoutingbeta does work! [host::myverylongservername] TRANSFORMS-routing=pciRoutingbeta ... View more

All, I am setting up asset center in Splunk ES/PCI. The idea of an Asset priority is sorta vague. Is it left that way on purpose? For me to define? "Example: Must be one of unknown, informational, low, medium, high, or critical" ... View more

All, I have data flowing through a heavy forwarder. Security wants a SECOND heavy forwarder that they manage to SEDCMD out certain PII. Is it possible to reprocess already cooked data? ... View more

All, Can I disable token/security for the http event collector? We have an internal app which has a log via http option. But there is no option in the app to put a token. I am GUESSING it should work if we can disable the token. thanks -Daniel ... View more

All, I have a dozen+ inputs I am creating. I feel there there should be a smarter way of doing this. As you can see, I am naming the sourcetype after the log. [monitor:///opt/paidsearch/autopilot/logs/collateral.log] index=paidsearch sourcetype=paidsearch:collateral [monitor:///opt/paidsearch/autopilot/logs/partner.log] index=paidsearch sourcetype=paidsearch:partner ... View more

All, We need to pull data from a platform called "SEC", Simple Event Correlator into Splunk. Any one familiar with this? Have any apps or experience in this matter they can share? thanks -Daniel ... View more

All, I am trying to understand why Splunk it opening a file here. When I run LSOF I see Splunk looking at a rolled over file "/opt/jboss-6.1.0.Final/server/default/log/jboss.log.2016-09-29" splunkd 29966 root 48u REG 253,0 3381522238 5800576 /opt/jboss-6.1.0.Final/server/default/log/jboss.log.2016-09-29 [root@lvsp01cat001 default]# But we don't monitor this file at all. It's not in our inputs.conf [root@lSERVER default]# /opt/splunkforwarder/bin/splunk btool inputs list | grep -i jboss [monitor:///opt/jboss/server/default/log/jboss.access.log.*] index = jbossweb sourcetype = jbossweb_access [monitor:///opt/jboss/server/default/log/jboss.log] [root@SERVER default]# My theory is logrotate is rotating the file, the iNode is the same and Splunk it not releasing it. /opt/splunkforwarder/bin/splunk -version Splunk Universal Forwarder 6.3.3 (build f44afce176d0) I am on 6.3.3 but I have seen thie behavior on 6.4x as well. Any idea what is going on here? Getting hounded by our Java support guys saying this isn't "permitted of Splunk". ... View more

All, Say I query Splunk and get a list of 1000 users today. And tomomorrow I do the same thing and get 1002 users. How can I get the 2 new users appeared over night? Basically diff the results of two searches? ... View more

All, I am currently playing with some line breaking. But in order to test it I need to update my crcSalt, and restart Splunk. A little time consuming. Is there a faster way of doing this I am unaware of? ... View more