Are you using this regex on the search bar with the rex command? If so, you have to use max_match. The default for max_match is 1. Your regex is matching more than one value in an event. Set max_match=0 for unlimited matches. ... View more

yes, you can set a Heavy Forwarder to index locally and forward data by setting IndexAndforward=true in your outputs.conf but this is NOT recommended behavior in a prod Splunk environment https://docs.splunk.com/Documentation/Splunk/7.2.4/Forwarding/Routeandfilterdatad ... View more

Sending syslog directly to Splunk is against best practice. You should send it to a syslog server like Rsyslog or Syslog-ng. Regardless, you would have to install certificates on the receiving host, otherwise handshake will fail. Check out the answer here: https://answers.splunk.com/answers/658055/setup-secure-encrypted-syslog.html But the splunkd.log would have any errors. /opt/splunk/var/log/splunk |grep -i error ... View more

Stagger your reports so they don't all run at midnight. Set them on cron schedules to run 11:50, 11:55, 12:05, etc. Also, make sure your searches are optimized. (i.e. you are specifiying your index in every search, not search All Time, not searching Real time, etc.) https://docs.splunk.com/Documentation/Splunk/7.2.4/Search/Quicktipsforoptimization ... View more

GMT and UTC are the same thing, are they not? ... View more

I downvoted this post because it is not true, per splunk docs ... View more

Are you writing the hostname to the directory where logs are being written in Kiwi? For example, C:\Program Files\Kiwi\firewall\HOSTNAME\ logfile.txt If so, you can set host_segment in your inputs.conf. In my example, your host_segment would be 4. So it would take the 4th entry from your path and make that the hostname ... View more

The OOTB Brute Force Behavior correlation rule with ES already does this. Why are you trying to create a custom one? ... View more

It sounds like the timestamp coming over is in UTC. Change your props to account for the difference. [source::/var/tmp/log/*] TZ = UTC ... View more

It is best practice to index _internal logs. Your Distributed Monitoring Console won't fully work without it and you won't really be able to remotely troubleshoot any problems on your UFs. If you are worried about the amount of data in the index, just adjust frozenTimePeriodinSecs on those indexes so you don't hang on to them for too long ... View more

I downvoted this post because op is asking about tuning a correlation search in es. not creating a brute force rule in core. ... View more

Eval is your friend. Try this: index=akamai_pi_prod message.reqHost=rpama message.status IN ("200","201","204","206","302","304","403","404","500","502") | eval Status=case(message.status == 200, "Success", message.status == 206, "Partial Content", message.status == 304, "Cache", message.status == 502, "Bad Gateway", true(), "Unknown") | stats count by message.reqHost,Status ... View more

Is the data routed through a syslog server or is it going to a network port open directly on Splunk? Can you share your inputs.conf stanza for your Bluecoat data? ... View more

Usually this is just an issue with your TaskServer port. Make sure whatever port you entered for TaskServer port when configuring DBconnect is open and available. ... View more

This is a really broad question. What is the data source? Is it coming from a Universal Forwarder? Syslog? API pull? If it is a UF, is it collected with a TA or is it via custom inputs? Assuming it is a file being monitored by a UF with an inputs.conf, then just adjust the index there. Set index=new index ... View more

No, adjusting the search window will not help. It will actually make it worse in this case. That correlation rule alerts when anomalous number hosts are detected with a new process. Local Processes tracker contains destination, first and last timeseen and process. If any data is returned, add it to the localprocesses_tracker file. Evaluate the time range and return values where the firsttime is between the evaluated time fields. Return a distinct count of destination grouped by process when the count is greater than 9. Is this a new deployment of ES? If so, the 'noise' should reduce overtime. ... View more

Cold path or cold volume? Is your hot/warm storage and cold storage on the same partition or do they each have their own? Judging by that app naming convention, it sounds like you had Professional Services help at some point. They should have set a parameter called maxVolumeDataSizeMB for that partition in indexes.conf. The max size should be set close to the total amount of storage available on that partition with a little bit of buffer. Essentially, when your cold storage reaches that point, it will begin rolling the oldest data to frozen. By default, rolling to Frozen just deletes the data. If you did not specify a coldToFrozenScript or coldToFrozenDir for Frozen data, then that is what would happen. I would not change your current cold directory. I would add a partition for Frozen data if you do not want data to be deleted, then just set the coldToFrozenDir. Or if you have no reason to retain the data, then just let Splunk roll the oldest stuff to frozen. ... View more

You can troubleshoot why your scripts are not working, but it is more than likely a permissions issue if you enabled inputs in your inputs.conf and you still do not see your data. You can do this: Navigate to $SPLUNK_HOME/etc/apps/Splunk_TA_nix/bin. Run sh --debug to run the script in debug mode. The debug output is saved in debug----. This file contains the command that was executed, and its output or the failure reason. Use this information to resolve the missing data issue. Also, for what it is worth, it is NOT recommended to run a Deployment Server and an Indexer on the same server. Especially a Windows box. ... View more

Those two indicators at the top look for 'action=allowed.' Your action is deferred so it does not match the search that the Key Indicators are looking for ... View more