You have encryption enabled on your Cloudtrail logs. http://docs.aws.amazon.com/awscloudtrail/latest/userguide/encrypting-cloudtrail-log-files-with-aws-kms.html The boto library that the Splunk add on uses does not pass the correct version of the AWS signature required by SSE-enabled S3 buckets by default: https://forums.aws.amazon.com/thread.jspa?threadID=165286 You can, however force boto to use the correct version of the signature, see the section titled "Specifying Signature Version in Request Authentication" for Python boto sdk. http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html You'll need to add the following line to the boto config file. [s3] use-sigv4 = True The doc below lists your options for the boto.cfg file. I'd suggest either /etc/boto.cfg or the .boto file in the home directory of your Splunk user (the account you run splunk as). http://boto.cloudhackers.com/en/latest/boto_config_tut.html /etc/boto.cfg - for site-wide settings that all users on this machine will use (if profile is given) ~/.aws/credentials - for credentials shared between SDKs (if profile is given) ~/.boto - for user-specific settings ~/.aws/credentials - for credentials shared between SDKs ~/.boto - for user-specific settings ... View more

Yes you probably do need to add more storage. If you are writing ~2.5 GB to the default index, and you plan to retain data for 365 days, your storage requirements would be: 2.5 GB * 365 * 50% (on-disk size) = 456 GB You can use the Splunk sizing tool to help calculate your disk requirements: https://splunk-sizing.appspot.com/ If you are running Splunk v6.3 you can look at the DMC to get a better idea of how well data is compressing in your index, the time range represented by your index, and how often your buckets are freezing. http://docs.splunk.com/Documentation/Splunk/6.3.3/DMC/IndexingIndexesandvolumes You can also pretty easily verify how data is expiring by searching for index=_internal sourcetype=splunkd bucketmover will attempt to freeze These messages indicate that Splunk is deleting data (moving from cold to frozen). The log message will also indicate the reason why the data is rolling off; either due to aging out (frozenTimePeriodInSecs) or because of the storage limit (maxTotalDataSizeMB). ... View more

How about this? The result should be "/appliance/detail/v3.0/vendor/XXX/fridge". | gentimes start=-1 | eval uri="/appliance/detail/v3.0/vendor/6757dfs32/refrigerator/small" | rex mode=sed field=uri "s/(.*\/vendor\/)[^\/]+(\/.*)/\1XXX\2/" | table uri ... View more

So, you may need to set: syslogSourceType = <string> In your outputs.conf syslog stanza. The string value should match the sourcetype of your Cisco data, so that Splunk knows this is syslog data and doesn't need to add a timestamp/hostname to the beginning of the log entry. From http://docs.splunk.com/Documentation/Splunk/6.3.2/admin/Outputsconf: "Data which does not match the rules has a header, optionally a timestamp (if defined in 'timestampformat'), and a hostname added to the front of the event. This is how Splunk causes arbitrary log data to match syslog expectations." There is a Splunk wiki article that might help explain what is happening when your data is being processed and passed on to a syslog destination: https://wiki.splunk.com/Community:Test:How_Splunk_behaves_when_receiving_or_forwarding_udp_data It sounds like you are passing data directly to splunk via syslog. I prefer to have a syslog server (syslog-ng or rsyslog) setup to receive my syslog data and write to a file. Then I use a Splunk forwarder to read the files and forward them to my indexer. This also gives you the advantage of routing data directly via syslog-ng if you need to. There's a discussion of the pros/cons here: https://answers.splunk.com/answers/103295/pros-cons-of-using-syslog-ng-or-other-syslog-file-receiver-vs-direct-tcp-udp-514-to-splunk.html ... View more

Can you give us some information, ie, what is the configuration you are using to forward your data, and what kind of system is receiving it? ... View more

I'm not sure that you can modify that setting in the log.cfg file. You can easily change it directly in the code though. In SA-Eventgen/lib/eventgenconfig.py, change the value of backupCount on line 402: fileHandler = logging.handlers.RotatingFileHandler(os.environ['SPLUNK_HOME'] + '/var/log/splunk/eventgen.log', maxBytes=25000000, backupCount=5) ... View more

This search should combine the events from the two indexes together by their ID fields, then finds only entries where both id fields are present. (index=indexA plugin_id=*) OR (index=indexB id=*) | eval match_id=coalesce(id,plugin_id) | stats values(*) AS * by match_id | search plugin_id=* id=* ... View more

The fact that the stanzas are the same in both btool outputs points towards a bug, though I'm really surprised this wasn't seen before. I can tell you we do this on linux forwarders and it works fine. ... View more

What output do you get from running btool? With/without the default stanza? splunk cmd btool inputs list The output should be identical regardless of how you set the index; ie, your default stanza should be interpreted correctly and each input statement should show the "index = windesktops" setting, even though its not set explicitly. Do you have different input types, or are they all WinEventLog? You might try creating a monitor input and see if it behaves differently. ... View more

If myIDField is unique across all files, do you really need to join the two files together? source=file_A* OR source=file_B* | stats values(*) AS * by myIDField If the myIDField value is not unique, you could also extract the id number from the file. source=file_A* OR source=file_B* | rex field=source "_(?<file_id>\d+)" | stats values(*) AS * by myIDField, file_id ... View more

You should be able to specify a default stanza in an inputs.conf file like this: [default] index=windesktops [WinEventLog://Application] disabled = 0 ... rest of your stanzas ... ... View more

Let's cover both network related questions together: Splunk forwarders send data to Splunk servers via TCP. All communication from the forwarders are initiated from the forwarders. If you are using the deployment server to manage your config, your Splunk forwarder will initiate a TCP connection on port 8089 to the deployment server to update its configuration. To forward log data, your Splunk forwarder initiates a connection to your indexer(s) on port 9997. You can remap these ports if necessary. https://answers.splunk.com/answers/63646/what-port-does-the-forwarder-need-opened-to-the-indexers.html By permission, do you mean the user that the Splunk forwarder runs as? This will depend on if you are running your forwarders on unix or windows. Generally speaking, you can run as any user as long as that user has read/write access to the Splunk installation directory and has read access to the files you would like to index. This document goes into detail about Windows permissions to run Splunk (including your forwarders): http://docs.splunk.com/Documentation/Splunk/6.3.2/Installation/ChoosetheuserSplunkshouldrunas There is a similar document for Unix systems. http://docs.splunk.com/Documentation/Splunk/6.3.2/Installation/RunSplunkasadifferentornon-rootuser ... View more

Indexed extractions aren't available for modular inputs. That's why your settings worked for a file, but not the JMS input. You'll have to rely on search time json field extraction. This is also why your timestamp isn't parsing; since "stop" isn't seen as a field, the timestamp isn't extracted. Just use the normal timestamp extraction settings like TIME_PREFIX, and it should work fine. ... View more

That isn't my understanding of how it is supposed to work. If you have a scheduled search, and the search has no actions, the search artifacts will stick around for 2x the search period. For example, for an hourly search, the artifacts are deleted after 2 hours. For a daily search, the artifacts are deleted after 2 days. Check out this link, under the setting dispatch.ttl: http://docs.splunk.com/Documentation/Splunk/6.3.2/Admin/Savedsearchesconf ... View more

Try using the Timewrap app. You can use it to compare two time ranges by adding it your search. https://splunkbase.splunk.com/app/1645/ t_done=* earliest=-2w@w latest=@w | timechart avg(t_done) | timewrap w ... View more

Yes, if you have your search head set to forward to your indexers (which is a best practice), then the data will be stored on the indexers. Same goes for your forwarder. ... View more

Hey John, You might want to take a look at https://answers.splunk.com/answers/337992/maxsearches-limit-reached.html and https://answers.splunk.com/answers/297030/what-is-the-max-value-for-maxsearches-is-there-a-w.html Also, since you are running 6.3, you are using the new scheduler. So take a look at https://conf.splunk.com/session/2015/conf2015_PLucas_Splunk_SplunkEntWhatsNew_MakingTheMostOf.pdf. ... View more