I used alert_logevent alert action to ingest events from custom alert action, you may try below : from future.moves.urllib.parse import urlencode from future.moves.urllib.request import urlopen, Request from future.moves.urllib.error import HTTPError, URLError from splunk.util import unicode def log_event(helper, event, source, sourcetype, host, index): if event is None: helper.log_error("ERROR No event provided\n") return False query = [('source', source), ('sourcetype', sourcetype), ('index', index)] if host: query.append(('host', host)) url = '%s/services/receivers/simple?%s' % (helper.settings['server_uri'], urlencode(query)) try: encoded_body = unicode(event).encode('utf-8') req = Request(url, encoded_body, {'Authorization': 'Splunk %s' % helper.settings['session_key']}) res = urlopen(req) if 200 <= res.code < 300: helper.log_debug("receiver endpoint responded with HTTP status=%d\n" % res.code) return True else: helper.log_error("receiver endpoint responded with HTTP status=%d\n" % res.code) return False except HTTPError as e: helper.log_error("Error sending receiver request: %s\n" % e) except URLError as e: helper.log_error("Error sending receiver request: %s\n" % e) except Exception as e: helper.log_error("Error %s\n" % e) return False def your_def_like_to_ingest_events(): success = log_event( helper, event=data_url_final, source=source, sourcetype=sourcetype, host=host, index=index ) if not success: sys.exit(2) Note: only one event per call to log_event will be ingested.  if you want to ingest multiple events then you need to call log_event multiple times.   ... View more

@vik Did you try my solution? ... View more

Congratulations to all the Splunk MVP members.  ... View more

Yes. ... View more

From UI: you can try dump command. choose the time range for which you want to export the logs. Make sure you have enough storage available in the server.  index=<index_name> | dump basefilename=dump_<index_name>   basefilename will be saved under $SPLUNK_HOME/var/run/splunk/dispatch/<search_job_id>/dump/basefilename_<some_number>_<hour>.raw.gz Note: Splunk generates 1 dump for 1 hour. if you have selected 24hours time range then you will see 24 or 25 files.  ... View more

Splunk will migrate and upgrade kvstore automatically if the instance is running in standard alone mode.  if Splunk instance is involved in any clustering ( search head , indexer ) then kvstore has to be manually migrated and upgraded. this applies to Deployer as well.  there is no need of changing storageEngineMigration = true if you follow below link. https://docs.splunk.com/Documentation/Splunk/9.0.2/Admin/MigrateKVstore?ref=hk In search head cluster, its 3 step process.   #verify if kvstore can be migrated by running below command splunk start-shcluster-migration kvstore -storageEngine wiredTiger -isDryRun true #initiate migrate /opt/splunk/bin/splunk start-shcluster-migration kvstore -storageEngine wiredTiger #wait until it compltes, you can check status splunk show shcluster-kvmigration-status #initiate upgrade splunk start-shcluster-upgrade kvstore -version 4.2 #wait until it completes, you can check status splunk show kvstore-status --verbose output once it's completed KVStore Migration Status: migrationStatus: kvstore_migration_completed Start Time: Fri Nov 4 08:50:16 2022 End Time: Fri Nov 4 08:55:06 2022 Migration ID: 1667537416 Peer Retries: 1 Cluster Percentage: 100 Storage Engine: wiredTiger Peer Status List: https://sh1:8089: migrationStatus: succeeded https://sh2:8089: migrationStatus: succeeded https://sh3:8089: migrationStatus: succeeded #finally remove unnecessary files Delete the following files from the $SPLUNK_HOME/bin directory: mongod-3.6 mongod-4.0 mongodump-3.6 mongorestore-3.6       ... View more

SQS notification can be created from both the ways. But Splunk add-on for aws is designed in way that has dependency to validate signature of the object. ... View more

CURRENT APPLICATION Splunk Add-on for AWS Version: 6.2.0 Build: 1658820915   Simple Architecture: Expected process by Splunk Add-on for AWS is as below: 1. Enable/configure cloud trail logs to s3 bucket and enable SNS topic  2. create standard SQS queue ( This queue will be used as Dead Letter queue in SQS queue creation of 3rd step) 3. Create standard SQS queue with below configuration Make sure your SQS queues have same configuration as below (except name, because your SQS will have different name) DLQ Configuration for SQS created in step 3 : Choose queue created in 2 step.  after creating the queue from step 3 . open queue created from step 3 and subscribe to SNS topic Modify SQS created in step 3 Policy   { "Version": "2008-10-17", "Id": "__default_policy_ID", "Statement": [ { "Sid": "__owner_statement", "Effect": "Allow", "Principal": "*", "Action": "SQS:*", "Resource": "<arn:aws:sqs:us-east-1:000000000000:this-sqs-queue>", "Condition": { "ArnEquals": { "aws:SourceArn": "<arn:aws:sns:us-east-1:000000000000:your-sns-topic" } } }   4. Above enabled SNS topic will be subscribed by SQS Below is the SNS Access policy : easiest way to get this policy created is to create SNS topic while enabling SNS in cloud trail log setup.      { "Version": "2008-10-17", "Id": "__default_policy_ID", "Statement": [ { "Sid": "__default_statement_ID", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": [ "SNS:GetTopicAttributes", "SNS:SetTopicAttributes", "SNS:AddPermission", "SNS:RemovePermission", "SNS:DeleteTopic", "SNS:Subscribe", "SNS:ListSubscriptionsByTopic", "SNS:Publish" ], "Resource": "<arn:aws:sns:us-east-1:0000000000:sns-topic>", "Condition": { "StringEquals": { "AWS:SourceOwner": "<account_id>" } } }, { "Sid": "AWSCloudTrailSNSPolicy20150319", "Effect": "Allow", "Principal": { "Service": "cloudtrail.amazonaws.com" }, "Action": "SNS:Publish", "Resource": "<arn:aws:sns:us-east-1:0000000000:sns-topic>", "Condition": { "StringEquals": { "AWS:SourceArn": "<arn:aws:cloudtrail:us-east-1:0000000000:trail/cloudtrail-events>" } } } ] }   5. Create IAM Policy ( Added permissions based on my best knowledge and keeping least privilege in mind)  using below: Note: don't forget to change SQS and S3 ARNs   { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "sqs:ListQueues", "s3:ListAllMyBuckets" ], "Resource": "*" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "sqs:DeleteMessage", "sqs:GetQueueUrl", "s3:GetLifecycleConfiguration", "s3:GetBucketTagging", "sqs:ReceiveMessage", "s3:GetBucketLogging", "sqs:SendMessage", "sqs:GetQueueAttributes", "s3:ListBucket", "s3:GetAccelerateConfiguration", "s3:GetObject", "s3:GetBucketCORS", "s3:GetBucketLocation" ], "Resource": [ "<arn:aws:sqs:us-east-1:00000000:sqs-name>", "<arn:aws:s3:::bucket_name>", "<arn:aws:s3:::bucket_name>/*>" ] } ] }​   6. Create a new role and choose above IAM policy while creating. 7. create a user 8. In the role trust relationships add below policy to allow user to assume this role.    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "<arn:aws:iam::00000000000:user/created_in_step7>" }, "Action": "sts:AssumeRole", "Condition": {} } ] }   Note: don't try to copy and paste the above policy. some characters are will not be copied properly. especially IAM policy where it gives s3 permission to resource object "buket_arn/*" ... View more

same in version 4.1.1 ... View more

what you are assuming is correct for hashing not for encyrpting and decrypting. ... View more

@jeffland Thanks for the solution. That's exactly what I was looking for.  https://docs.splunk.com/Documentation/Splunk/8.0.0/Viz/PanelreferenceforSimplifiedXML#input_.28form.29 ... View more

@suzakutheknight @HiMAT-Mode  appreciate if you post new question about AWS TA. The current question is all about ec2 and s3. ... View more

index=_internal host IN ("yourheavyforwarders") group=per_index_thruput | stats sum(kb) as kb by host | eval GB=round(kb/1024/1024,2) | fields - kb Note: if you want to exclude internal indexes which are starting with _(underscore), you can use below search index=_internal host IN ("yourheavyforwarders") group=per_index_thruput series!=_* | stats sum(kb) as kb by host | eval GB=round(kb/1024/1024,2) | fields - kb   ... View more

Yes, this is really pain job updating lookups manually on search head members.  At least lookups should be deployed from the deployer to search head members if the lookup doesn't exist on searh head members even if we use -preserve-lookups true  may be we need to create an idea for Splunk to consider this build a solution around this issue.  ... View more

No, Splunk doesn't support at least before or on 8.0.6 we observed that the below mentioned AWS service control policy (SCP) Sid was enforced at root organization level part of enhancing security and our current Splunk version is 8.0.6 which uses security credentials retrieved from IMDSv1. We have performed below actions to confirm that Splunk uses security credentials retrieved from IMDSv1: 1. Added below SCP Sid to AWS account where our ec2 instances hosted and running Splunk. Splunk started logging s3 “access denied” errors 2. listed directories of the configured S3 bucket from one of the ec2 instances was success when we made s3api call from CLI 3. to confirm CLI by default used security credentials retrieved from IMDSv2, we configured keys retrieved from IMDSv1 in CLI environment variables and made same request as above but this time we observed access denied errors. 4. Reverted the change performed in step 1 then Splunk stopped logging S3 access denied errors, this confirmed that Splunk uses security credentials retrieved from IMDSv1. ... View more

@the_wolverine  This answer is old. but its still valid as of today.  ... View more

@Georges1  If you have inputs.conf.spec then there should be inputs.conf under default directory. if your app doesn’t have default inputs.conf then also you can just create same stanza name like inputs.conf.spec in default inputs.conf and specify python version. ... View more

Below is endpoint where all checkpoints are saved for an app   curl -v https://127.0.0.1:8089/servicesNS/nobody/<applicationname>/storage/collections/data/<applicationname>_checkpointer/ -k -u admin:password   don't forget to change <applicationame> in the above url.  To call specific checkpoing just add checkpoint key at the end of the above URL as show below:   curl -v https://127.0.0.1:8089/servicesNS/nobody/<applicationname>/storage/collections/data/<applicationname>_checkpointer/<samplecheckpoint> -k -u admin:password   don't forget to change <applicationame> and <samplecheckpoint> in the above url.  To update checkpoint: curl -v -X "POST" https://127.0.0.1:8089/servicesNS/nobody/<applicationname>/storage/collections/data/<applicationname>_checkpointer/batch_save -d '[{"_key": "<samplecheckpoint>", "state": "{\"end_date\": \"2022-01-01T00:00:00.000\"}"}]' -H "Content-Type: application/json" -k -u admin:password reference: https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/kvstore/usetherestapitomanagekv/ be cautious with double quotes and escape characters used in -d parameter in the above curl statement.  ----------- I also tried calling same endpoint from GUI as below, for some reason GUI is returning 0 results.  | rest splunk_server=local /servicesNS/nobody/<applicationame>/storage/collections/data/<applicationame>_checkpointer/ ... View more

is there any way to forward events based on db input stanza.? I wanted to forward all db inputs which have same source type to third party and below is the configuration I have made on Heavy forwarder. outputs.conf [tcpout:thirdparty] server = server1:9997,server2:9997 props.conf [oracle:audit:db] TRANSFORMS-routing=route_db_thirdparty transforms.conf [route_db_thirdparty] REGEX=.* DEST_KEY=_TCP_ROUTING FORMAT=thirdparty   ... View more

@vijaysri  I haven't tested the one I have mentioned in my previous answer.  I had followed the below document to integrate mcas logs to Splunk and its working fine.  https://docs.microsoft.com/en-us/defender-cloud-apps/siem Note: Step 1: Set it up in the Defender for Cloud Apps portal In the point 5. you need to enter your HF/indexer/syslog IP and port.  if you run java jar on the host which you entered in step 1 and point 5 then you don't need to open any firewall connections, otherwise remember to open firewall connection from the host where you install java jar file and host you entered in step 1 and point 5.    ... View more