I found follow logs in _audit logs.  The user who run this search cannot access internal logs, so I assume the underline part is added by Splunk system.  Could anyboda explain follow 2 questions? What does the underline part mean? what does the field _cd mean? search='search (index=* OR index=_*) _time>=1661000447 _time<1661000460 host="XXX" source="XXX" | eval _DBID = replace(_cd, "(\d+):\d+", "\1") | eval _OFFSET = replace(_cd, "\d+:(\d+)", "\1")'] ... View more

I can get cloudwatch log and description successfull but not S3 buckets. My splunk EC2 has full access right to the s3 buckets but they are not listed in the Add-on setting page. I tried to get the s3 buckets from Add data, but I got internal logs like: "The last data ingestion itertion hasn't been completed yet" I attcheched the internal logs, hope soemone can help. ---------------internal logs----------------------------------------------2021-01-18 18:57:59,217 level=ERROR pid=11508 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:index_data:91 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="Failed to collect data through generic S3." start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066" Traceback (most recent call last):   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 86, in index_data     self._do_index_data()   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 107, in _do_index_data     self.collect_data()   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 153, in collect_data     self._discover_keys(index_store)   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 224, in _discover_keys     bucket = self._get_bucket(credentials)   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 370, in _get_bucket     bucket = conn.get_bucket(self._config[asc.bucket_name])   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/s3/connection.py", line 509, in get_bucket     return self.head_bucket(bucket_name, headers=headers)   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/s3/connection.py", line 528, in head_bucket     response = self.make_request('HEAD', bucket_name, headers=headers)   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/common/boto2_s3_patch.py", line 12, in wrapper     response = func(*args, **kwargs)   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/s3/connection.py", line 671, in make_request     retry_handler=retry_handler   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/connection.py", line 1084, in make_request     retry_handler=retry_handler)   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/connection.py", line 1043, in _mexe     raise ex   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/connection.py", line 956, in _mexe     request.body, request.headers)   File "/opt/splunk/lib/python3.7/http/client.py", line 1244, in request     self._send_request(method, url, body, headers, encode_chunked)   File "/opt/splunk/lib/python3.7/http/client.py", line 1290, in _send_request     self.endheaders(body, encode_chunked=encode_chunked)   File "/opt/splunk/lib/python3.7/http/client.py", line 1239, in endheaders     self._send_output(message_body, encode_chunked=encode_chunked)   File "/opt/splunk/lib/python3.7/http/client.py", line 1026, in _send_output     self.send(msg)   File "/opt/splunk/lib/python3.7/http/client.py", line 966, in send     self.connect()   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/boto/https_connection.py", line 119, in connect     sock = socket.create_connection((self.host, self.port), self.timeout)   File "/opt/splunk/lib/python3.7/socket.py", line 727, in create_connection     raise err   File "/opt/splunk/lib/python3.7/socket.py", line 716, in create_connection     sock.connect(sa) socket.timeout: timed out 2021-01-18 18:57:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:57:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:56:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:56:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:55:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:55:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:54:53,444 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:54:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:53:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:53:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:52:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:52:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:51:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:51:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:50:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:50:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:49:53,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:49:23,442 level=INFO pid=11508 tid=Thread-13 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:96 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test" | message="The last data ingestion iteration hasn't been completed yet." 2021-01-18 18:48:53,545 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_get_bucket:365 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066", phase="discover_key" | message="Create new S3 connection." 2021-01-18 18:48:53,545 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.common.aws_credentials pos=aws_credentials.py:load:163 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066", phase="discover_key" | message="load credentials succeed" arn="arn:aws:sts::052086164386:assumed-role/splunk_ec2_access/i-000c869a7adf815b0" expiration="2021-01-18 15:23:20+00:00" 2021-01-18 18:48:53,444 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.common.aws_credentials pos=aws_credentials.py:_load_source_credentials:195 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066", phase="discover_key" | message="fetch ec2 instance credentials" 2021-01-18 18:48:53,444 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.common.aws_credentials pos=aws_credentials.py:load:156 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066", phase="discover_key" | message="begin loading credentials" aws_account="splunk_ec2_access" aws_iam_role=None 2021-01-18 18:48:53,444 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_discover_keys:220 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066", phase="discover_key" | message="Start of discovering S3 keys." 2021-01-18 18:48:53,443 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:collect_data:143 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066" | message="Start processing" last_modified="2020-12-17T07:30:59.000Z" latest_scanned="" 2021-01-18 18:48:53,442 level=INFO pid=11508 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:105 | datainput="s3_test_122401" bucket_name="aws-sb-dev-ia0767-splunk-test", start_time=1610963333 job_uid="96ba7b5d-6dff-4d81-bc6b-b4581d4d3066" | message="Start processing."   ... View more

We have a file sever which generates about 7G windows Event Log a day. Windows Event Log is rotated as soon as the size reach to 200MB. We want to use splunk UF to get the logs, but we have follow concern: Is it possible that splunk UF cannot get the log right before the rotation happened ? (we don't know how UF handle event logs, we just assume UF might not get the one right before the rotation before it is moved to backup so fast) We only need to know what happen in the general situation but not in the case such like UF service is down or Indexer server is down.) ... View more

I know both of the two settings can help me to index the whole file, What the difference between the two? Is there some thing one can do but the other cannot? ... View more

Can UF be restart via REST API? What other things can be done to UF via REST API? ... View more

We want to run linux command via splunk web to linux servers in which UF is installed. For example, top, ps. I found there are some build-in scripts such like ps.sh in Splunk Add-on for Unix and Linux. I wonder if there is any method to use theses build-in scripts to run custom search command via splunk web? I know we can install Splunk add-on in linux UF and use [script:xxxx] stanza to check result of linux commands, however, we want to run command to get real-time result. ... View more

Is there any risk to monitor .sh or .bat files? ... View more

My inputs.conf is as follow: [batch://C:\Splunk\2.txt] index = netiq move_policy = sinkhole sourcetype = shinsei_db_audit_utf8 [monitor://C:\Splunk\log_SME*.log] disabled = false followTail = 0 ignoreOlderThan = 100d index = netiq sourcetype = shinsei_common_shift_jis With this inputs.conf, the batch stanza object 2.txt is indexed twice every time. If I remove the whole monitor part, the 2.txt is indexed once. What is the reason of it being indexed twice? ... View more