All Topics

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Topics

Hello,   We upgraded Microsoft Azure Add on for Splunk to the latest version 3.2.0 After the upgrade, we started seeing the following errors: From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/... See more...
Hello,   We upgraded Microsoft Azure Add on for Splunk to the latest version 3.2.0 After the upgrade, we started seeing the following errors: From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: "Failed to get password of realm=%s, user=%s." % (self._realm, user) From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/aob_py3/solnlib/utils.py", line 148, in wrapper From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: . From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: WARNING:root:Run function: get_password failed: Traceback (most recent call last): From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: solnlib.credentials.CredentialNotExistException: Failed to get password of realm=__REST_CREDENTIAL__#TA-MS-AAD#configs/conf-ta_ms_aad_settings, user=proxy.   I tried to add again the credentials and re-create the inputs, but still getting them.   We are getting the logs, but I'm not sure if this errors is impacting us/if we are getting all the logs or how should we correct it.   Thank you,   Andreea
Hi,   I 'm new to Splunk, but I need some answers pretty fast. We are invited to insource Infrastructure monitoring and control from a high secure environment. As we are outside customers domain, o... See more...
Hi,   I 'm new to Splunk, but I need some answers pretty fast. We are invited to insource Infrastructure monitoring and control from a high secure environment. As we are outside customers domain, obviously the dashboard runs on servers outside customers infrastructure. Of course there needs to be communications between agents running in the infrastructure and the dashboard to upload events and monitoring data.  However,  it is absolutely a requirement from customer there is NO traffic from the dashboad to the agents on his infrastructure. Upload of data is no problem, but any packet downstream will be blocked. Even "keep alive" traffic. Is anyone experienced to give me an answer on this?   Thanks, Wim
Hi, We are having bunch of HFs in our environment from HFs we have confusion from which HF is getting the data, so to find easily we have to right stanza in props.conf in defaults/local. So based on... See more...
Hi, We are having bunch of HFs in our environment from HFs we have confusion from which HF is getting the data, so to find easily we have to right stanza in props.conf in defaults/local. So based on this what is the stanza i can right Example fields: splunk_HF                                 indextime  
Hello, I am trying to timechart two event types ONLY: heartbeat and start.  However, every event in our Splunk is also mapped as nix-all-logs and few other events by the system admin.  Attached are... See more...
Hello, I am trying to timechart two event types ONLY: heartbeat and start.  However, every event in our Splunk is also mapped as nix-all-logs and few other events by the system admin.  Attached are screenshots.  How can I timechart these 2 event types only.
Hello everyone, I found due DMC console, that splunk stopped get logs into _introspection index. I open search like this "index=_introspection sourcetype=splunk_resource_usage component=PerProcess ... See more...
Hello everyone, I found due DMC console, that splunk stopped get logs into _introspection index. I open search like this "index=_introspection sourcetype=splunk_resource_usage component=PerProcess host=*" and see that there were events, but they stopped. Can anybody help me with this problem?
I am forwarding some json files from a splunk forwarder on linux, example file below: { "dateTime" : "04/11/2021 08:22:30", "functionName" : "ZAUTOPSRALL", "userId" : "sanchez", "issueCategory" ... See more...
I am forwarding some json files from a splunk forwarder on linux, example file below: { "dateTime" : "04/11/2021 08:22:30", "functionName" : "ZAUTOPSRALL", "userId" : "sanchez", "issueCategory" : "PSR", "issueType" : "HDRUNKNOWN", "issueSummary" : "PSR File Processing â\u0080\u0093 Cannot match to original file", "issueDescription" : "The received PSR file "PSR_CBD174.PAIN001_DTLRJCT3.xml" refers to an unknown original file.\n\nPSR file\nName: PSR_CBD174.PAIN001_DTLRJCT3.xml\nCreated: 2021-10-08T12:09:43+01:00\nMessage ID: LBG/0000000027834/003\n\nReference to original file\nMessage ID: MSGID/PAIN001/20210913T100930/1\nStatus: RJCT\nControl sum: 38965.82\nNumber of transactions: 86", "exceptionType" : null, "notificationId" : null, "timeStamp" : 1636014150661056 } Its not being indexed, i found the following errors for this fle in the splunkd.log   I ran the json through a json checker and it was valid so not sure why splunk is complaining.  Any help would be much apreciated. 11-05-2021 15:48:57.625 +0000 ERROR JsonLineBreaker [10224113 structuredparsing] - JSON StreamId:14224088848725967690 had parsing error:Unexpected character while parsing backslash escape: 'x' - data_source="/sanchez/instances/beta/log/splunk/splunk_1636014150661056_19399032.json", data_host="pbasalsldw002", data_sourcetype="_json" 11-05-2021 15:48:57.625 +0000 ERROR JsonLineBreaker [10224113 structuredparsing] - JSON StreamId:14224088848725967690 had parsing error:Unexpected character: ':' - data_source="/sanchez/instances/beta/log/splunk/splunk_1636014150661056_19399032.json", data_host="pbasalsldw002", data_sourcetype="_json" 11-05-2021 15:48:57.625 +0000 ERROR JsonLineBreaker [10224113 structuredparsing] - JSON StreamId:14224088848725967690 had parsing error:Unexpected character: ':' - data_source="/sanchez/instances/beta/log/splunk/splunk_1636014150661056_19399032.json", data_host="pbasalsldw002", data_sourcetype="_json" 11-05-2021 15:48:57.625 +0000 ERROR JsonLineBreaker [10224113 structuredparsing] - JSON StreamId:14224088848725967690 had parsing error:Unexpected character: ':' - data_source="/sanchez/instances/beta/log/splunk/splunk_1636014150661056_19399032.json", data_host="pbasalsldw002", data_sourcetype="_json" 11-05-2021 15:48:57.625 +0000 ERROR JsonLineBreaker [10224113 structuredparsing] - JSON StreamId:14224088848725967690 had parsing error:Unexpected character in string: '\0A' - data_source="/sanchez/instances/beta/log/splunk/splunk_1636014150661056_19399032.json", data_host="pbasalsldw002", data_sourcetype="_json"
Hi All, I am using Website Monitoring in one of our HF.But whenever i run sourcetype=web_ping query in the search bar, splunk PID's increases suddently and it stops the splunk service on HF. Please ... See more...
Hi All, I am using Website Monitoring in one of our HF.But whenever i run sourcetype=web_ping query in the search bar, splunk PID's increases suddently and it stops the splunk service on HF. Please suggest me where am i going wrong/Help me to fix the issue We are monitoring around 114 URL's with below sample of inputs.conf [web_ping://SOMAN] interval = 2m title = SOMAN url = http://sapsoman.www.com:5030/startPage user_agent = Splunk Website Monitoring (+https://splunkbase.splunk.com/app/1493/) configuration = default website_monitoring.conf [default] max_response_body_length = 1000 proxy_port = 312 proxy_server = proxy.conexus.svc.local proxy_type = http thread_limit = 100 Error: ERROR [618cf90fac7eff7b2b1290] config:146 - [HTTP 401] Client is not authenticated Traceback (most recent call last): File "/opt/app/splunk/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/config.py", line 144, in getServerZoneInfoNoMem return times.getServerZoneinfo() File "/opt/app/splunk/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/times.py", line 163, in getServerZoneinfo serverStatus, serverResp = splunk.rest.simpleRequest('/search/timeparser/tz', sessionKey=sessionKey) File "/opt/app/splunk/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 553, in simpleRequest raise splunk.AuthenticationFailed splunk.AuthenticationFailed: [HTTP 401] Client is not authenticated   @LukeMurphey @Anonymous 
We tried to install splunk 8.1.0 and after untarring the file tried to start splunk both as root and splunk user via /opt/splunk/bin/splunk start  Error comes up as execve: Operation not permitted ... See more...
We tried to install splunk 8.1.0 and after untarring the file tried to start splunk both as root and splunk user via /opt/splunk/bin/splunk start  Error comes up as execve: Operation not permitted while running command /opt/splunk/bin/splunkd Any urgent help is appreciated
Hi I have an issue that Splunk might be help to solve it. Here is scenario: Need to find unusual send and receive patterns in huge log file, here is the example:   00:00:01.000     S-001 00:00:... See more...
Hi I have an issue that Splunk might be help to solve it. Here is scenario: Need to find unusual send and receive patterns in huge log file, here is the example:   00:00:01.000     S-001 00:00:01.000     S-002 00:00:01.000     S-003 00:00:01.000     S-004 00:00:01.000     S-005 00:00:01.000     R-005 00:00:01.000     S-006 00:00:01.000     R-006 00:00:01.000     S-007 00:00:01.000     S-008 00:00:01.000     R-008 00:00:01.000     R-007 00:00:01.000     S-009 00:00:01.000     S-010 00:00:01.000     S-011 00:00:01.000     S-012 00:00:01.000     S-013 00:00:01.000     R-009 00:00:01.000     R-010 00:00:01.000     R-011 00:00:01.000     R-012 00:00:01.000     R-013 00:00:01.000     S-014 00:00:01.000     R-014 00:00:01.000     R-001 00:00:01.000     R-002 00:00:01.000     R-003 00:00:01.000     R-004   red line need to detect and show on chart. FYI1: Duration is not good way to find them because some of them occurred at the exact time. FYI2: ids are different not in order as i write above like this 98734543 or 53434444   any idea? Thanks,
Hi, I have to run python script as an alert action. My Splunk is on windows. I tried my script running like this and its working. Its very basic hello world script. C:\Program Files\Splunk\bin... See more...
Hi, I have to run python script as an alert action. My Splunk is on windows. I tried my script running like this and its working. Its very basic hello world script. C:\Program Files\Splunk\bin>splunk cmd python hello_world.py This message will be displayed on the screen. commands.conf   [hello_world] filename = hello_world.py I have placed commands.conf in C:\Program Files\Splunk\etc\apps\search\local and C:\Program Files\Splunk\etc\system\local when I am trying running this script from command line  its not working. | script python hello_world  OR | script hello_world Error Message:  Error in 'script' command: Cannot find program 'hello_world' or script 'hello_world'. Not sure why its not be able to find the script. I have placed it to multiple location. $SPLUNK_HOME$\etc\apps\search\bin\scripts\hello_world.py $SPLUNK_HOME$\bin\hello_world.py   (from command line it take this script) My ultimate goal is to run this script as an alert action. but I dont think there is option to run python script. I have option as run a script but seems like that is only for shell script. Thanks  
When I upgrade from Splunk 7.3.9 to 8.0.0 or 8.1, it failed with below message. "The program can't start because LIBEAY32.dll is missing from your computer. Try reinstalling the program to fix this ... See more...
When I upgrade from Splunk 7.3.9 to 8.0.0 or 8.1, it failed with below message. "The program can't start because LIBEAY32.dll is missing from your computer. Try reinstalling the program to fix this problem" "The program can't start because SSLEAY32.dll is missing from your computer. Try reinstalling the program to fix this problem"        
Hi, For registration of devices on splunk mobile instance we used to register by Splunk Cloud Gateway. But after upgrading splunk to 8.1 version, We have to register the devices through Splunk Secur... See more...
Hi, For registration of devices on splunk mobile instance we used to register by Splunk Cloud Gateway. But after upgrading splunk to 8.1 version, We have to register the devices through Splunk Secure Gateway.  Splunk Cloud Gateway is no longer valid on new version of Splunk. But now I am unable to register my device on Splunk Mobile instance using Splunk Secure Gateway.  I am getting Error 503 while registering. I have also copied the data from Cloud gateway to Secure gateway. Please feel free to provide your input on this problem.
Hello Experts,  We are trying to integrate Sailpoint with Splunk. We used the required add-on and all the necessary information for API however, after putting all the required information we are get... See more...
Hello Experts,  We are trying to integrate Sailpoint with Splunk. We used the required add-on and all the necessary information for API however, after putting all the required information we are getting the certificate error that stops the complete integration. Below are some of the sample error logs of Sailpoint integration.    File "/data/splunk/etc/apps/Splunk_TA_sailpoint/bin/splunk_ta_sailpoint/aob_py3/requests/api.py", line 60, in request     return session.request(method=method, url=url, **kwargs)   File "/data/splunk/etc/apps/Splunk_TA_sailpoint/bin/splunk_ta_sailpoint/aob_py3/requests/sessions.py", line 533, in request     resp = self.send(prep, **send_kwargs)   File "/data/splunk/etc/apps/Splunk_TA_sailpoint/bin/splunk_ta_sailpoint/aob_py3/requests/sessions.py", line 646, in send     r = adapter.send(request, **kwargs)   File "/data/splunk/etc/apps/Splunk_TA_sailpoint/bin/splunk_ta_sailpoint/aob_py3/requests/adapters.py", line 514, in send     raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='#hostname', port=8443): Max retries exceeded with url: /identityiq/oauth2/token (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1106)'))) Can someone please provide some input on the same so that we can proceed with the integration.  Thanks in advance  
11-09-2021 07:21:11.662 +0000 ERROR ExecProcessor [19962 ExecProcessor] - Invalid user admin, provided in passAuth argument, attempted to execute command /opt/splunk/bin/python3.7 /opt/splunk/etc/app... See more...
11-09-2021 07:21:11.662 +0000 ERROR ExecProcessor [19962 ExecProcessor] - Invalid user admin, provided in passAuth argument, attempted to execute command /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/python_upgrade_readiness_app/bin/pura_send_email.py
Hello all,   I am trying to extract the below highlighted fields, but the extractions at time is failing to get the required values, can you please help me get this working. 1) 537654 High 2021.11... See more...
Hello all,   I am trying to extract the below highlighted fields, but the extractions at time is failing to get the required values, can you please help me get this working. 1) 537654 High 2021.11.10 10:53:50 RDS_Failure_notification01 prd-Server2 127.0.0.1 sns.event EventSource : db-instance IdentifierLink : https://console.aws.amazon.com SourceId : prd-Server2 EventId : http://docs.aws.amazon.com EventMessage : DB instance restarted TopicArn : arn:aws:sns:ap-northeast-1:123456789:Lambda-PRD-Server1-SSS 2) 536465 High 2021.11.09 23:07:33 Server just booted [prd-Server1] prd-Server1 127.0.0.1 Server Status 00:04:44 3) 536438 High 2021.11.09 23:01:02 App Proxy: Utilization of unreachable poller processes over 80% prd-Server3 127.0.0.1 Utilization of unreachable poller data collector processes, in % 100 % 4) 448232 Average 2021.11.09 09:56:02 App Proxy: Utilization of unreachable poller processes over 70% prd-Server4 127.0.0.1 Utilization of unreachable poller data collector processes, in % 100 %   BOLD - Field1 Underlined -Field2 Strikethrough - Field3   @ITWhisperer @javiergn @richgalloway  Please have a look at this.   Thank you
Hey There,  Below I have a field in where ABC > 2500 cuz the value is actually 2800. So then If ABC>than 2500 add 1 day to the Human_readable field. I have already created the logic to adding 1 day ... See more...
Hey There,  Below I have a field in where ABC > 2500 cuz the value is actually 2800. So then If ABC>than 2500 add 1 day to the Human_readable field. I have already created the logic to adding 1 day to the Human_readable field.... Question now is how can I write the logic for it in a nested loop? So If ABC>2500 add 1 day to human readable.  This is my logic that I have thus far: | eval Then_Set=if(ABC>2500,strftime(strptime(Human_readable,"%B %d, %Y") +86400, "%B %d, %Y") This is what I have so far: | makeresults | eval ABC="2800", DEF="3", GHI="5" | eval rel_Time="11102021" | eval Epoch_Time=strpTime(rel_Time,"%m%d%Y") | eval Human_readable=strfTime(Epoch_Time, "%B %d, %Y") | eval Service=if(ABC>2500, "Send Alert", "No Alert") | eval Add_1Day=strftime(strptime(Human_readable,"%B %d, %Y") +86400, "%B %d, %Y") | eval Then_Set=if(ABC>2500,strftime(strptime(Human_readable,"%B %d, %Y") +86400, "%B %d, %Y") | table Service Epoch_Time Human_readable Add_1Day Then_Set
I have Splunk 7.3.6 with ES 6.0.2 on an on-prem Linux VM. I have an EC2 instance already setup with Splunk Core 8.1.5 where I want to migrate the ES app. Looking at various docs like Migrate from st... See more...
I have Splunk 7.3.6 with ES 6.0.2 on an on-prem Linux VM. I have an EC2 instance already setup with Splunk Core 8.1.5 where I want to migrate the ES app. Looking at various docs like Migrate from standalone searchheads and How to migrate, First doc is more about migrating from a standalone search head to an SHC, where it suggests to only migrate /etc/apps and /etc/users directory, however in the 2nd doc, which seems more closely relevant to what I want to achieve, it states, first I should copy over entire $SPLUNK_HOME directory on new system and then install Splunk on top of that. Not sure which one to follow. Also, incase of 2nd doc, I have done the opposite, I have installed Splunk first and now looking to copy existing ES SH's $SPLUNK_HOME, on top of that, but dont know if it would work ? Any suggestion ideas thoughts ?
Hi All, I have recently upgraded Splunk HF from 7.3.x to 8.1.2 and also the Cisco eStreamer (Encore) app from 3.6.x to 4.8.1. Both upgrades went fine and cisco:estreamer:data logs were coming in fine... See more...
Hi All, I have recently upgraded Splunk HF from 7.3.x to 8.1.2 and also the Cisco eStreamer (Encore) app from 3.6.x to 4.8.1. Both upgrades went fine and cisco:estreamer:data logs were coming in fine till 1.5 hours post-upgrade after which logs stopped coming in. The file  estreamer.log in /opt/splunk/etc/apps/TA-eStreamer/bin/encore doest show any ERROR ( INFO     Running. 3500 handled; average rate 4.86 ev/sec;). Also, I'm able to see logs populating in /opt/splunk/etc/apps/TA-eStreamer/data. However, it appears logs are not getting updated in cisco:estreamer:data sourcetype. There are other log sources relayed from the HF to cloud which do not have any issues (ruled out any network connectivity issues between HF and splunkcloud). Has anyone else seen similar issues?
Hi SMEs,   Greeting, i am seeking help to configure splunk to start at boot while SELinux is in enforcing mode. We are running with latest version 8.2.0  Many thanks in advance
I have a stacked bar chart.  The user wanted dark colors (which I did using the code on the bottom) .  However, the labels barely show on the bars.  I would like to change the font to white.  I have ... See more...
I have a stacked bar chart.  The user wanted dark colors (which I did using the code on the bottom) .  However, the labels barely show on the bars.  I would like to change the font to white.  I have googled with no luck and tried a few option changes with no luck.   <option name=“charting.fontColor”>“#FFFFFF”</option> <option name=“charting.backgroundColor”>{“#FFFFFF”}</option> <option name=“charting.backgroundColor”>#FFFFFF</option> <option name=“charting.fieldColors”>{“% Achievement”: #407294, “% Misses”: #A7090F}</option> Please tell me there is an easy solution for what seems to be a simple fix..  Thanks!