All Topics

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Topics

Hi, My sample code looks like below : Mon Mar 9 14:18:14 2020: Unknown trap (.1.1.1.1.1..1) received from hostname.abcd.com at: Value 0: hostname.abcd.com Value 1: 1.2.3.4 Value 2: 11.22.33.... See more...
Hi, My sample code looks like below : Mon Mar 9 14:18:14 2020: Unknown trap (.1.1.1.1.1..1) received from hostname.abcd.com at: Value 0: hostname.abcd.com Value 1: 1.2.3.4 Value 2: 11.22.33.44 Another sample value is Mon Mar 9 13:38:23 2020: Unknown trap (.1.2.3.4.5.6) received from 19.19.19.19 at: Value 0: 19.19.19.19 Value 1: 4.4.4.4.4 Value 2: 12.13.14.15 Value 0: give me either the IP or the hostname . I need to extract field called machine_name with the value of Value 0:
How to hide and remove Create New Dashboard button only for user Role.
I've installed the forwarder on Ubuntu and it did get the apps from the deployment server right after the install. But it does not get any updates - it cannot phone home. What the root cause is, I'm ... See more...
I've installed the forwarder on Ubuntu and it did get the apps from the deployment server right after the install. But it does not get any updates - it cannot phone home. What the root cause is, I'm still not yet sure, but for some reason, whenever I type a command on the forwarder that requires you to provide a valid username/password nothing happens. The cli does not show the request for the Splunk username: as it does on any of the Windows forwarders we have installed. I'm pretty sure I will solve the phone home issue if I can solve the issue with username/password not showing, but no luck yet doing that.
I have a TSV file im uploading into Splunk, I'd like to be able to group by a column in the file itself. So far I'm using the Splunk Source Type UI to create the custom source type, used the field... See more...
I have a TSV file im uploading into Splunk, I'd like to be able to group by a column in the file itself. So far I'm using the Splunk Source Type UI to create the custom source type, used the field tab to label the columns and I'm missing the step to group by a column. Can anyone please advise? Screen shot shows the setup below, I'd like to group by Session
Hi, I am working on a query where I need to join some events using a transaction command in Splunk. Below is my query where I am joining a particular web service request with a response using a "requ... See more...
Hi, I am working on a query where I need to join some events using a transaction command in Splunk. Below is my query where I am joining a particular web service request with a response using a "requestID" and then extracting the data. Now what happens is if the response contains an error, it will again try and may be next time it gets success. requestID remains same here. Right now this query is just using the 1st response event which it encounters and reported as error even if it is not. How do i extract the last of the responses so that i get the updated data whether it resulted in an error or success. index=temp ("SoapMessage" "GetCustomerRequest") OR ("SoapMessage" "GetCustomerResponse") | rex field=_raw ">(?[^<]+)<\/\w+?:requestID>" | transaction requestID startswith="GetCustomerRequest" endswith="GetCustomerResponse" keepevicted=true | search eventcount > 1
When I try to create a connection jdbc url returns the following error: [jcc] [t4] [10380] [11951] [4.26.14] Required property "b01avi12446653.ahe.pok.ibm.com" is unknown host. ERRORCODE = -4222, S... See more...
When I try to create a connection jdbc url returns the following error: [jcc] [t4] [10380] [11951] [4.26.14] Required property "b01avi12446653.ahe.pok.ibm.com" is unknown host. ERRORCODE = -4222, SQLSTATE = 08001 I already tried it with hostname and IP. The server ping both the ip and the hostname
Hi! I have worked for a while to make Splunk use TLS and PKI as much as possible. At present the system contains of version 8.0.1 components only. I have managed to get Splunk Indexer to require c... See more...
Hi! I have worked for a while to make Splunk use TLS and PKI as much as possible. At present the system contains of version 8.0.1 components only. I have managed to get Splunk Indexer to require client certificate from the UFs, and it seems to work. The Splunk serverCert is in a file containing the certificate, private key, issuer certificate and the root-CA certificate (the issuer of the issuer certificate). The trusted root-CA -certificates are in a separate text file. For this connection requiring requireClientCert = true works fine. With the web-UI things are not going as elegantly. My web.conf looks like this: [settings] enableSplunkWebSSL = 1 privKeyPath = etc/auth/splunkweb/splunk.pki.key serverCert = etc/auth/splunkweb/splunk.pki.txt requireClientCert = false sslVersions = tls1.2 loginBackgroundImageOption = none login_content = This is a <b>Test installation</b>. With the above, things are just fine. Changing requireClientCert to true breaks everything and the webGUI is not started. The splunkd.log gets populated with lines like: 03-09-2020 12:45:04.223 +0200 ERROR X509Verify - X509 certificate (CN=Root CA,O=X,C=Y) failed validation; error=19, reason="self signed certificate in certificate chain" I get exactly the same error message, if I connect using openSSL to the indexer port, where UFs connect: openssl s_client -connect splunk:9998 -state -prexit * Certificate chain 0 s:/C=Y/O=X/CN=Test Splunk Indexer i:/C=Y/O=X/CN=TestCA-1 1 s:/C=Y/O=X/CN=TestCA-1 i:/C=Y/O=X/CN=Root CA 2 s:/C=Y/O=X/CN=Root CA i:/C=Y/O=X/CN=Root CA * SSL-Session: Protocol : TLSv1.2 * Verify return code: 19 (self signed certificate in certificate chain) My point here is that the connection between UF and indexer still works fine. The question is: Is requireClientCertificate simply not supported in the web-GUI, or is there something in the documentation I have not understood correctly? If it is possible to require a certificate from the client (i.e. a web browser), is there a way to define the trusted CA-certificates and should any intermediate CA certifictes be included as well? Another thing I have been wondering about is certificate validation and CRLs. Is there a way to make Splunk actually validate the certificates it is presented? Best regards, Petri
Hi, Continuously getting Splunk forwarder service agent alerts in ticketing tool for every week. and then I tried to check-in Splunk site and it was showing as server was not shut down or rebooted... See more...
Hi, Continuously getting Splunk forwarder service agent alerts in ticketing tool for every week. and then I tried to check-in Splunk site and it was showing as server was not shut down or rebooted means Splunk agent able to get logs then I need to close that request. like that, getting too many requests. what might be the issue of whether servers are restarting or Splunk agent issue? I have checked in Splunkd logs also but no error information. for some servers, I don't have privileges to check the logs. please provide some error information as you know. which errors we can get for Splunk Universal Forwarder usually(send me some examples of errors). can someone please suggest to me what I can do. Thanks in Advance.
hello i'm running splunk with Kubernetese and Ansible from time to time im getting this error : [SPLUNKD] Error in 'inputlookup' command: External command based lookup 'kv_alerts_prod' i... See more...
hello i'm running splunk with Kubernetese and Ansible from time to time im getting this error : [SPLUNKD] Error in 'inputlookup' command: External command based lookup 'kv_alerts_prod' is not available because KV Store initialization has failed. Contact your system administrator. {"message":"{\"response\":{\"headers\":{\"date\":\"Mon, 09 Mar 2020 07:44:04 GMT\",\"expires\":\"Thu, 26 Oct 1978 00:00:00 GMT\",\"cache-control\":\"no-store, no-cache, must-revalidate, max-age=0\",\"content-type\":\"application/json; charset=UTF-8\",\"x-content-type-options\":\"nosniff\",\"content-length\":\"215\",\"vary\":\"Cookie, Authorization\",\"connection\":\"Close\",\"set-cookie\":[\"splunkd_8089=S8owSsAcljUIFXeya8Nhkk9y^cqA^qGsZi2mnFodHbZzb51KqkZIsqrtkEp1RVvwejUi1ADnoVtJaqV859dCuoZX^WkIKg6ZWDWM_h0Ks1lhSMRXKgpZ323DKC; Path=/; Secure; HttpOnly; Max-Age=3600; Expires=Mon, 09 Mar 2020 08:44:04 GMT\"],\"x-frame-options\":\"SAMEORIGIN\",\"server\":\"Splunkd\"},\"statusCode\":400},\"status\":400,\"data\":{\"messages\":[{\"type\":\"FATAL\",\"text\":\"Error in 'inputlookup' command: External command based lookup 'kv_alerts_prod' is not available because KV Store initialization has failed. Contact your system administrator.\"}]},\"error\":null}","level":"ERROR","logger":"argus:aviation-splunk-rest-apis:services:splunkService","timestamp":"2020-03-09T07:44:04.451Z"} {"message":"{\"response\":{\"headers\":{\"date\":\"Mon, 09 Mar 2020 07:44:04 GMT\",\"expires\":\"Thu, 26 Oct 1978 00:00:00 GMT\",\"cache-control\":\"no-store, no-cache, must-revalidate, max-age=0\",\"content-type\":\"application/json; charset=UTF-8\",\"x-content-type-options\":\"nosniff\",\"content-length\":\"215\",\"vary\":\"Cookie, Authorization\",\"connection\":\"Close\",\"set-cookie\":[\"splunkd_8089=S8owSsAcljUIFXeya8Nhkk9y^cqA^qGsZi2mnFodHbZzb51KqkZIsqrtkEp1RVvwejUi1ADnoVtJaqV859dCuoZX^WkIKg6ZWDWM_h0Ks1lhSMRXKgpZ323DKC; Path=/; Secure; HttpOnly; Max-Age=3600; Expires=Mon, 09 Mar 2020 08:44:04 GMT\"],\"x-frame-options\":\"SAMEORIGIN\",\"server\":\"Splunkd\"},\"statusCode\":400},\"status\":400,\"data\":{\"messages\":[{\"type\":\"FATAL\",\"text\":\"Error in 'inputlookup' command: External command based lookup 'kv_alerts_prod' is not available because KV Store initialization has failed. Contact your system administrator.\"}]},\"error\":null}","level":"ERROR","logger":"argus:aviation-splunk-rest-apis","timestamp":"2020-03-09T07:44:04.451Z"} KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details. removing mongod.lock fix the problem but it is happening again . im wondering if there is another way to solve it thanks !
I have set up alerts in Splunk and usually I hard-code the recipients email id in the TO field, and it works flawlessly. But in this case , I cannot hardcode the user email id in the alert's TO fiel... See more...
I have set up alerts in Splunk and usually I hard-code the recipients email id in the TO field, and it works flawlessly. But in this case , I cannot hardcode the user email id in the alert's TO field, because the user ID has to be extracted from the event (from the event that satisfies the alert condition). Example (sample event that will satisfy the alert query): 40.145.234.438 329x399740x1 PERSON1 [09/Mar/2020:05:29:23 -0400] "DELETE /rest/api/2/issue/TES1-2/butchers?username=PERSON2 HTTP/1.1" 204 - 40 "https://phutan-dev.mayhem.com/browse/RES1-2" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "1k35v6f" IF the word butcher is identified then the event should be picked (I can handle until this) and from the event extract PERSON1 and PERSON2 field and trigger email to these two PERSONS as PERSON1@mayhem.com. PERSON2@mayhem.com through the alert. I have extracted PERSON1 and PERSON2 from the event. I'm just looking to append @mayhem.com to them and trigger alert emails to these two persons only.
After the configuration of Okta SPlunk TA app I see the following error in _internal HTTPError: 401 Client Error: Unauthorized for url: https://XXXYYZZZ.com/api/v1/logs I verified the token u... See more...
After the configuration of Okta SPlunk TA app I see the following error in _internal HTTPError: 401 Client Error: Unauthorized for url: https://XXXYYZZZ.com/api/v1/logs I verified the token used is active .
we have created dashboard in splunk using tabs as per below URL and its perfectly working fine. https://www.splunk.com/en_us/blog/tips-and-tricks/making-a-dashboard-with-tabs-and-searches-that-run... See more...
we have created dashboard in splunk using tabs as per below URL and its perfectly working fine. https://www.splunk.com/en_us/blog/tips-and-tricks/making-a-dashboard-with-tabs-and-searches-that-run-when-clicked.html Dashboard was created in 7.0 version but recently splunk has been upgraded to 7.3. Since upgrade we have been facing issue with the tab focus. That means, when we click on any tab, a blue line is shown under the tab to know, user is on which tab but when user clicks on another tab, blue line remain on previous tab as well as comes on new tab. That means if user clicks on multiple tabs one by one then for each clicked tab there are blue lines which becomes confusing for user. Ideally, when user clicks on new tab, blue line(focus) should get removed from previous tab and should be always be on the latest clicked tab. Issue seems to be either with tabs.css or tabs.js but we are unable to identify it. Can some one look into these two files from the above link and suggest what can be modified to rectify this error. Thanks in advance!
Hi, I am trying to build an alert from the following query. The query collects the counters for memory usage, especially the free amount. It plots a time chart of the last 21 days and performs a p... See more...
Hi, I am trying to build an alert from the following query. The query collects the counters for memory usage, especially the free amount. It plots a time chart of the last 21 days and performs a prediction over the coming 14 days. The graph itself is perfect. It also shows in the prediction that in the next 14 days we run out of memory. index=xxxxx host=xxxxx source="Perfmon:Memory" counter="Available MBytes" | eval Value=(Value/1024) | timechart span=1d avg(Value) as "Available MBytes", latest(host) as host, latest(counter) as counter | lookup resource_thresholds.csv resource_name AS host, resource_metric AS counter OUTPUTNEW resource_threshold_warning,resource_threshold_critical | eval Warning=resource_threshold_warning | eval Critical=resource_threshold_critical | predict "Available MBytes" as Prediction future_timespan=14 | eval Prediction = round(Prediction,0) | fields - lower95(Prediction), upper95(Prediction) resource_threshold_warning resource_threshold_critical host counter I want to run this as a scheduled alert (email, MS Teams) every night and be informed when the prediction hits 0 or lower somewhere in the future 14 days in this case. For some reason I cannot seem to get my head around the logic here to trigger the alert. Any suggestions?
Is there a provision to create custom AppDynamics agent to monitor a custom data that has been aggregated manually from an application data.
SPL: "(index=3y OR index=3mon) (host=x OR host=y) name="RegisteredUserLog" actionType=egg pointGet=true (platform=0 OR platform=1) | eval earned_date=strftime(_time, "%Y-%m-%d") | stats coun... See more...
SPL: "(index=3y OR index=3mon) (host=x OR host=y) name="RegisteredUserLog" actionType=egg pointGet=true (platform=0 OR platform=1) | eval earned_date=strftime(_time, "%Y-%m-%d") | stats count by event_id earned_date | rename event_id as easy_id | table easy_id earned_date" Notes - The data I am seeing today is different from when i saw and exported same data before 1 moth providing same date range. - To give you idea, I am seeing 20K less results as compared to 1L events before one month for exact SPL and exact time range. - Retention of index is not issue - Date range is not issue Please help Thanks
we have created dashboard in splunk using tabs as per below URL and its perfectly working fine. https://www.splunk.com/en_us/blog/tips-and-tricks/making-a-dashboard-with-tabs-and-searches-that-run... See more...
we have created dashboard in splunk using tabs as per below URL and its perfectly working fine. https://www.splunk.com/en_us/blog/tips-and-tricks/making-a-dashboard-with-tabs-and-searches-that-run-when-clicked.html Dashboard was created in 7.0 version but recently splunk has been upgraded to 7.3. Since upgrade we have been facing issue with the tab focus. That means, when we click on any tab, a blue line is shown under the tab to know, user is on which tab but when user clicks on another tab, blue line remain on previous tab as well as comes on new tab. That means if user clicks on multiple tabs one by one then for each clicked tab there are blue lines which becomes confusing for user. Ideally, when user clicks on new tab, blue line(focus) should get removed from previous tab and should be always be on the latest clicked tab. Issue seems to be either with tabs.css or tabs.js but we are unable to identify it. Can some one look into these two files from the above link and suggest what can be modified to rectify this error. Thanks in advance!
Notes - Our retention policy is 3 years for that abc index. - When I exported the result of that query before 1 month, I was able to see that particular data - Today when I run exact same query... See more...
Notes - Our retention policy is 3 years for that abc index. - When I exported the result of that query before 1 month, I was able to see that particular data - Today when I run exact same query, I can see some missing data. - To give you the detail, today I am seeing approx 20K less events out of 1L events. - The date range is exact same
How to enable just one tag name from the CIM model? eg. I just want to use network tag from Inventory model. But the data model gives error saying other tags names are not included.
when i input related email address into the dashboard, it shows the error message "command="sendemail", 'rootCAPath' while sending mail to:" how to solve?
i used the following command index=ABC | stats values(L) AS USER i need the output like below user usercount Ramesh 54 Somesh 12 Kamlesh 27