All Topics

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Topics

03 Mar 2022 10:08:18,188 GMT ERROR [dbdiNotificationService,ServiceManagement] {} - Caught Runtime exception at service dbdiNotificationService java.lang.IllegalArgumentException: No enum constant co... See more...
03 Mar 2022 10:08:18,188 GMT ERROR [dbdiNotificationService,ServiceManagement] {} - Caught Runtime exception at service dbdiNotificationService java.lang.IllegalArgumentException: No enum constant com.db.fx4capi.Fx4cApiLocal.TradeProcessingStatus.TRADE_STATUS_CANCELLED at java.lang.Enum.valueOf(Enum.java:238) ~[?:1.8.0_311] at com.db.fx4capi.Fx4cApiLocal$TradeProcessingStatus.valueOf(Fx4cApiLocal.java:10) ~[trade-22.1.1-8.jar:?] at com.db.fx4cash.trade.step.GetTradeReferenceAndStatusStep.step(GetTradeReferenceAndStatusStep.java:24) ~[step-22.1.1-8.jar:?] at com.db.servicemanagement.TransactionDispatchService.executeIteration(TransactionDispatchService.java:275) [servicemanagement-22.1.1-8.jar:?] at com.db.servicemanagement.TransactionDispatchService.startDispatch(TransactionDispatchService.java:673) [servicemanagement-22.1.1-8.jar:?] at com.db.servicemanagement.TransactionDispatchService.run(TransactionDispatchService.java:91) [servicemanagement-22.1.1-8.jar:?] at com.db.servicemanagement.ServiceThread.run(ServiceThread.java:36) [servicemanagement-22.1.1-8.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_311]     ---------------------------------------------------------------------------------------------------------------------------- in above string i need to capture string in bold ,  basically whatever comes after first line ERROR would like to capture  using below command ,  index=app_events_fx4cash_uk_prod source=*STPManager-servicemanagement.20220303-100818.log* | rex field=_raw "^[^\-\n]*\-\s+(?P<Error>.$)" |table error   am getting blank record, please help 
hi again after upgrading our 26 linux universal forwarders from 7.x to 8.2.5, one of them will not run anymore. it immediately shuts down itself after start. splunkd.log shows nothing special ... See more...
hi again after upgrading our 26 linux universal forwarders from 7.x to 8.2.5, one of them will not run anymore. it immediately shuts down itself after start. splunkd.log shows nothing special for me. here some lines: 03-30-2022 13:03:03.587 +0200 INFO WatchedFile [110115 tailreader0] - File too small to check seek crc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunk_instrumentation_cloud.log'. 03-30-2022 13:03:04.440 +0200 INFO AutoLoadBalancedConnectionStrategy [110109 TcpOutEloop] - Connected to idx=172.17.1.91:9997, pset=0, reuse=0. using ACK. 03-30-2022 13:03:05.997 +0200 INFO loader [110063 HTTPDispatch] - Shutdown HTTPDispatchThread 03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - Shutting down splunkd 03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - shutting down level "ShutdownLevel_Begin" 03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - shutting down level "ShutdownLevel_NoahHealthReport" any ideas? thanks...
Hello,  I am trying to do the following: I need to add all the times in ELAPSED and give the result in the Total Time Elapsed row. I also am trying to remove the sort_field and Total from my table.... See more...
Hello,  I am trying to do the following: I need to add all the times in ELAPSED and give the result in the Total Time Elapsed row. I also am trying to remove the sort_field and Total from my table. I have tried to use the stats, count and addtotal commands, but I am not finding the right way to use them. Can anyone help? Regards, Celine BATCH_NAME ELAPSED STAGE START_TIME sort_field Total _raw _time MCO/COB.INITIALISE 00:00:00 Application 20:18:34 1 1 20220321,A000,MCO/COB.INITIALISE,COB.EXECUTE.API,20:18:34,20:18:34,00:00:00 2022-03-21T20:18:34.200+0200 MCO/SYSTEM.SECURITIES 00:00:00 System Wide 20:35:46 2 2 20220321,S001,MCO/SYSTEM.SECURITIES,SC.EOD.SUB.ACC.CHG.POST,20:35:46,20:35:46,00:00:00 2022-03-21T20:35:46.200+0200 MCO/DATE.CHANGE 00:00:00 Start of Day 21:30:26 4 4 20220321,D000,MCO/DATE.CHANGE,B.DATE.CHANGE,21:30:26,21:30:26,00:00:00 2022-03-21T21:30:26.200+0200 MCO/RESET.CO.STATUS 00:00:00 Online 21:34:47 5 5 20220321,O000,MCO/RESET.CO.STATUS,RESET.CO.STATUS,21:34:47,21:34:47,00:00:00 2022-03-21T21:34:47.200+0200 MCO/BATCH.DATE.RESET 00:00:00 End of COB 21:35:59 6 6 20220321,O999,MCO/BATCH.DATE.RESET,BATCH.DATE.RESET,21:35:59,21:35:59,00:00:00 2022-03-21T21:35:59.200+0200 Total Time Elapsed  XX:XX:XX     18 18    
Hi Experts, I want to know regarding license usage dashboard for Splunk APM , wherein we can view license consumtion details and analytics. Thanks.
Hello. I'm trying to view Splunk configuration, but getting a very odd error: splunk@test1:/> /opt/splunk/bin/splunk show config authentication Splunk username: admin Password: Can't create dire... See more...
Hello. I'm trying to view Splunk configuration, but getting a very odd error: splunk@test1:/> /opt/splunk/bin/splunk show config authentication Splunk username: admin Password: Can't create directory "/opt/splunk/splunk/.splunk": No such file or directory  Changing dir to /opt/splunk/bin/ and running command from it also doen't help. Doesn't matter if I run the command under splunk user (this user owns files in /opt/splunk) or with sudo. This Splunk instance was upgraded from 7.0.0 to 8.1.5 and then to 8.2.5, so maybe it affected somehow. How can I fix this?
Hello, Recently we have started working with AppDynamics as we are taking it over from another team. There have been a few HealthRules already created and we are trying to understand them all. The... See more...
Hello, Recently we have started working with AppDynamics as we are taking it over from another team. There have been a few HealthRules already created and we are trying to understand them all. There is one that we are not sure about and it seems to generate a handful number of alerts: App Starts's value was not within baseline-based calculated value by x standard deviation(s) for x times in the last x minutes What does exactly App Starts means here? The application failed to start? Couldn't find such information in the AppDynamics docs and when following the link via alert email, it just opens the Mobile Application dashboard and I don't see any connection. Is it possible to check more details regarding this alert? Thank you in advance!
Hi, Is Apache camel framework is supported in AppDynamics? ^ Post edited by @Ryan.Paredez to move contents into the body of the post. Please make sure to include a title and a body in your disc... See more...
Hi, Is Apache camel framework is supported in AppDynamics? ^ Post edited by @Ryan.Paredez to move contents into the body of the post. Please make sure to include a title and a body in your discussion post. 
Hi splunk experts, Can anyone elaborate this below event and tell me why this event is getting triggered? the user name in this event has left the organization and we removed his access and transfe... See more...
Hi splunk experts, Can anyone elaborate this below event and tell me why this event is getting triggered? the user name in this event has left the organization and we removed his access and transferred the knowledge objects to other person also but we are getting his name in the below event. and please help me how to avoid this type type of alerts also. 127.0.0.1 - **User name*** [30/Mar/2022:09:29:54.891 +0000] "POST /servicesNS/nobody/search/saved/searches/Single%20User%20Failed%20Attempt/notify?trigger.condition_state=1 HTTP/1.1" 200 1933 "-" "Splunk/8.1.0 (Linux 4.15.0-1023-azure; arch=x86_64)" - 2ms   this event is getting displayed when we search by using the query: index=_internal sourcetype= splunkd_access user=*.   thanks in advance.
  Hello everyone,    We recently upgraded our Splunk Enterprise from 7.x to 8.1.7.2 and we noticed some changes when we search on sourcetype. Some of our sourcetypes can contain different... See more...
  Hello everyone,    We recently upgraded our Splunk Enterprise from 7.x to 8.1.7.2 and we noticed some changes when we search on sourcetype. Some of our sourcetypes can contain different types of data. Let's say we have data A stored in sourcetype SRCT with some fields corresponding to this type of data: field1_A, field2_A. And data B is also stored in sourcetype SRCT with its own fields: field1_B, field2_B. If we do a simple search:   index=index sourcetype=SRCT field1_A=value1 | table * In Splunk Enterprise 7.x, the table only shows the fields that concern data A, that is to say, field1_A, field2_A. field1_A field2_A value1 value2 Now, since the upgrade, the table shows all the fields from data A and data B, even if the data we are looking for is a data A. In this case, field1_B and field2_B are empty. field1_A field2_A field1_B field2_B value1 value2     BUT: if we do not specify sourcetype=SRCT in our search:     index=index field1_A=value1 | table * It only shows the fields field1_A and field2_A. field1_A field2_A value1 value2 It's as if, searching on sourcetype makes it retrieve all the fields that this sourcetype has encountered without discarding null fields. The same can be noticed when we search on source.   Does anyone have seen this before?  What can explain this change of behavior?
Hi. I'm brand new to splunk.  I've created a table report that shows counts of columns: Time (Hour) Result: Good Result: Bad Result: Ugly After the Index and source type, my query has     |... See more...
Hi. I'm brand new to splunk.  I've created a table report that shows counts of columns: Time (Hour) Result: Good Result: Bad Result: Ugly After the Index and source type, my query has     |chart count over _time span=hour by data.result What do I need to add to query to calculate the Total of Good, Bad, Ugly and then use that to add a column with the percentage of Total Result to the table.  Appreciate your help.
Hello there, I am new to Splunk. I had configured my universal forwarder in order to send data to the indexer. The universal forwarder is a Linux server and running the command netstat -an | grep 9... See more...
Hello there, I am new to Splunk. I had configured my universal forwarder in order to send data to the indexer. The universal forwarder is a Linux server and running the command netstat -an | grep 9997 I can see that tcp packages are being sent to the indexer, but the status is 'TIME_WAIT'. While my indexer is a windows 10 desktop, I have added permission to accept tcp and ICMP packages, but still, I can't find the data I want on the splunk instance installed on the indexer (or any other data concerning the forwarder).  My question is then, what can I do in order to receive the packages on the indexer please? PS: I have another indexer which is a Linux desktop, and it works just fine, I can find the forwarder data. PS': Here is the link for the tutorial I've been following in order to configure the splunk instences I'm using Using the Universal Forwarder to gather data | Splunk Operational Intelligence Cookbook (packtpub.com) Any help would be appreciated ! Regards,
Can I use Splunk REST API to get data from Splunk Cloud? Can someone give me some examples? I have read some documents on how to get data from Splunk Cloud but not really understand? Any help is hi... See more...
Can I use Splunk REST API to get data from Splunk Cloud? Can someone give me some examples? I have read some documents on how to get data from Splunk Cloud but not really understand? Any help is highly appreciated.
index=* namespace="dk1017-j" sourcetype="kube:container:kafka-clickhouse-snapshot-writer" message="*Snapshot event published*" AND message="*dbI-WAR*" AND message="*2022-03-29*" AND message="*" |fiel... See more...
index=* namespace="dk1017-j" sourcetype="kube:container:kafka-clickhouse-snapshot-writer" message="*Snapshot event published*" AND message="*dbI-WAR*" AND message="*2022-03-29*" AND message="*" |fields message |rex field=_raw "\s+date=(?<BusDate>\d{4}-\d{2}-\d{2})" |rex field=_raw "sourceSystem=(?<Source>[^,]*)" |rex field=_raw "entityType=(?<Entity>\w+)" |rex field=_raw "\"timestamp\":\"(?<Time>\d{4}-\d{2}-\d{2}[T]\d{2}:\d{2})" |sort Time desc |dedup Time |table Source, BusDate, Entity, Time     output of above command is below ------------------- Source BusDate Entity Time dbI-WAR 2022-03-29 BOOKING 2022-03-30T02:05 dbI-WAR 2022-03-29 DATA_QUALITY_REPORTS 2022-03-30T02:04 dbI-WAR 2022-03-29 DATA_QUALITY_ENTITIES 2022-03-30T02:03 dbI-WAR 2022-03-29 COMBINED_POSITION_NORMALIZED 2022-03-30T01:40 dbI-WAR 2022-03-29 COMBINED_POSITION 2022-03-30T01:36 dbI-WAR 2022-03-29 DATA_QUALITY_ENTITIES 2022-03-30T01:35 dbI-WAR 2022-03-29 DEPOSIT 2022-03-30T01:34 dbI-WAR 2022-03-29 DATA_QUALITY_REPORTS 2022-03-30T01:33 dbI-WAR 2022-03-29 DATA_QUALITY_ENTITIES 2022-03-30T00:43 dbI-WAR 2022-03-29 NEXT_BUSINESS_DAYS 2022-03-29T23:49     question - i would like to line chart  x axis should be date as date is same for all  &  y axis should be time 
I have a blacklist.csv file that looks like the following,   IP domain 1.0.136.29 # 2018-11-12, node-1lp.pool-1-0.dynamic.totbb.net, THA, 2 1.0.136.215 # 2018-10-06, node-... See more...
I have a blacklist.csv file that looks like the following,   IP domain 1.0.136.29 # 2018-11-12, node-1lp.pool-1-0.dynamic.totbb.net, THA, 2 1.0.136.215 # 2018-10-06, node-1qv.pool-1-0.dynamic.totbb.net, THA, 2   i want to scan my syslog events and see if any IP address match the IPs in this blacklist. a syslog event looks like this: Feb 7 03:32:31 Router kernel: [WAN_IN-3009-A]IN=eth0 OUT=eth1.100 MAC=18:e8:29:44:40:ac:00:1d:aa:a2:78:axxxxxx src=128.199.123.0 DST=192.168.100.207 LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=52834 DF PROTO=TCP SPT=38290 DPT=8194 WINDOW=29200 RES=0x00 SYN URGP=0 MARK=0x64800000 i already set up a lookup definition and lookup table, but i dont know exactly how to put up a search to display if a syslog even matches an IP in the blacklist.csv
I am looking forward to creating a table for system metrics values like "cpu", "memory" and "swap", now if run the below search it works, but it will get all hosts available while I want my search to... See more...
I am looking forward to creating a table for system metrics values like "cpu", "memory" and "swap", now if run the below search it works, but it will get all hosts available while I want my search to be specific to some hosts. 1) | mstats max(cpu.idle) AS "CPU_IDLE" avg(memory.free) as "MEMORY_FREE" avg(swap.used) as "SWAP_USED" WHERE `sai_metrics_indexes` earliest=-30m@m by host | eval "cpu_active"=100-cpu_idle | fillnull value=0 | foreach CPU* MEM* SWAP* [| eval "<<FIELD>>"=round('<<FIELD>>',2)] 2)Where if i try like below then i get an error as i am beginner and not getting the right approach to get it . | mstats max(cpu.idle) AS "CPU_IDLE" avg(memory.free) as "MEMORY_FREE" avg(swap.used) as "SWAP_USED" WHERE `sai_metrics_indexes` earliest=-30m@m by ("host"="host1.example.com" OR "host"="host2.example.com" OR "host"="host3.example.com" ) | eval "cpu_active"=100-cpu_idle | fillnull value=0 | foreach CPU* MEM* SWAP* [| eval "<<FIELD>>"=round('<<FIELD>>',2)] 1) working screen shot  2)  trial but not working  Would appreciate to get any help or direction on this.
Hello, I have a alert dump data Horizon.csv having important columns like below: Alert   GRN   Type .... PNC/hz-hfp-l-abc[MAXRUN]      PNC/hz-hfp-l-abc   Autosys Filesystem[ivp1234.xy.com] [9... See more...
Hello, I have a alert dump data Horizon.csv having important columns like below: Alert   GRN   Type .... PNC/hz-hfp-l-abc[MAXRUN]      PNC/hz-hfp-l-abc   Autosys Filesystem[ivp1234.xy.com] [91>90]   ivp1234.xy.com   Application Filesystem[ivp1244.xy.com] [91>90]   ivp1244.xy.com   Application p.start.script.pl is down     Process down   API which I need to merge with Mapping.csv but on a condition that if Type=Autosys then merge on GRN else merge on Type details of Mapping.csv Type   Name   Module    Header Autosys   hz-hfp-l-abc   HF   EOD Job Application   <blank>   Eng   Server alerts API   <blank>   LF   Service alerts    I need output as  Alert   GRN   Type   Module   Header PNC/hz-hfp-l-abc[MAXRUN]      PNC/hz-hfp-l-abc   Autosys   HF   EOD Job Filesystem[ivp1234.xy.com] [91>90]   ivp1234.xy.com   Application   Eng   Server alerts Filesystem[ivp1244.xy.com] [91>90]   ivp1244.xy.com   Application   Eng   Server alerts p.start.script.pl is down     Process down   API   LF Service alerts
We have a distributed architecture  Search head cluster with 6 hosts across 3 data centres Index cluster with 6 index peers and 1 index master  Forwarders on all servers in environment  - web t... See more...
We have a distributed architecture  Search head cluster with 6 hosts across 3 data centres Index cluster with 6 index peers and 1 index master  Forwarders on all servers in environment  - web tier, app tier, load balancer tier Few months back , web tier stopped sending - log stopped coming to splunk ; but other tiers are are working  When checked the activity on web-tier , there was a patching happened and splunkd was restarted -after that forwarding stopped in web-tier  But splunkd process came up fine - still running in those  And observed below WARN messages started coming exactly same time  [ See the highlighted in red starting from 10 seconds it grows ] WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group index_peers has been blocked for 10 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. ---------- ------ +0000 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group index-peers has been blocked for 9725460 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. +0000 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group index-peers has been blocked for 9725470 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. =============================================================================  Why we picked this WARN message may be cause - as same happened in other tier recently  load lancer tier stopped stopped forwarding recently. Above WARN started showing  same time onwards - starting with  "blocked for 10 seconds  "   splunk forwarder is running fine in all these  App tier still working -sending data , so indexers are fine  not disk space or memory issue in any of these  No config changes done any where ( inputs or outputs conf or any file that matter) -its same , just that stopped working suddenly    What could have caused this sudden stopping of forwarding ? Splunk Enterprise Version:7.2.1Build:be11b2c46e23    
Hi All,  I need to filter my search based on the condition if the values of 2 fields are equal or not.  The 2 fields in question are actor.alernateID  and src_user_email and both fields are visible ... See more...
Hi All,  I need to filter my search based on the condition if the values of 2 fields are equal or not.  The 2 fields in question are actor.alernateID  and src_user_email and both fields are visible in the same event. For example:  Raw data shows value of actor.alternateID is   anand.pandey@company.com   Likewise, Raw data shows value or src_user_email is also same:  anand.pandey@company.com If i run the following search,  the value of the field match  comes out to be "No match" .  Why is eval showing them to be not a match if both field values are the same ?      index=xxx sourcetype=xxxx .... | eval match=if(actor.alternateId=src_user_email,"Match","No Match")       Likewise, instead  if i  use the where condition instead of eval  ,  this shows NO results to display;   meaning  even the where clause thinks both fields are different .       |where src_user_email = actor.alternateID     The same is happening for other email IDs and other fields even though their values are same. What am i doing wrong here? How to compare fields then?  Both are strings.
Hello,   I am trying to add a background cover for the panels within a dashboard.   I have attached a photo of what we currently have and then a photo example of what I am trying to do.   ... See more...
Hello,   I am trying to add a background cover for the panels within a dashboard.   I have attached a photo of what we currently have and then a photo example of what I am trying to do.   Including is the source code for these panel sections.   <row> <panel> <title>DoS Protection Count and Distribution Aggregate - ALL DATACENTER FIREWALLS</title> <table> <title>DoS Protection Count and Distribution - ALL DATACENTER FIREWALLS</title> <search> <query>index=pan_logs threat_category=flood dvc="FD0*.*" rule="* DoS Protection" vendor_action="random-drop" OR vendor_action="drop" action!=unknown | stats sparkline sum(repeat_count) by vsys_name dvc dest_ip action | sort by sum(repeat_count) desc limit=10 | rename sparkline AS "Distribution", sum(repeat_count) AS "Count", dest_ip AS "Destination", vsys_name AS "Virtual System", dvc AS "Device", action AS "Action" | table Device "Virtual System" Action Count Destination Distribution</query> <earliest>$field1.earliest$</earliest> <latest>$field1.latest$</latest> <refresh>2m</refresh> <refreshType>delay</refreshType> </search> <option name="dataOverlayMode">none</option> <option name="drilldown">none</option> <option name="percentagesRow">false</option> <option name="refresh.display">progressbar</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> </table> </panel> <panel> <title>ALL DATACENTERS | Resources Unavailable</title> <chart> <title>ALL DATACENTERS | Resources Unavailable</title> <search> <query>index=pan_logs dvc="*" session_end_reason="resources-unavailable" | timechart span=5h count by dvc</query> <earliest>$field1.earliest$</earliest> <latest>$field1.latest$</latest> <refresh>2m</refresh> <refreshType>delay</refreshType> </search> <option name="charting.chart">column</option> <option name="charting.drilldown">none</option> <option name="refresh.display">progressbar</option> </chart> </panel> </row> <row>  
  after trying to open my Splunk enterprise on my pc I am getting this page, please help me out here