Hello, I am running splunk enterprise 9.0.0.1 and attempting to run the upgrade readiness app. The jquery test for TA_cisco-ACI is failing, but the version installed is 5.0.0, the latest one on splunkbase, which claims to be fully compatible. I checked for any custom code within the app's directory and only found an app.conf and passwords.conf files. So i am reasonably sure it's probably not in the app installation directory.   What could be wrong?  could this be a false positive? --jason   ... View more

Hello, I am thinking about the splunk add-on for cisco ucs, but this might apply to add-ons in general. it's easy to visualize installing it onto a heavy forwarder, and configure it to talk to all of my ucs managers. BUT this implies a single system that does this, and I need redundancy. My cluster of forwarders would seem to be the answer here, but if I install the add-on onto all three heavy forwarders, will I be ingesting replicated data?   Is the add-on or splunk smart enough to prevent this? Thank you!   --jason     ... View more

Hello,  I have a splunk heavy forwarder (splunk 9.0.0.1, centos 7) configured as a heavy forwarder.  When I issue "splunk stop" as either the splunk user or user root, it will restart within a minute. This system has been configured to start as a systemd service and user splunk is allowed to issue systemctl commands to stop and restart. just now, I completely disabled the systemd service and then rebooted.  After about a minute of being fully booted, splunk automatically started up.  Its as if there's a parasitic cron or anacron job that starts it if its not running. What could be causing this? I would really rather splunk stayed down especially since I am usually altering the configuration files at the time and it starts up before I can complete the task. --jason ... View more

Hello, I have something strange going on.  I need to monitor logs from three different systems.  thus far I have only built one system, so there are only logs from one system present. the three systems are: sldvuspeedtest01p ptdvuspeedtest01p tsdvuspeedtest01p while it's always tempting to go crazy with regex's, I tried the more simple version first: [monitor:///opt/syslog/*speedtest*] index=isp sourcetype=speedtest whitelist= \.log$ blacklist = (default[a-zA-Z0-9\_\-]+)\.log #host_regex = \/opt\/syslog\/(.*)/ host_segment=3   but splunk will only load the syslog files if the stanza reads: [monitor:///opt/syslog/sldvuspeedtest01p].   so when I tried the previous version, and then run  "splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus"   I see the following lines: <s:key name="/opt/syslog/sldvuspeedtest01p/syslog_2023-02-24.log"> <s:dict> <s:key name="parent">/opt/syslog/zayo_devices_new</s:key> <s:key name="type">File did not match whitelist '^\/opt\/syslog/[^/]*\.docker/syslog_[^/]*\.log$'.</s:key> </s:dict> </s:key>   There is a stanza for that "parent", but why would splunk even confuse the two?  is there a heirarchy in which monitor stanzas are loaded in that I am running awry of? There is also a separate stanza [monitor:///opt/syslog/*.docker/syslog_*.log] but it doesn't make sense why it would be referred to here either.   [monitor:///opt/syslog/zayo_devices_new] whitelist = \.log$ blacklist = (Health[a-zA-Z0-9\_\-]+)\.log index=z_catchall sourcetype = zayo_routing host_regex = zayo_devices_new/(.*)\_     ... View more

Hello I am perplexed: when I run firebrigade, and choose "detail | index detail" and having chosen a host, my index list is incomplete. I see in the source where it gets input on line 20: "inputlookup fb_hostname_index_cache". I see the contents of the fb_hostname_index_cache.csv file is the same incomplete list I found the periodic search where it extracts the data to put into "fb_hostname_index_cache.csv", the search command being: index=summary search_name="DB inspection" | dedup orig_host, orig_index | sort orig_host, orig_index | table orig_host,orig_index | outputlookup fb_hostname_index_cache When I run this search, I get "Error in 'outputlookup' command: The lookup table 'fb_hostname_index_cache' is invalid." when I run the search without "| outputlookup fb_hostname_index_cache", I get an incomplete list of my indexes. so a few things might be happening that I don't know how to determine. splunk doesn't like something about the "fb_hostname_index_cache.csv" file not all of the indexes are being returned from the query something isn't right about the contents of "index=summary" This issue appeared shortly after i upgraded from 8.0.3 to 8.1.6 to 9.0.0.1. My current system has six dedicated indexers, and an independent search head.  there are also three heavy forwarders. Can someone shed some light on this?   Thank you!     ... View more

Hello, I have a tcp stream incoming with xml Call Data Records (CDR).  enclosed at the end is an example. The CDR contains information about the caller and destination phone.  There are several SEDCMDs in the props.conf to take lines like <party type="orig"... and convert them into <orig_party... The problem is with lines whose only differentiation is their position in the data structure.  With the xml lines that start with "<RTCPstats>  I need to modify their fields:  I need the first line to be ingested as "<orig_PS>31367</orig_PS> <orig_OS>6273400</orig_OS>..." and the second line as "<term_PS>31366</term_PS>, <term_OS>6273200</term_OS>...". <flowinfo>     <RTCPstats>PS=31367, OS=6273400,..>   </flowinfo> <flowinfo>     <RTCPstats>PS=31366, OS=6273200,...>    </flowinfo> The actual sed command "sed -r -e '0,/ ?([A-Z_]*)=([0-9]*)/s//<ORIG_\1>\2<\/ORIG_\1>/g'" will do this, but the same entry in the SEDCMD will not.   Altering a line of input is easy: altering only the FIRST instance in a record with embedded newlines is not. What are my options?   SEDCMD  in props.conf? strip out all newlines so sedcmd treats it all as one? (can't really do it w. sed...) regex in transforms.conf? pass input through a script that CAN do this?  Some details: The linebreaker is two carriage returns (literally \r\r, or 0d0d).   There are embedded newlines in each record so any SEDCMD will be applied to each line, not the entire record all at once. I get 60,000 records per minute: this transformation needs to be fast. Sample record: (spaced out neatly)         <e> <st>Perimeta XML CDR</st> <h>the perimeta hostname</h> <t>1664838107186</t> <tid>2814754955435820</tid> <sid>2082</sid> <eid>CDR</eid> <call starttime="1664837476918" starttime_local="2022-10-03T15:51:16-0700" endtime="1664838107179" endtime_local="2022-10-03T16:01:47-0700" duration="630261" release_side="term" bcid="a big string of numbers and letters"> <party type="orig" phone="caller phonenumber" domain="ipaddr1" sig_address="ipaddr1" sig_port="5060" sig_transport="udp" trunk_group="trunkgroupname" trunk_context="nap" sip_call_id="anumber@adestination"/> <party type="term" phone="destination phonenumber" domain="0.0.0.0" routing_number="a routing number" sig_address="<an ip addr>" sig_port="5060" sig_transport="udp" trunk_group="6444" trunk_context="itg" edit_trunk_group="" edit_trunk_context="" sip_call_id="adifferentnumber@adifferentdestination"/> <adjacency type="orig" name="orig_adjacency_system" account="" vpnid="0X00000001" mediarealm="CoreMedia1"/> <adjacency type="term" name="dest_adjacency_system" account="" vpnid="0X00000004" mediarealm="CoreMedia1"/> <category name="cat.sbc.redirected"/> <connect time="1664837483144" time_local="2022-10-03T15:51:23-0700"/> <firstendrequest time="1664838107158" time_local="2022-10-03T16:01:47-0700"/> <disconnect time="1664838107179" time_local="2022-10-03T16:01:47-0700" reason="0"/> <redirector bcid="another string of letters and numbers" editphone="a phone number"/> <post_dial_delay duration="2895"/> <QoS stream_id="1" instance="0" reservetime="1664837476918" reservetime_local="2022-10-03T15:51:16-0700" committime="1664837483144" committime_local="2022-10-03T15:51:23-0700" releasetime="1664838107184" releasetime_local="2022-10-03T16:01:47-0700"> <gate> <flowinfo> <local address="an ip address" port="63130"/> <remote address="another ip address" port="36214"/> <sd>m=audio 0 RTP/AVP 0 a=rtpmap:0 PCMU/8000 a=ptime:20 </sd> <RTCPstats>PS=31367, OS=6273400, PR=31366, OR=6273200, PD=0, OD=0, PL=0, JI=0, TOS=0, TOR=0, LA=0, PC/RPS=31165, PC/ROS=4986400, PC/RPR=31367, PC/RPL=0, PC/RJI=0, PC/RLA=0, RF=91, MOS=43, PC/RAJ=0, PC/RML=0</RTCPstats> </flowinfo> <flowinfo> <local address="an ip address" port="19648"/> <remote address="a diffrent ip address" port="26046"/> <sd>m=audio 0 RTP/AVP 0 a=rtpmap:0 PCMU/8000 a=ptime:20 </sd> <RTCPstats>PS=31366, OS=6273200, PR=31367, OR=6273400, PD=0, OD=0, PL=0, JI=0, TOS=0, TOR=0, LA=0, PC/RPS=0, PC/ROS=0, PC/RPR=0, PC/RPL=0, PC/RJI=0, PC/RLA=0, RF=82, MOS=41, PC/RAJ=0, PC/RML=0</RTCPstats> </flowinfo> </gate> </QoS> </call> </e>           ... View more

Hello,  I have splunk starting up with systemd, and running as user splunk.    I went to run the performance tasks on my indexers.  Each of them failed.  under triggered collectors, it reads the collector stack trace failed. I logged into the system in question, and looked at the splunk_rapid_diag.log file.   tools_collector ERROR 139880958523200 - Error occurred for collector tcpdump while running `/usr/sbin/tcpdump -i any -w /tmp/tmpbkxib485/tcpdump_All_All.pcap` Process finished with code=1 how do I run diagnostic tools without root access?   I expect this would affect any collectors using strace as well.   --jason       ... View more

Hello, I am looking at https://docs.splunk.com/Documentation/Splunk/9.0.0/Capacity/Parallelization and was wondering which systems to make changes on. For instance: batch parallelization: should the limits be changed on the search heads, indexers, or both? same question for datamodels, report acceleration and indexer parallelization. Oh for what it's worth, I am running splunk enterprise 9, on a C1/C11 deployment. -jason   ... View more

Hello, I have inherited a set of splunk servers, and three are search heads.  Some of the apps on the search heads  are installed directly.  I would prefer to manage those apps from the deployment server.   I have found lots of information about deploying apps via single-instance/distributed, but not a lot on how to move an app from a single instance to a deployment server. In my brain, the procedure looks like this: On the search head, tar up the app from $SPLUNK/etc/apps/<appname> copy the tar file to the deployment server untar the file into the $SPLUNK/etc/deployment-apps uninstall or disable the original app on the original searchhead Map the new  app to serverclass and to the client system (original search head) Is this correct?  --jason     ... View more