Warning:  Long, detailed explanation ahead. 🙂   Summary version is that I have a nested json arrays and fields that I am having an issue with extracting properly into individual fields.  The chosen fields will change over time, based on external factors so I need to be able to extract and report on all of them, with the ability to identify the array index (i.e. {0}, or {1}, etc).   No solution that I have looked at or come up with is working for me, so I am turning to you smarter folks to help. Detail: I have a nested json arrays and fields that I am having an issue with extracting properly into individual fields.   The end result is that I want to be able to place alerts or report on various fields that are deemed interesting. These are "request" and "response" arrays in each transaction (think checking items in a shopping cart for various flag and indicators).    The chosen fields will change over time, based on external factors so I need to be able to extract them from the array  and report on all of them at some point. Here is a sample request and response As you can see the the request array is market_basket.request{} and the response is market_basket.response{}. Focusing on the response portion, the first response has an "02" field and a "dataset". The next response{1} has fields 02,03,04,05,08,etc etc., same with response{2} and response{3} if I do a simple rename | rename market_basket.response.* to Resp_* the fields don't line up. The contents of "Resp_19" should be down 1 line as there was no field 19 in market_basket.response{0}.  See here:  If I change the query to this   | spath path=market_basket.response{} output=Response | spath input=Response | table tran_id 0* 1* 2* dataset   Then I only get the first row, the other 3 rows don't show up. The only way that I have been able to get it to work is to address each indices and  field individually   | spath path=market_basket.response{0} output=Resp_0 | spath path=market_basket.response{0}.dataset output=Resp_0_dataset | spath path=market_basket.response{0}.02 output=Resp_0_02 | spath path=market_basket.response{1} output=Resp_1 | spath path=market_basket.response{1}.dataset output=Resp_1_dataset | spath path=market_basket.response{1}.01 output=Resp_1_01 | spath path=market_basket.response{1}.02 output=Resp_1_02 | spath path=market_basket.response{1}.03 output=Resp_1_03 | spath path=market_basket.response{1}.04 output=Resp_1_04 | spath path=market_basket.response{1}.05 output=Resp_1_05 | spath path=market_basket.response{1}.06 output=Resp_1_06 | spath path=market_basket.response{1}.07 output=Resp_1_07 | spath path=market_basket.response{1}.08 output=Resp_1_08 | spath path=market_basket.response{1}.09 output=Resp_1_09 | spath path=market_basket.response{1}.10 output=Resp_1_10 | spath path=market_basket.response{1}.11 output=Resp_1_11 | spath path=market_basket.response{1}.12 output=Resp_1_12 | spath path=market_basket.response{1}.13 output=Resp_1_13 | spath path=market_basket.response{1}.14 output=Resp_1_14 | spath path=market_basket.response{1}.15 output=Resp_1_15 | spath path=market_basket.response{1}.16 output=Resp_1_16 | spath path=market_basket.response{1}.17 output=Resp_1_17 | spath path=market_basket.response{1}.18 output=Resp_1_18 | spath path=market_basket.response{1}.19 output=Resp_1_19 | spath path=market_basket.response{1}.20 output=Resp_1_20 | spath path=market_basket.response{1}.21 output=Resp_1_21 ...   But with up to 60 responses with 20 fields per transaction, that many spaths would be a non-starter. Especially considering that I need to factor in the request portions too at some point. Finally, to give an example use case, I want to be able to check field 19 on the response  and if the flag starts with "NN" or "NY", then put out an alert: "Item".market_basket{whatever #}.02." has been not been cleared for sale". Flags are:".market_basket{whatever #].19 I know that was a lot of detail, but I wanted to make sure that I put down the different ways that I tried. Any help would be much appreciated! ... View more

Hi,  I have a report that pulls daily transaction counts from a summary index.  Running the report for "month to date", I don't get results from every day.   My search is this:    index=summary search_name=Summarization_Daily_Txn App IN ("XXX")) endpoint="ZZZZ" | bin _time span=1d | stats sum(Count) AS Txn_Count by _time | addcoltotals     Gives me this output: The Dec 2, 4th and 5th totals are missing.    Yes,  I have verified that there are counts for those days.  If I run the report so that it spans just Dec 4th and 5th, the counts show up.   Just not if I run it using earliest=@mon latest=@d Any ideas on what I am doing wrong? ... View more

I saw this one of the monitoring console dashboard and I have been wracking my brain trying to figure out how it gets done.      By "it" I mean the border around the table rows and the bar inside.  I looked at the MC dashboard for it, but I don't see anything special about the search or dashboard (no js or other formatting tricks) that I could discern.       ... View more

If you read the title, you are going "well of course it does", but hear me out.   (This will be a long explanation that will hopefully answer the immediate questions)... Background: We have some on-prem UFs that forward "everything"  to our on-prem enterprise indexers AND specific logs  to our splunk cloud instance indexer.    In case you are wondering,  the cloud instance is where our customer can look at their data without needing access to our internal systems. Problem: Splunk did some maintenance on our cloud instance and when they did so, forwarding  from the UFs also stopped coming into our on-prem Splunk.    I can't figure out why cloud being down would stop the forwarders from sending to enterprise.     Checking the documentation here: https://docs.splunk.com/Documentation/Splunk/8.1.0/Forwarding/Setuploadbalancingd#Configure_universal_forwarder_load_balancing_for_horizontal_scaling It reads like the UFs should switch to the next indexers when it goes down.  But it didn't.  Instead we saw this in the internal logs when the cloud instance was taken down for maintenance  11-25-2020 21:59:48.139 -0600 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group splunkcloud has been blocked for 1200 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.   Looking at the inputs.conf and outputs.conf,  I can see nothing wrong with them to have the data blocked from these UFs Sanitized inputs.conf, with the log that gets sent to both the on-prem instance  (PP_indexers) and cloud instance  bolded  [monitor://C:\blahblahblah\q2.log] _TCP_ROUTING = pp_indexers index = fsd sourcetype = q2 [monitor://C:\blahblahblah\wrapper.log] _TCP_ROUTING = pp_indexers index = fsd_sandbox sourcetype = wrapper [monitor://C:\blahblahblah\metrics.log] _TCP_ROUTING = pp_indexers,splunkcloud index = fsd_sandbox sourcetype = metrics Sanitized outputs.conf:  defaultGroup = pp_indexers forceTimebasedAutoLB = true autoLBFrequency = 15 [tcpout:pp_indexers] server = indexer1.ip.address.here:9997, indexer2.ip.address.here:9997 [tcpout:splunkcloud] compressed = false disabled = false server = our_domain_name.cloud.splunk.com:9997 sslCommonNameToCheck = our_domain_name.cloud.splunk.com sslCertPath = $SPLUNK_HOME/etc/apps/sanitized/client.pem sslPassword = sanitized sslRootCAPath = $SPLUNK_HOME/etc/apps/sanitized/cacert.pem sslVerifyServerCert = true useACK = true Oh and just in case you need it... UF versions are 7.1.2 and 7.2.3 enterprise version is 7.3.4,  cloud is 7.3. ... View more

When I run a report of roles that have been modified in our splunk cloud instance I see these entries Date Time user action info role_name 04-06-20 16:50:18 _ops_admin edit_user granted _ops_admin 04-06-20 16:50:15 _ops_admin edit_user granted _ops_admin 04-06-20 16:49:52 _ops_admin edit_user granted _ops_admin 04-06-20 16:44:06 _ops_admin edit_user granted _ops_admin 04-06-20 16:44:03 _ops_admin edit_user granted _ops_admin We dont have that user or role name defined in our instance. Anyone have an idea of what this is and if I should be concerned? ... View more

Dec 24th is the biggest day for us. During the afternoon, a summary job that populates a summary index for sales reports got delayed due to all of the searches that were happening. Consequently the job ran twice during the next hour. So now I have doubled up sales and transaction amounts for that hour. Is there a way to find and back out the double entries from the summary index? ... View more