I have seen several posts that seem to dance around this use case. I'm writing into a summary index (si_sum_data), summarizing by minute the number of transactions APPROVED, STAGED, DECLINED, and TOTAL. I run this ever 5 minutes to summarize the last 5 minutes, skipping the past 5 minutes to account for any queued data that may be 1 - 3 minutes late. I need to add a sum of each RESPONSE_CODE to the data, we have a USE case that needs each RESPONSE_CODE to be a KPI. I can add RESPONSE_CODE to the stats BY however this creates a new event for each RESPONSE_CODE for the minute time. It roughly in creases the number of events by 30%, and we already generate 3.9 million summary events a day. The Question : How can I create a new field for each RESPONSE_CODE (eval RESPONSE_CODE{RESPONSE_CODE}=RESPONSE_CODE as an example) and have it populate the column headers? This will allow a single event to look like the following (example1). EXAMPLE: _time CUSTOMER_NUMBER CUSTOMER_SESSION APPROVED STAGED DECLINED TOTAL RESPONSE_CODE_01 RESPONSE_CODE_02 2019-11-08 15:35:00 12345 XI298Y1 20 2 6 28 11 6 index=raw_data_index earliest=-10m@s latest=-5m@s | bucket _time span=1m | eval RESPONSE_CODE = coalesce(PLATFORM1_CODE,PLATFORM2_CODE,PLATFORM3_CODE) | eval CUSTOMER_NUMBER = coalesce(PLATFORM1_CUSTOMER_NBR,PLATFORM2_CUSTOMER_NBR) | eval CUSTOMER_SESSION = coalesce(PLATFORM1_CUSTOMER_SESSION,PLATFORM2_CUSTOMER_SESSION,PLATFORM2_CUSTOMER_SESSION) | stats count(eval(RESPONSE_CODE == "00" OR RESPONSE_CODE == "08" OR RESPONSE_CODE == "10" OR RESPONSE_CODE == "11" OR RESPONSE_CODE == "85")) as "APPROVED", count(eval(RESPONSE_CODE == "04" OR RESPONSE_CODE == "41" OR RESPONSE_CODE == "43")) as "STAGED" count(eval(1=1)) as "TOTAL" by _time,CUSTOMER_NUMBER,CUSTOMER_SESSION | eval DECLINED=(TOTAL-(APPROVAL+STAGED)) | table _time CUSTOMER_NUMBER CUSTOMER_SESSION APPROVED STAGED DECLINED TOTAL ... View more

We currently index logs into index=indexY at a rate of 2G – 5G a day with the retention set to 12 months. One day last week, a code update started logging the user and password into the log in the form of ‘password=changeme’. Our Splunk environment is certified non-pci/non-private data and needs to remain so. We need to retain the event data due to security best practices, it is the event for access to a critical services, however the field needs to be redacted. What are the best options for retaining the data while scrubbing the field? For instance 'password=change' should be redacted to 'password=SCRUBBED' or removed entirely. Please consider that that data at rest cannot contain confidential data, so masking it at the Search Head is not a viable solution. ... View more