About actionabledata

actionabledata · ‎07-31-2022

In this case, you are not using Data Model (and Acceleration) syntax, so I will ignore that use case. A tstats command uses data from the tsidx file(s). One of the means that data is put into the tsidx file(s) is index-time extractions. If the data has NOT been index-time extracted, tstats will not find it. https://docs.splunk.com/Documentation/Splunk/9.0.0/SearchReference/Tstats https://docs.splunk.com/Documentation/Splunk/9.0.0/Data/Aboutindexedfieldextraction#:~:text=Search%2Dtime%20field%20extraction%2C%20which,in%20the%20Knowledge%20Manager%20Manual. The index=xxx | stats ... function first of all reads ALL data from the index=xxx and THEN performs the stats function on that resulting data. https://docs.splunk.com/Documentation/Splunk/9.0.0/SearchReference/Stats https://community.splunk.com/t5/Splunk-Search/What-is-tstats-and-why-is-so-much-faster-than-stats/m-p/116960 My theory is that the other 5 host regions do not have index-time extractions performed. I recommend that you check your props.conf and accompanying transforms.conf file to determine which hosts have index-time field extractions. I suspect you will only find the one host. Best of luck to you. This type of question and investigation is critical to more efficient uses of Splunk. So keep asking!!

actionabledata · ‎06-10-2022

Thanks Jamie. Yeah, I saw that and made the same assumptions but couldn't find a definitive answer. Through a few not-so-scientific experiments, I found the TTL to be longer than 15 min but less than 3 hrs ... but the files DO go away. Thanks, Greg (ActionableData)

actionabledata · ‎06-06-2022

Do the resulting files from a "dump" command have a TTL? I think they must since the files I created on Friday no longer exist. Here is the search I am using to create the files. index = “myIndexName” sourcetype=”mySourcetype” myFilterField IN(123ABC, 456DEF, 789GHI) | dump basefilename= ABCCorp_06-06-22_0800_01330_ rollsize=1000 compress=5 format=raw | table * Thank you.

actionabledata · ‎05-24-2022

Hello All, I am wanting to create a user-defined "dictionary" for a dashboard and would desire for the user to click on a link within the dashboard to launch the Lookup Editor app on a specific *.csv file. I would seed the dictionary with specific columns. Is this possible? I tried something like the following but got an error ... "The lookup could not be loaded from the server" <drilldown> <link> http://myServer:8000/en-US/app/lookup_editor/lookup_edit?owner=nobody&namespace=myAppName&type=csv&Name=client_information.csv </link> </drilldown> Appreciate the help.

actionabledata · ‎04-25-2022

Which takes precedence: the props.conf setting (CHECK_METHOD) or the inputs.conf setting (crcSALT)? CHECK_METHOD = modtime # Notes from props.conf spec Set CHECK_METHOD to "modtime" to check only the modification time of the file. crcSalt = <SOURCE> # Notes from inputs.conf spec # If set to the literal string "<SOURCE>" (including the angle brackets), the # full directory path to the source file is added to the CRC. This ensures # that each file being monitored has a unique CRC. When 'crcSalt' is invoked, # it is usually set to <SOURCE>.

actionabledata · ‎03-09-2022

Hello All, Can someone please help me understand the functionality of AUTO_KV_JSON? Does it modify the functionality of KV_MODE? If so, under what scenario? From my experiments it doesn't seem to do anything when traversing the permutations of INDEX_EXTRACTIONS, KV_MODE and AUTO_KV_JSON. We typically just use the following settings in our sourcetype definitions: KV_MODE=json AUTO_KV_JSON=true Here is the table of results actionabledata_1-1646862753236.png Appreciate a response. Thank you.

actionabledata · ‎02-23-2022

Turns out that the app containing my master props.conf had been disabled within an app.conf file. Not sure how, but it was. Once enabled, the INDEXED_EXTRACTIONS - json worked well on our all in one development platform. Special thanks to Woodock.

actionabledata · ‎02-23-2022

All, Will INDEXED_EXTRACTIONS = JSON perform the extractions on an All-in-One platform? Here is my props.conf The sourcetype was executed but none of the fields were extracted. I can see the fields & values in _raw but they are not listed as fields. Here is what I see with an adhoc search. The time "field" within _raw is Jan 5, 2022 I did index the data on 2/22/22 but I am uncertain where the _time field came from. It matches nothing in the data. actionabledata_2-1645632966977.png props.conf (no transforms.conf) # created on 2/22/2022 for a test case using INDEXED_EXTRACTIONS=JSON # The non-highlighted settings are identical to a known working stanza for the exact same data [allfields_index_extracted] INDEXED_EXTRACTIONS = JSON NO_BINARY_CHECK = true LINE_BREAKER = ([\r\n]+) EVENT_BREAKER = ([\r\n]+) EVENT_BREAKER_ENABLE = true SHOULD_LINEMERGE = false TIME_PREFIX = ^"?{""?time""?: TIME_FORMAT = %s.%6N MAX_TIMESTAMP_LOOKAHEAD = 17 category = Structured description = INDEXED_EXTRACTIONS eq JSON pulldown_type = 1 # Search Time stuff # Disable search time field extractions since INDEXED_EXTRACTIONS=JSON KV_MODE = none AUTO_KV_JSON = false disabled = false Appreciate the help!

actionabledata · ‎01-28-2022

This was incredibly useful. Thank you!

actionabledata · ‎08-09-2021

@ITWhisperer Good questions. Let me clarify. The dataset this algorithm is operating on is fixed. The data is indexed and does not change. This is why my algorithm should be very deterministic. Consequently, the time periods you see in the images are really just the different times of the day that I executed the code. The code is quite complex, not rocket science, but sequentially arduous ... creating correlations and then sub-correlations in some places and then aggregating those pre-results for summary level dashboard graphics. I need to obfuscate the code sufficiently while leaving the command sets in place such that you can better follow the functionality I am performing. Truly appreciate the help.

actionabledata · ‎08-09-2021

Update 8/9/21 I reran one of the tests just to see if adding the | stats count at the bottom made any difference in the resulting count. It did. Nothing is different except for the | stats count line. Here are some images showing the results ... which I cannot explain. Without the | stats count actionabledata_0-1628535101294.png With the | stats count actionabledata_1-1628535121102.png

actionabledata · ‎08-09-2021

@bowesmana , I apologize for the delay in responding. Had deadlines to meet. Good catch on my bad math. I also responded to @ITWhisperer and the bottom line is that after rerunning the tests this morning, the results were identical. I must have been doing something odd the other day. I do not know what the difference in my tests was and this bothers me, but for now, all is well. My foreach is merely appending a numerical prefix to two (2) separate fields to retain the time ordered sequence through the big stats. The time frame is the same for both tests. I did use your suggestion of adding a | eval dummy=1 and then the | search NOT dummy=* and at least today, the answers are consistent. Unfortunately, I cannot show the algorithm, at least at this juncture. Thank you for responding. The community supporting Splunk Answers is phenomenal!!

actionabledata · ‎08-09-2021

ITWhisperer, I apologize for the delay in responding. Had deadlines to meet. Your question made me think. I reran the tests this morning with ... | stats count ... at the bottom of each set of code and the results were identical ... 351,426 I have to admit that I am puzzled because I thought this was the process I used recently; however, at this time, it looks like a false alarm. Thank you again.

actionabledata · ‎08-04-2021

My long set of SPL starts with the typical filtering on the primary search line. It then uses various eval, foreach, streamstats and eventstats commands to process the data for a big stats aggregation command. Here is the problem or at least a gap in my misunderstanding: Early in the SPL I use a "| where" command to eliminate events not containing a specific value. This works great. The results filter down to 351,513 events. However, between this where command and the line just before the big stats command, I only use eval, foreach, streamstats and eventstats commands ... and the search results increase by 29. I thought each of these commands merely modified / created fields within the events. There are now 351,532 events instead of the 351,513 events. So the question is ... Can an eval, foreach, streamstats or eventstats ever INCREASE the number of search results or am I just misinterpreting the results.

actionabledata · ‎07-24-2021

I have a single algorithm with 2 methods. Each method produces the same type of data but with different fields names to keep them separated. The dashboard charts depend on which method the user selects in a menu. Essentially I create interim results for both methods but desire to change the names to the field names used in the subsequent code. [Q] What is a more efficient method of performing the "Big Switch" in the run anywhere code below? | makeresults 5 | rename comment AS "-----------------------------------------------------------------" | rename comment AS "User Menu Selection" | eval switch="A" | rename comment AS "-----------------------------------------------------------------" | rename comment AS "Algorithm element2" | eval calcMethod1_field1="1" | eval calcMethod1_field2=2 | eval calcMethod1_field3=3 | eval calcMethod1_field4=4 | eval calcMethod1_field5=5 | rename comment AS "-----------------------------------------------------------------" | rename comment AS "Algorithm element2" | eval calcMethod2_field1="1sub" | eval calcMethod2_field2="2sub" | eval calcMethod2_field3="3sub" | eval calcMethod2_field4="4sub" | eval calcMethod2_field5="5sub" | rename comment AS "-----------------------------------------------------------------" | rename comment AS " Big Switch " | rename comment AS "-----------------------------------------------------------------" | rename comment AS "This is the big switch before entering a stats command" | rename comment AS "Intent is to rename several fields depending on switch value" | eval fieldnameforstats_field1=case(switch=="A",calcMethod1_field1,switch=="B",calcMethod2_field1) | eval fieldnameforstats_field2=case(switch=="A",calcMethod1_field2,switch=="B",calcMethod2_field2) | eval fieldnameforstats_field3=case(switch=="A",calcMethod1_field3,switch=="B",calcMethod2_field3) | eval fieldnameforstats_field4=case(switch=="A",calcMethod1_field4,switch=="B",calcMethod2_field4) | eval fieldnameforstats_field5=case(switch=="A",calcMethod1_field5,switch=="B",calcMethod2_field5) | fields - _time | table fieldnameforstats_field*

actionabledata · ‎07-15-2021

I know it was many years ago, but your answer helped, so thank you for your response somesoni2. I did have to update the XML code for compliance with the recent version of Splunk, but that was simple. My situation is a bit different in that I need to extract a segment from a clicked field (UID) to become the form.field input for my drilldown page. This doesn't seem hard, but I cannot make it work. Example: My UID field looks like ABC123:456456:99887766 and I need to pass the first segment (ABC123) to the drilldown page when a user clicks on that table row. To ensure uniqueness, this UID field is a concatenation of several fields within my algorithm. The original field is not available since it gets consumed with stats commands. I have attempted within the <drilldown> section of the main page to perform a rex extraction and then use the resulting token within the same <drilldown> section. But the rex extraction does not seem to execute. I also tried performing the rex extraction within the drilldown page using the passed UID value. I can extract the value but can't seem to populate the form with the value. Thank you for any help you can provide.

actionabledata · ‎07-09-2021

Update to the appendpipe version of code I eliminated stanza2 and the final aggregation SPL reducing the overall code to just the pre-appendpipe SPL and stanza 1 but leaving the appendpipe nomenclature in the code. Total execution time = 486 sec Then for this exact same search, I eliminated the appendpipe syntax. Everything is the same except for the | appendpipe and [ ] syntax. Total execution time = 77 sec The overhead to using appendpipe is HUGE. I suspect the same is true for using multireport.

actionabledata · ‎07-09-2021

Code Architecture: common code generating initial results; generate unique key for each grouping multireport or appendpipe stanza 1 with its own set of stats, evals, etc.; uses the key from the common code stanza 2 with its own set of stats, evals, etc.; uses the key from the common code Aggregates the resulting data from the common, stanza1, stanza2 results using the key Issue Statement common code + stanza 1 takes about 1 min to execute common code + stanza 2 takes about 1 min to execute common code + stanza 1 + stanza2 using either a multireport or appendpipe takes about 17 min [Q] Does this huge execution time difference make sense? I have attached a few images to show how I think multireport and appendpipe work. [Q] Is my understanding accurate? actionabledata_0-1625861746744.png The pre-appendpipe SPL reads the data from the index, filters the data, creates some initial fields using streamstats and eventstats and creates a key that is unique per the overall groupings correlated within this code. Lines 1 and 2 are identical and originate from the pre-multireport SPL. These results are presented to the stanza1 and stanza2 SPL. Lines 3 and 4 are independent results from stanza1 and stanza2 respectively stanza1 and stanza2 execute mutually exclusive from one another The sort and stats clauses within stanza1 and stanza2 are quite different but the one does NOT impact the other. The final aggregation software ties all the data together based on a common key. actionabledata_1-1625861773920.png The pre-appendpipe SPL reads the data from the index, filters the data, creates some initial fields using streamstats and eventstats and creates a key that is unique per the overall groupings correlated within this code. Lines 1 and 2 and 3 are identical and originate from the pre-appendpipe SPL. These results are presented to the stanza1 and stanza2 SPL. Lines 3 and 4 CAN be removed if I filter the input data with a where clause and the flag I called "which" associated with each set of data. Lines 5 and 6 are independent results from stanza1 and stanza2 respectively stanza1 and stanza2 execute mutually exclusive from one another. The stats clauses within stanza1 and stanza2 are quite different but the one does NOT impact the other. The final aggregation software ties all the data together based on a common key.

actionabledata · ‎06-18-2021

I apologize for not seeing your previous answer ... I still do not see it although it may be to another person. I will research REST calls.

actionabledata · ‎06-18-2021

How do I search for all apps and dashboards on a server and yield a table with author, app name, description, actual linux file name of dashboards within each app, revision #, and date last updated? Thank you I played around with the following code but it doesn't give everything I need | rest /services/apps/local | search disabled=0 | table author label version description title id label version I think I need to go learn REST APIs for Splunk

actionabledata · ‎06-16-2021

Turns out that you don't really need the sameFlag and prev_error fields and associated calculations. I just change the BY clause on the streamstats | streamstats count as repCount BY error reset_on_change=true | makeresults count=10 | rename COMMENT AS "------------------------------------------------------------" | rename COMMENT AS "Generate fake data" | eval group = "GROUP" | eval r = random() % 10 + 1 | eval num = case( r=1,18, r=2,10, r=3,99, r=4,82, r=5,95, r=6,54, r=7,4, r=8,40, r=9,83, r=10,33) | rename COMMENT AS " eval num = random() % 100" | eval error = case( num < 10, 404, num >= 10 AND num < 13, 500, num >= 13 AND num < 40, 200, num >=40, 333) | eval error_msg = case(error == 404, "Not found", error == 333, "Made Up Error Code", error == 500, "Internal Server Error", error == 200, "OK") | rename COMMENT AS "------------------------------------------------------------" | rename COMMENT AS "Calculate repCount" | streamstats count as repCount BY error reset_on_change=true | table group repCount error

actionabledata · ‎06-16-2021

Thank you ITWhisperer ! However, I need to clarify my requirements a bit. Sorry for not clarifying previously. I need to count the number of sequential replications of a field. Eventually, I will aggregate the fields into a smaller list where only the unique fields are included with a suffix of the max(replication factor). The replication factor (repCount) looks like this: Note: The first 3 values or "error" are 333 and the repCount increments to 3, which is correct. Note: The next 2 values also have a repeat of 222 and repCount increments to 2 which is correct. Note: The repCount value resets every time the error value changes state. Note: The subsequent 333's are an independent set of repeats, so the repCount begins fresh and is not related to the previous count of 333's. actionabledata_0-1623865482693.png Here is my code that produced the above results but there has to be a more elegant way to do this. I experimented with the streamstats "reset_on_change" option but had to add an eval command afterward to force the results. | eval repCount = if(sameFlag=="0", 1, repCount+1) | makeresults count=10 | rename COMMENT AS "------------------------------------------------------------" | rename COMMENT AS "Generate fake data" | eval group = "GROUP" | eval r = random() % 10 + 1 | eval num = case( r=1,18, r=2,10, r=3,99, r=4,82, r=5,95, r=6,54, r=7,4, r=8,40, r=9,83, r=10,33) | rename COMMENT AS " eval num = random() % 100" | eval error = case( num < 10, 404, num >= 10 AND num < 13, 500, num >= 13 AND num < 40, 200, num >=40, 333) | eval error_msg = case(error == 404, "Not found", error == 333, "Made Up Error Code", error == 500, "Internal Server Error", error == 200, "OK") | rename COMMENT AS "------------------------------------------------------------" | rename COMMENT AS "When sameFlag=1, increment repCount; When sameFlag=1, repCount=1" | rename COMMENT AS "------------------------------------------------------------" | rename COMMENT AS "------------------------------------------------------------" | rename COMMENT AS "Calculate sameFlag" | streamstats window=1 current=f last(error) AS prev_error BY group | eval sameFlag = if(like(error,prev_error), 1, 0) | rename COMMENT AS "------------------------------------------------------------" | rename COMMENT AS "Calculate repCount" | streamstats window=1 current=f last(sameFlag) AS prev_sameFlag BY group | streamstats count as repCount BY sameFlag reset_on_change=true | eval repCount = if(sameFlag=="0", 1, repCount+1) | table group repCount sameFlag error prev_error Thank you in advance for any help on optimizing this code.

actionabledata · ‎06-15-2021

My objective is to increment ReplicaCount if the previous event's field value matches the current event's field value. | makeresults count=10 | eval num = random() % 100 | eval error = case( num < 10, 404, num >= 10 AND num < 13, 500, num >= 13 AND num < 40, 200, num >41, 333 ) | eval error_msg = case(error == 404, "Not found", error == 333, "Made Up Error Code", error == 500, "Internal Server Error", error == 200, "OK") | sort 0 - _time | streamstats current=f last(error) AS prev_error BY error | eval ReplicaCount=1 | eval ReplicaCount =if(like(error,prev_error), ReplicaCount+1,ReplicaCount) | table _time error prev_error ReplicaCount error_msg num actionabledata_0-1623786960099.png Sorry, cannot figure this out. What am I missing? I don't think last() is doing what I think it's doing which is ... obtain the field value from the previous event. [1] Why are some of my prev_error values blank? [2] why does ReplicaCount max out at 2? Thank you in advance.

actionabledata · ‎06-08-2021

All, Hopefully a straightforward question. Is it possible to increase the following setting in a .../appname/local/limits.conf such that the scope remains local and pertinent only to that appname and NOT global thus impacting other Apps and their searches? FROM: max_mem_usage_mb = 200 TO: max_mem_usage_mb = 2000 We have an algorithm that must process a very large data ball before aggregating and reducing the data and we keep hitting this limit. We partition the data to get around the limit but I am hoping that this partitioning process can be eliminated. Thank you

actionabledata · ‎01-21-2021

Follow on question to https://community.splunk.com/t5/Getting-Data-In/Can-batch-read-a-partial-file-such-that-the-of-events-indexed-is/m-p/536557#M89931 [Q] For inputs.conf, can the time_before_close setting be used with [batch]? The inputs.conf file specifically indicates which [monitor] settings are compatible with [batch] and this setting is not included. https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf From inputs.conf The following settings work identically as for [monitor::] stanzas, documented above host_regex = <regular expression> host_segment = <integer> crcSalt = <string> recursive = <boolean> whitelist = <regular expression> blacklist = <regular expression> initCrcLength = <integer>

Posts	28
Solutions	6
Karma Given	77
Karma Received	4
Member Since	‎12-20-2019

Online Status	Offline
Date Last Visited	‎03-22-2023 04:27 PM

Do the resulting files from a "dump" command have ...

Lookup Editor: Can I open a specific lookup file f...

Can someone please help me understand the function...

Will INDEXED_EXTRACTIONS = JSON perform the extrac...

Can an eval, foreach, streamstats or eventstats ev...

How do I implement multiple rename commands based ...

multireport vs. appendpipe

Search for all apps and dashboards on a server

Calculation based on comparison of previous event'...

limits.conf, non-global settings local to specific...

Re: Why do stats and tstats give different set of ...

Re: Do the resulting files from a "dump" command h...

Do the resulting files from a "dump" command have ...

Lookup Editor: Can I open a specific lookup file f...

Re: Indexing same file: crcSalt = <SOURCE> not wor...

Can someone please help me understand the function...

Re: Will INDEXED_EXTRACTIONS = JSON perform the ex...

Will INDEXED_EXTRACTIONS = JSON perform the extrac...

Re: Change & Condition within From Text Inputs

Re: Can an eval, foreach, streamstats or eventstat...

Re: Can an eval, foreach, streamstats or eventstat...

Re: Can an eval, foreach, streamstats or eventstat...

Re: Can an eval, foreach, streamstats or eventstat...

Can an eval, foreach, streamstats or eventstats ev...

How do I implement multiple rename commands based ...

Re: How to pre-populate text boxes by passing toke...

Re: multireport vs. appendpipe

multireport vs. appendpipe

Re: Search for all apps and dashboards on a server

Search for all apps and dashboards on a server

Re: Calculation based on comparison of previous ev...

Re: Calculation based on comparison of previous ev...

Calculation based on comparison of previous event'...

limits.conf, non-global settings local to specific...

For inputs.conf, can the time_before_close setting...

Join the Conversation