About yuanliu

yuanliu · ‎01-26-2015

Say, I have a series of jobs involving a certain number of members, _time MemberCount JobRunTime (min) JobName 01:00:00 100 15 Job1 01:05:00 200 30 Job2 01:15:00 300 50 Job3 01:30:00 80 10 Job4 I want to show that during the first 5 minutes, total number of members is 100, between 5 and 15 minutes, total is 300 (100+200), between 15 and 30 minutes, total is 500 (300 + 300 - 100), between 30 and 35 minutes, total is 580 (500 + 80), between 35 and 40 minutes, total is 380 (580 - 200), between 40 and 75 minutes, total is 300 (380 - 80), and so on. So this is like the task of calculating concurrency, but instead of mere concurrency of events, I want concurrent total of a field value. The closest I have got is [ original search] | appendpipe [ eval _time=_time + JobRunTime*60 ] | timechart max(MemberCount) by JobName This will give me an "ending" event for each job start time, so I can connect the two dots as a straight line, and see that it overlaps with some other lines on time axis. What I want is a column chart (or area chart as resolution increases) that stacks Job1's MemberCount on top of Job2's when it starts, and so on. How do I do that?

yuanliu · ‎12-19-2014

ScanCount and eventCount are very useful tools! I think performance degradation after Criteria2 is due to shear event volume increase. Here are a few data points. 1. Criteria2 returns >10x more events than Criteria1 does. 2. Criteria1 has a scan to event overhead of ~100%, whereas Criteria2 has <1%. (When they are OR'd, overall overhead is <1%.) 3. Adding one = sign does not change overhead, nor does adding additional field selectors. As I explained in the original problem, the goal is to detect delays. Only delayed events give logs matching Criteria1, but all events give a log matching Criteria2. In my data set, even though each delayed event can contribute to more than one Criteria1 log, the total is still small compared to total matching Criteria2. I cannot think of any way to narrow down the event-end log to select only delayed events - that's the whole point of using Criteria1. Now if I can use Criteria1 as a subsearch to Criteria2 as in the SQL sense, that might make search less painful.

yuanliu · ‎12-19-2014

Wow - I initially thought that long literals (no punctuation) will lessen the impact of wildcard. I'll do more inspections as you suggested. Maybe it's better to just use two literals broken down at the wildcard? I can possibly go with just one of the literals - just not enough confidence in how distinct are log messages.

yuanliu · ‎12-19-2014

This is getting interesting. Criteria1 has the form field=value1 "literal string" . Criteria2 has the form field=value2 "string * with wildcard" . In the data set under test, fewer events match Criteria2. (field= is unnecessary when the string portion match; individually, field=value1 would return more events.) Thanks @somesoni2 for the min-max suggestion.

yuanliu · ‎12-18-2014

I have a large set of logs and two sets of mutually exclusive criteria, one signifies beginning and progression of an "event", the other signifies the end of the event, linked by a unique identifier. I can use any of three approaches to extract the duration of the event: (Criteria1 OR Criteria2) | stats latest(_time) as end earliest(_time) as begin by EventId | eval duration=end-begin (Criteria1 OR Criteria2) | transaction EventId Criteria1 | stats earliest(_time) as begin by EventId | join EventId [search Criteria2 | stats latest(_time) as end by EventId] | eval duration=end-begin Which method is more speedy/less taxing? Is there a speedier method? I have learned to avoid transaction, so I used method 1 with Criteria1 alone, before discovering Criteria2, i.e., Criteria1 | stats latest(_time) as end earliest(_time) as begin by EventId | eval duration=end-begin . When I added Criteria2, the search slowed to a screech. That was when I started to contend with method 3. Because the observation that adding an "OR" criteria drastically slows down search is anecdotal and because apparent speed in Splunk is affected greatly by cache hit, I want to have a realistic assessment about efficiency. (Obviously I had done searches with only Criteria1 on the same data set before noticing slowdown when Criteria2 was added.) Also note that the actual output is more than just duration - I want to preserve event start time, for example.

yuanliu · ‎12-16-2014

addinfo is it. Thanks @Ayn

yuanliu · ‎12-16-2014

field=value earliest=-7d@d latest=@d When there are many events in these 7 days, I can use earliest(_time) and latest(_time) (or if I'm lucky with data, even last(_time), first(_time)) to determine the time range, thereby determine frequency. But when events are sparse, these functions differ too much from the specified earliest and latest time boundaries. I initially thought _span was available in stats, but it is not. How can I access the time range in stats?

yuanliu · ‎10-15-2014

In stats, values() can be used to enumerate values fitting the stats criteria. Is there a similar function to do this for individual events? For example, a meal can involve fork, knife and spoon multiple times, but I only want an unordered list of utensils have been used, not the full sequence in which they are used. Given inputs 07:10 food=milk utensil=bowl utensil=spoon food=cereal utensil=spoon food="scrambled eggs" utensil=fork food=milk utensil=bowl food=banana 12:00 food="hot dog" 18:00 food=salad utensil=fork food=bread utensil=knife food=soup utensil=spoon food=steak utensil=knife utensil=fork food=bread food=apple utensil=knife I want to have the following utensils values attached to each event: Breakfast: utensils="bowl, fork, knife, spoon" Lunch: Dinner: utensils="fork, knife, spoon" If there is a field "meal", I can perhaps eventstats values(utensil) as utensils by meal . But meal is not always present. Even _time may not be distinct enough. Besides, when dealing with millions of events, eventstats for values in each events looks insane. Is there a better way?

yuanliu · ‎10-14-2014

As @jrodman said, timechart by User does not give you a field named la but fields named after each User. The "as" clause is used for legend only. I think what you wanted is to find Maximum value of each User ; Round the maximum value to the 2nd decimal place; For those users whose maximum value in a given time span is greater than or equal to 10 and only for those, display users and their respective maximum values. Note the above also imply two logical consequences: Users whose maximum value has never reached 10 in the entire search period will never be shown. For users appearing on the chart, their results will show 0 in spans of time in which their maxima don't reach 10. The logic can be reversed to produce the exact same output, i.e., by limiting timechart only to those users who ever showed a value or values reaching or exceeding 10. | where value >= 10 | timechart max(value) as la by User | eval la=round(la,2) Because you are seeking maxima and not average, where you perform rounding doesn't affect the outcome. In other words, you can do | where value >= 10 | eval value=round(value,2) | timechart max(value) as la by User to get the same results although the first form is more efficient.

yuanliu · ‎10-14-2014

@adamw the link points to this question itself. This said, I have seen this before 6, too. Does it have to do with overload? I am under the impression that this tends to happen when volume is extremely high, although I do not have direct measurement.

yuanliu · ‎10-14-2014

If all you need is a gauge, you could then count the inactive, then compare to total as counted in inputlookup: | set diff [ inputlookup thingamajig | fields hostname ] [ activesearch |fields hostname | fields - _* ] | stats count as Active | join [ inputlookup thingamajig | stats dc(hostname) as Total ] | stats values(eval(1 - Active/Total)) as Ratio The way set works, it is important that you count Total after it.

yuanliu · ‎10-10-2014

So it is unrelated to curly bracket, but triggered by left-aggressiveness before field extraction. If I add left-aggressiveness to the first rex, the first rex will fail the long string, too. E.g., | rex field=emul "Begi.+:(?<DATA>.*)" | rex field=emul "Begin:.+}} (?<DATA>.+)" Output becomes count DATA short bar /stuff long

yuanliu · ‎10-09-2014

Let me rephrase: You have a lookup table that contains more unique hosts than your search result; you want to output hosts that are not found in your search result. If this is the case, this pseudo search will show you the hosts that do not show in your active search: | set diff [ inputlookup thingamajig | fields hostname ] [ activesearch |fields hostname | fields - _* ] Here, "thingamajig" is the lookup table containing all hosts in field hostname ; "activesearch" is a search that will return all active hosts as field hostname . No need to pipe into stats . Caveat: If "activesearch" comes back with a hostname that is not in "thingamajig", it will also show in results. (Worse, if this extraneous hostname is contained in more than one event, it will be repeated more than once unless you do stats in search.)

yuanliu · ‎10-08-2014

If the first rex is removed, the second rex indeed extracts no data with the long string.

yuanliu · ‎10-08-2014

When input length exceeds a certain threshold, it seems that some rex match will fail while others do not. Consider the following emulation: index=none |stats count | eval count=mvappend("short","long") | mvexpand count | eval emul="Begin:foo}} bar " + if(count="short",mvjoin(mvrange(10000,15000),"-"), mvjoin(mvrange(10000,20000),"-")) | rex field=emul "Begin:(?<DATA>.*)" | rex field=emul "Begin:.+}} (?<DATA>.+)" | eval DATA=replace(DATA,"[\d-]+","/stuff") | fields - emul Note: The first two lines simply produces a two-event sample. Field emul is populated using numerals between 10000 and 20000. The first event will contain a string roughly 5,000x6 byte=30,000 byte long, whereas the second event contains a string roughly 10,000x6 byte=60,000 byte long. The intent of the two cascaded rex commands is to utilize text behind double curly braces whenever possible. Because the two events are structurally identical, I expect both to produce bar /stuff . Such is the case when the second mvrange() is up to mvrange(10000,19000) , or ~9,000x6 byte=54,000 byte. Not much longer than that, output becomes count DATA short bar /stuff long foo}} bar /stuff In other words, the second rex fails to take effect. (The last two commands simply shorten output and have no effect on whether the second rex fails or not.) I see no error in job inspector and such. In my real-world search, complicated subsequent commands, including rex, do not appear affected, even after the equivalent of "Begin:.+}} (?.+)" fails. Is this a bug or is there some parameter I need to tweak? What is special about "Begin:.+}} (?.+)" ?

yuanliu · ‎09-22-2014

The side effect actually is not from fields alone, but has to be combined with subsequent transaction . In other words, if you have | fields x y z | transaction x you'll end up having fields x, y, and z whose null values are replaced with string "NULL". However, simply using |fields x y z would not have this effect.

yuanliu · ‎09-19-2014

As it turns out, I use fields user before performing transaction in order to reduce data rate. In other words, the field is forced to acquire a non-null value as a side effect of fields command. The value fields forces for an event that doesn't contain a listed field is string NULL.

yuanliu · ‎09-10-2014

This is not the case in my data. When I do |search user="NULL" after transaction, it returns transactions in which any constituent event is missing user , i.e., field user doesn't exist. In fact, my data has no user named NULL. Maybe this is special to transaction results?

yuanliu · ‎09-10-2014

For example, if all events in | transaction ID contain ID but only some carry user , I want to capture those transactions in which user is completely absent. | where user="NULL" AND mvcount(user)=1 gives the answer, but | where isnull(user) AND mvcount(user)=1 returns nothing. I thought "NULL" was a string representation of null value (therefore carries a higher intrinsic overhead when used for selection), so isnull(user) would be a more natural representation. Why wouldn't it work?

yuanliu · ‎09-08-2014

I'm looking at software behavior, akin to user behavior. A user loads any number of products into cart=>takes a series of steps (such as login=>open wallet), some have retries => then checks out or abandons cart. So "type" would be user session, and "mydata" would be user's steps. Final stats enumerates paths and their respective popularities. Unfortunately, |transaction type mvlist=t | mvexpand mydata |reverse |dedup type mydata gives 5 transactions, 3 of type 9, 2 type 15. Stats are also incorrect. compact type 2.10.8=>3.2.2 15... 3.4.6001=>2.14.1=>2.10.8 9...

yuanliu · ‎09-05-2014

Not having IP for reference really sucks if a user can connect from multiple locations using simultaneous VPN sessions. If user is allowed only one VPN session at a time, using user alone should suffice as you have startswith and endswith. bucket is perhaps not needed. BTW, if no other source has field src, | rename src as VPNExternalIP will do. http://docs.splunk.com/Documentation/Splunk/6.1.3/SearchReference/Rename

yuanliu · ‎09-05-2014

Thanks for help, @Tom_frosscher. Unfortunately, simulated data in the original question are inadequate. So I updated with real sample data. Oddly, test code below interferes with transaction itself. Instead of two transactions, verbose mode lists 4 transactions, two for each type. | mvexpand mydata | dedup mydata | mvcombine mydata | eval compact=mvjoin(mydata,"=>") | stats sum(eventcount) by compact type Stats show strange output: compact type count 2.10.8=>3.2.2 15 15 15 15 15 15 15 15 15 15 15 15 12 2.14.1=>3.4.6001 9 9 9 9 9 9 9 9 9 9 9 9 9 9 14

yuanliu · ‎09-04-2014

I wonder if you can combine xenapp:65:session in the transaction itself. It is unclear to me what src represents in juniper_sa_log. If it is the source IP and if that is the same as ClientAddress in xenapp:65:session, the following should work: sourcetype=juniper_sa_log eventtype=juniper_sa_authentication OR sourcetype=xenapp:65:session | rename ClientAddress as src | rename UserName as user | transaction user src mvlist=t startswith="eventtype=juniper_sa_authentication_success" endswith="eventtype=juniper_sa_authentication_logout" keepevicted=t

yuanliu · ‎09-04-2014

I have a transaction in which field mydata contains repeating values like ("xyz","ijk","ijk","abc","abc","abc","abc","abc","lmn","def","def"). I want to compact this list by representing repeating elements only once, but preserving the order in which each repetition occurs. In effect, this is the output of transaction mvlist=true . (Default transaction implies mvlist=false . The output is a compact, but unordered list.) Is there a list command to do this? The end goal is to illustrate a chain of events like "xyz=>ijk..=>abc..=>lmn=>def.." The closest discussion was http://answers.splunk.com/answers/95363/perform-transaction-for-only-repeating-values-of-field. But the answer there was partial, and does not apply to my use case. (Updated) Sample data are like: 16-May-2014 00:50:10.386 type=9,mydata=2.10.8 16-May-2014 00:55:23.205 type=9,mydata=2.10.8 16-May-2014 00:59:39.760 type=9,mydata=2.10.8 16-May-2014 01:12:26.410 type=9,mydata=2.10.8 16-May-2014 01:19:55.528 type=9,mydata=2.10.8 16-May-2014 01:41:33.508 type=9,mydata=2.10.8 16-May-2014 01:43:54.872 type=9,mydata=2.10.8 16-May-2014 11:53:43.119 type=9,mydata=2.14.1 16-May-2014 11:53:44.121 type=15,mydata=2.10.8 16-May-2014 11:55:46.376 type=15,mydata=3.2.2 16-May-2014 11:57:09.548 type=15,mydata=3.2.2 16-May-2014 11:58:03.658 type=15,mydata=3.2.2 16-May-2014 11:59:03.782 type=15,mydata=3.2.2 16-May-2014 11:59:06.788 type=15,mydata=3.2.2 16-May-2014 11:59:45.870 type=15,mydata=3.2.2 16-May-2014 12:00:07.914 type=15,mydata=3.2.2 16-May-2014 12:01:25.073 type=15,mydata=2.10.8 16-May-2014 17:01:07.343 type=9,mydata=3.4.6001 16-May-2014 17:19:41.923 type=9,mydata=3.4.6001 16-May-2014 17:20:58.090 type=15,mydata=2.10.8 16-May-2014 17:21:32.159 type=15,mydata=2.10.8 16-May-2014 17:21:51.198 type=15,mydata=2.10.8 16-May-2014 19:48:41.102 type=9,mydata=3.4.6001 16-May-2014 19:49:15.172 type=9,mydata=3.4.6001 16-May-2014 20:35:44.316 type=9,mydata=3.4.6001 16-May-2014 21:15:31.373 type=9,mydata=3.4.6001 With help from Perl community, I came up with the following string method. Though usable, I feel it is lame to use string to compact a list, given that Splunk is list oriented. source=mydata | transaction type mvlist=true | eval flatten=mvjoin(mydata,"=>") | eval compact=replace(flatten,"([^=]+)(?:=>\1)+","\1..") | stats max(eventcount) as count by compact type compact type count 2.10.8..=>2.14.1=>3.4.6001.. 9 14 2.10.8=>3.2.2..=>2.10.8.. 15 12 (Update: max(eventcount) gives the correct count, not sum(eventcount).) Actual chain of events can easily be tested in shell: $ for t in type=9, type=15,; do fgrep $t < mydata |cut -d\ -f3-|uniq -c; done 7 type=9,mydata=2.10.8 1 type=9,mydata=2.14.1 6 type=9,mydata=3.4.6001 1 type=15,mydata=2.10.8 7 type=15,mydata=3.2.2 4 type=15,mydata=2.10.8 Note: The original example used mvappend to simulate output from transaction, as listed below. ( source=* can be a search that returns at least one event.) But this simulation apparently lacks some important aspect of a transaction. source=* | eval mydata=mvappend("xyz","ijk","ijk","abc","abc","abc","abc","abc","lmn","def","def") | eval flatten=mvjoin(mydata,"=>") | eval compact=replace(flatten,"([^=]+)(?:=>\1)+","\1..") | stats values(mydata) by compact flatten compact flatten values(mydata) xyz=>ijk..=>abc..=>lmn=>def.. xyz=>ijk=>ijk=>abc=>abc=>abc=>abc=>abc=>lmn=>def=>def abc def ijk lmn xyz

yuanliu · ‎08-13-2014

I used Splunk search to feed FFT here: http://answers.splunk.com/answer_link/149675/. A really basic test to understand input from Splunk is to run | r "output=input" input is a data frame composed of your search results as well as some Splunk implicit fields. (Data frame is the biggest revelation to me, thanks to @rfujara_splunk.) All fields are prefixed with "X". For example, X_time is Splunk _time , X_span is Splunk _span if you used timechart or bucket ; if your search outputs a field host , R sees it as Xhost . output is the data frame to send back to Splunk. Each component name is used as a field name by Splunk.

Posts	2165
Solutions	335
Karma Given	190
Karma Received	731
Member Since	‎11-12-2013

Online Status	Offline
Date Last Visited	13 hours ago

How to prevent Splunk on MacOS from "freezing"

How to fix wandering trellis order?

How to force chart over _time to show trailing zer...

How to fix CSV ingestion random cutoffs?

How to set "abnormal" tallies to null after timech...

How to insert hyperlink in Dashboard Studio?

Why the error "The lookup could not be loaded from...

How to fix weird futuristic time range in timechar...

Any way to filter multiple wildcard lookup matches...

Why does lookup return null when there are multipl...

How to compute concurrent members in events?

Re: "OR" and subsearch, which one is more efficien...

Re: "OR" and subsearch, which one is more efficien...

Re: "OR" and subsearch, which one is more efficien...

"OR" and subsearch, which one is more efficient?

Re: How to use search time range in stats?

How to use search time range in stats?

How to enumerate values in an individual event?

Re: Why am I unable to use comparison operators "g...

Re: Why is Splunk not maintaining line breaks as i...

Re: How to compare output of a search to a lookup ...

Re: Why does rex match fail when input length exce...

Re: How to compare output of a search to a lookup ...

Re: Why does rex match fail when input length exce...

Why does rex match fail when input length exceeds ...

Re: What is the difference between user="NULL" and...

Re: What is the difference between user="NULL" and...

Re: What is the difference between user="NULL" and...

What is the difference between user="NULL" and isn...

Re: How to compact a list of repeating field value...

Re: How to combine transaction search results of o...

Re: How to compact a list of repeating field value...

Re: How to combine transaction search results of o...

How to compact a list of repeating field values bu...

Re: Simple example of passing Splunk search result...