About doweaver

doweaver · ‎12-04-2017

User 1 has logged in already and we are collecting the pages he / she has visited during the session I didn't realize this was a possible distinction. Someone can save their login, or something, so they can have a new session without logging in?

doweaver · ‎12-04-2017

Oh! I meant "streamstats", not "eventstats". Updating my original answer (and to incorporate the sessioni identifier) What that will do is "count" the pages post login, so the first page they visit will be the first event for that clientSessionId, and will get PageNumVisited set to 1, the second set to 2, etc. By filtering out anything above 5, you should filter out the sixth pages visited and beyond.

doweaver · ‎12-04-2017

index=wholesale_app buildTarget=blah product=product1 Activity Properties.index=5 | sort 0 _time | streamstats count AS PageNumVisited BY clientSessionId | search PageNumVisited<=5 | top activitytype What's happening here: - Sorting the events by _time, increasing (oldest events first) - Using streamstats to tag the event with the order in which it was visited (the first page will get 1, the second page will get 2, etc.) - Filtering out pages beyond the first five - Using top to see the top pages visited after filtering out anything beyond the first 5

doweaver · ‎12-04-2017

Thanks! Yeah, "map" does work here, although it's really slow, and doesn't appear to work for more than a few hundred files (well, it sat at "Finalizing job" for a few hours before I killed it). I've upvoted but I won't mark the question as solved yet, per @niketnilay's advice in the comments on his answer.

doweaver · ‎12-02-2017

Definitely! Thanks for all your help today 🙂

doweaver · ‎12-02-2017

This did work! It takes a really long time to run (10+ minutes) with no intermediate results, because of the embedded search, but it works! Thanks for the help! Map is a really powerful command I haven't explored in the past. If I discover any faster way of doing this, I'll definitely update.

doweaver · ‎12-02-2017

Yes, this runanywhere search does add the dummy events.

doweaver · ‎12-02-2017

The desired outcome for stats p50(SizeMB) BY File Would be: Foo: 10 Bar: 0

doweaver · ‎12-02-2017

Adding dummy events like that would be one way to solve it, yes. Not sure if there are others. Each event is from a single machine, and contains information on a single file. As the events are currently constructed, there aren't any mvfields.

doweaver · ‎12-02-2017

Unfortunately, it's impossible for me to statically define my list of possible machines in a lookup file - there's not a predetermined set. The list of files is statically defined... let me play around with your suggestion and see if I can get it to work with the single lookup. Thanks!

doweaver · ‎12-02-2017

To clarify - a machine can have many files. In this example, Machine A has files "Foo" and "Bar", and the rest only have file "Foo".

doweaver · ‎12-02-2017

I have a set of events that look something like the following: Machine: A, File: Foo, SizeMB: 10 Machine: A, File: Bar, SizeMB: 100 Machine: B, File: Foo, SizeMB: 10 Machine: C, File: Foo, SizeMB: 10 Machine: D, File: Foo, SizeMB: 10 Not all machines contain all files (Machine A is the only one with File "Bar" in this example). I'd like to calculate the percentile size for each file, but I want the file to be treated as size 0 when it's not present. Given the above example, if I tried to calculate the 50th percentile size for each file, the calculation would get the 50th percentile of (100), since we only have the one event. What I really want is to get the 50th percentile of (100, 0, 0, 0), but I've yet to come up with a reasonable way to do that. Anyone have any ideas?

doweaver · ‎12-02-2017

I don't think filldown works - that would blindly insert the most recently seen value for "FieldToInject", which means it would fail when we don't have A followed by all its matching Bs, and repeat. selfjoin doesn't seem like the answer here either, since I just want to join A to all Bs, and not Bs to Bs. I'm not seeing anything giving me that flexibility in the doc, but I'll keep looking.

doweaver · ‎12-02-2017

Updated with desired output.

doweaver · ‎11-30-2017

Yeah, my phrasing here is a product of my lack of understanding of what's happening under the covers. It sounds like you're suggesting the aggregation approach, which should work. The main reason I asked this question is because of a "eureka" moment I had a few months ago. I was doing a lot of gross "stats first(x), first(y) BY Z" on things when I only expected a single event per Z... and then I discovered that the "xyseries" was the "right" way to do that. I was wondering if there's a "right" way to do what I wanted above, but it sounds like aggregating and using "first" is the best there is.

doweaver · ‎11-30-2017

I have a set of events with the pattern that there's a single event A that pairs with many event Bs (based on a field let's call CorrelationId). The event A has a field I want on all of the B events. The events can come in in any order. We might have the following (event type followed by CorrelationId): A (Correlation ID: 1, FieldToInject: 10) B (Correlation ID: 1) B (Correlation ID: 1) B (Correlation ID: 1) B (Correlation ID: 1) B (Correlation ID: 2) B (Correlation ID: 2) B (Correlation ID: 3) A (Correlation ID: 3, FieldToInject: 100) A (Correlation ID: 2) FieldToInject: 50 B (Correlation ID: 2) B (Correlation ID: 2) B (Correlation ID: 3) B (Correlation ID: 3) The new output should look like: A (Correlation ID: 1, FieldToInject: 10) B (Correlation ID: 1, FieldToInject: 10) B (Correlation ID: 1, FieldToInject: 10) B (Correlation ID: 1, FieldToInject: 10) B (Correlation ID: 1, FieldToInject: 10) B (Correlation ID: 2, FieldToInject: 50) B (Correlation ID: 2, FieldToInject: 50) B (Correlation ID: 3, FieldToInject: 100) A (Correlation ID: 3, FieldToInject: 100) A (Correlation ID: 2) FieldToInject: 50 B (Correlation ID: 2, FieldToInject: 50) B (Correlation ID: 2, FieldToInject: 50) B (Correlation ID: 3, FieldToInject: 100) B (Correlation ID: 3, FieldToInject: 100) There are a couple of ways I can think of to do this. I could use an aggregation: eventstats first(FieldToInject1) AS FieldToInject1, first(FieldToInject2) AS FieldToInject2 BY CorrelationId That works, but I don't imagine is very efficient - I know all of the events will come in in a short window, but this call will keep looking for events from a given CorrelationId throughout the entire search. The other obvious option is a transaction: transaction CorrelationId maxspan=1m The problem here is that, because we more than one B event, I need to play games of zipping of multivalue fields and then mvexpanding to make any sense of things. Is there a more natural way folks would recommend attempting to do something like this?

doweaver · ‎11-21-2017

Got it! One more question, I think 🙂 Looking at transformations, I'm not understanding how "Format" comes into play. I have the following regex: (?:\d*\.){2}(?<BuildNumber>\d+)\.(?<BuildRevision>\d+)\.(?<BuildArch>[^\.]*)\.(?<BuildBranch>[^\.]*) Format says to "Specify the event format in terms of field names and values", but I'm already naming the fields in my regex extraction.

doweaver · ‎11-21-2017

(Using a CSV with headers as input data) The default construction of the regex using the field extractor UI was to essentially skip the first two columns of the CSV, and look for the fields to extract in the third column. If I added another column of data that went before the field I wanted to use for the regex, it would break, unlike if I'm scoping it right to the field name (which will be correctly identified by the headers in the CSV).

doweaver · ‎11-21-2017

I see - and it can only work on _raw? I was hoping I could use a specific field to pull from, so I'm not married to some strict ordering in the future...

doweaver · ‎11-21-2017

I want to do this automatically at every search, though, rather than stapling the "rex" command to every search. My understanding was that field extractions were the right way to do this, but maybe not?

doweaver · ‎11-21-2017

I'm attempting to create a field extraction from the web UI (I'm not an admin and don't have access to "*.conf" files) using regex on a field that's not "_raw"). I can't seem to find any documentation on it, but I see some existing extractions with the format: "FieldToExtractFrom":"(?<ExtractedField>\d+)" Using that same format, though, I don't get the automatic extraction, even though the regex DOES work if I run it manually at search time with "rex". Is there some way to specify an inline regex field extraction from a field other than "_raw"? EDIT: Field transformations did the trick, since it looks like extractions can only work on "_raw".

doweaver · ‎06-07-2017

Thank you! Exactly what I was looking for!

doweaver · ‎06-07-2017

I have some data I'm trying to rearrange into an appropriate table for visualization. It starts out like this: Group Subgroup Value 1 A 100 1 B 300 2 A 500 2 B 700 3 A 1000 3 B 2000 I want to transform it to look like this: 1 2 3 A 100 500 1000 B 300 700 2000 (Where the column headers are the group, and the rows are the subgroups) I can accomplish this by calling: chart first(Value) BY Subgroup, Group ...but that doesn't seem like the right approach. i'm calling an aggregation method when I'm not actually DOING any aggregating, just transforming. Is there a better way to handle this?

doweaver · ‎11-03-2016

Hmm - I tried to post your comment as the answer, but Splunk is saying I can't make more than 2 posts per day until I hit 40 points. Pretty sure I've only made one post today, but... /shrug If you paste that same thing as the answer, I'll mark it solved 🙂

doweaver · ‎11-03-2016

Yes! Perfect! Didn't realize appendpipe was a thing. Thanks for your help!

Posts	30
Solutions	0
Karma Given	16
Karma Received	8
Member Since	‎10-04-2016

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

Calculating percentiles and including "missing" da...

Injecting fields into other events?

Field regex extractions on non-raw fields from web...

Transforming tables without hacky chart(first) usa...

How to create a multistage Sankey diagram with a s...

Re: first 5 pages visited post login

Re: first 5 pages visited post login

Re: first 5 pages visited post login

Re: Calculating percentiles and including "missing...

Re: Calculating percentiles and including "missing...

Re: Calculating percentiles and including "missing...

Re: Calculating percentiles and including "missing...

Re: Calculating percentiles and including "missing...

Re: Calculating percentiles and including "missing...

Re: Calculating percentiles and including "missing...

Re: Calculating percentiles and including "missing...

Calculating percentiles and including "missing" da...

Re: Injecting fields into other events?

Re: Injecting fields into other events?

Re: Injecting fields into other events?

Injecting fields into other events?

Re: Field regex extractions on non-raw fields from...

Re: Field regex extractions on non-raw fields from...

Re: Field regex extractions on non-raw fields from...

Re: Field regex extractions on non-raw fields from...

Field regex extractions on non-raw fields from web...

Re: Transforming tables without hacky chart(first)...

Transforming tables without hacky chart(first) usa...

Re: How to create a multistage Sankey diagram with...

Re: How to create a multistage Sankey diagram with...