I currently have a search that is looking at firewall data that looks something like this:
index=my_index sourcetype=fw_data action=drop src_ip!="10.0.0.0/8"
| fields src_ip src_port dst_ip dst_port proto action
| my_special_command dst_ip
| search my_command_output_field=true
| dedup src_ip dst_ip dst_port
| table _time src_ip src_port dst_ip dst_port proto action my_command_output_field
The problem I'm having is that when I pass my list of dst_ip (destination IP) addresses to my custom command (my_special_command), I'd like to pass the list dedup'd instead of as-is, so my custom command isn't running across the same values multiple times. However, I want that done in such a way that the original list of dst_ip addresses is not impacted/dedup'd once I continue my search. In other words, I want the list dedup'd, but only for the purpose of invoking my custom command.
For a little background, in this instance, my custom command takes a destination IP when it is invoked and returns an additional field (my_command_output_field) back to Splunk from an external API call that adds context to the data in my initial search. That piece works fine, however, when I pass that list to the custom command, I don't want the list to have any duplicates, as it really impacts performance. I've tried re-writing the invocation of the custom command part in my search a few different ways (I tried copying dst_ip to a new value via eval in one instance, and "join" in another) and haven't found a solution that works the way I want it to.
I know I could probably get a join statement to work if I re-ran the entire search up to that point in a sub-search, but to me that seems way less efficient since I'm then running the same search twice. What I really want to do is just take dst_ip, dedup it just for the purpose of passing it to the custom command, and then taking the additional field it returns and applying it to a non-dedup'd copy of dst_ip addresses without having to re-run the whole thing from the start.
Is this possible in Splunk, or is there maybe a better way/different way of looking at this that I haven't considered?
... View more
I'm looking to create a report that lists out the occurrences of a given event, but also includes information about the previous instance of the event for a given user. Let's start with some sample data. Say this is all stored in myindex1 with the sourcetype mysourcetype1:
Let's say I'm making the report about Auth Success events. Say I iterate through a span of time in my index listing out all success events. So, my search would be something like:
index=myindex1 sourcetype=mysourcetype1 Event="Auth Success"
| table _time Event Event_ID User MyValue1
Which essentially returns the "Auth Success" events listed above in the sample data. However, for each of those lines/occurrences, I'd like to include two additional values-- one being the time that event last occurred prior to the given instance, and the other being the "MyValue1" field from that previous occurrence. So in this case, for the Auth Success event for mbojangles at 11:07 (event #5 in the first table), the line in my report would look like this:
So not only am I showing the info for event #5 from the first table (since it was an instance of a successful login), I'm also going back and finding the last successful auth event for mbojangles, which happens to be event #1 in the first table, and adding that time in another column, along with the value of "MyValue1" for event #1, and combining them all on to the same line.
What is the best way for me to achieve this? I've seen a few hints/ideas in other questions, but I'm still not sure what the best approach is for this specific situation. I'd also like the search to be able to handle an instance where no previous information is available for a given event. So, for example, let's say I had event #1 from the first table on my report... there is no previous "Auth Success" event for mbojangles prior to that, so my last two columns would either show empty or "N/A" or something along those lines.
... View more
Wow, I did an even worse job of explaining this than I originally thought, but that's ok, and I'm glad you mentioned that. So the biggest part that I failed to mention is that for sourcetype BCD, X exists in both myindex2 AND myindex3. That's what I'm trying to do the conditional join for, but I haven't found a way that I can write "pull fields X and Y from sourcetype ABC in myindex1, and where you see value X in myindex2 for sourcetype BCD, join that with myindex3 on value X to grab value Y. Sorry, it's late in the day and my brain isn't totally functional. I hope that makes more sense.
... View more
Sorry, I should have explained that more clearly-- X always uniquely corresponds to Y (consider X a unique code for Y) for both sources. X and Y always exist in a pair like that, however, each pair does not always necessarily exist in both sources (sometimes it does, sometimes it doesn't). In no case should a value of X in the one source correspond to a different value for Y in the other (or vice-versa), it should always be consistent.
... View more
So I have two log sources-- one that stores values X and Y together in the same index, and the second which stores value X in one index and value Y in another. I need to figure out a way to have Splunk get both X and Y together for both sources in a search so I can work with that data further.
So for example, sourcetype ABC has the following data in myindex1:
sourcetype BCD has the following data in myindex2:
and the following data in myindex3:
Both sourcetypes are the same data, they're just coming from two different places and being stored differently.
I'm sure the first question that might come up is "why are you storing the data in a way that makes no sense," but unfortunately, that is well outside the scope of this and something that cannot be controlled in this particular situation. I need to find a way to pull the data from both sourcetypes together in one search so I have values X and Y for both, together, and can manipulate the data from there.
I tried doing a conditional join inside an eval statement, but every way I wrote it seemed to produce an error.
I'd love to hear any suggestions anyone has on other ways of attacking this problem-- I've seen some other threads on conditional searching and the like, but I haven't been able to find syntax that would work for this particular situation.
... View more
So right now I have a summary index that is being populated by the following command:
earliest=-20m latest=-5m | bucket _time span=5m | sistats count by _time host sourcetype index
The idea is that I'd like to have a record of how many new events were added for each host/sourcetype/index in five minute increments. I'm running that every 15 minutes, using the time window specified (going back -20/-5 in case anything is a little slow to be indexed - I care more about the time the event was generated than when it made it into the index in this case). I'd like to be able to take this data and do a number of different things with it, and I'm wondering what's doable and what isn't:
I'd like to be able to sum counts together using the stored values in the summary index to give me a count of events for each host, or sourcetype, or index, over various spans of time. So I'd like to be able to say, using that stored data, tell me how many events were generated for each sourcetype for the last week. Or each index, or each host, etc. Is this doable using the way I'm storing the data above?
I'd like the ability to timechart against the stored summarized data, too. So I'd like to be able to create a timechart showing me counts per sourcetype over time (or index, or host, etc.). Is this doable given how the data is being stored above, especially since I'm not using sitimechart to store it?
The main reason I'm asking is I don't have the best grasp of what is doable and what is not via the sistats/sitimechart/etc. commands, and the best way of populating summary indexes that will give me the flexibility I need when it comes time to report on that data.
... View more
I'm looking to read in a set of field name/value pairs from a given lookup table (using inputlookup) and then use that as a set of parameters in a search. Specifically, I'm looking to search for "field_name = X" value in a given index and then use that list of field/value pairs as an exclusion list to what I'm searching for. I could also have multiple sets of field/value pairs. So, in other words, in plain english:
Search for FieldName = X in ABC Index
But not if (A=blah AND B=doh AND C=meh) OR (B=meh AND C=blah)
and the A/blah, B/doh, C/meh etc. values would all be field/value pairs in the lookup table that are read in via inputlookup.
So far I've successfully used inputlookup w/csv tables for filtering out/including single values where I pre-define the field name in the search, but I haven't found a good way of doing more complicated logic where the field name is actually being read from a table and you can have multiple groupings. I'd appreciate any feedback/help anyone has to offer. Thanks!
... View more