About bowesmana

bowesmana · ‎06-21-2020

Here's an example dashboard that allows you to set the number of days before the click date to then show the table for <form> <label>testtc</label> <search id="base"> <query>| makeresults | eval x=mvrange(1,100) | mvexpand x | eval val=random() % 100 | eval _time=_time-(x*86400) | timechart span=1d values(val)</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <fieldset submitButton="false"> <input type="text" token="days" searchWhenChanged="true"> <label>Show days before click</label> <default>5</default> </input> </fieldset> <row> <panel> <chart> <search base="base"> <query/> </search> <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option> <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option> <option name="charting.axisTitleX.visibility">visible</option> <option name="charting.axisTitleY.visibility">visible</option> <option name="charting.axisTitleY2.visibility">visible</option> <option name="charting.axisX.abbreviation">none</option> <option name="charting.axisX.scale">linear</option> <option name="charting.axisY.abbreviation">none</option> <option name="charting.axisY.scale">linear</option> <option name="charting.axisY2.abbreviation">none</option> <option name="charting.axisY2.enabled">0</option> <option name="charting.axisY2.scale">inherit</option> <option name="charting.chart">line</option> <option name="charting.chart.bubbleMaximumSize">50</option> <option name="charting.chart.bubbleMinimumSize">10</option> <option name="charting.chart.bubbleSizeBy">area</option> <option name="charting.chart.nullValueMode">gaps</option> <option name="charting.chart.showDataLabels">none</option> <option name="charting.chart.sliceCollapsingThreshold">0.01</option> <option name="charting.chart.stackMode">default</option> <option name="charting.chart.style">shiny</option> <option name="charting.drilldown">all</option> <option name="charting.layout.splitSeries">0</option> <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option> <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option> <option name="charting.legend.mode">standard</option> <option name="charting.legend.placement">right</option> <option name="charting.lineWidth">2</option> <option name="trellis.enabled">0</option> <option name="trellis.scales.shared">1</option> <option name="trellis.size">medium</option> <drilldown> <set token="latest">$click.value$</set> <eval token="relative_days">"-".$days$."d"</eval> <eval token="earliest">relative_time($click.value$,$relative_days$)</eval> <eval token="from">strftime($earliest$,"%F")</eval> <eval token="to">strftime($latest$,"%F")</eval> </drilldown> </chart> </panel> <panel> <table> <title>Showing results from $from$ to $to$</title> <search base="base"> <query> | where _time>=$earliest$ AND _time<=$latest$ </query> </search> </table> </panel> </row> </form> It's a run anywhere so you can see how the eval token statements in the drilldown use relative_time to calculate the number of days prior to the clicked date to set the from token. You can see I have used the where clause to do the date filtering in the second search as I have used a base search, but in your case you could just do this in you original search you_search earliest>=$earliest$ latest<=$latest$ for the same effect. Hope this helps.

bowesmana · ‎06-19-2020

I believe you will have to mvexpand out the fields first to get the mv field into separate rows and then run streamstats. Take this for example | makeresults | eval _raw="sysmodtime,idnumber,epoch time 05/03/20 12:40 PM,1,1588502400 05/01/20 12:01 AM,1,1588284060 05/01/20 12:02 AM,1,1588284120 05/01/20 12:02 AM,1,1588284120 05/02/20 12:00 PM,2,1588413600 04/02/20 12:00 AM,2,1585778400 04/02/20 01:00 AM,2,1585782000 04/02/20 02:00 AM,3,1585785600" | multikv | stats list(*) as * by idnumber | eval x="So, up to here is your original data" | eval base=mvzip(sysmodtime,epoch_time) | fields - _raw linecount sysmodtime epoch_time | mvexpand base | rex field=base "(?<sysmodtime>[^,]*),(?<epoch_time>.*)" | fields - base | streamstats window=2 range(epoch_time) as time_diff by idnumber | stats list(*) as * by idnumber | eval time_diff=mvindex(time_diff,1,-1) That gives you what you want, but will be expensive to do the mvexpand and re-aggregations, but you can measure that. As to calculating values inside MV fields, if you have Splunk 8 I think, then mvmap may be able to get you to where you want, I couldn't get it to work though - I've not used it before.

bowesmana · ‎06-18-2020

| makeresults | eval Message="The following client performed a SASL (Negotiate/Kerberos/NTLM/Digest) LDAP bind without requesting signing (integrity verification), or performed a simple bind over a cleartext (non-SSL/TLS-encrypted) LDAP connection. Client IP address: 172.4.5.6:57157 Identity the client attempted to authenticate as: DELTA\Kelly Binding Type: Fixed..... Please close" | rex field=Message "(?ms).*Client IP address:[^\d]+(?<ip>\d+\.\d+\.\d+.\d+).*authenticate as:[\r\n\s]+(?<domain>\w+)\\\(?<user>[\w ]+)" This should work and will extract the fields ip, domain and user. If you want to include the user and domain as a single value, then get rid of the (?<domain>\w+) \\\ and add change the user extraction to (?<user>[\w\\\ ]+)

bowesmana · ‎06-18-2020

Sure, I wasn't suggested you use a table, just pointing out Splunk ways to achieve what you were trying to do

bowesmana · ‎06-17-2020

Thinking out loud, but this could be done easily if the first chart was a <table> and you could number the rows with | eval row=1 | accum row and in the drilldown have <drilldown> <eval token="even_click_version">if($row.row$%2=0,$row.row$,$even_click_version$)</eval> <eval token="odd_click_version">if($row.row$%2=1,$row.row$,$odd_click_version$)</eval> </drilldown> The full example, but with searches that work in my data <row> <panel> <title>Here is the chart</title> <table> <search> <query>index=location | chart values(tid) as v by type | eval row=1 | accum row </query> <earliest>$time_range.earliest$</earliest> <latest>$time_range.latest$</latest> </search> <option name="drilldown">cell</option> <option name="refresh.display">progressbar</option> <drilldown> <eval token="even_click_version">if($row.row$%2=0,$row.type$,$even_click_version$)</eval> <eval token="odd_click_version">if($row.row$%2=1,$row.type$,$odd_click_version$)</eval> <set token="type">$row.type$</set> </drilldown> <fields>"type","v"</fields> </table> </panel> </row> <row> <panel depends="$type$"> <title>LEFT PANEL</title> <table> <search> <query>index=location type=$even_click_version$ | table type lat lon</query> <earliest>$time_range.earliest$</earliest> <latest>$time_range.latest$</latest> </search> <option name="refresh.display">progressbar</option> </table> </panel> <panel depends="$type$"> <title>RIGHT PANEL</title> <table> <search> <query>index=location type=$odd_click_version$ | table type lat lon</query> <earliest>$time_range.earliest$</earliest> <latest>$time_range.latest$</latest> </search> <option name="refresh.display">progressbar</option> </table> </panel> </row> Problem is that <chart> does not support <fields> so you can't hide the row value used for the drilldown. What you could do using the above principle, is to, for example, add the row to the version, with something like | eval row=1 | accum row | eval version=version."/".row and then in the drilldown eval do <eval token="even_click_version">if(tonumber(replace($row.version$,".*/(\d+)","\1"))%2=0,tonumber(replace($row.version$,".*/(\d+)","\1")),$even_click_version$)</eval> <eval token="odd_click_version">if(tonumber(replace($row.version$,".*/(\d+)","\1"))%2=0,tonumber(replace($row.version$,".*/(\d+)","\1")),$odd_click_version$)</eval> although I didn't manage to make that work, it _should_ be possible to eval that out

bowesmana · ‎06-17-2020

The same principle applies for all 'tokens' you want to pass in to the Splunk dashboard, they are just url parameters prefixed with 'form.'. For example this url https://yoursplunkhost/en-GB/app/app_name/dashboard?form.time_range.earliest=-30d%40d&form.time_range.latest=now&form.level=*&form.first_token=ValueOfFirstToken&form.second_token=ValueOfSecondToken will set the time picker token 'time_range' for earliest=-30d@d latest=now and it will set the token named 'first_token' and 'second_token' as above. Sorting is just managed in the search. By default Splunk will show you indexed events in reverse chronological order, so depending on what visualisation you are doing, you may not need to do any sorting, but Splunk sort is in a simple form | sort fieldname but check the docs for full details https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Sort

bowesmana · ‎06-17-2020

The mobile app does have limitations. I am only familiar with the Android version, but I know the development team has been working heavily to improve the app and there has been a lot of work over the last few months to address early issues from the initial release. There are still some limitations, e.g. some visualisation types that cannot yet be shown referred to in the other post. Also, drilldown is not currently working, so if your app uses any kind of conditional panels and drilldown, then they will also not work. However, the team is definitely working towards improving the experience - but I am not privy to any timescales within Splunk.

bowesmana · ‎06-17-2020

A few points/questions 'and' between the tests should be capitalised AND https://docs.splunk.com/Documentation/Splunk/8.0.4/Search/Booleanexpressions - you say 'creation hybris' is the field (2 words separated with space), but it's a single word in the test. - your example search query is quoted in its entirety, is that what it really is or is it quoted here for example Can you provide the <drilldown> section from the panel and the <query> section from your subsequent search.

bowesmana · ‎06-17-2020

If you have a Splunk dashboard with an input text box with a token name 'search_data', then you can link to this dashboard with https://yoursplunkhost/en-GB/app/search/web_dashboard?form.search_data=Hello%20World where in the above URL, 'search' is the name of the Splunk app, web_dashboard is the name of your dashboard and all tokens you are passing in to that dashboard are prefixed with 'form.' Your search in the dashboard would already need to have that search filter enabled as part of the search, so it would look something like your search $search_data|s$ so here your input search filter token is added as part of the search query - note the |s at the end of the name will cause it to double quote the value of the search string, effectively the same as doing "$search_data$" Hope this helps.

bowesmana · ‎06-17-2020

Thanks for the update. Happy to submit a fix - but was really looking to understand if what I deem a fix is actually correct under the original implementation, as it looks like you've taken pains to do the json.loads and catch exceptions and I don't get why.

bowesmana · ‎06-17-2020

Please provide a data sample indicating how - from the data - you would be able to tell which logs relate to which context. If you have nothing in the data that says what application or environment those logs come from, then you cannot make a search that will perform that. If your only link from data to environment is by host, then you will need to maintain a lookup of hosts that represent each environment and you can then use that lookup to construct the hosts that will be used in the search. However, that still leaves you with how to determine the application for a particular a log entry. Data samples with explanation of how this is represented is needed. Thanks

bowesmana · ‎06-16-2020

Here's a basic dashboard with static values for what you want and a basic search panel, but you have not explained how you can correlate the app and the environment against your data. So, when the example search is searching for data, it is searching app and env fields, which may or may not exist in your data. <form> <label>Test</label> <fieldset submitButton="true" autoRun="false"> <input type="dropdown" token="app" searchWhenChanged="true"> <label>Application</label> <choice value="*">All</choice> <choice value="abc">ABC</choice> <choice value="xyz">XYZ</choice> </input> <input type="dropdown" token="env" searchWhenChanged="true"> <label>Environment</label> <fieldForLabel>envs</fieldForLabel> <fieldForValue>envs</fieldForValue> <choice value="*">All</choice> <choice value="prod">Prod</choice> <choice value="qa">QA</choice> <choice value="dev">Dev</choice> </input> <input type="dropdown" token="sourcetype"> <label>Sourcetype</label> <choice value="*">All</choice> <choice value="catalina_logs">Catalina Logs</choice> <choice value="access_logs">Access logs</choice> </input> </fieldset> <row> <panel> <event> <search> <query>index=* app=$app|s$ env=$env|s$ sourcetype=$sourcetype|s$ | stats values(host) by sourcetype </query> <earliest>-24h@h</earliest> <latest>now</latest> </search> <option name="list.drilldown">none</option> </event> </panel> </row> </form>

bowesmana · ‎06-16-2020

Good point @DalJeanis about the wildcards - you're right, that particular construct is not something you're ever likely to want to do on _raw data given all the additional fields you'd collect on the way, so worth pointing out.

bowesmana · ‎06-16-2020

Can you provide a sample of the data so I can see how you expect to find the data rows for each tool/environment? Dashboards can use cascading tokens in a search that populates the contents of the dropdown, for example <form> <label>Test</label> <fieldset submitButton="false"> <input type="dropdown" token="tool" searchWhenChanged="true"> <label>Tool</label> <choice value="tool1">Tool 1</choice> <choice value="tool2">Tool 2</choice> </input> <input type="dropdown" token="env" searchWhenChanged="true"> <label>Environment</label> <fieldForLabel>envs</fieldForLabel> <fieldForValue>envs</fieldForValue> <search> <query> | makeresults | eval envs=case($tool|s$="tool1","Dev:QA:Prod",$tool|s$="tool2","Dev:QA:Prod") | makemv delim=":" envs | mvexpand envs</query> <earliest>-24h@h</earliest> <latest>now</latest> </search> </input> </fieldset> </form> shows how in the second dropdown it uses the value of the selected tool token to build a list of environments (in this case they are identical - but if your data supports it you can get this from the data or take it from a lookup as other options.

bowesmana · ‎06-16-2020

Is the time range different in each search? In the UI, the time would be applicable to the time zone of the UI, if your timestamps are in UTC for eaxmple. Whereas, I'm not sure how the timezone for the search would be interpreted running as the Python script, but probably the system time zone. Not sure if this will help, but I have had a similar issue with different results for different searches based on time zone settings.

bowesmana · ‎06-16-2020

bowesmana · ‎06-15-2020

Maybe start with something like base search | streamstats time_window=100ms values(*) as * by user where the field 'user' is your username field, but at this point, it very much depends on what you want to do with that data

bowesmana · ‎06-15-2020

You can use autoregress. index=abc <jobname> | stats earliest(_time) AS begin, latest(_time) AS end count by source | sort 0 begin | autoregress end as prev_end p=1 | where begin<prev_end | convert ctime(begin), ctime(end) | sort - count If that doesn't give you what you want, then consider using streamstats to calculate the window I am not sure of the relevance of count in your scenario. Hope this helps.

bowesmana · ‎06-15-2020

You can use streamstats to capture window type behaviour, e.g. | streamstats time_window=5m avg("% _total Prcessor times") as CPUMetric by host | where CPUMetric>80 | stats values(host) That's using average CPU over the 5 minute window, but you could also use any of the stats aggregations, e.g. min() to find servers that never reported <80% perc90() to find servers that reported over 80% for 90% of the reports during the window and so on. Hope this helps

bowesmana · ‎06-15-2020

Is there any way to determine the group based on the server name or something else. If the host name contains the environment in some form, e.g. WBPRDSRV01 WBDEVSRV01 and so on, then you could derive group from the host. In the absence of that, you could create a lookup table with server name and group, but that would need to be maintained Are you looking to be able to select the group in the dashboard and then query according to that group or show all groups on the same dashboard. If you just want to select a group, then you just an a dropdown input, which could be populated from the groups on your lookup. A little more detail would help.

bowesmana · ‎06-15-2020

In a query that culminates in a curl command on a resuult, when the result set if empty, it's not possible to prevent the curl command from trying to execute. A workaround I am using is to set the urifield rather than using uri, as then when there are not results and no urifield, the curl call just returns an error saying no URL specified. However, it would be nice to be able to indicate the curl command do nothing when the results.len = 0. Currently when results is 0, it just goes and tries to run the curl. My search is running as a saved search and all iterations will record an error when there are no results.

bowesmana · ‎06-15-2020

I think there may be a bug with the use of datafield in the POST operation with curl. Currently the curl.py python (line 259) code does data = json.loads(result[options['datafield']]) which will create a Python object called ‘data’ containing the parsed JSON fragment. In my example, my JSON payload is {"devices": ["arn:aws:sns:ap-southeast-2:000000000000:endpoint/GCM/bcrm/cabf12dc-ec12-3af4-12e1-121a193ecc7b"], "message": {"title": "Reminder to complete your self checkup", "body": "Your last completed self checkup is to old. Please perform the self checkup before attending the workplace"}} The curl_data_payload is then output with the debug=true setting as {u'devices': [u'arn:aws:sns:ap-southeast-2:000000000000:endpoint/GCM/bcrm/cabf12dc-ec12-3af4-12e1-121a193ecc7b'], u'message': {u'title': u'Reminder to complete your self checkup', u'body': u'Your last completed self checkup is to old. Please perform the self checkup before attending the workplace'}} When this gets passed into the requests.post call, it appears to then send the string representation of the Python object rather than the string data itself. The receiving system fails to parse the received object as it is not JSON any more. If I change the code to be data = str(result[options['datafield']]) then this will send OK as the curl_data_payload is then {"devices": ["arn:aws:sns:ap-southeast-2: 000000000000:endpoint/GCM/bcrm/cabf12dc-ec12-3af4-12e1-121a193ecc7b"], "message": {"title": "Reminder to complete your self checkup", "body": "Your last completed self checkup is to old. Please perform the self checkup before attending the workplace"}} I am not sure what the intention of the datafield option is if it always tried to JSON parse the data with json.loads() and it seems to me that it will always fail if the data is valid JSON. Is this a bug or just a problem with how I am using curl?

bowesmana · ‎06-15-2020

The rex statement below will work | makeresults | eval result=split("TRERY\j2874ac,TRERY\k5846de",",") | mvexpand result | rex field=result mode=sed "s/^TRERY\\\//" You can run the above example

bowesmana · ‎06-15-2020

Note that running all time searches is not a great idea, you would be better off collecting 'first time seen' events to either a summary index or a lookup file to add that into the results, but to achieve the result you are after you can do index="firewall" "confidence_level=high" action=Detect |stats count by protection_name | append [ search index="firewall" "confidence_level=high" action=Detect earliest=0 latest=now |stats earliest_time(protection_name) as firstEvent by protection_name | eval firstEvent=strftime(firstEvent, "%F %T.%Q") ] | stats values(*) as * by protection_name Note that depending on your data volume, the all time search may be really slow and I recommend looking at collecting the summary information as for example a saved search that runs regularly, or if you do not expect to see new protection_names very often, just collect the information to a lookup occasionally, e.g. index="firewall" "confidence_level=high" action=Detect earliest=0 latest=now | stats earliest_time(protection_name) as firstEvent by protection_name | sort protection_name | outputlookup protection_names.csv to collect the data then in the first search rather than doing the append and the last stats, you can do | lookup protection_names.csv protection_name | eval firstEvent=strftime(firstEvent,"%F %T.%Q") Note that keeping the time value in the lookup rather than stringifying it allows you to calculate with time and format when necessary. Hope this helps.

bowesmana · ‎06-15-2020

One possible avenue you could explore is to do the following 1. Have a lookup file containing the host words, e.g. name,val pepsi,1 coke,2 dr_pepper,3 sprite,4 then do something along the lines of | makeresults | eval t="id1:coke-combo-pepsi:pepsi-public-dmz,id2:sprite_internet_transit:test_drpepper_internet_transit,id3:sprite_drpepper_internet_transit:coke-combo-pepsi,id4:coke-combo-pepsi:sprite_drpepper_internet_transit,id5:pepsi-public-dmz:pepsi_internet_transit,id4:coke-combo-pepsi:sprite_coke_internet_transit" | makemv delim="," t | mvexpand t | rex field=t "(?<id>[^:]*):(?<src>[^:]*):(?<dest>.*)" | fields - t | eval n_src=src, n_dest=dest | makemv tokenizer="([^-_]+)[-_]?" n_src | makemv tokenizer="([^-_]+)[-_]?" n_dest | mvexpand n_src | lookup hosts.csv name as n_src OUTPUT val as src_val | where !isnull(src_val) | mvexpand n_dest | lookup hosts.csv name as n_dest OUTPUT val as dest_val | where !isnull(dest_val) | table _time, id, src, dest, n_src, n_dest, src_val, dest_val | stats values(n_src) as n_src values(n_dest) as n_dest values(*_val) as *_val by id, src, dest | where isnull(mvfind(n_src, n_dest)) NB: replace the < and > with the chevrons - seems to cause a problem with this forum if chevrons used. Anyway, that will give you a list of entities that do not talk to the correct domain, but I am not sure what you want to do when the dual token (pepsi-combo-coke) talks to pepsi or coke - are both allowed or only one? Anyway, hopefully this gives you something to get started with

Posts	3424
Solutions	747
Karma Given	288
Karma Received	1316
Member Since	‎09-05-2011

Online Status	Offline
Date Last Visited	38 seconds ago

Windows TA not extracting key_path as registry_pat...

Why does FIELDALIAS work or doesn't depending on t...

How can I write this regex for rex to extract mult...

How to rename fields in a table without changing c...

Why doesn't drilldown eval token statement work un...

Performance fields/stats/table

Infoblox TA not extracting response code

Is there any way to create CSS that can be dynamic...

How to break up groups of <input> elements onto th...

tokenlinks.js in Splunk 8.24 broken- working dashb...

Re: Drill down time from drill down editor to anot...

Re: Summing values inside mvfield

Re: Field extract

Re: Can i display panel with Even or Odd Click on ...

Re: Can i display panel with Even or Odd Click on ...

Re: Create Dynamic URL for Splunk Search Dashboard...

Re: Splunk Mobile App Visualizions Not Displaying

Re: Single value drill down search query in the da...

Re: Create Dynamic URL for Splunk Search Dashboard...

Re: Problem with TA-Webtools curl POST method with...

Re: Grouping hosts based on their environment and ...

Re: Grouping hosts based on their environment and ...

Re: How to search for requests from the same sourc...

Re: Grouping hosts based on their environment and ...

Re: Different results when search is run in web UI...

Re: Group by repeating value at value change

Re: How to search for requests from the same sourc...

Re: How to find concurrent run of processes?

Re: find servers in 24 hour period that have susta...

Re: Grouping hosts based on their environment and ...

How to prevent TA-webtools curl command from runni...

Problem with TA-Webtools curl POST method with JSO...

Re: remove characters in field results

Re: Show event count from two different times

Re: Compare words in a string against all other wo...

Join the Conversation