About tscroggins

tscroggins · ‎02-20-2021

@fdevera A typical recurring lookup generator would append to the existing lookup: index=azuread | rex field=initiatedBy.user.userPrincipalName "ex(?<GUID>\d+)z\@" | search GUID=* | table initiatedBy.user.userPrincipalName, GUID | inputlookup append=t zguids.csv | dedup initiatedBy.user.userPrincipalName | outputlookup zguids.csv The initial lookup can be populated by executing the search manually over a sufficient range of time. Subsequent searches should be limited to the schedule window. I.e. For an hourly search, set earliest=-1h@h latest=@h. Be sure to account for input lag and adjust scheduling as needed.

tscroggins · ‎02-20-2021

@Joe20 Using an example time window of 24 hours, try: element=* errortext=* earliest=-24h latest=now NOT [ element=* errortext=* earliest=-48h latest=-24h | table element errortext ] This will return events over the last 24 hours with key field combinations (element AND errortext) that were not present between 48 and 24 hours ago. You may want to review timestamp extractions and time zone offsets if you frequently need to search for future times, e.g. +7d@w6, and those future times are not be design.

tscroggins · ‎02-20-2021

@edfigue Assuming a historical baseline from a source that's stable on five minute intervals (add earliest and latest values to cover your baseline time range): index=data | bin time span=5m | stats count by _time | eval time_bucket=stftime(_time, "%M") | stats avg(count) as avg_count stdev(count) as stdev_count by time_bucket | eval lower_control_limit=avg_count-3*stdev_count | outputlookup baseline.csv you can compare the current measurement to the threshold in a search that executes on */5 * * * *: index=data earliest=-5m@m latest@m | bin time span=5m | stats count by _time | eval time_bucket=strftime(_time, "%M") | lookup baseline.csv time_bucket output lower_control_limit | where count<lower_control_limit

tscroggins · ‎02-20-2021

@bowesmana You could try something like this: | foreach * [ | eval {index}_<<FIELD>>=<<FIELD>> | fields - <<FIELD>> ] Although you'll want to be more selective in the fields you iterate over and delete. Rather than deleting a field, you could just force its value to null in an if or case function.

tscroggins · ‎02-20-2021

@klim The double quote is a major breaker, so PREFIX won't work. You might consider extracting and indexing the acct_id field, but it won't help with already indexed events.

tscroggins · ‎02-20-2021

@tod_s If you use proxies and can log response content types, you could look for e.g. application/x-shockwave-flash or other Flash related content types.

tscroggins · ‎02-20-2021

If the events look something like this: Sat Feb 20 20:13:00 EST 2021 acct_id=123 your tstats search could look like this: | tstats count where index=foo TERM(acct_id=*) by PREFIX(acct_id=) The PREFIX() function must contain the entire string prefix, including the "=" character. If the events look something like this, and you're using search time field exaction: Sat Feb 20 20:13:00 EST 2021 123 then you need to fall back to search time aggregations. If you plan to summarize data, compare the "cost" of summarizing with the cost of simply indexing the acct_id field in the primary index.

tscroggins · ‎02-20-2021

@klim EDIT: Updated. Using top returns both the field value and the count, so it's not necessary to run additional searches: index=i | top limit=1 acct_id acct_id count percent 1 123 98.6 If you're running Splunk 8+ and your events contain field names and values with only minor breakers, e.g. acct_id=xxx, you can produce similar results with tstats: | tstats count where index=i TERM(acct_id=*) by PREFIX(acct_id=) | eventstats sum(count) as total | eval percent=100*count/total | sort 1 - count | fields - total | rename acct_id= as acct_id On my test system, using tstats with TERM and PREFIX followed by subsequent commands is about 390% faster. You can nest subsearches in tstats just as you can with other searches. Instead of using format (either implicitly or explicitly), you can return a field value directly using $field: | tstats count where index=unified_tlx [| tstats count where index=i TERM(acct_id=*) by PREFIX(acct_id=) | sort 1 - count | rename acct_id= as acct_id | eval acct_id="TERM(acct_id=".acct_id.")" | return $acct_id ] by PREFIX(acct_id=) If you're running an earlier version of Splunk or if your raw data does not contain field names, you can fall back to nesting top, but your final output will not contain the acct_id value: | tstats count where index=unified_tlx [ index=i | top limit=1 acct_id | eval acct_id="TERM(".acct_id.")" | return $acct_id ] If acct_id is an indexed field, you can reference it directly in tstats: | tstats count where index=unified_tlx [ | tstats count where index=i by acct_id | sort 1 - count | table acct_id ] by acct_id

tscroggins · ‎02-20-2021

@deaseec You can query searches and actions using the REST API. | rest /servicesNS/-/-/saved/searches count=0 splunk_server=local | foreach action.* [| eval alert_actions=mvappend(alert_actions, case('<<FIELD>>'==1 AND match("<<FIELD>>", "^action\.[^.]+$"), "<<FIELD>>"))] | fields splunk_server eai:acl.app title author alert_actions | search alert_actions=* Actions have field names like action.foo. Action parameters have field names like action.foo.bar. In this example, I've used foreach to iterate over field names and add them to a multi-valued field named alert_actions if 1) the value is 1 and 2) the field is not a parameter.

tscroggins · ‎02-20-2021

@ForeverNoob2 Pie charts only require two fields: a category and a count. E.g.: * | eval os=if(like(_raw, "%Linux%"), "Linux", "Other") | stats count by os

tscroggins · ‎02-20-2021

@tonymaibox Your condition has no match attribute, so it will match all events, including form initialization. For example, foo_tok is cleared on init in this Simple XML: <form> <init> <set token="foo_tok">initialized</set> </init> <label>Test</label> <fieldset submitButton="false"> <input type="multiselect" token="field1" searchWhenChanged="true"> <label>Sample Input</label> <fieldForLabel>count</fieldForLabel> <fieldForValue>count</fieldForValue> <search> <query>| stats count</query> </search> <change> <condition> <unset token="foo_tok"></unset> </condition> </change> <delimiter> </delimiter> </input> </fieldset> <row> <panel> <title>$foo_tok$</title> </panel> </row> </form> You can control when unset is invoked by adding constraints to the condition element: <condition label="0">

tscroggins · ‎02-20-2021

@hishamjan RTP and SIP are application layer protocols that may use either TCP or UDP as a transport. I'm not familiar with Asterisk, but it presumably includes functionality to log session and call metrics. You can use Splunk App for Stream to monitor network traffic on the forwarders and send cooked traffic events to the indexer. Both RTP and SIP are supported. The implementation of Splunk App for Stream can be non-trivial. If you're unfamiliar with packet capture and protocol analysis concepts, you may prefer to enlist Splunk Professional Services or another qualified consultant.

tscroggins · ‎02-20-2021

You can review the Splunk implementation of dnslookup in $SPLUNK_HOME/etc/system/bin/external_lookup.py. The configuration is documented at https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Configureexternallookups#External_lookup_example. On the Splunk server, you can validate the functionality of external_lookup.py from e.g. a Bash shell: $ cd $SPLUNK_HOME/etc/system/bin $ ./etc/system/bin/external_lookup.py host src_ip <<EOF > host,src_ip > www.splunk.com , > ,8.8.8.8 > EOF host,src_ip www.splunk.com,23.12.144.246 dns.google,8.8.8.8 In this example, I've provided three input lines: 1) the CSV header matching the fields provided on the command line; 2) the host www.splunk.com and no src_ip; and 3) the src_ip 8.8.8.8 and no host. On my test system, both the forward and reverse lookup were successful. If the command fails or returns the wrong result, validate your host's DNS configuration, including your DNS servers, domain search list, and resolver cache if enabled. If you're having name resolution problems with TCP or UDP inputs, i.e. you see IP addresses in the host field but expected to see host names, confirm the connection_host setting on the input. If connection_host is set to dns, Splunk uses FCrDNS to validate resolved names. E.g.: Source is 8.8.8.8. Reverse lookup returns dns.google. Forward lookup returns 8.8.8.8. => PASS, host=dns.google Source is 10.0.0.1. Reverse lookup returns foo.example.com. Forward lookup returns 10.254.254.254. => FAIL, host=10.0.0.1

tscroggins · ‎02-20-2021

@splunkcol If your search is auto-finalizing before it completes, you'll need to adjust srchMaxTime for your role in authorize.conf. Also review srchTimeWin (maximum time range), srchDiskQuota, and other role-based limits, depending on the errors/messages shown.

tscroggins · ‎02-20-2021

@rneel You can use streamstats to generate rolling summarizations. I'll use the _internal index in this example, but you can modify it to use your base search: index=_internal sourcetype=splunkd source=*/splunkd.log* earliest=-14d@d latest=@d | timechart fixedrange=f span=1d count as subtotal by log_level | untable _time log_level subtotal | streamstats time_window=7d sum(subtotal) as total by log_level | timechart span=1d max(total) as total by log_level | where _time>relative_time(relative_time(now(), "@d"), "-7d@d") I want a rolling count for the last 7 days, so I've expanded my time range to 14 days to ensure day 1 includes 7 days of prior data. From there, I've reduced the initial result set to a daily summary using timechart followed by untable. Then I've used streamstats to generate a rolling 7 day total from the daily subtotal. Finally, I've summarized the total by the field of interest and truncated the results to the last 7 days. (I've used the max aggregation for simplicity. There should only be one total value per log_level per day.)

tscroggins · ‎02-20-2021

@flyingpiglet If you have timestamp extraction configured correctly, the _time field should equal epochtime, but if it doesn't, you assign the value directly to _time for ease of use in other commands: host=host1 sourcetype IN (sourcetype1 sourcetype2) | eval _time=epochtime | bin _time span=5m | stats values(c0) as c0 values(c1) as c1 values(c2) as c2 values(c3) as c3 values(c3) as c4 by _time | eval totals=coalesce(c0,0)+coalesce(c1,0)+coalesce(c2,0)+coalesce(c3,0)+coalesce(c4,0) | delta totals as dtot | timechart fixedrange=f span=5m per_second(dtot) The base search returns all events with host1 and source type of sourcetype1 or sourcetype2. The first eval command assigns the value of epochtime to the field _time. The bin command changes all _time values to the nearest 5 minute boundary: :00, :05, :10, etc. E.g. 00:23 => 00:20, 3:34 PM => 15:30. The stats command collates the values of c0, c1, c2, c3, and c4 by _time. Note that if you multiple values of c0, c1, c2, etc. with the same epochtime value, the result will contain multi-valued fields. This example assumes you have no duplication of epochtime values at the cN level. After stats, the events should be sorted by _time. The second eval command sums the cN fields, defaulting to a value of 0 for null (missing) fields. You may want to additionally validate that cN is numeric. The delta command computes the difference between totals on events 2, 3, 4, ... n. The first event will have a null dtot value. The timechart command converts the 5 minute subtotals into per second rates. Please do validate that the combination of time bins and per_second aggregation returns the values you expect.

tscroggins · ‎02-20-2021

@REACHGPRAVEEN Assuming a base URL of https://splunk:8000/en-us/app/search, i.e. the search app on your Splunk search head, you can construct basic search URLs with the earliest, latest, and q query parameters: https://splunk:8000/en-US/app/search/search?earliest=-24h%40h&latest=now&q=search%20index%3Dx%20employeeid%3D123 https://splunk:8000/en-US/app/search/search?earliest=-24h%40h&latest=now&q=search%20index%3Dx%20app%3Dabc Basic URL encoding rules apply.

tscroggins · ‎02-20-2021

@alexspunkshell Can you provide sanitized sample events? I'm assuming you want to join Microsoft Graph Security API Add-On for Splunk events with Zscaler Technical Add-On for Splunk events.

tscroggins · ‎02-20-2021

It's still an excellent suggestion, but you can summarize over smaller time ranges. If your summary populating search runs every ~15 minutes, you're counting by acct_id over ~730,000 events. When a user needs to aggregate over larger time ranges, they can search the summary index. The available precision will just be limited to your aggregation window. If you're running Splunk 8+ and your raw events contain acct_id=x, you can also take advantage of PREFIX in tstats: | tstats count where index=some_index TERM(acct_id=*) by PREFIX(acct_id=) _time span=1d Check out Richard Morgan's excellent "TSTATS and PREFIX" presentation from .conf20 at https://conf.splunk.com/files/2020/slides/PLA1089C.pdf.

tscroggins · ‎02-20-2021

I've used the trellis option to achieve this. For example: index=_internal sourcetype=splunkd source=*/splunkd.log* | bin _time span=10m | stats count(eval(case(log_level=="ERROR", log_level))) as ERROR count(eval(case(log_level=="WARN", log_level))) as WARN count(eval(case(log_level=="INFO", log_level))) as INFO by _time component will produce a table of log_level counts by _time and component: _time component ERROR WARN INFO 2021-02-20 11:00 AdminManager 0 1 0 ... ... ... ... ... In the chart configuration, enable trellis and split by the desired field. I split by component in this example to display counts of events by log_level over time per component.

tscroggins · ‎02-20-2021

Hi Alex, Your fields--c0, c1, c2, and c3--are likely in different events: Sat Feb 20 12:47:40 EST 2021 c0=123 host=host1 sourcetype=st1 Sat Feb 20 12:47:40 EST 2021 c1=456 host=host1 sourcetype=st2 Sat Feb 20 12:47:40 EST 2021 c2=789 host=host1 sourcetype=st3 Sat Feb 20 12:47:40 EST 2021 c3=234 host=host1 sourcetype=st4 You'll need to group your events in some way to sum field values. If your events have synchronized time stamps, for example, you may be able to group by time: host=host1 sourcetype IN (st1,st2,st3,st4,st5,st6) | bin _time span=5m | stats values(c0) as c0 values(c1) as c1 values(c2) as c2 values(c3) as c3 by _time | eval totals=coalesce(c0,0)+coalesce(c1,0)+coalesce(c2,0)+coalesce(c3,0) | delta totals as dtot | timechart fixedrange=f span=5m per_second(dtot) There are many other ways to aggregate/group events. The chart, stats, timechart, and tstats commands are usually the most efficient. The transaction command is very flexible, but it doesn't scale over large result sets. -Trev

tscroggins · ‎02-20-2021

In Splunk Enterprise 8.1, when using chart with spans containing fractional values of 0.54, 0.95, and others that result in rounding errors, duplicate bins are created. For example: | makeresults count=1000 | eval x=(random()/2147483647)*20 | chart count over x span=0.54 generates a duplicate bin at 9.72-10.26 with a count of 0. | chart count over x span=1.54 generates a duplicate bin at 15.40-16.94 with a count of 0. | chart count over x span=2.54 generates a duplicate bin at 15.24-17.78 with a count of 0. Changing the x values results in different outcomes, of course, but rounding appears to be the cause. | makeresults count=1000 | eval x=(random()/2147483647)*1000 | chart count over x span=10.54 generates duplicate bins at 31.62-42.16, 52.70-63.24, 94.86-105.40, and 642.94-653.48 with a count of 0. | makeresults count=1000 | eval x=(random()/2147483647)*1000 | chart count over x span=10.95 generates duplicate bins at 32.85-43.80 and 153.30-164.25 with counts of 0. | makeresults count=1000 | eval x=(random()/2147483647)*200000 | chart count over x span=100.54 generates 258 duplicate bins with counts of 0. I haven't tested earlier versions of Splunk yet, but I'm curious if others are seeing the same issue. My personal Splunk account isn't attached to a support agreement, so I can't submit a bug report.

tscroggins · ‎01-20-2021

I'm not sure how you're parsing the data, of course, but I'm not seeing any issues with a combination of spath and eval: | makeresults | eval json="{ \"paused_time\" : \"14585\", \"completed_calls\" : \"8\", \"paused_since\" : \"2021-01-20 13:20:23\", \"talking_to_name\" : \"\", \"login_type\" : \"login\", \"order\" : \"1\", \"login_time\" : \"24593\", \"extension\" : \"4826\", \"max_talk_time\" : \"2045\", \"time_of_last_call\" : \"2021-01-20 13:12:23\", \"paused\" : \"1\", \"account_id\" : \"1503\", \"missed_calls\" : \"1\", \"logged_in_status\" : \"logged_in\", \"fullname\" : \"*******\", \"talking_to_number\" : \"\", \"avg_talk_time\" : \"896\" }" | spath input=json | eval login_time=tostring(login_time, "duration") | table _time login_time _time login_time 2021-01-21 00:44:26 06:49:53

tscroggins · ‎01-20-2021

CPU and memory use vary with input volume and complexity, but local disk use of 1500 MB to 2000 MB is normal given how Splunk manages the fishbucket directory.

tscroggins · ‎01-20-2021

Hi @splunkcol, There are other ways to get data into Splunk, the most common being raw TCP or UDP ports, i.e. syslog, and the HTTP Event Collector. Do you have a use case that precludes or prohibits using Splunk Universal Forwarder?

Posts	793
Solutions	127
Karma Given	106
Karma Received	295
Member Since	‎08-13-2014

Online Status	Offline
Date Last Visited	a week ago

json_extend Scalar Bug, or Coercing JSON Scalars i...

Chart command generates duplicate bins when span a...

Re: Best way to dump Azure UPNs into scheduled loo...

Re: Find unique events in one search and NOT the o...

Re: Standard deviation in a period of time

Re: Field renaming of some but not all fields

Re: How to optimize this timechart query

Re: dnsrequest url or query help

Re: How to optimize this timechart query

Re: how to use format in a subsearch to make a tst...

Re: Cataloging Report Notification Actions

Re: How to create a Pie Chart of particular events...

Re: <change> event firing before the user ever mak...

Re: Nature of traffic

Re: Ip resolving from host name in events

Re: limits.conf modify time out search

Re: Recursive Query over Time

Re: performing calculations on fields from differe...

Re: how to create hyperlink that directs to search...

Re: Splunk query to find IP attack in Geo Map

Re: How to optimize this timechart query

Re: Multiple Timecharts by a field in single searc...

Re: performing calculations on fields from differe...

Chart command generates duplicate bins when span a...

Re: Splunk Convert Duration in Seconds to HH:MM:SS

Re: Universal Forwarder hardware requirements

Re: The only way is to use UF to send logs?

Join the Conversation