About tscroggins

tscroggins

Hi @siva_kumar0147, No, I only use makeresults to generate sample data. The logic from the sort command down drives the visualization.

tscroggins

This was a fun thread! I upvoted https://ideas.splunk.com/ideas/EID-I-2176.

tscroggins

Hi @zerocoolspain, I would use separate but similar radio inputs. Each radio input has its own set of tokens; however, updating a radio input also updates the global trobots token. The currently selected trobots1 and trobots2 values are preserved across changes to the tintervalo token. <form version="1.1" theme="light"> <label>intervalo</label> <init> <unset token="trobots"></unset> </init> <fieldset> <input type="dropdown" token="tintervalo" searchWhenChanged="true"> <label>Intervalo</label> <choice value="|loadjob savedsearch="q71139x:vap:precalculoVAPusoultimasemana">Última semana completa</choice> <choice value="|loadjob savedsearch="q71139x:vap:precalculoVAPusoultimomes">Último mes completo</choice> <choice value="|loadjob savedsearch="q71139x:vap:precalculoVAPusoultimotrimestre">Último trimestre completo</choice> <choice value="|loadjob savedsearch="q71139x:vap:precalculoVAPusoultimoaño">Último año completo</choice> <choice value="|loadjob savedsearch="q71139x:vap:precalculoVAPusomescurso">Mes en curso</choice> <choice value="|loadjob savedsearch="q71139x:vap:precalculoVAPusoañoencurso">Año en curso</choice> <choice value="7">Otros</choice> <change> <condition match="'tintervalo'==7"> <set token="show_trobots1">true</set> <unset token="show_trobots2"></unset> <set token="trobots">$trobots1$</set> </condition> <condition match="'tintervalo'!=7"> <unset token="show_trobots1"></unset> <set token="show_trobots2"></set> <set token="trobots">$trobots2$</set> </condition> </change> </input> <input type="radio" token="trobots1" depends="$show_trobots1$" id="inputRadioRI1" searchWhenChanged="true"> <label>Robots</label> <choice value="| eval delete=delete">Yes</choice> <choice value="`filter_robots` `filter_robots_ip`">No</choice> <initialValue>`filter_robots` `filter_robots_ip`</initialValue> <change> <set token="trobots">$trobots1$</set> </change> </input> <input type="radio" token="trobots2" depends="$show_trobots2$" id="inputRadioRI2" searchWhenChanged="true"> <label>Robots</label> <choice value="conBots">Yes</choice> <choice value="sinBots">No</choice> <initialValue>sinBots</initialValue> <change> <set token="trobots">$trobots2$</set> </change> </input> </fieldset> <row> <html> <table> <tr> <td><b>tintervalo:</b></td><td>$tintervalo$</td> </tr> <tr> <td><b>trobots1:</b></td><td>$trobots1$</td> </tr> <tr> <td><b>trobots2:</b></td><td>$trobots2$</td> </tr> <tr> <td><b>trobots:</b></td><td>$trobots$</td> </tr> </table> </html> </row> </form>

tscroggins

Hi @venal, Experience querying and pivoting relational data provides a solid foundation for analyzing data in Splunk; however, Splunk data is stored as a time series and streamed to clients as events. "Splunk SPL for SQL Users" is a good starting point, although it can lead to bad habits if not supplemented by other documentation and formal training. If you're a database administrator with combined Microsoft SQL Server and Microsoft Windows experience, you'll find many parallels between how a Microsoft SQL Server failover cluster is managed and how a simple Splunk Enterprise indexing cluster is managed; however, the product architectures diverge very quickly. Experience with Microsoft Windows and a server-oriented Linux distribution, e.g. Red Hat Enterprise Linux, are important for supporting Splunk Enterprise on-premises or in a self-managed hybrid/cloud environment. If your target is Splunk Cloud, operating system experience is still required to effectively support Splunk Universal Forwarder on endpoints/servers and Splunk Enterprise configured as a heavy forwarder. Splunk administrators install, configure, and maintain all aspects of Splunk Enterprise or configure and maintain customer-accessible aspects of Splunk Cloud. A Splunk "developer" can be a content developer (searches, reports, and dashboards), an app developer (add-ons and technology integrations), a back-end developer (primarily web services), a front-end developer (Splunk SDK and third-party clients), or any combination of the four. Splunk certification paths are published at https://www.splunk.com/en_us/training/certification.html. The Splunk Success Framework describes the typical roles associated with Splunk deployments and provides corresponding learning paths: https://lantern.splunk.com/Splunk_Success_Framework/People_Management/Setting_roles_and_responsibilities?mt-learningpath=ssfpeople

tscroggins

tscroggins

Hi @munang, The set command and the join command perform overlapping but different functions. set diff returns the symmetric difference of the subsearches: I.e. set diff returns all events in either subsearch A or subsearch B but not both: A url A_field https://www.splunk.com/ A_value https://www.appdynamics.com/ A_value B url A_field https://www.appdynamics.com/ A_value https://www.cisco.com/ A_value diff url A_field https://www.splunk.com/ A_value https://www.cisco.com/ A_value Both join type=left and join type=outer perform a left outer join by joining all fields in all events in the base search with all fields from the first (default: max=1) matching event in the subsearch: I.e.: A url A_field https://www.splunk.com/ A_value https://www.appdynamics.com/ A_value B url B_field https://www.appdynamics.com/ B_value1 https://www.appdynamics.com/ B_value2 join url A_field B_field https://www.splunk.com/ A_value (null) https://www.appdynamics.com/ A_value B_value1 As written, your join search is equivalent to join type=inner. The where command removes all events from the base search that were not joined to an event in the subsearch. To return the difference using the join command, the command would need to support a full outer join, and it does not.

tscroggins

Hi @danielbb, As an alternative in Simple XML, you can stack visualizations vertically by including more than one visualization in a panel: <row>  <panel> </panel> </row> <row> <panel>  </panel> <panel> <chart>  </chart> <chart>  </chart> <chart>  </chart> </panel </row> Some visualizations may require an empty spacer, e.g. an <html/> section, to defeat the auto-layout code. Here's an extended example using an inverse normal macro (not shown) to generate a bit of random data: <dashboard version="1.1" theme="light"> <label>danielbb_layout</label> <search id="base"> <query>| gentimes start=11/29/2024:00:00:00 end=11/30/2024:00:00:00 increment=1m | eval _time=starttime | fields + _time | sort 0 - _time | eval x=`norminv("`rand()`*(0.9999999999999999-0.0000000000000001)+0.0000000000000001", 1.3, 10)`</query> </search> <search id="stats" base="base"> <query>| stats avg(x) as u stdev(x) as s</query> </search> <row> <panel> <html> <h1>Some Random Data</h1> </html> </panel> </row> <row> <panel> <chart> <title>Histogram</title> <search base="base"> <query>| chart count over x span=0.5</query> </search> <option name="charting.legend.placement">none</option> <option name="height">600</option> </chart> </panel> <panel> <chart> <title>Samples</title> <search base="base"></search> <option name="charting.legend.placement">none</option> </chart> <single> <title>Mean</title> <search base="stats"> <query>| fields u</query> </search> <option name="numberPrecision">0.000</option> </single> <html/> <single> <title>Standard Deviation</title> <search base="stats"> <query>| fields s</query> </search> <option name="numberPrecision">0.000</option> </single> </panel> </row> </dashboard>

tscroggins

Hi @Priya70, Without seeing your dashboard source code, I'm not sure there's more we can recommend. Everyone's addressed the most common causes of issues. To look for specific causes, you can open the problem panel's search in a separate tab to evaluate the generated SPL by clicking the Open in Search (magnifying glass) link. You can inspect search logs for warnings and errors by clicking the Inspect (lowercase "i") link.

tscroggins

Hi @robertlynch2020, I'm not aware of a native method, although you could override the Splunk bar's behavior with a browser extension, assuming you control the browser. My preference is to customize the launcher or provide a default app with customized navigation menus.

tscroggins

Hi @Priya70, Have you verified the panel search references the base search? As in the example above, the search element should have a base attribute with the same value as the id attribute of the base search: The |s in $token|s$ is a token filter. The |s filter sanitizes the token as a quote-enclosed string. Apart from string validation, it helps prevent SPL injection attacks and inadvertent mistakes. For example, a token named foo with the value bar"baz would produce the following searches: index=main foo=$foo$ => index=main foo=bar"baz index=main foo="$foo$" => index=main foo="bar"baz" index=main foo=$foo|s$ => index=main foo="bar\"baz" The first two searches are malformed and generate an "Unbalanced quotes" error. The third search is well-formed. The |s filter also prevents foo values like bar" OR baz="qux and bar" | delete | eval baz="qux from being executed: index=main foo="$foo$" => index=main foo="bar" OR baz="qux" => index=main foo="bar" | delete | eval baz="qux" index=main foo=$foo|s$ => index=main foo="bar\" OR baz=\"qux" => index=main foo="bar\" | delete | eval baz=\"qux"

tscroggins

Hi @siva_kumar0147, The simplest solution is to use the Timeline visualization. You'll need to calculation durations in milliseconds between transitions: | makeresults format=csv data="_time,direction,polarization 1732782870,TX,L 1732782870,RX,R 1732781700,TX,R 1732781700,RX,L" | sort 0 - _time + direction | eval polarization=case(polarization=="L", "LHCP", polarization=="R", "RHCP") | streamstats global=f window=2 first(_time) as end_time by direction | addinfo | eval duration=if(end_time==_time, 1000*(info_max_time-_time), 1000*(end_time-_time)) | table _time direction polarization duration

tscroggins

Hi @MMMM, Is that the size of the evtx files on disk or the size of the events after they're indexed? Have you confirmed you haven't reached the Splunk Free license limit?

tscroggins

Hi @Priya70, Can you provide a deidentified sample of the raw events? How is _time extracted? Does the _time value match the *time value(s) in your raw events? How are ApplicationName, SoftwareVersion, and endpoint extracted? If _time and *time have the same values, you can configure your base search to use the time input directly: <form version="1.1" theme="light"> <label>Dashboard</label> <fieldset submitButton="false"> <input type="text" token="selected_software"> <label>Application Name</label> </input> <input type="time" token="time_selector"> <label></label> <default> <earliest>-24h@h</earliest> <latest>now</latest> </default> </input> </fieldset> <search id="base"> <query>index=software_inventory sourcetype=software:inventory ApplicationName=$selected_software|s$</query> <earliest>$time_selector.earliest$</earliest> <latest>$time_selector.latest</latest> </search> <row> <panel> <table> <search base="base" /> </table> </panel> </row> </form> You have enough fields in common across the first three searches that you can aggregate using stats in the base search and then aggregate again in the post-processes searches. Base (id="base"): index=software_inventory sourcetype=software:inventory ApplicationName=$selected_software|s$ | stats dc(endpoint) as TotalEndpoints by ApplicationName SoftwareVersion First Panel (base="base"): | stats sum(TotalEndpoints) as TotalEndpoints dc(SoftwareVersion) as VersionCount by ApplicationName | sort - TotalEndpoints Second Panel (base="base"): | stats sum(TotalEndpoints) as EndpointCount by SoftwareVersion | sort - EndpointCount The third panel is the same as the second. The fourth panel is just a table of endpoint values for the selected software and version. If you're lucky, your raw data is structured in a way that will allow you to use tstats. If not, you can use summary indexing or an accelerated data model to make aggregations significantly faster. You previously mentioned you do not want to use scheduled searches, but scheduled summary searches may solve your problem.

tscroggins

Hi @kbrisson, Yes, it's possible, although the "how" is a long answer, and I don't have any active Tenable.sc or Tenable.io data to demo with. A few key points to remember: Tenable data is relational, but the Splunk data will be a point-in-time snapshot of assets and scan results represented as a time series. Each query returns the latest scan results from all repositories the configured account can access. You'll need to deduplicate assets and vulns using time ranges that cover the span of first seen and last seen timestamps for the assets and vulns of interest. UUIDs may be globally unique, but if you have multiple repositories and/or multiple Tenable instances, you'll need to deduplicate by Tenable instance, repository, and UUID*. * UUID isn't the only field used to uniquely identify assets. Check the uniqueness/hostUniqueness field to see which fields create a composite key that uniquely identifies a host. Some apps, e.g. Tenable's, attempt to work around these issues by storing data in a kvstore collection; however, the collection can grow quite large, limiting its usefulness as a search tool. It doesn't scale. You may have better luck defining reports in Tenable and pulling the report results into Splunk.

tscroggins

Hi @MMMM, Splunk Free is limited to 500 MB of ingest per day. How large are the indexed events? You can check for license alerts under Settings > Licensing, although an alert should also appear under Messages. You can run a simple search to see daily usage over time: | tstats sum(PREFIX(b=)) as bytes where index=_internal source=*license_usage.log* TERM(type=RolloverSummary) earliest=-7d@d latest=now by _time span=1d | eval MB=round(bytes/1024/1024) If your daily usage is over 500 MB, Splunk Free will stop indexing new data, i.e. your evtx files, when the limit is reached.

tscroggins

Hi @fatsug, Adding to what was previously discussed, you can break down the behavior and where it occurs by merging $SPLUNK_HOME/etc/system/default/props.conf and $SPLUNK_HOME/etc/apps/Splunk_TA_nix/default/props.conf. @PickleRick's use of btool in the last question hints at how to do this: $SPLUNK_HOME/bin/splunk btool --debug props list lastlog The source type will also inherit [default] settings provided by any app and any additional [lastlog] settings you may have. Looking at the settings relevant to the question: [lastlog] KV_MODE = multi LINE_BREAKER = ^((?!))$ SHOULD_LINEMERGE = false TRUNCATE = 1000000 The LINE_BREAKER, SHOULD_LINEMERGE, and TRUNCATE settings tell Splunk the boundaries of the event. These settings are used by the linebreaker and aggregator on a heavy forwarder or indexer; they are not, except under specific conditions, used by a universal forwarder. The SHOULD_LINEMERGE setting disables reassembly of multiple lines (delineated by the LINE_BREAKER setting) into a single event; in this case, the LINE_BREAKER setting does that work for us more efficiently. As @PickleRick noted, the regular expression ^((?!))$ matches nothing. When lastlog.sh is executed, its entire output up to TRUNCATE = 1000000 bytes (~1 MB) is indexed as one event: USERNAME FROM LATEST user2 10.0.0.1 Wed Oct 30 11:20 another_user 10.0.0.1 Wed Oct 30 11:21 discovery 10.0.0.2 Tue Oct 29 22:19 scanner 10.0.0.3 Mon Oct 28 21:39 admin_user 10.0.0.4 Mon Oct 21 11:19 root 10.0.0.1 Tue Oct 1 08:57 If Splunk_TA_nix is not installed on the search head, then the sample above is what you would see as a single event in your search results. What happens if the lastlog.sh output is longer than 1,000,000 bytes? All data after the 1,000,000th byte is simply truncated and lost. For example: USERNAME FROM LATEST user2 10.0.0.1 Wed Oct 30 11:20 another_user 10.0.0.1 Wed Oct 30 11:21 discovery 10.0.0.2 Tue Oct 29 22:19 scanner 10.0.0.3 Mon Oct 28 21:39 admin_user 10.0.0.4 Mon Oct 21 11:19 root 10.0.0.1 Tue Oct 1 08:57 ... ~1,000,000 bytes of data jsmit If the "t" in "jsmit" is the 1,000,000th byte, then that's the last byte indexed, and everything after "jsmit" is truncated. If Splunk_TA_nix is installed on the search head, then the KV_MODE = multi setting tells Splunk to pipe the events through the multikv command before returning the results. On disk, there is only one event, and that event has been split into six separate events using the first line for field names. If Splunk_TA_nix is not installed on the search head, you can include the multikv command in your search to produce the same results: index=main sourcetype=lastlog | multikv You can also test multikv directly using sample data: | makeresults | eval _raw="USERNAME FROM LATEST user2 10.0.0.1 Wed Oct 30 11:20 another_user 10.0.0.1 Wed Oct 30 11:21 discovery 10.0.0.2 Tue Oct 29 22:19 scanner 10.0.0.3 Mon Oct 28 21:39 admin_user 10.0.0.4 Mon Oct 21 11:19 root 10.0.0.1 Tue Oct 1 08:57" | multikv If you have Splunk_TA_nix installed on your forwarders, your heavy forwarders, your indexers, and your search heads, then everything is "OK." Technically, your heavy forwarders are doing the parsing work instead of your indexers; however, if your indexers are Linux, you probably want to run lastlog.sh on them, too. Eventually, though, you'll realize that auditd, /var/log/auth.log, /var/log/secure, etc. are better sources of login data, although /var/log/wtmp and lastlog.sh are useful if you want regular snapshots of login times relative to wtmp retention on your hosts. You may find that log rotation for wtmp is either misconfigured or not configured at all on older Linux hosts.

tscroggins

Hi @anna, The chart, timechart, xyseries, and tstats commands all produce output suitable for a run chart, depending on our source events. The chart and timechart commands bin _time into spans automatically or using the span size we specify: index=web sourcetype=access_common | chart limit=0 usenull=false count over _time span=1h by status index=web sourcetype=access_common | timechart limit=0 span=1h usenull=false count by status Note that the timechart command pads the results with empty bins spanning our search time range. To remove extraneous bins, set the fixedrange argument to false: | timechart fixedrange=false limit=0 span=1h usenull=false count by status The xyseries command first requires binning _time using the bin command and then aggregating status values using the stats command. Like the chart and timechart commands, the bin command bins the target field into spans automatically or using the span size we specify; however, the stats command does not generate empty bins. We use the makecontinuous command after the xyseries command to add missing _time bins: index=web sourcetype=access_common | bin _time span=1h | stats count by _time status | xyseries _time status count | makecontinuous _time | fillnull value=0 The tstats command is similar to the stats command but works with indexed fields or terms and data models. We can pipe the output through either the timechart command or the xyseries command to group events by status over _time. | tstats prestats=true count from datamodel=Web.Web by _time Web.status | rename Web.* as * | timechart limit=0 span=1h usenull=false count by status Like the chart and timechart commands, the tstats command normally bins _time into spans automatically or using the span size we specify; however, when using a datamodel without the prestats argument, specify a span: | tstats count from datamodel=Web.Web by _time span=1h Web.status | rename Web.* as * | xyseries _time status count | makecontinuous _time | fillnull value=0 If the status field is indexed, the tstats command can reference it directly: | tstats prestats=true count where index=web sourcetype=iis by _time sc_status | rename sc_status as status | timechart limit=0 span=1h usenull=false count by status If the status field and value appear in _raw as a key-value pair not separated by a major breaker, we can use the PREFIX() directive to access the status value as if it were indexed. Given events like the following: Nov 23 12:00:00 my_server my_app[1234]: c_ip=192.0.2.1 request="GET /favicon.ico" status=500 bytes=1024 we can execute: | tstats prestats=true count where index=web sourcetype=access_common by _time PREFIX(status=) | rename status= as status | timechart limit=0 span=1h usenull=false count by status The prestats argument in these examples instructs the tstats command to produces results suitable for the chart, stats, and timechart commands. The prestats argument is not required, but it allows subsequent commands to work as they would following other generating commands. The most common generating command is the search command, which we use implicitly in every search that doesn't begin with a pipe. We can find more information on breakers, the tstats command, and the PREFIX() directive in the Splunk documentation. Irrespective of the method used to count events, we can add a total field by piping the results through the addtotals command: index=web sourcetype=access_common | timechart limit=0 span=1h usenull=false count by status | addtotals fieldname="total requests" We can filter status values while retaining a total request count in two (or more!) ways: 1) group status values before aggregating or 2) filter status values after aggregating. We can use the eval and appendpipe commands to group status values and calculate total requests over _time: index=web sourcetype=access_common | eval status=if(status>=400, status, "other") | bin _time span=1m | stats count by _time status | appendpipe [| stats sum(count) as count by _time | eval status="total requests" ] | where status!="other" | xyseries _time status count | fillnull value=0 We can use the untable command to filter status values after aggregating: index=web sourcetype=access_common | timechart limit=0 span=1h usenull=false count by status | addtotals fieldname="total requests" | untable _time status count | where status>=400 OR status=="total requests" | xyseries _time status count | makecontinuous _time | fillnull value=0 The makecontinuous command isn't required following the timechart command, but it's included here as a failsafe. As @gcusello and @PickleRick noted, we use the Line Chart visualization to produce a run chart and then save the visualization to a new or existing dashboard. We can also create the Line Chart directly using the classic and Dashboard Studio editors. All of the examples used above produce a chart similar to the following (shown in log scale): In the Simple XML Line Chart visualization, the points on the lines and the legend entries are drill-down targets. We can access the status value using the click.name2 token and the context-sensitive time range using the earliest and latest tokens: <form version="1.1" theme="light"> <label>Drilldown Example</label> <fieldset submitButton="false"> <input type="time" token="time_tok" searchWhenChanged="true"> <label>Time</label> <default> <earliest>-60m@m</earliest> <latest>now</latest> </default> </input> </fieldset> <row> <panel> <title>Volumes</title> <chart> <search> <query>index=web sourcetype=access_common | timechart limit=0 span=1h usenull=false count by status | addtotals fieldname="total requests" | untable _time status count | where status>=400 OR status=="total requests" | xyseries _time status count | makecontinuous _time | fillnull value=0</query> <earliest>$time_tok.earliest$</earliest> <latest>$time_tok.latest$</latest> <sampleRatio>1</sampleRatio> </search> <option name="charting.axisTitleX.text">t</option> <option name="charting.axisTitleY.text">#</option> <option name="charting.axisY.scale">log</option> <option name="charting.chart">line</option> <option name="charting.drilldown">all</option> <option name="refresh.display">progressbar</option> <drilldown> <link target="_blank">search?q=index%3Dweb%20sourcetype%3Daccess_common%20status%3D%22$click.name2$%22&earliest=$earliest$&latest=$latest$</link> </drilldown> </chart> </panel> </row> </form> Clicking the "total requests" series will generate the following search, which will return no results: index=web sourcetype=access_common status="total requests" We can handle this case using condition elements within the drilldown element: <drilldown> <condition match="$click.name2$ == "total requests""> <link target="_blank">search?q=index%3Dweb%20sourcetype%3Daccess_common%20status%3D*&earliest=$earliest$&latest=$latest$</link> </condition> <condition match="$click.name2$ != "total requests""> <link target="_blank">search?q=index%3Dweb%20sourcetype%3Daccess_common%20status%3D%22$click.name2$%22&earliest=$earliest$&latest=$latest$</link> </condition> </drilldown> Clicking the "total requests" series will now generate the following search: index=web sourcetype=access_common status=* Note that using a custom link target allows us to use any of the example searches shown above assuming the underlying events share the same index and source type. Data models, tags, default search indexes, and other configuration techniques can help standardize or simplify our approach to searching data. Dashboard Studio (not shown) provides similar functionality; however, conditional drilldowns to custom URLs are not supported. (Thank you for coming to my TED Talk. Disclaimer: Not a TED Talk. I hope this was helpful! I write these responses as an exercise for myself as well.)

tscroggins

Not directly, no. Even if the source, e.g. a web server, and the destination, e.g. a Splunk indexer, have perfectly synchronized clocks (they do not), the time (latency) it takes to share information between the source and the destination is greater than zero. That time is composed of reading the source clock, rendering the source event, writing the event to storage, reading the event from storage, serializing the event to the network, transmitting the event across the network, deserializing the event from the network, reading the destination clock, rendering the destination event, and writing the destination event to storage. The preceding list is not exhaustive and may vary. Just note that it takes time to go from A to B. There are delays everywhere! You can search by _indextime instead of _time using _index_earliest and _index_latest and very wide earliest and latest parameters: index=web status=400 earliest=0 latest=+1d _index_earliest=-15m@m _index_latest=@m however, it's still possible to miss events that have been given an _indextime value of T but aren't synchronized to disk until after your search executes. You can use real-time searches to see events as they arrive at indexers (or as they're written to storage, depending on the type of real-time search), but for your use case, time windows are still required, and events may still be missed.

tscroggins

Hi @Karthikeya, False positive, false negative, etc. have the same definitions in Splunk that they have in statistics. I'm in the United States, and I find NIST/SEMATECH e-Handbook of Statistical Methods, Chapter 6, "Process or Product Monitoring and Control," a useful day-to-day reference: https://www.itl.nist.gov/div898/handbook/index.htm. In your example, you're counting events. For example, a basic search scheduled to run every minute: index=web status=400 earliest=-15m@m latest=@m | status count | where count>5 gives you the count of status=400 events over the prior 15 minutes. In this context, false positive and false negative could relate to the time the events were generated and the delay between that time and the index time. If a status=400 event occurred at 00:14:59 but was indexed by Splunk at 00:15:04, then a search that executes at 00:15:01 for the interval [00:00:00, 00:015:00) would not count the event because it has not been indexed by Splunk. This is a false negative. You can reduce the probability of false negatives by adding a backoff to your search--1 minute in this example: index=web status=400 earliest=-16m@m latest=-1m@m | status count | where count>5 However, that will not eliminate all false negatives because there is still a non-zero probability that an event will be indexed outside your search time range. False positives are more typically associated with measuring against a model. Let's say you've modeled your application's behavior and determined that more than 5 status=400 events over a 15 minute interval likely indicates a client-side code deployment issue as opposed to "normal" client behavior. "More than 5" is associated with a control limit, for example a deviation from a mean; however, the number of status=400 events is a random variable. A bad client-side code deployment may trigger 4 status=400 events, which is a false negative, and a good client-side deployment may trigger 6 status=400 events, which is a false positive. Several Splunk value-added products like Splunk Enterprise Security and Splunk IT Service Intelligence provide ready-to-run modeling and monitoring solutions, but in general, you would model your application's behavior using either traditional methods outside Splunk or statistical functions or an add-on like the Splunk Machine Learning Toolkit inside Splunk. You would then apply your model using custom Splunk searches.

tscroggins

Hi @NavS, Refer to https://docs.splunk.com/Documentation/SplunkCloud/latest/Service/SplunkCloudservice for supported data egress methods: Data Egress Dynamic Data Self-Storage export of aged data per index from Splunk Cloud Platform to Amazon S3 or Google Cloud Storage No limit to the amount of data that can be exported from your indexes to your Amazon S3 or Google Cloud Storage account in the same region. Dynamic Data Self-Storage is designed to export 1 TB of data per hour. Data Egress Search results via UI or REST API Recommend no more than 10% of ingested data For optimal performance, no single query, or all queries in aggregate over the day from the UI or REST API, should return full results of more than 10% of ingested daily volume. To route data to multiple locations, consider solutions like Ingest Actions, Ingest Processor, or the Edge Processor solution. Data Egress Search results to Splunk User Behavior Analytics (UBA) No limit Data as a result of search queries to feed into Splunk User Behavior Analytics (UBA). To stream events to both Splunk Cloud and another destination, an intermediate forwarding solution is required. You should contact your client's Splunk account team for confirmation, but your Splunk Cloud native options are likely limited to the table above.

tscroggins

Ha. I do think splunkd etc. should require it when acting as a server, especially given that it requires it when acting as a client! You're right about lazy CA policies, though.

tscroggins

Hi @shai, Looking at https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/custominputs/modinputsconfspec/, the interval setting should "just work;" however, the use_single_instance scheme parameter controls its behavior. Is use_single_instance set to false in your modular input's scheme? E.g.: <scheme>  <use_single_instance>false</use_single_instance>  </scheme>

tscroggins

Speaking broadly (and without citing anything--ipse dixit), the consensus is an application (client and server) should require the presence of extended key usages to keep implementers from harming themselves, but it's not required. If you identify a scenario in which Splunk Enterprise/Splunk Universal Forwarder are vulnerable to some attack independent of the implementer's choices, it would be wise to disclose the vulnerability privately to Splunk through https://advisory.splunk.com/report. Otherwise, https://ideas.splunk.com/ is the best place to request new features, i.e. requiring extended key usages. I would upvote such an idea.

tscroggins

From RFC 5280 section 4.2.1.12: If the extension is present, then the certificate MUST only be used for one of the purposes indicated. If multiple purposes are indicated the application need not recognize all purposes indicated, as long as the intended purpose is present. Certificate using applications MAY require that the extended key usage extension be present and that a particular purpose be indicated in order for the certificate to be acceptable to that application. "If" and "MAY"--the easy way out. At a glance, OpenSSL's libssl only rejects unsupported certificate purposes if extended key usages are present [https://github.com/openssl/openssl/blob/master/crypto/x509/v3_purp.c] (the implementation may vary by version, of course; I'm making an assumption that earlier versions are similar): /* ... */ #define xku_reject(x, usage) \ (((x)->ex_flags & EXFLAG_XKUSAGE) != 0 && ((x)->ex_xkusage & (usage)) == 0) /* ... */ /* * Key usage needed for TLS/SSL server: digital signature, encipherment or * key agreement. The ssl code can check this more thoroughly for individual * key types. */ #define KU_TLS \ KU_DIGITAL_SIGNATURE | KU_KEY_ENCIPHERMENT | KU_KEY_AGREEMENT static int check_purpose_ssl_server(const X509_PURPOSE *xp, const X509 *x, int non_leaf) { if (xku_reject(x, XKU_SSL_SERVER | XKU_SGC)) return 0; if (non_leaf) return check_ssl_ca(x); if (ns_reject(x, NS_SSL_SERVER)) return 0; if (ku_reject(x, KU_TLS)) return 0; return 1; }

tscroggins

I just tested forwarder version 9.1.1 on Windows with outputs.conf [tcpout] sslVerifyServerCert = true, and key usage is checked: 11-09-2024 20:57:30.517 -0500 ERROR X509Verify [85708 TcpOutEloop] - Server X509 certificate (CN=splunk.example.com,O=Example,L=Washington,ST=District of Columbia,C=US) failed validation; error=26, reason="unsupported certificate purpose" Edit: I tested in 9.0.3, and key usage is verified there as well. splunkd, splunkweb, and mongod on Splunk Enterprise 9.3.0 do happily load and use the certificate, though. My last conversation with Splunk on the topic was circa 8.2. I haven't had cause to test this specifically since then. Good to see!

Posts	667
Solutions	112
Karma Given	83
Karma Received	247
Member Since	‎08-13-2014

Online Status	Offline
Date Last Visited	yesterday

Chart command generates duplicate bins when span a...

Re: Using Chart Visualisation in classic dashboard...

Re: non english words length function not working ...

Re: Problem with token input radio options

Re: Suggestion on certification

Re: Why are the results of set diff and join diffe...

Re: Why are the results of set diff and join diffe...

Re: How do I use HTML on multiple panels?

Re: Base Search for Splunk Dashboard

Re: Multi-drop-down on the App menu

Re: Base Search for Splunk Dashboard

Re: Using Chart Visualisation in classic dashboard...

Re: Unable to load all evtx files in a folder

Re: Base Search for Splunk Dashboard

Re: Tenable Data - Combining Sourcetypes

Re: Unable to load all evtx files in a folder

Re: Possible lineabreaker issue with lastlog in Sp...

Re: Create a view in dashboard like below

Re: False positives

Re: False positives

Re: Best Practices for Streaming Logs from Splunk ...

Re: Splunk forwarder is not validating ExtendedKey...

Re: interval as a data input parameter

Re: Splunk forwarder is not validating ExtendedKey...

Re: Splunk forwarder is not validating ExtendedKey...

Re: Splunk forwarder is not validating ExtendedKey...