About cstump_splunk

cstump_splunk · ‎02-02-2021

Double-check the searchFilter associated with role that is producing the error. The key::value syntax can cause problems with metrics data. For example: [role_test] srchFilter= index::* Any time a user int the 'test' role runs one of these metrics searches, "index::*" will be appended to (or at least included in) the search. This is what the "Term based search is not supported" message is referring to. It is not expecting the key=value syntax. From Add and Edit Roles docs : For search filters with metrics data, use the key=value to specify search restrictions to metrics fields. This is because the key::value syntax does not work for searches over metrics data. In this case, you can safely disregard syntax warnings about the = operator that the search filter dialog box presents.

cstump_splunk · ‎12-07-2020

The limitation is set in visualizations.conf: [area] data_sources.annotation.params.count = 1000 [column] data_sources.annotation.params.count = 1000 [line] data_sources.annotation.params.count = 1000 Notice there is limit for each chart type that supports annotations. Since you are working with a line graph, you will want raise the limit in the '[line]' stanza.

cstump_splunk · ‎04-10-2020

At this time _indextime fields are not included in the datamodel accelerations. I imagine this is due to optimization with regards to both disk space and memory usage during the acceleration process. If you are interested in having a field that tracks the time the accelerated event gets written to disk, then I encourage you to submit the idea to the ideas portal at https://ideas.splunk.com/.

cstump_splunk · ‎04-10-2020

First off, if you have not upgraded to either the latest core version of 7.2, 7.3, or 8.0, I recommend you do so as there have been updates that optimize the acceleration process. Secondly, you should check to see if your datamodels have high cardinality fields, i.e fields with a high number of unique events. This leads to additional memory consumption on the Search Peers. One way I like to do this is by using: |datamodel <DM_name> acceleration_search | stats estdc(*) | transpose Running this for a short 15-minute window in the past will tell you if any of the fields in the DM have high cardinality. If possible, either filter out some of the values in these fields or remove them for the DM definition.

cstump_splunk · ‎03-14-2020

One minor thing I want to point out about the tstats command: | tstats count where earliest=-5m by splunk_server By default, this tstats command will only search default indexes. If there are any data imbalances across the cluster and one of the indexers does not have any data from a default index, it may not appear in the results. You can look for all internal indexes by adding "AND index=_internal" | tstats count where earliest=-5m AND index=_internal by splunk_server

cstump_splunk · ‎02-12-2019

| eval today=now(), nextMonth=relative_time(today,"+1Mon@Mon"), minusOneDay=relative_time(nextMonth,"-1d@d"), daysInMonth=strftime(minusOneDay,"%e") daysInMonth will have the value you seek

cstump_splunk · ‎07-25-2018

| rex field=_raw "\[(?<my_field>.+)(SFP|XFP)" Also, I recommend that you play around with your regexes on regex101.com

cstump_splunk · ‎07-24-2018

One more thing about this splunkd.log error message: ERROR loader - Failed to download bundle from master, err="App='system' with replicated index='session_end' is neither in the bundle downloaded from master nor managed by local deployment client. Either define this index at the master or specify repFactor=0 on peer to skip replication. If you see this message on your search peer and it is causing the indexer to crash, the issue may be with locally defined indexes. If an index is created through the GUI, the index definition is saved in $SPLUNK_HOME/etc/system/local/indexes.conf. Clustering of the Indexers becomes a problem because of these local indexes. An index that needs to be a clustered index needs to be in the master_apps directory on the Cluster Master. If there are local index definitions on the clustered indexer, remove them and make sure they are defined on the Cluster Master

cstump_splunk · ‎07-23-2018

one the PowerShell comment, you make sure you cd to the Splunk bin directory and that you dot source the Splunk binary: ./splunk reload deploy-server Also, run PS with elevated permissions

cstump_splunk · ‎07-23-2018

For indexed data, you will want to calculate you anticipated capacity. Start by determining how much you will index each day. Then take into account your retention settings. For instance, let's say that you will be ingesting a 1 GB of data each day and that you want a retention period of 2 years. This means that in two years you will be storing 730GB of data. I would get some estimates on what you expect/ need and go from there.

cstump_splunk · ‎07-19-2018

This is where you will want to use the 'where' command: http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Where Inequalities (such as '<' and '>') cannot be used with the 'search' command, but the can be used with 'where' <inital_search> | where (OverallAvgNetworkMOS < 3.5 AND isnotnull(OverallAvgNetworkMOS)) OR (Stream_1_PacketLossRate> 0.1 AND isnotnull(Stream_1_PacketLossRate)) OR (Stream_2_PacketLossRate>0.1 AND isnotnull(Stream_2_PacketLossRate)) OR (Stream_1_RoundTrip>500 AND isnotnull(Stream_1_RoundTrip)) OR (Stream_2_RoundTrip>500 AND isnotnull(Stream_2_RoundTrip)) OR (Stream_1_JitterInterArrival>30 AND isnotnull(Stream_1_JitterInterArrival)) OR (Stream_2_JitterInterArrival>30 AND isnotnull(Stream_2_JitterInterArrival) | table tartTime, EndTime, MediaTypesDescription, FromUri, ToUri, FromIPAddr, ToIPAddr, Stream_1_PacketLossRate, Stream_1_RoundTrip, Stream_1_JitterInterArrival, Stream_2_PacketLossRate, Stream_2_RoundTrip, Stream_2_JitterInterArrival, OverallAvgNetworkMOS One thing to point out, that is an awful lot of 'OR' statements. Be sure the statement that is most likely to evaluate as true first. Also, I don't believe you need the isnotnull statements. For example, if Stream_2_JitterInterArrival is greater than 30, how can the value be null? Test it out and see what you think.

cstump_splunk · ‎05-25-2018

Quick Update for Splunk 7.1.0. To get the Time_Format to extract a time stamp in 7.1.0 you can use @dineshraj9 answer if you lose the escape characters (). A colleague and I were trying to extract a timestamp across multiple xml tags: <fiveminute id="28"><date><year>2018</year><month>05</month><day>22</day></date><time><hour>17</hour><minute>05</minute></time><rx>578489</rx><tx>1800555</tx></fiveminute> We were able to get the timestamp extracted in 7.1.0 with the following configurations: [sourcetype] TIME_PREFIX = year\> TIME_FORMAT = %Y%m%d%H%M Notice that the escape character was used in the TIME_PREFIX, that value is still regex, but the TIME_FORMAT is not regex and therefore does not make use of the escape character.

cstump_splunk · ‎04-17-2018

When troubleshooting any data ingestion issue, track down where the data is being transmitted and received. For instance, in this scenario, we know that the Tenable add-on needs to be installed on the Search Head and a Heavy Forwarder, and can be installed on the indexer (http://docs.splunk.com/Documentation/AddOns/released/Nessus/InstalltoSearchHead#Where_to_install_this_add-on). We should first check to see if the tenable data is leaving the Heavy Forwarder: index=_internal host=<Heavy_Forwarder> source=*metrics* group=per_sourcetype_thruput series=<Tenable_sourcetype> | timechart sum(kb) by series span=15min The visualization here we show you when and if your tenable data is being sent from the forwarder. If there are no results from this search, this is an indication that there is something wrong with the input. In that case, check out the heavy forwarder's splunkd.log file. If there are results with this search, then all is good on the Forwarder side. In this case, run a similar search for the Indexer : index=_internal host=<Indexer> source=*metrics* group=per_sourcetype_thruput series=<Tenable_sourcetype> | timechart sum(kb) by series span=15min If there is no data here then it could be an indication that there the data is getting lost in transmission (possibly by a Firewall). If there are results here, then check to see that the tenable data is going into the index you expect it to and that you are searching for it correctly. There are other things that could be going wrong in the process but start there.

cstump_splunk · ‎04-17-2018

does the following search work? index=abcd proto=TCP If so there may not be a problem. The behavior you are describing, with spath being added to the search, is the default behavior when Splunk detects JSON or XML events. If there is a way to turn it off, you may not want to as it will turn off the behavior for all JSON or XML inputs.

cstump_splunk · ‎04-16-2018

What is most likely happening here is that each time the SHC member comes back up, it is performing a checksum against the Deployer and incorrectly assessing that it does not have the current bundle. This causes the bundle to be redeployed to this SHC Member which then results in another restart. You may find the following log entry in splunkd.log on the Search Head in question: <mm-dd-YYYY HH:MM:S.%3N -TMZ> ERROR DistributedBundleReplicationManager - HTTP response code 409 (HTTP/1.1 409 Conflict). Checksum mismatch: received copy of bundle="/opt/splunk/var/run/searchpeers/<DeployerServer>-<bundlename>.bundle" has transferred_checksum=0 instead of checksum=<value_other_than_zero> The reason bringing down the Deployer stops the restart loop is because the checksum matches (0 compared to 0). The most likely reason for the checksum mismatch is a problematic app installed somewhere in the Cluster that does not allow the checksum to be properly calculated. You can go through and remove apps one by one to see which of them may be causing the problem or you could prevent the restart loop from occuring at all. To avoid the restart loop, we can stop the SH member from performing the checksum when it comes back. server.conf [shclustering] conf_deploy_fetch_mode = none Here is what the spec file for server.conf says about 'conf_deploy_fetch_mode: conf_deploy_fetch_mode = auto|replace|none * Controls configuration bundle fetching behavior when the member starts up. * When set to "replace", a member checks for a new configuration bundle on every startup. * When set to "none", a member does not fetch the configuration bundle on startup. * Regarding "auto": * If no configuration bundle has yet been fetched, "auto" is equivalent to "replace". * If the configuration bundle has already been fetched, "auto" is equivalent to "none". * Defaults to "replace". As you can see, setting this value to none will prevent the configuration bundle from being fetched thus preventing the restart loop. Keep in mind that when you make this change you will have to manually push out each config bundle, which is what most admins do anyhow.

cstump_splunk · ‎04-16-2018

It has been reported by some that members of a Search Head Cluster (SHC) will sometimes get caught in a restart loop after the Deployer has pushed a out a new configuration bundle. Here is what happens 1). The Deployer pushes out a new bundle 2). One SHC member restarts 3). Immediately after coming back up, the SHC member restarts again. 4). This member continues to restart until the Deployer is restarted. 5). After the Deployer is bounced, the next member in the cluster get's caught in a restart loop. 6). The Deployer is continually restarted as each member goes through it's own restart loop. 7). The Captain is the last member to restart. Oddly enough, it does not get caught in the restart loop. It is strange that bringing the Deployer down stops the restart loop since restarts are handled by the Search Head Cluster Captain. Each time the Search Head Member comes back up (within its restart loop), the following is logged into splunkd.log: <mm-dd-YYYY HH:MM:S.%3N -TMZ> INFO loader - Downloaded new baseline configuration; restarting ...

cstump_splunk · ‎04-10-2018

The information you are after may be here: https://docs.splunk.com/Documentation/Splunk/7.0.3/AdvancedDev/Whatsinthismanual Also, try searching around on the Splunk Dev page http://dev.splunk.com/

cstump_splunk · ‎12-28-2017

The first parenthesis is escaped: ( The regex engine thinks that there is an unmatched closing parenthesis at the end of your expression.

cstump_splunk · ‎08-31-2017

When you need authentication to run a splunk command from the CLI, the credentials you will need are for actual Splunk users, not a service account. Typically, you will need a Splunk admin account to add data or, in this case, a forwarder. You can bypass the credential prompting by passing the credentials through the command line: ./splunk add forward-server servername.domainname:9997 -auth admin:changeme

Posts	19
Solutions	1
Karma Given	66
Karma Received	25
Member Since	‎08-23-2017

Online Status	Offline
Date Last Visited	‎03-31-2021 12:26 AM

Why is the Search Head Cluster getting Caught in t...

Re: mcatalog causing ERROR SearchPhaseGenerator - ...

Re: Event annotations getting truncated?

Re: Accelerated data model and _indextime field

Re: Optimizing Accelerated Data Models

Re: What do I validate after I upgrade Splunk Ente...

Re: Number of day per month

Re: How to create a regex to extract data?

Re: Introspection Replication

Re: Why is blacklisting Windows event logs on a de...

Re: Does anyone know the base recommended amount o...

Re: How to filter as long as one of fields meets t...

Re: timestamp from XML multivalue

Re: Why is the data not writing to my index after ...

Re: How to not automatically invoke the spath comm...

Re: Why is the Search Head Cluster getting Caught ...

Why is the Search Head Cluster getting Caught in t...

Re: Is there support for the latest version of Cus...

Re: Regex issue

Re: Best practices: Can a su command be used to lo...