Agree, your issue now deals with a non-root user trying to send an snmptrap. Check out the following: https://superuser.com/questions/795889/snmp-with-not-root-user ... View more

Ensure you did the following: Create the script in the $SPLUNK_HOME/bin/scripts directory. Create the directory if it doesn't already exist. Copy the code listed below into sendsnmptrap.pl. Run chmod on sendsnmptrap.pl to make the script executable. In the scrpt, change the Host:Port of the SNMP trap handler, the paths to the external commands splunk and snmptrap, and the user/password if necessary. If those are good, you can test the script by executing, it's expecting 9 arguments: $SPLUNK_HOME/bin/scripts/snmptrap.pl "9" "test" "test" "test" "test" "test" "test" "test" "tes" The script should run, or output some sort of errors. If the script runs correctly and you don't see the events in CA: - tcpdump to verify you see trap being sent, if you don't see trap sent verify you have the snmptrap installed correctly - Verify CA knows how to handle the configured OID and its not filtered. - Verify the trap is not blocked between host -> CA If the trap makes it to CA in test mode, but not when configured as an alert action: - Verify the Splunk user has permissions to execute the script. - Checked the savedsearch logs to see if any suppressions are being applied. - Check internal logs for any errors related to the script or the search * Check the search logs for any errors ... View more

If you want to use tstats, I recommend creating a data model: https://docs.splunk.com/Documentation/Splunk/7.3.1/Knowledge/Aboutdatamodels That includes at minimum, the fields specified in your search. ... View more

Check out the following using extreme search to create a baseline and identify when performance doesn't match the baseline: http://www.georgestarcher.com/splunk-getting-extreme-part-one/ ... View more

I recommend you check out: https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Indexesconf A volume can be used for a single index or multiple indexes. We have hot and cold volumes defined, and pointed to hot (SSD) and cold (HDD) storage. We have sized the storage based on our expected usage for a given time period (hot/warm - 30 days, cold - 90 days). We then use the volume to control the hot/warm/cold. Since the indexes can fluctuate in usage, we have had to increase the maxTotalDataSizeMB for some indexes which are going to exceed the default setting. We set frozenTimePeriodInSecs to control the data retention. If you are only using one index, you can also apply sizing controls to it: [main] homePath = homePath.maxDataSizeMB = coldPath.maxDataSizeMB = maxTotalDataSizeMB = frozenTimePeriodInSecs = One more thing u missed while suggesting is that transfer of frozen data to my Azure blobs. [main] coldToFrozenScript = Specifies a script to run when data is to leave the splunk index system. * Essentially, this implements any archival tasks before the data is deleted out of its default location. * Add "$DIR" (including quotes) to this setting on Windows (see below for details). * Script Requirements: * The script must accept one argument: * An absolute path to the bucket directory to archive. You can check the following for tips on creating the script using theAWS CLI https://aws.amazon.com/getting-started/tutorials/backup-to-s3-cli/ ... View more

Do you need to use transactions? Are the id's re-used? If not, you might be able to just use stats: your search | stats min(eval(match(msg,"(trying|calling service) to get",_time,NULL))) as start, max(eval(match(msg,"(got|returning) info",_time,NULL))) as stop, values(msg) as msg by id ... View more

| tstats values(SUM_BILL_TM_CNT) where index=ndspr | stats sum(SUM_BILL_TM_CNT) as SUM_BILL_TM_CNT by DPC_CARRIER_LONG_NM,CALLED_PARTY_NOA_CD DEST_POINT_CD, DPC_CLLI_CD, OPC_CLLI_CD,TERM_SUBGRPG_CD | table SUM_BILL_TM_CNT As written the tstats command should return one statistic, the list of values(SUM_BILL_TM_CNT) as "values(SUM_BILL_TM_CNT)" Therefore the stats command is not getting any of the fields referenced. If the recommended tstats search doesnt return results: | tstats sum(SUM_BILL_TM_CNT) as SUM_BILL_TM_CNT,sum(SUM_NTWK_DURTN_CNT) as SUM_NTWK_DURTN_CNT, sum(TOTAL_CALL_CNT) as TOTAL_CALL_CNT by DPC_CARRIER_LONG_NM,CALLED_PARTY_NOA_CD DEST_POINT_CD, DPC_CLLI_CD, OPC_CLLI_CD,TERM_SUBGRPG_CD where index=xyz sourcetype=SUMMARY I would verify that all of the fields specified are INDEXED fields. ... View more

The following regex : | rex "(?<new_fieldname>\d+\.\d+\.\d+)" Should produce new_fieldname=01.2.3 \d -> only matches number characters \. -> matches a literal period (.) ... View more

I recommend configuring a single Volume for your Hot/Warm data: [volume:hot_volume] path=/path/to/2TBHDD maxVolumeDataSizeMB = Max size splunk will use on disk Then for each index: [sample_index] homePath = volume:hot_volume/sample_index/db As long as all index on the drive use volume:hot_volume, Splunk will roll buckets as needed to ensure the volume remain below the maxVolumeDataSize. ... View more

Did you see any WatchedFile or TailingProcessor events for the path? To be clear, you replaced $file_location$ with a regular expression that would match the path? If so, here's some steps: Make sure you're receiving events to the internal index ( index=_* ) Make sure you're getting metrics (index=_* component=Metrics) Make sure you're getting events from the host in question ( index=_* host=$hostname$ ) ... View more

If the search head is forwarding events to the indexing tier, and the index exists, the notable should get created. That said, if you don't have the notable index, you are probable missing other ES specific indexes (https://docs.splunk.com/Documentation/ES/5.3.1/Install/Indexes), which could impact your correlation searches. You may want to revisit your ES install, https://docs.splunk.com/Documentation/ES/5.3.1/Install/DeploymentPlanning#Using_the_deployment_server_with_Splunk_Enterprise_Security ... View more

Do you have a separate search head(s) and indexers? Are your search head(s) configured to forward events to your indexers? Do your indexers have a notable index created? ... View more

If you see a result_count>0 and a suppressed<1 you should see a notable event for the search (index=notable). ... View more

result_count=1 <- the number of results returned by the correlation search. suppressed=0 <- This is true (1) or false (0) to indicate if any of the results were suppressed due to throttling. ... View more

Splunk will not recognize the hostname change, it will continue to use the outputs.conf hostname value. If the outputs.conf hostname changes, I believe you will get see two records on the deployment server and will need to delete the old entry. (Or not, it really doesnt impact anything). If you use the hostname to identify which serverclass a host falls into, you would need to fix that. ... View more

Have you checked the _internal indexes for the startup logs from the UF? It should show the UF reading in the Stanza's, do you see the input stanza in the logs? 08-04-2019 11:37:07.050 -0500 INFO WatchedFile - Will begin reading at offset=123 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\conf.log'. 08-04-2019 11:37:02.409 -0500 INFO TailingProcessor - Adding watch on path: C:\Program Files\SplunkUniversalForwarder\var\log\splunk Do you see any error logs, related to access reading from the file location? Check the metrics logs? index=_* component=metrics group=per_source_thruput series=$file_location$ Do you see this source getting forwarded? Check permissions on the file location and file, make sure the Splunk process can access it. ... View more

Interesting fields, are just the other fields extracted from the events. Information on the fields extracted from an event, if available, would be found in the documentation of the add-on used to extract the fields. You may need to go back to the documentation of the application/appliance/etc producing the events. NOTE: Some sourcetype, like weblogs that contain text that gets recognized as key=value pairs. For example a CGI web request : /en-US/app/search/search?display.general.type=statistics&display.page.search.tab=statistics&display.page.search.mode=fast&dispatch.sample_ratio=1 Splunk by default will extract these key=value pairs, and if enough events contain them, they would show up under interesting fields. Despite the fact that the fields are not actually valid fields for that event. ... View more