SQL Monitoring - I'd like to know how to write a Splunk SPL query to alert on the top users running long running SQL queries on my databases.  I'm currently using the MS SQL add-on for Splunk and monitoring the included monitors for  Perfmon:sqlserver:* and sourcetypes "mssql:agentlog" and "mssql:errorlog"   Thank you in advance! ... View more

Hello, I'm trying to create a working props/transforms to separate standard events from json formatted logs (by filtering/resetting the json logs to their own sourcetype).  Here's what I've tried so far and I am able to do most of what I want with the exception of timestamp recognition of the json events..  The below trims my json event headers only and filters/resets the json events to their own separate sourcetype.  Since the header is trimmed splunk is doing a great job auto extracting my json field value pairs.  I'm looking for help on getting the timestamp or _time value to match my json field "log_time".   PROPS.conf       [mainlog] MAX_TIMESTAMP_LOOKAHEAD = 25 TIME_PREFIX = (?=[20])|log_time: SEDCMD-remove-jsonheader = s/^[0-9T\:Z]*.*?\s*{/{/g TRANSFORMS-set_sourcetype = example_json [mainlog:json] TIME_PREFIX = log_time: #TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ MAX_TIMESTAMP_LOOKAHEAD = 3 INDEXED_EXTRACTIONS = json       TRANSFORMS.conf       [example_json] REGEX = \{\"json\"\: FORMAT = sourcetype::mainlog:json DEST_KEY = MetaData:Sourcetype       sample log:       2023-08-21 11:59:10 TRACE [pool-12-thread-1] c.a.l.m.e.AbstractElasticSearchBatch$ElasticSearchBatch [Slf4jLogging.scala:13] Deadline time left is 302ms and record count is 72 2023-08-21 11:11:41 TRACE [pool-11-thread-1] c.a.l.m.e.AbstractElasticSearchBatch$ElasticSearchBatch [Slf4jLogging.scala:13] Indexing {"json":"s3://example/logs/2023/08/21/0111111a-2222-33ff-9e4e-c1a01dfdf448.gz","phase":"ingest","log_time":"2023-08-21T15:11:31.073Z","tick":"7777777777","id":"0111111a-2222-33ff-9e4e-c1a01dfdf448","source_time":"2023-08-21T11:11:25Z","status":"submitted","client":"555555","environment":"test","category":"changestream","account":"9","level":7}         ... View more

Hello, I have an existing high volume index and have discovered a chunk of event logs within the index that would be a great canidate to convert to metrics.  Can you filter these type of events to send to the metrics index and then convert the events to metrics at index time all using props/transforms? I have this props.conf  [my_highvol_sourcetype] TRANSFORMS-routetoIndex = route_to_metrics_index Transforms.conf [route_to_metrics_index] REGEX = cpuUtilization\= DEST_KEY=_MetaData:Index FORMAT = my_metrics_index But now what sourcetype do I use to apply the event log to metrics conversion settings?  Should I filter this dataset to a new sourcetype within my high volume index so I can apply my event log to metrics to all events matching the new sourcetype then filter to the metrics index? Any thoughts would be helpful to see if something like this is possible to do using props/transforms. ... View more

I'm looking for help on how to output the contents of my dashboard textbox to a kv lookup.  I'm hoping to display and collect the timestamp, user who created the note entry, and the notes in the panel below. As a bonus, I would also like to make the existing notes entries editable in my dashboard panel and capture the user who edited and timestamp in another column. ... View more

I'm looking for help to filter my mstats data using eventtype OR tag I've created for groups of hosts.. Here's an example of my CPU metrics dashboard panel    | mstats avg(_value) as value where `nmon_metrics_index` metric_name=os.unix.nmon.cpu.cpu_all.Sys_PCT OR metric_name=os.unix.nmon.cpu.cpu_all.User_PCT OR metric_name=os.unix.nmon.cpu.cpu_all.Wait_PCT host=$host$ groupby metric_name, host span=1m | `def_cpu_load_percent` | timechart `nmon_span` avg(cpu_load_percent) AS cpu_load_percent by host useother=false     I've tried appending a non-metrics subsearch to search against the metric data using my tag AND host so that only the selected hosts return in my panel    index = example_index (eventtype=test1 OR eventtype=test2 OR eventtype=test3) | search (host=* AND tag = test2) | append [ | mstats avg(_value) as value where `nmon_metrics_index` metric_name=os.unix.nmon.cpu.cpu_all.Sys_PCT OR metric_name=os.unix.nmon.cpu.cpu_all.User_PCT OR metric_name=os.unix.nmon.cpu.cpu_all.Wait_PCT host=dac51elo.pjm.com groupby metric_name, host span=1m | `def_cpu_load_percent` ] | timechart `nmon_span` avg(cpu_load_percent) AS cpu_load_percent by host useother=false   ... View more

Hello, I'm looking for help showing the Uptime/downtime percentage for my Universal Forwarders (past 7 days) : I've seen many people trying to solve a similar use case on Answers but haven't quite seen what I'm looking for yet..  I've been testing the below query and my thinking was to calculate the difference in minutes between a host's timestamp for eval field Action = "Splunkd Shutdown" - "Action = "Splunkd Starting".  Then sum the total in minutes divided by the total minutes in 1 week (10080) to get the uptime?  There are problems with this logic though because if the last time a host shutdown is not within your search window you won't get an accurate metric.  I'm open to a discussion to see how this can be monitoring most accurately. This query returns the host and timestamp for when splunkd shut down and another event with timestamp when Splunkd started. index=_internal source="*SplunkUniversalForwarder*\\splunkd.log" (event_message="*Splunkd starting*" OR event_message="*Shutting down splunkd*") | eval Action = case(like(event_message, "%Splunkd starting%"), "Splunkd Starting", like(event_message, "%Shutting down splunkd%"), "Splunkd Shutdown") | stats count by host, _time, Action ... View more

I'm trying to work with a data input using DB Connect version 3.0 and I cannot get the below input to save using the field alias 'time' that using this format : 2020-03-21 00:11:12.387 Based off this article I added these configurations to my stanza to help DB Connect identify the correct timestamp format : input_timestamp_format = yyyy-MM-dd HH:mm:ss.SSS output_timestamp_format = yyyy-MM-dd HH:mm:ss.SSS *The LogEntryId is my rising column and returns as column #1 *The time column/Timestamp returns as column #2 I've also uses the below Answers suggestion to try to resolve the NULL values possible issue : https://answers.splunk.com/answers/616150/how-to-force-dbconnect-to-send-fields-with-null-va.html [TestDB_2] connection = TestDB description = Test Query disabled = 0 index = main interval = */5 * * * * max_rows = 1000 mode = advanced output_timestamp_format = yyyy-MM-dd HH:mm:ss.SSS query = SELECT le.LogEntryId AS [LogEntryId] , [Date] AS [time] , l.[Name] AS [Level] , at.Name AS [Application Source] , le.Logger AS [Logger] , le.[Message] AS [Message] , COALESCE(le.FullMessage, 'NA') AS [FullMessage] , COALESCE(le.Exception, 'NA') AS [Exception] , COALESCE(le.FullException, 'NA') AS [Full Exception] FROM "Logging"."dbo"."LogEntry" le JOIN "Logging"."dbo"."LevelType" l ON l.LevelTypeId = le.LevelTypeId JOIN "Logging"."dbo"."ApplicationSourceType" at ON at.ApplicationSourceTypeId = le.ApplicationSourceTypeId WHERE le.LogEntryId > '?' AND le.LevelTypeId IN (3,4,5) -- WARN, ERROR, FATAL AND at.[Name] != 'developer.example.com' ORDER BY le.LogEntryId DESC; sourcetype = Test tail_rising_column_number = 1 input_timestamp_column_number = 2 input_timestamp_format = yyyy-MM-dd HH:mm:ss.SSS index_time_mode = dbColumn ... View more