Hi @sandfly_dev, Can you add an option to your configuration to allow customers to provide a list of trusted certificates in PEM or some or other format? This could be a single self-signed certificate, a list of concatenated certificates in a certificate chain, etc. depending on what's supported by your code. ... View more

Hi @shocko, The typical approach discards lines at an intermediate heavy forwarder or indexer by sending them to nullQueue: # props.conf [my_sourcetype] # add line and event-breaking and timestamp extraction here TRANSFORMS-my_sourcetype_send_to_nullqueue = my_sourcetype_send_to_nullqueue # transforms.conf [my_sourcetype_send_to_nullqueue] # replace foo with a string or expression matching "keep" events REGEX = ^(?!foo). DEST_KEY = queue FORMAT = nullQueue As with @PickleRick, I've not seen a common use case for force_local_processing. I often say I don't want my application servers turning into Splunk servers, so I prioritize a lightweight forwarder configuration over data transfer. If CPU cores (fast growing files) and memory (large numbers of files) cost you less than network I/O, you may prefer the force_local_processing option; you won't save on disk I/O either way. If you need a refresher on the functions performed by the uft8, linebreaker, aggregator, and regexreplacement processors, see https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platform-the-Masa/m-p/590781/highlight/true#M103485. ... View more

Hi @Roy_9, Changes should be logged in index=_audit: index=_audit host IN (sh) action=modified info=succeeded savedsearch_name=xyz earliest=-30d Replace "sh" with a list of your search head host names and "xyz" with the name of the report. ... View more

I missed your comment re: accelerated data models earlier. The field values should be available at search time, either from _raw or tsidx, and then stored in the summary index. Off the top of my head, I don't know if the segmenters impact INDEXED_EXTRACTIONS = w3c, but they shouldn't impact transforms-based indexed extractions or search-time field extractions from other source types. ... View more

Also, yes, your proposal to install Splunk Enterprise on Server B and Splunk Universal Forwarder on Server C will allow you to run queries against Server A, assuming you have connectivity and a database account with appropriate access, and forward the evens to Server C and downstream to Splunk Cloud. Note, however, that sys.fn_get_audit_file does not scale. If you query .sqlaudit files through this function, your SQL Server administrator should store only the .sqlaudit files necessary to allow Splunk to execute queries and index events in a timely manner. I.e. Rotation and retention of live .sqlaudit files should be configured with Splunk and fn_get_audit_file performance in mind. You'll need to test performance in your environment to understand its constraints. ... View more

This should work on Linux: Install PowerShell Core. As the Splunk Universal Forwarder user--splunk or splunkfwd--install the SqlServer module as before: $ /bin/pwsh PS> Install-Module SqlServer If Splunk Universal Forwarder runs as root, install the SqlServer module as root. Copy Stream-SqlAudit.ps1 to an appropriate directory, e.g. $SPLUNK_HOME/bin/scripts. Note the addition of the interpreter directive on the first line. #!/bin/pwsh $file = New-TemporaryFile $output = $file.Open([System.IO.FileMode]::Append, [System.IO.FileAccess]::Write) $stdin = [System.Console]::OpenStandardInput() $stdout = [System.Console]::Out $buffer = New-Object byte[] 16384 [int]$bytes = 0 while (($bytes = $stdin.Read($buffer, 0, $buffer.Length)) -gt 0) { $output.Write($buffer, 0, $bytes) } $output.Flush() $output.Close() Read-SqlXEvent -FileName "$($file.DirectoryName)\$($file.Name)" | %{ $event = $_.Timestamp.UtcDateTime.ToString("o") $_.Fields | %{ if ($_.Key -eq "permission_bitmask") { $event += " permission_bitmask=`"0x$([System.BitConverter]::ToInt64($_.Value, 0).ToString("x16"))`"" } elseif ($_.Key -like "*_sid") { $sid = $null $event += " $($_.Key)=`"" try { $sid = New-Object System.Security.Principal.SecurityIdentifier($_.Value, 0) $event += "$($sid.ToString())`"" } catch { $event += "`"" } } else { $event += " $($_.Key)=`"$([System.Web.HttpUtility]::JavaScriptStringEncode($_.Value.ToString()))`"" } } $stdout.WriteLine($event) } $file.Delete() Make sure the file is executable, e.g.: $ chmod 0750 $SPLUNK_HOME/bin/scripts/Stream-SqlAudit.ps1 Update props.conf: [source::....sqlaudit] unarchive_cmd = $SPLUNK_HOME/bin/scripts/Stream-SqlAudit.ps1 unarchive_cmd_start_mode = direct sourcetype = preprocess-sqlaudit NO_BINARY_CHECK = true [preprocess-sqlaudit] invalid_cause = archive is_valid = False LEARN_MODEL = false [sqlaudit] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) EVENT_BREAKER_ENABLE = true EVENT_BREAKER = ([\r\n]+) TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%N%Z MAX_TIMESTAMP_LOOKAHEAD = 30 KV_MODE = auto_escaped Update inputs.conf: [monitor:///tmp/*.sqlaudit] index = main sourcetype = sqlaudit /tmp is just an example. Use whatever file system and path makes the most sense for your deployment. The Splunk Universal Forwarder user must have read and execute access to all directories in the path and read access to the .sqlaudit files. As before, your temporary directory should have enough free space to accommodate your largest .sqlaudit file. Depending on your Splunk configuration, Splunk Universal Forwarder may process multiple files concurrently. If that's the case, ensure you have enough free space for all temporary files. Finally, let us know how it goes! ... View more

Maybe? The prototype is here for anyone to grab. I'd need to find the time and resources for long-term maintenance of an app: development, build and integration, support, etc. ... View more

The SqlServer module is available for PowerShell Core on various platforms. I can test a solution on Linux x86-64, but I don't have access to a macOS or ARM host. What platform is Server C? ... View more

A custom segmenter has merit in this case, but globally, folks will recommend tagging events with an appropriate add-on (or custom configuration) and using an accelerated Web or other data model to find matching URLs, hostnames, etc. ... View more

Note that I used System.Web.HttpUtility.JavaScriptStringEncode as a shortcut for encoding/escaping strings. KV_MODE = auto_escaped only handles a few escape sequences. If you prefer, you can simply replace \ and " with \\ and \", respectively, in strings before writing them. ... View more

Hi @Ismail_BSA, We can use the SqlServer PowerShell module to read SQL Server audit files. As an administrator, install the SqlServer PowerShell module under PowerShell 5.1, which should be installed by default on all modern Windows releases: PS> Install-Module SqlServer With the module installed, we can read .sqlaudit files created by SQL Server using Read-SqlXEvent. Column/field information is available at https://learn.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-records. Columns with type bigint or varbinary will be read as byte arrays and must be converted to strings using a .NET object of the appropriate type. We can write a small PowerShell script to act as a stream reader for .sqlaudit files read by Splunk's archive processor (see below). Note that Read-SqlXEvent uses System.IO.Stream internally and calls Stream.Length, which throws "Stream does not support seeking" for forward-only streams. We'll work around this isssue by copying the stream to a temporary file, reading the temporary file, and finally, deleting the temporary file. C:\Temp\Stream-SqlAudit.ps1 $file = New-TemporaryFile $output = $file.Open([System.IO.FileMode]::Append, [System.IO.FileAccess]::Write) $stdin = [System.Console]::OpenStandardInput() $stdout = [System.Console]::Out $buffer = New-Object byte[] 16384 [int]$bytes = 0 while (($bytes = $stdin.Read($buffer, 0, $buffer.Length)) -gt 0) { $output.Write($buffer, 0, $bytes) } $output.Flush() $output.Close() Read-SqlXEvent -FileName "$($file.DirectoryName)\$($file.Name)" | %{ $event = $_.Timestamp.UtcDateTime.ToString("o") $_.Fields | %{ if ($_.Key -eq "permission_bitmask") { $event += " permission_bitmask=`"0x$([System.BitConverter]::ToInt64($_.Value, 0).ToString("x16"))`"" } elseif ($_.Key -like "*_sid") { $sid = $null $event += " $($_.Key)=`"" try { $sid = New-Object System.Security.Principal.SecurityIdentifier($_.Value, 0) $event += "$($sid.ToString())`"" } catch { $event += "`"" } } else { $event += " $($_.Key)=`"$([System.Web.HttpUtility]::JavaScriptStringEncode($_.Value.ToString()))`"" } } $stdout.WriteLine($event) } $file.Delete() We can use the invalid_cause and unarchive_cmd props.conf settings to call the PowerShell script. Note that unarchive_cmd strips or escapes quotes depending on the value unarchive_cmd_start_mode, so we've stored the PowerShell script in a path without spaces to avoid the use of quotes. If PowerShell can't find the path specified in the -File argument, it will exit with code -196608. Sample props.conf on forwarders, receivers (heavy forwarders or indexers), and search heads: [source::....sqlaudit] unarchive_cmd = powershell.exe -ExecutionPolicy RemoteSigned -File C:\Temp\Stream-SqlAudit.ps1 unarchive_cmd_start_mode = direct sourcetype = preprocess-sqlaudit NO_BINARY_CHECK = true [preprocess-sqlaudit] invalid_cause = archive is_valid = False LEARN_MODEL = false [sqlaudit] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) EVENT_BREAKER_ENABLE = true EVENT_BREAKER = ([\r\n]+) TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%N%Z MAX_TIMESTAMP_LOOKAHEAD = 30 KV_MODE = auto_escaped We can use a batch or monitor stanza to monitor the directory containing .sqlaudit files. Use a batch stanza if the files are moved to the monitored directory atomically to allow Splunk Universal Forwarder to delete the files after they're indexed. Sample inputs.conf: [monitor://C:\Temp\*.sqlaudit] index = main sourcetype = sqlaudit The script can be refactored as a scripted input; however, using the archive processor allows Splunk to perform file and change tracking on our behalf. ... View more

Now that is a valid use case for modifying segmentation; however, the impact is wide-reaching. You may also want to look at setting INTERMEDIATE_MAJORS = true, although that could result in a significant indexing performance impact. Which access log formats and source types do you most commonly use? ... View more

It was just a Wikipedia joke: "In Latin script, the double hyphen ⹀ is a punctuation mark that consists of two parallel hyphens. It was a development of the earlier double oblique hyphen ...." I'm assuming an early developer analyzed a suitable corpus of log content and determined a double hyphen or long dash should be considered a major breaker. ... View more

Somewhere in Splunk history, there's a developer who did the lexicographically correct thing knowing it would stymy future Splunkers. Let's raise a glass to the double oblique hyphen (thanks, Wikipedia)! ... View more

Hi @YuriSpirin, When we define our lookup with time_format = %s, the time field in our collection should have type number: # collections.conf [dhcp_timebased_lookup] enforceTypes = true field.time = number field.ip = string field.hostname = string field.mac = string # transforms.conf [dhcp_timebased_lookup] collection = dhcp_timebased_lookup external_type = kvstore fields_list = time,ip,hostname,mac max_offset_secs = 691200 min_offset_secs = 0 time_field = time time_format = %s We can populate the lookup with test data: | makeresults format=csv data="time,ip,hostname,mac 1709251200,1.2.3.4,host-43,aa:bb:cc:dd:ee:ff 1709164800,1.2.3.4,host-42,aa:bb:cc:dd:ee:fe 1709078400,1.2.3.4,host-41,aa:bb:cc:dd:ee:fd" | outputlookup dhcp_timebased_lookup and validate it with an additional test: | makeresults format=csv data="_time,dest_ip 1709208000,1.2.3.4" | lookup dhcp_timebased_lookup ip as dest_ip output hostname _time dest_ip hostname 2024-02-29 07:00:00 1.2.3.4 host-42 We can also experiment with accelerated fields to improve performance, although we may not see the performance returns we expect: [dhcp_timebased_lookup] enforceTypes = true field.time = number field.ip = string field.hostname = string field.mac = string accelerated_fields.ip = {"time": -1, "ip": 1} Compare with a similar file-based lookup with a size less than or equal to the configured max_memtable_bytes limit: # limits.conf [lookup] # default 25 MB; increase max_memtable_bytes to a value greater than our # largest lookup file, assuming we have adequate phyiscal memory available max_memtable_bytes = 26214400   ... View more

Hi, Ironstream isn't a Splunk product, and IBM mainframes and minis are protected by a rather large paywall. Have you talked to your Ironsteam account/support team? Ironstream documentation is open, but I don't see a reference to ESDS or other data sets. If Ironstream can read a data set and deserialize its records to UTF-8, it should be technically possible for Splunk to receive the data. ... View more

Hi, Can you provide a sample of the raw data? It's probably JSON assuming what you've posted is from Splunk's "List" view. The spath command in your search also expects _raw (by default) to be JSON. If that's the case, the fields aren't empty. They have a literal hyphen as their value. For example: {"body_bytes_sent": "0", "bytes_sent": "0", "host": "nice_host", "http_content_type": "-", "http_referer": "-", "http_user_agent": "-", "kong_request_id": "8853b73ffef1c5522b4a383c286c825e", "log_type": "kong", "query_string": "-", "remote_addr": "10.138.100.153", "request_id": "93258e0bc529fa9844e0fd2d69168d0f", "request_length": "1350", "request_method": "GET", "request_time": "0.162", "scheme": "https", "server_addr": "10.138.100.151", "server_protocol": "HTTP/1.1", "status": "499", "time_local": "25/Feb/2024:05:11:24 +0000", "upstream_addr": "10.138.103.157:8080", "upstream_host": "nice_host", "upstream_response_time": "0.000", "uri": "/v1/d5a413b6-7d00-4874-b706-17b15b7a140b"} {"body_bytes_sent": "0", "bytes_sent": "0", "host": "nice_host", "http_content_type": "-", "http_referer": "-", "http_user_agent": "-", "kong_request_id": "89cea871feba9f2d5216856f7a884223", "log_type": "kong", "query_string": "productType=ALL", "remote_addr": "10.138.100.214", "request_id": "9dbf69defb49a3595cf1040e6ab5d4f2", "request_length": "1366", "request_method": "GET", "request_time": "0.167", "scheme": "https", "server_addr": "10.138.100.151", "server_protocol": "HTTP/1.1", "status": "499", "time_local": "25/Feb/2024:05:11:24 +0000", "upstream_addr": "10.138.98.140:8080", "upstream_host": "nice_host", "upstream_response_time": "0.000", "uri": "/v1/a8b7570f-d0af-4d0d-bd6d-f6cf31892267"} You can search for the literal value directly: query_string=- or query_string="-" There is a caveat: the hyphen is a minor breaker and isn't indexed by Splunk as a term. All events will be returned initially, the query_string field will be extracted, and its value will be scanned for a hyphen to filter results. If your JSON fields aren't auto-extracted, we can investigate your inputs.conf and props.conf settings. ... View more

Hi, We can use the cmdb_ci and cmdb_rel_ci tables to analyze CI relationships. For this example, we'll use Splunk Add-on for ServiceNow 7.7.0 with the cmdb_ci and cmdb_rel_ci inputs configured and enabled. The number and types of relationships will vary depending on our model. We'll use the relationships described in the ServiceNow Common Service Data Model at https://docs.servicenow.com/bundle/washingtondc-servicenow-platform/page/product/csdm-implementation/concept/ci-relationships.html: Application Service -[ Depends on::Used by ]-> Application Application -[ Runs on::Runs ]-> Infrastructure CIs If we're not using Service Mapping, the CI classes and relationships may differ. We'll create several sample CIs with appropriate relationships: Splunk::Application Service -[ Depends on::Used by ]-> Splunk Enterprise::Application Splunk Enterprise::Application -[ Runs on::Runs ]-> splunk-cm-1::Linux Server Splunk Enterprise::Application -[ Runs on::Runs ]-> splunk-idx-1::Linux Server Splunk Enterprise::Application -[ Runs on::Runs ]-> splunk-idx-2::Linux Server Splunk Enterprise::Application -[ Runs on::Runs ]-> splunk-idx-3::Linux Server Splunk Enterprise::Application -[ Runs on::Runs ]-> splunk-sh-1::Linux Server We'll start our search with the required relationships: index=snow sourcetype=snow:cmdb_rel_ci dv_type IN ("Depends on::Used by" "Runs on::Runs") earliest=0 latest=now If we have more than one ServiceNow instance, we can add endpoint=https://xxx to our searches, where xxx is the fully-qualified domain name of our instance. sourcetype=snow:cmdb_rel_ci includes the following fields of interest: sys_id parent dv_type child illustrated by: index=snow sourcetype=snow:cmdb_rel_ci dv_type="Depends on::Used by" earliest=0 latest=now | stats latest(parent) as parent latest(child) as child by sys_id Using sourcetype=snow:cmdb_ci_list and sourcetype=snow:cmdb_rel_ci, we can graph relationships using join: index=snow sourcetype=snow:cmdb_ci_list dv_sys_class_name="Mapped Application Service" name=Splunk earliest=0 latest=now | stats latest(name) as name by sys_id | rename name as service_name, sys_id as service_sys_id | join type=left max=0 service_sys_id [ search index=snow sourcetype=snow:cmdb_rel_ci dv_type="Depends on::Used by" earliest=0 latest=now | stats latest(parent) as service_sys_id latest(child) as application_sys_id by sys_id | fields service_sys_id application_sys_id ] | join type=left max=0 application_sys_id [ search index=snow sourcetype=snow:cmdb_ci_list earliest=0 latest=now | stats latest(name) as name by sys_id | rename name as application_name, sys_id as application_sys_id ] | join type=left max=0 application_sys_id [ search index=snow sourcetype=snow:cmdb_rel_ci dv_type="Runs on::Runs" earliest=0 latest=now | stats latest(parent) as application_sys_id latest(child) as server_sys_id by sys_id | fields application_sys_id server_sys_id ] | join type=left max=0 server_sys_id [ search index=snow sourcetype=snow:cmdb_ci_list earliest=0 latest=now | stats latest(name) as name by sys_id | rename name as server_name, sys_id as server_sys_id ] | stats values(server_name) as server_name by service_name We can add search predicates to the sourcetype=snow:cmdb_ci_list subsearches, e.g. dv_operational_status=Operational, to limit the CIs returned. Note that Splunk doesn't "know" if a CI is deleted. If we delete a CI or have multiple CIs with the same name but different sys_id values, invalid or duplicate CIs by name will appear in the search results. Given the searches above, we should highlight: 1) earliest=0 latest=now will return all currently available events. This is not only inefficient for a large number of static CIs or a moderate number of frequently updated CIs, it's also subject to the limits of our indexer cluster and index configurations: SmartStore cache may be exceeded, older CIs may be in frozen buckets, etc. 2) The join command can be inefficient and is subject to subsearch limits in limits.conf. What are the alternatives? We can refactor the searches using transaction, stats, etc. and creative logic, but we'll still be subject to index lifecycle limits and the frequency of CI updates. We can create KV store collections to store CIs, but do we want to clone our CMDB in both indexes and KV store collections? KV store collections also have limits. If we're in a Splunk Cloud environment, for example, increasing instance disk space to store large collections is a challenge. In my own work, I've replicated CMDB data to Neo4j and used Cypher to query and analyze CI relationships. You may be interested in the Common Metadata Data Model (CMDM) https://splunkbase.splunk.com/app/5508 app by @lekanneer. The app implements much of what's required to use Neo4j with Splunk. ... View more

Hi @abhi04, To filter with the where or search commands at the end of the pipeline, try the untable command instead of the transpose command:   | mstats rate_avg(abc*) as abc* where index=def span=3m | untable _time instance MessagesRead | eval MessagesRead=round(MessagesRead, 0) | where ...     ... View more

Hi @faiq1999, The ObjectDN data element should contain the distinguished name of the object. If that's not present in your data, can you provide a deidentified example of the event? ... View more

Hi @tonishantsms, The functionality is deprecated, but the single value visualization still supports automatic color-coding using rangemap and the range values severe (red), high (orange), elevated (yellow), guarded (blue), and low (green). You can take advantage of this functionality by combining the rangemap and chart commands with a trellised single value visualization: | makeresults format="csv" data="cf_app_name,error_rate foo,0 bar,6 baz,51" | rangemap field=error_rate UP=0-5 ISSUE=6-50 default=DOWN | rename range as status | rangemap field=error_rate low=0-5 elevated=6-50 default=severe | chart values(status) as status values(range) as range over cf_app_name You can technically use any method to generate a field named range with the correct values. To use trellis, though, you must use chart, timechart, xyseries, etc. to add hidden field metadata required by the visualization code. <dashboard version="1.1" theme="light"> <label>tonishantsms_single</label> <row> <panel> <single> <search> <query>| makeresults format="csv" data="cf_app_name,error_rate foo,0 bar,6 baz,51" | rangemap field=error_rate UP=0-5 ISSUE=6-50 default=DOWN | rename range as status | rangemap field=error_rate low=0-5 elevated=6-50 default=severe | chart values(status) as status values(range) as range over cf_app_name</query> </search> <option name="colorBy">value</option> <option name="colorMode">block</option> <option name="drilldown">none</option> <option name="trellis.enabled">1</option> <option name="trellis.size">medium</option> </single> </panel> </row> </dashboard> Older documentation is still available through archive.org, e.g. https://web.archive.org/web/20150831233457/http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rangemap, but Splunk may remove the functionality in a future release. ... View more