Hi all fellow Splunkthiasts, I need your help in understanding what could possibly go wrong in HFW upgrade. We are collecting some logs through HFW and rely on search-time extractions for this sourcetype.  After upgrading HFW, extractions stopped working. Data format is basically several Key=Value pairs separated by newline. Investigating this, I found out: There is no props.conf configuration for this sourcetype on upgraded HFW (and it wasn't there even before upgrade). All relevant configuration is on another instance serving as both indexer and search-head. Props.conf in relevant sourcetype on Search-head has AUTO_KV_JSON=true, I don't see KV_MODE in splunk show config props / splunk btool props list (I suppose it takes default value "auto") I have already realized that upgrade didn't take the right path (it was 7.2.6 upgraded to 9.2.1, without gradually upgrading through 8.2). Except search-time extractions, everything seems to work as expected (data is flowing in, event breaking and timestamp extraction seems to be correct). What I don't understand is how can HFW upgrade even affect search-time extractions on another instance. From there I am a bit clueless on what to focus on to fix this. What I don't understand ... View more

Good morning fellow Splunkthiasts! I have an index with 100k+ events per minute (all of them having the same sourcetype), approximately 100 fields are known in this dataset. Some of these events are duplicit, while others are unique. My aim is to understand the duplication and be able to explain what events exactly get duplicated. I am detecting duplicities using this SPL:   index="myindex" sourcetype="mysourcetype" | eventstats count AS duplicates BY _time, _raw   Now I need to identify what fields or their combination make the difference, under what circumstances the event is ingested twice. I tried to use predict command, however it is somehow producing new values for "duplicates" field, but it does not disclose the rule by which it makes the decision. In other words, I am not interested in prediction itself, I want to know the predictors. Is something like that possible in SPL? ... View more

Hello fellow Splunkthusiasts! TL;DR: Is there any way to connect one indexer cluster to two distinct license servers?   Our company has two different licenses: one acquired directly by the company (we posses the license file) the other was acquired by a corporate group to which our company belongs, it is provided to us through group's license server (it is actually some larger license split to several pools, one of them being available to us). The obvious solution is to have one IDXC for each license with SHs searching both clusters. However, both licenses together are approximately 100GB/day, therefore building two independent indexer clusters feels like a waste of resources. What is the best way to approach this? ... View more

Hello fellow Splunkthiasts! I need some insights to understand how comparison functions in mstats could be used. Consider the following query:   | mstats latest(cpu_metric.*) as * WHERE index="osnix_metrics" sourcetype=cpu_metric CPU=all BY host | where pctUser > 50   As expected, it returns a list of hosts having latest CPU usage value higher than 50%. However, according to mstats command reference, I can have comparison expression within WHERE clause and I'd expect it would be more efficient to rewrite the above query like this:   | mstats latest(cpu_metric.*) as * WHERE index="osnix_metrics" sourcetype=cpu_metric CPU=all pctUser > 50 BY host   Unfortunately, this doesn't return any results. I tried to refer to metric before aggregation with no luck:   | mstats latest(cpu_metric.*) as * WHERE index="osnix_metrics" sourcetype=cpu_metric CPU=all cpu_metric.pctUser > 50 BY host   What am I missing here? ... View more

Hello fellow Splunthusiasts! I have some applications running on classic VMs, I am happily splunking their logs and everything works fine. Recently we started to deploy the same applications to Docker containers. To collect logs, I use Docker's native Splunk logging driver and receive the data through HEC. The logging driver adds its own stuff to app's log (either prepends a prefix or it wraps the app's log into JSON, the extra information identifies the container instance). Due to this, some of my field extractors stopped working, as the format of the data actually ingested has changed. What are the best practices for writing extractors universally, so one configuration works with all ways of collecting logs? Just a side note: the point is to have extractors in props.conf in an app distributed from DS, therefore my question is about what should be addressed in regular expression itself. Using | rex field=xxx is not an option here. ... View more

Dear fellow Splunkthusiasts! I have found out one of old scheduled searches in my installation is failing with this error message: Invalid value "+18y@y" for time term 'latest' Looking closer, it turned out the search fails with any value beyond  latest=01/19/2038:04:14:07 . I have noticed this value as expiration date for perpetual licenses as well. I understand this is the maximum time that could be represented by four-byte signed integer as a number of seconds since 1970-01-01 00:00:00. My question is: how do I specify - using time modifiers in SPL - that my time range includes future with no upper limit? I don't want to hard-code the above-mentioned time into my search, as that limit may (and surely will) change in the future, not to mention it is not very self-explanatory. ... View more

Good morning fellow Splunkthiasts! I am trying to build some dashboard using Splunk REST, unfortunately I can not get the data from certain endpoints when using | rest SPL command, while CURL approach returns what is expected. To be specific, I want to read /services/search/jobs/<SID>/summary endpoint. Following SPL returns 0 results:       | rest /services/search/jobs/1648543133.8/summary       When called externally, the endpoint works as expected:       [2022-03-29 10:46:25] root@splunk1.lab2.local:~# curl -k -u admin:pass https://localhost:8089/services/search/jobs/1648543133.8/summary --get | head % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 15578 100 15578 0 0 661k 0 --:--:-- --:--:-- --:--:-- 661k <?xml version='1.0' encoding='UTF-8'?> <results preview='0'> <meta> <fieldOrder> <field>_bkt</field> <field>_cd</field> <field>_eventtype_color</field> <field>_indextime</field> <field>_kv</field> <field>_raw</field>        The same happens with /services/search/jobs/<SID>/results and /services/search/jobs/<SID>/events. When I call /services/search/jobs/ or /services/search/jobs/<SID>, data is returned by both SPL and CURL. I tried this on several Splunk instances with versions ranging from 8.2.3 back to 7.3.3, always using account with admin role - the behavior is always exactly the same. Any hints what I might be missing? ... View more

Good afternoon fellow splunkthiasts, I need your help with data anonymization. Situation: Application on server with UFW produces a log - most of it is a boring operational stuff, however certain records contain a field considered to be sensitive. Log records are necessary for ordinary Ops admins (who need to see all records, but don't need to see the actual sensitive field value) and privileged troubleshooters, who need to see the sensitive data, too. Architecture: data is produced on a server with UFW, will be stored on indexer cluster and there is one heavy-forwarder available in my deployment. Limitations: 1. Due to limited bandwidth between UFW and Splunk servers, it is preferred not to increase volume of data transferred from UFW (bandwidth between HFW and indexers is fine). 2. Due to time-constrained validity of the sensitive field, delays introduced by search->modify->index again every few minutes are not acceptable. 3. Indexing the sensitive records twice is OK. Indexing whole log twice would be too expensive fun. Proposed solution: UFW will forward the log to heavy-forwarder where it should be duplicated. One copy of the data should be anonymized and forwarded to index "operational", while the other one should be filtered (only records with sensitive field are kept) and then forwarded to index "sensitive". Problem: I know how to route data, how to anonymize data, how to filter data before routing, but I am not sure how to connect the dots in described manner. To be specific, I don't know how to duplicate the data on HFW and make sure each copy is treated differently. Can you help, or possibly propose some better solution? ... View more

Dear fellow Splunkthusiasts, is there a way to put my own script manipulating the data in between the forwarder and indexer? To be specific: I have XML logs from SmartMeter/jMeter looking like this: <?xml version="1.0" encoding="UTF-8"?> <testResults version="1.2"> <httpSample t="86" it="0" lt="37" ts="1553000000000" s="true" lb="openLoginPage" rc="200" rm="OK #subresults:3" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="ISO-8859-1" by="3999" sc="1" ec="0" ng="2" na="2" hn="sm-generator2"> <httpSample t="37" it="0" lt="37" ts="1553000000000" s="true" lb="openLoginPage-0" rc="200" rm="OK" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="ISO-8859-1" by="1578" sc="1" ec="0" ng="2" na="2" hn="sm-generator2"> <responseHeader class="java.lang.String"></responseHeader> <requestHeader class="java.lang.String"></requestHeader> <responseData class="java.lang.String"></responseData> <responseFile class="java.lang.String"></responseFile> <cookies class="java.lang.String"></cookies> <method class="java.lang.String">GET</method> <queryString class="java.lang.String"></queryString> <java.net.URL>https://some.host/path/</java.net.URL> </httpSample> <httpSample t="17" it="0" lt="17" ts="1553000000001" s="true" lb="openLoginPage-1" rc="200" rm="OK" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="" by="578" sc="1" ec="0" ng="2" na="2" hn="sm-generator2"> <responseHeader class="java.lang.String"></responseHeader> <requestHeader class="java.lang.String"></requestHeader> <responseData class="java.lang.String"></responseData> <responseFile class="java.lang.String"></responseFile> <cookies class="java.lang.String">some_cookie_name=some_cookie_value</cookies> <method class="java.lang.String">GET</method> <queryString class="java.lang.String"></queryString> <java.net.URL>https://some.host/path/</java.net.URL> </httpSample> </httpSample> ... That is way too verbose for my needs, so I wrote a script transforming the XML to the following: httpSession sessionId="123" t="86" it="0" lt="37" ts="1553000000000" s="true" lb="openLoginPage" rc="200" rm="OK #subresults:3" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="ISO-8859-1" by="3999" sc="1" ec="0" ng="2" na="2" hn="sm-generator2" httpRequest sessionId="123" t="37" it="0" lt="37" ts="1553000000000" s="true" lb="openLoginPage-0" rc="200" rm="OK" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="ISO-8859-1" by="1578" sc="1" ec="0" ng="2" na="2" hn="sm-generator2" method="GET" url="https://some.host/path/" httpRequest sessionId="123" t="17" it="0" lt="17" ts="1553000000001" s="true" lb="openLoginPage-1" rc="200" rm="OK" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="" by="578" sc="1" ec="0" ng="2" na="2" hn="sm-generator2" cookies="some_cookie_name=some_cookie_value" method="GET" url="https://some.host/path/" Please note the output is enriched by sessionId field holding the relationship of session and requests, which can't be simply done by sed. I would like to collect the original log in XML format by universal forwarder, have it processed by my script (possibly on HFW?) and finally index the simplified output. Is something like that possible? Scripted outputs are not exactly what I am looking for as this method would introduce data lags and a need to prevent re-reading the same data (both is solved with monitor:// input method). ... View more