Hello fellow Splunthusiasts! I have some applications running on classic VMs, I am happily splunking their logs and everything works fine. Recently we started to deploy the same applications to Docker containers. To collect logs, I use Docker's native Splunk logging driver and receive the data through HEC. The logging driver adds its own stuff to app's log (either prepends a prefix or it wraps the app's log into JSON, the extra information identifies the container instance). Due to this, some of my field extractors stopped working, as the format of the data actually ingested has changed. What are the best practices for writing extractors universally, so one configuration works with all ways of collecting logs? Just a side note: the point is to have extractors in props.conf in an app distributed from DS, therefore my question is about what should be addressed in regular expression itself. Using | rex field=xxx is not an option here. ... View more

Dear fellow Splunkthusiasts! I have found out one of old scheduled searches in my installation is failing with this error message: Invalid value "+18y@y" for time term 'latest' Looking closer, it turned out the search fails with any value beyond  latest=01/19/2038:04:14:07 . I have noticed this value as expiration date for perpetual licenses as well. I understand this is the maximum time that could be represented by four-byte signed integer as a number of seconds since 1970-01-01 00:00:00. My question is: how do I specify - using time modifiers in SPL - that my time range includes future with no upper limit? I don't want to hard-code the above-mentioned time into my search, as that limit may (and surely will) change in the future, not to mention it is not very self-explanatory. ... View more

Good morning fellow Splunkthiasts! I am trying to build some dashboard using Splunk REST, unfortunately I can not get the data from certain endpoints when using | rest SPL command, while CURL approach returns what is expected. To be specific, I want to read /services/search/jobs/<SID>/summary endpoint. Following SPL returns 0 results:       | rest /services/search/jobs/1648543133.8/summary       When called externally, the endpoint works as expected:       [2022-03-29 10:46:25] root@splunk1.lab2.local:~# curl -k -u admin:pass https://localhost:8089/services/search/jobs/1648543133.8/summary --get | head % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 15578 100 15578 0 0 661k 0 --:--:-- --:--:-- --:--:-- 661k <?xml version='1.0' encoding='UTF-8'?> <results preview='0'> <meta> <fieldOrder> <field>_bkt</field> <field>_cd</field> <field>_eventtype_color</field> <field>_indextime</field> <field>_kv</field> <field>_raw</field>        The same happens with /services/search/jobs/<SID>/results and /services/search/jobs/<SID>/events. When I call /services/search/jobs/ or /services/search/jobs/<SID>, data is returned by both SPL and CURL. I tried this on several Splunk instances with versions ranging from 8.2.3 back to 7.3.3, always using account with admin role - the behavior is always exactly the same. Any hints what I might be missing? ... View more

Good afternoon fellow splunkthiasts, I need your help with data anonymization. Situation: Application on server with UFW produces a log - most of it is a boring operational stuff, however certain records contain a field considered to be sensitive. Log records are necessary for ordinary Ops admins (who need to see all records, but don't need to see the actual sensitive field value) and privileged troubleshooters, who need to see the sensitive data, too. Architecture: data is produced on a server with UFW, will be stored on indexer cluster and there is one heavy-forwarder available in my deployment. Limitations: 1. Due to limited bandwidth between UFW and Splunk servers, it is preferred not to increase volume of data transferred from UFW (bandwidth between HFW and indexers is fine). 2. Due to time-constrained validity of the sensitive field, delays introduced by search->modify->index again every few minutes are not acceptable. 3. Indexing the sensitive records twice is OK. Indexing whole log twice would be too expensive fun. Proposed solution: UFW will forward the log to heavy-forwarder where it should be duplicated. One copy of the data should be anonymized and forwarded to index "operational", while the other one should be filtered (only records with sensitive field are kept) and then forwarded to index "sensitive". Problem: I know how to route data, how to anonymize data, how to filter data before routing, but I am not sure how to connect the dots in described manner. To be specific, I don't know how to duplicate the data on HFW and make sure each copy is treated differently. Can you help, or possibly propose some better solution? ... View more

Dear fellow Splunkthusiasts, is there a way to put my own script manipulating the data in between the forwarder and indexer? To be specific: I have XML logs from SmartMeter/jMeter looking like this: <?xml version="1.0" encoding="UTF-8"?> <testResults version="1.2"> <httpSample t="86" it="0" lt="37" ts="1553000000000" s="true" lb="openLoginPage" rc="200" rm="OK #subresults:3" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="ISO-8859-1" by="3999" sc="1" ec="0" ng="2" na="2" hn="sm-generator2"> <httpSample t="37" it="0" lt="37" ts="1553000000000" s="true" lb="openLoginPage-0" rc="200" rm="OK" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="ISO-8859-1" by="1578" sc="1" ec="0" ng="2" na="2" hn="sm-generator2"> <responseHeader class="java.lang.String"></responseHeader> <requestHeader class="java.lang.String"></requestHeader> <responseData class="java.lang.String"></responseData> <responseFile class="java.lang.String"></responseFile> <cookies class="java.lang.String"></cookies> <method class="java.lang.String">GET</method> <queryString class="java.lang.String"></queryString> <java.net.URL>https://some.host/path/</java.net.URL> </httpSample> <httpSample t="17" it="0" lt="17" ts="1553000000001" s="true" lb="openLoginPage-1" rc="200" rm="OK" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="" by="578" sc="1" ec="0" ng="2" na="2" hn="sm-generator2"> <responseHeader class="java.lang.String"></responseHeader> <requestHeader class="java.lang.String"></requestHeader> <responseData class="java.lang.String"></responseData> <responseFile class="java.lang.String"></responseFile> <cookies class="java.lang.String">some_cookie_name=some_cookie_value</cookies> <method class="java.lang.String">GET</method> <queryString class="java.lang.String"></queryString> <java.net.URL>https://some.host/path/</java.net.URL> </httpSample> </httpSample> ... That is way too verbose for my needs, so I wrote a script transforming the XML to the following: httpSession sessionId="123" t="86" it="0" lt="37" ts="1553000000000" s="true" lb="openLoginPage" rc="200" rm="OK #subresults:3" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="ISO-8859-1" by="3999" sc="1" ec="0" ng="2" na="2" hn="sm-generator2" httpRequest sessionId="123" t="37" it="0" lt="37" ts="1553000000000" s="true" lb="openLoginPage-0" rc="200" rm="OK" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="ISO-8859-1" by="1578" sc="1" ec="0" ng="2" na="2" hn="sm-generator2" method="GET" url="https://some.host/path/" httpRequest sessionId="123" t="17" it="0" lt="17" ts="1553000000001" s="true" lb="openLoginPage-1" rc="200" rm="OK" tn="10.20.30.40:1234_TestCases 1-2" dt="text" de="" by="578" sc="1" ec="0" ng="2" na="2" hn="sm-generator2" cookies="some_cookie_name=some_cookie_value" method="GET" url="https://some.host/path/" Please note the output is enriched by sessionId field holding the relationship of session and requests, which can't be simply done by sed. I would like to collect the original log in XML format by universal forwarder, have it processed by my script (possibly on HFW?) and finally index the simplified output. Is something like that possible? Scripted outputs are not exactly what I am looking for as this method would introduce data lags and a need to prevent re-reading the same data (both is solved with monitor:// input method). ... View more