isn't what I needed (the last column is still empty), but gave me other ideas which kind of solved my problem 🙂 ... View more

as you said, it works for a single row; however, I am interested to make it work over the whole timeframe where the count() applied; what I want to achieve is to have two more columns (warning + error) in the table below - is it possible?   _time info 1 2020-09-22 14:00:00 66 2 2020-09-22 14:30:00 56 3 2020-09-22 15:00:00 64 4 2020-09-22 15:30:00 56 5 2020-09-22 16:00:00 66 ... View more

i did manage to fix it with EVAL-_raw = replace(_raw, "^\{\"log\"\:\"(.+?)\",\"stream.*", "\1") looks like SEDCMD works only at index time, I was operating at search time level SEDCMD- = * Only used at index time. ... View more

following your example I have tried with the format below, but somehow it interprets the string between square brackets as splunk command | makeresults | eval _raw="{"log":"[2020/02/28 18:38:00] [error] [out_fw] no upstream connections available","stream":"stderr","docker":{"container_id":"736f7b10a0bda6b97267d8f51e9a8c1fbf8f8f41edea0f3d79b174b7dc5f48cb"},"kubernetes":{"container_name":"billing","namespace_name":"pks-system","pod_name":"telemetry-agent-77f797c749-qj9rv","container_image":"pkstelemetrybot/telemetry-agent:latest","container_image_id":"docker://sha256:c9dbff3df8b19ee2b91df08982cbdb0047bd8e0d830acba73e03959db80c6928","pod_id":"dbbeee1f-63a4-4b03-a74b-0a53b5db44e8","labels":{"app":"telemetry-agent","pod-template-hash":"77f797c749"},"host":"77141ae5-920f-4f65-bd0a-ba78a85c157d","master_url":"https://10.111.192.1:443/api","namespace_id":"6e78cc1d-b345-44c3-b8e3-5c1af076afc5"},"K8Cluster":"k8s-cluster-test-kr-01"}" | rex mode=sed "s/\{("log"\:\".+?)\","stream.*/\1/" ... View more

I would like to do this transparently for the user, without piped commands the events come via HEC, and when I remove the EVAL-_raw = log line everything is displayed correctly in json format; I want to display only the content of the "log" field INDEXED_EXTRACTIONS is not set, and removing KV_MODE on the SH doesn't get rid of the additional line looks like EVAL doubles the values of all fields (excepting the default ones), not only log ... View more

I'm answering myself --> splunk must "see" the new app, in order for restart deploy-server to work --> after that it works as it should so - create the app on the deployment server, reload (restart) the splunk daemon - than edit serverclasses and reload deploy-server as you need ... View more

ldd /opt/splunk/bin/splunkd and see if all required lib dependencies are met; ... View more

curl -ku user:pass https://localhost:8089/servicesNS/admin/search/search/jobs/export -d search="search 2016/02/26 03:00:01,some_search_string" -d output_mode=csv ... View more

I suppose you mean curl -ku user:pass https://localhost:8089/servicesNS/admin/search/search/jobs/export -d search="search 2016/02/26 03:00:01,some_search_string" -d output_mode=csv this "spreads" the search across the indexers and gives all available results (see below) many many thanks "_serial","_time",source,sourcetype,host,index,"splunk_server","_raw" 0,"2016-02-26 03:00:01.000 CET",www,"web_event","10.104.176.7","p_uno","splunk01-indexer","2016/02/26 03:00:01,some_search_string" 0,"2016-02-26 03:00:01.000 CET",www,"web_event","10.104.176.7","p_uno","splunk02-indexer","2016/02/26 03:00:01,some_search_string" 1,"2016-02-26 03:00:01.000 CET",www,"web_event","10.104.176.7","p_uno","splunk02-indexer","2016/02/26 03:00:01,some_search_string" ... View more

sorry, my fault, I have thought search head and wrote indexers the indexes are defined ONLY on the indexers and not on the searchhead curl -k -u user:pass "https://searchhead:8089/services/receivers/simple?source=www&sourcetype=web_event&index=p_uno" -d "2016/02/26 03:00:01,some_random_string,0,2367,84032" answer <msg type="WARN">supplied index 'p_uno' missing</msg> answer for index main works <result> <field k="_index"> <value> <text>main</text> </value> </field> <field k="bytes"> <value> <text>60</text> </value> </field> <field k="host"> <value> <text>10.134.222.99</text> </value> </field> <field k="source"> <value> <text>www</text> </value> </field> <field k="sourcetype"> <value> <text>web_event</text> </value> </field> </result> basically he doesn't find the p_uno on the searchhead, because the searchhead doesn't have the indexes defined what happens if I copy the indexes.conf on the searchhead ? will it also save/store data locally ? thanks for your patience ... View more

exactly, und according to one question I found here (the link is in my OP) the indexes should be defined also on the indexer (which isn't, in my case) ... View more

yes, the indexers are configured as distributed search peers and they are working; the forwarders are configured to send data only to the indexers and the searchhead reads the data (correctly) from both ... View more

Unfortunately yes, it gets to "dynamic" indexex; it's about an IBM Middleware, where several applications (with the corresponding processes), on several environments (test/dev/inte/prod). From the Splunk point of view, it is unknown when an application is deployed and what name shall it have; what we did, as prerequisite, is every application should write its logs in a folder with its name and defined a whole nomenclature. Basically now I have a structure like this (what is below IBM, except apps/, is not such a big problem) /var/log/IBM/apps/App1 /var/log/IBM/apps/App2 ... /var/log/IBM/apps/AppX the App* Folder is created automatically everytime a new app is deployed. What I did: - on the forwarder - inputs.conf --> all files under /var/log/IBM/app/ get the app index - once they reach the indexer, the index is rewritten, taking the string from the path SOURCE_KEY = MetaData:Source REGEX = /var/log/IBM/apps/([^\/]+)/([^\/]+)\.([^\/]+).log DEST_KEY = _MetaData:Index FORMAT = $1 (I take it that this operation wil always take precedence and my index will always be the one defined in this step) now I have to manually add the index, and, until I do that, the logs go nowhere, because of the missing index. Is it a possibility, that when the index defined on step 2 before the indexing doesn't exist, the one defined on step 1 on the forwarder will be used ? hopefully I made sense 🙂 back to your questions: Is the name of the app truly completely dynamic and a total surprise? Yes, unfortunately, I can see it only when I get the "index missing" error pops out Can you explain what you DO know in advance about the data? Nothing, except the path and the structure/nomenclature of this path, on which I have to build my extraction regexes; I know where that the fourth position is the index, then also a few fields based on the filename. But, as earlier said, it is more a theoretical question, based on the similarity with ACL's/iptables rules (for example), where the last rule takes precedence, if somehow satisfied. Thanks for your time ... View more