We need to monitor 300 devices for up and down state and the customer would like to have a tight SLA such as 3 - 4 minutes reporting on a down device. I have the following scripted input working - ping.sh - date; echo ip=<ip1> ; ping -c 4 <ip1> ; date; echo ip=<ip2> ; ping -c 4 <ip2> ; date; echo ip=<ip3> ; ping -c 4 <ip3> ; date; echo ip=<ip4> ; ping -c 4 <ip4> ; ... for 300 lines Is this a right approach? This script as is, is probably taking over 10 minutes to run. Should I spawn all 300 lines in the background? Is it reasonable to spawn 300 commands in parallel?   ... View more

We are trying to to extract the fields from Message in WinEventLog in the Avecto data. The data looks like -   Process Id: 21592 Parent Process Id: 24704 Workstyle: Avecto Defendpoint.Systems Employees Application Group: Avecto Defendpoint.Add Admin - Privileged Users - Applications Reason: <None> File Name: <file name> Hash: 4478EBABE67B50EB111D59F95FE029D31329F1FC Certificate: <name> Description: Command line runner Application Type: exe Product Name: IntelliJ Platform Product Code: <None> Upgrade Code: <None> ....   Each line in Message has a name value pair, separated by a colon. The documentation at https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Rex shows -   | makeresults | eval test="a$1,b$2" | rex field=test max_match=0 "((?<field>[^$]*)\$(?<value>[^,]*),?)"   which works. The similar one I did for Avecto works fine -   index = <avecto index> Message=* | rex field=Message max_match=0 "((?<field>.+)\:(?<value>.+),?)" | table Message field value   We end up with field a and value, each is a multi-value field. Is there a way to change so, we'll have multiple fields, each with its own name/value pair, such as Process_Id having 21592 as its value.             ... View more

We have the varonis ta and its props has the following section -   [varonis:ta]   However, each varonis server that sends us data has a different time zone and the data doesn't have the time zone as part of it. Therefore, can I have also?   [host::tkvar*] TZ = <Tokyo Time Zone>   Will it work? ... View more

We see the following -   /opt/apps/splunk/etc/apps/FireEye_iSIGHT_Splunk_App/bin [splunk@qcsplapps bin]$ python tail_f.py Traceback (most recent call last): File "tail_f.py", line 2, in <module> from common import ISIGHT_ADDON_LOGS, get_curr_epoc_time, SPLUNK_LOG_DIR, LOCAL_CONFIG, DEFAULT_CONFIG File "/opt/apps/splunk/etc/apps/FireEye_iSIGHT_Splunk_App/bin/common.py", line 23, in <module> import splunk.entity as entity ImportError: No module named splunk.entity   What do we need to do in order to import the splunk.entity module? ... View more

I have the EVENT_TIMESTAMP_UTC field with the values of -   2020-11-19 13:50:08.393085 2020-11-19 13:50:08.3517 2020-11-19 13:50:08.306023 2020-11-19 13:50:08.238995 2020-11-19 13:50:08.16885   I would like to create a new time field and treat the data as in the UTC time-zone.  ... View more

We have data such as -     EVENT_TIMESTAMP="2020-11-09 11:12:30.617896 America/New_York",     How can I handle the America/New_York part?     TIME_FORMAT = %Y-%m-%d %H:%M:%S.%6N %Z     The %Z part doesn't seem right. Does this look correct?     TIME_PREFIX = .*EVENT_TIMESTAMP=\"                     ... View more

During an indexer cluster rolling restart we are missing events for a certain index and these events appear to be lost when one of the indexers just came back up after the bounce. Can it be? ... View more

When can we use SPL2 with the lovely comments options described at https://docs.splunk.com/Documentation/SCS/current/Search/Comments  ? ... View more

Couple of times the TA-ObserveIT caused Splunk to shut itself down. How can it be? We see the following in the splunkd.log -     10-02-2020 07:41:27.869 -0400 ERROR ExecProcessor - message from "python /opt/apps/splunk/etc/apps/TA-ObserveIT/bin/observeit_api.py" File "/opt/apps/splunk/etc/apps/TA-ObserveIT/bin/ta_observeit/solnlib/packages/splunklib/binding.py", line 1221, in request 10-02-2020 07:41:27.869 -0400 ERROR ExecProcessor - message from "python /opt/apps/splunk/etc/apps/TA-ObserveIT/bin/observeit_api.py" raise HTTPError(response) 10-02-2020 07:41:27.869 -0400 ERROR ExecProcessor - message from "python /opt/apps/splunk/etc/apps/TA-ObserveIT/bin/observeit_api.py" HTTPError: HTTP 500 Internal Server Error -- {"messages":[{"type":"ERROR","text":"External handler failed with code '-1' and output: ''. See splunkd.log for stderr output."}]} 10-02-2020 07:41:27.903 -0400 ERROR ExecProcessor - message from "python /opt/apps/splunk/etc/apps/TA-ObserveIT/bin/observeit_api.py" ERRORHTTP 500 Internal Server Error -- {"messages":[{"type":"ERROR","text":"External handler failed with code '-1' and output: ''. See splunkd.log for stderr output."}]} 10-02-2020 07:41:27.946 -0400 INFO PipelineComponent - Performing early shutdown tasks 10-02-2020 07:41:27.950 -0400 INFO IndexProcessor - handleSignal : Disabling streaming searches. 10-02-2020 07:41:27.951 -0400 INFO IndexProcessor - request state change from=RUN to=SHUTDOWN_SIGNALED 10-02-2020 07:41:27.951 -0400 INFO IndexProcessor - handleSignal : Disabling streaming searches. 10-02-2020 07:41:27.951 -0400 INFO IndexProcessor - request state change from=RUN to=SHUTDOWN_SIGNALED 10-02-2020 07:41:27.951 -0400 INFO UiHttpListener - Shutting down webui 10-02-2020 07:41:27.961 -0400 INFO UiHttpListener - Shutting down webui completed 10-02-2020 07:41:28.689 -0400 INFO IndexProcessor - ingest_pipe=0: active realtime streams have hit 0 during shutdown 10-02-2020 07:41:28.849 -0400 INFO IndexProcessor - ingest_pipe=1: active realtime streams have hit 0 during shutdown   ... View more

We hit the 0.5 TB limit for _internal in our lower environment and we have barely 10 days of data. Unfortunately, we don’t have enough disk space to have the index for 60 days or so. We wonder if there is a way to log, let’s say 60 days from the Splunk servers but only 10 or so from the forwarders. Is it possible? Are there other ways to configure what goes to _internal ? ... View more

Over the weekend we bounce our indexers and we just found out that the data model accelerations take over an hour to stabilize after such bounces. Their cpu is close to 100% for a while, the time to complete the searches is very long and we don’t fully trust the system when the cpu is so high for quite a long time. Any thoughts how to improve the situation? ... View more

We are ingesting lots of Oracle audit data from various instances into Splunk. We wonder if there are any dashboards/apps for this audit data out there? ... View more

We see in the MC the message - Search peer <host> has the following message: Health Check: msg="A script exited abnormally with exit status: 3" input="../bin/mbbr.py" stanza="default" This script is part of the Malwarebytes Remediation App for Splunk. What can it be? ... View more