I have a RHEL 6.10 Splunk server and currently have the following configuration in /etc/security/limits.d/ for open file descriptors: root soft nofile 64512 root hard nofile 80896 But when I run the Monitoring Console Health Check, it reports that the current ulimit.open_files = 4096. I tried using * instead of root but that didn't change anything. I have seen this issue brought up but for RHEL 7.x that uses systemd instead of SysV. ... View more

I have been working with new inputs for a testing environment and I noticed that one point the Data Summary said that there are 10 events indexed with the earliest and latest event being 2 months ago. At first I thought that the data indexed had been erased, but after checking some custom dashboards, executing searches, and checking the server's storage, all the data is still there. Why is the Data Summary not reflecting the reality of the amount of data indexed? It was working fine earlier in the day and the only changes I did were in inputs.conf and props.conf, I didn't change the configuration of the server or the indexes. Running 7.3.4 on a single-instance deployment. ... View more

I am creating a dashboard to show all Linux command line history per user and I would like to create an input where you can type the user and if it matches anything in a case statement, it assigns a value to "source" and runs the search. For example, I have two sources: source=/root/.bash_history source=/opt/splunk/.bash_history I have a token $acct$ which holds the user that was typed in the input. I wrote this search: index=linux sourcetype=linux_cli | eval search_source=case($acct$ == root, "/root/.bash_history", $acct$ == splunk, "/opt/splunk/.bash_history") | search source=search_source But this returns no results. How can I do this assignment during search? ... View more

I have a script for Linux that executes "sar -n DEV" and formats the output to look like: Linux <kernel version> (<hostname>) <date> <arch> (<#> CPU) Average: <interface> <field1> <field2> <field3> Average: <interface> <field1> <field2> <field3> Average: <interface> <field1> <field2> <field3> Using Splunk Web's field extractor, I have a regex that applies field extraction to the first "Average:" line. How do I make it so the field is applied to as many "Average:" lines exist? ... View more

I have a bash script that queries audit.log using ausearch for events that I have configured in audit.rules to have attached a specific key. This is the general idea of the script: # Assign path variables # Capture saved timestamp from last execution # Save new timestamp for future execution # Execute query using ausearch # Redirect stdout and stderr to two different variables # Check stderr variable does not equal "<no matches>" and exit execution if true Now this is where I have tried multiple things and while all of them work when executed from a terminal, they don't generate any results when Splunk executes them. echo $stdout_var OR echo $stdout_var > /path/to/tmp cat /path/to/tmp I have even tried monitoring "/path/to/tmp", that's when I realized this might be a user permissions issue since the file is generated, but there is never any content in it. Currently, SPLUNK_OS_USER=root, but does that mean that the script is executed as SPLUNK_OS_USER? Or do I have to configure the script through Splunk to run as a specific user? Again, when I execute this command manually from the CLI as root, it works exactly as expected, but it generates nothing when executed through the scripted input. EDIT: So I continue to debug to find the issue. 1. Script is being executed as root (placed "echo $UID" at top of script, which showed on Splunk Web as an event that simply returned 0) 2. I have added "echo" commands at every step of the execution, and I have found that it actually keeps exiting execution at the stderr variable check. This makes no sense, because when I run exactly the same command with the same timestamp, on the command line it works as expected, but apparently when it is executed by Splunk as a scripted input, ausearch returns nothing. I know this is starting look like a bash script question, but from a Linux standpoint, the script works as it should. I don't know what else to do at this point to make it work through Splunk. ... View more

I am in the process of designing an implementation of Splunk using version 7.3.4 (for organizational reasons) and the first time I opened Splunk Web I received a notice about the Python 3.7 migration. Is this applicable to version 7.3.4 or only for 8.x? ... View more

I have a test environment with similar devices (Windows 10 laptops that serve the same purpose and have the same software) that I'm using to test the SSL configuration for the input-outputs of log receiving/forwarding between Universal Forwarders and one Indexer (both version 6.5). The issue is that for some devices, once the Universal Forwarder is installed and they point to the Indexer (which is set up as a Deployment Server as well) to pull the app that contains the testing SSL configuration for outputs.conf, certain devices do not hash the sslPassword when the UF restarts, while others do. So all devices are phoning home and are listed under Forwarder Management, but only some are sending logs, while the rest (that are not hashing the sslPassword) can't send any logs. What is going on? ... View more

I am currently monitoring a file that generates logs, but assigns the time in epoch format. Is there a way to transform/convert the epoch timestamp to a human readable format during index time? (I know there is a way to do this in a search query, but I would like to store the timestamp in a human readable format.) EDIT: The file I am monitoring is /root/.bash_history (I made system configuration changes to make sure that every command execution is stored immediately) and most of the time Splunk does a very good job at assigning timestamps to each command execution, but sometimes it will create one event with multiple commands and assigns one timestamp to all of them. So I decided to generate a timestamp that is appended to every command. The way they are listed in the file is as such: #1234567890 <command> Now, I've set the correct configuration in props.conf to ensure that every two strings is one event, but now I'm trying to assign the "1234567890" as the timestamp of the event and make sure it shows in human readable format for search results. ... View more

I will be installing version 7.3.4 in a testing environment and I am wondering if this version already has the patched datetime.xml that was released before the beginning of this year. The notes released under "Timestamp recognition of dates with two-digit years fails beginning January 1, 2020" list all the major/minor versions that have the patched datetime.xml, but does that mean that any version outside of those need to have the file updated manually (or follow the other options included in the released documentation)? ... View more

Is it possible to stop processing properties in props.conf if a condition is met? I've been running a lot of tests with props.conf-transforms.conf and how the indexer performs under different conditions and settings and I was wondering if there was a way to stop processing transforms assigned under a props.conf stanza if a transform was successful. ... View more