We have some issues with line breaking such that we have events that often consist of multiple logical events, or they might consist of fragments of logical records.  We've tried a variety of fixes but no joy so far.  Anyway, that's not really what my question is about.  I'm trying to do a Splunk search that finds only "good" events as in "Scenario 1" below, where the event begins with the XML tag <record> and ends with </record>.  There should be no other tags like this in the event, which would indicate an event like in "Scenario 2", which contains multiple logical events merged together.   Scenario 1:   <record> blah blah blah </record>   Scenario 2:   <record>blah blah blah</record> <record> blah blah blah</record> <record> blah blah blah</record>   I learned that this can be accomplished outside of Splunk using a "negative lookbehind".  I tried this in a Splunk search like the one below. | regex "(?s)^<record>((?!<record>)(\s|\S))*<\/record>$" I couldn't get this to work, however.  This does work as expected in regex101: Scenario 1 (match) Scenario 2 (no match) Any ideas?  I used the regex command instead of the rex command because I didn't need to extract anything.  Thanks in advance. ... View more

I'm trying to use a regex in a transforms.conf file on the Indexer to prevent indexing of informational and debug messages in specific files. The messages are in this format: 2018-11-30 13:10:55,474 INFO blah blah blah 2018-11-30 13:10:55,474 DEBUG blah blah blah There are three Indexers in our environment. I have this coded in the props.conf files on the Indexers: [source::...*(plain|debug|startup).log*] TRANSFORMS-null12 = setnull12 ... and this in the transforms.conf files: [setnull12] REGEX = ^\d+-\d+-\d+\s+\d+:\d+:\d+,*\d+\s+([a-zA-Z0-9]+\s+)?(INFO|DEBUG)\s+.* DEST_KEY = queue FORMAT = nullQueue The undesirable messages are no longer getting indexed by two of the Indexers, but they are still getting indexed on the third Indexer. I've used btool to compare props.conf and transforms.conf files on all three Indexers. There are no differences in transforms.conf files and only inconsequential differences in props.conf files ("pulldown_type = true" set for some sourcetypes for two of the three Indexers, and some unrelated learned sourcetypes defined on one of the Indexers where the rules are working). Does anyone have any ideas of what might be wrong or how I might go about troubleshooting this? I realize it's impossible for anyone to say for sure without a complete picture of our configuration files, but any ideas would be appreciated. ... View more

How can you determine the ages of events in frozen buckets? My first thought was to use dbinspect, but that only looks at hot, warm and cold buckets. I don't want to thaw the frozen buckets just to get info about them. The first two components of bucket names are apparently the newest and oldest event times in "UTC epoch time (in seconds)". I've seen various posts in Splunk Answers about converting epoch times to human readable time, but in this case I don't really have data to search. I just want to convert the epoch times in the bucket names to human readable times. I'm sure I could come up with an Excel formula that would do this easily enough, but I figured I would ask here in case someone already had something handy in this regard. Thanks. (I figured this would be a commonly asked question but I could find no evidence of this.) ... View more

$SPLUNK_DB for one of our Splunk Search Servers filled up recently. The root cause was apparently due to significant recent increases in the _audit index on this server due to recent increases in search activity. We have dedicated Indexers as well, but apparently indexing for the _audit and _internal indexes (and possibly other internal indexes) still occurs on the Search Servers. We had been getting by with the default indexes.conf file (/opt/splunk/etc/system/default/indexes.conf) on the Search Server, but apparently that will no longer be possible. We recently changed the indexes.conf files on our Indexers to use the volume based parameters ([volume:home], [volume:cold], etc.). That has worked well for the Indexers, where the hot/warm volume is a distinct volume and file system from the cold volume. For our Search servers, however, we just have all hot/warm and cold buckets in $SPLUNK_DB (/opt/splunk/var/lib/splunk). That being said, I would still like to use the volume based parameters in a new (non-default) indexes.conf file for the Search Servers, since I think it provides the best "set-it-and-forget-it" control, letting Splunk manage the size of all indexes based upon the volume size. Finally, to my question... What's not clear to me is how to specify the parameters in [volume:home] and [volume:cold] stanzas in this scenario. Specifically, do I specify exactly the same maxVolumeDataSizeMB parameter value for each stanza? For example, if the size of the $SPLUNK_DB volume is 14 GB (14336 MB), is this how I would specify these parameters, or is this saying that I have 14 GB of hot/warm storage, and another 14 GB of cold storage? [volume:home] path = $SPLUNK_DB maxVolumeDataSizeMB = 14336 [volume:cold] path = $SPLUNK_DB maxVolumeDataSizeMB = 14336 [_audit] coldPath = volume:cold/audit/colddb homePath = volume:home/audit/db thawedPath = $SPLUNK_DB/audit/thaweddb [_internal] coldPath = volume:cold/_internaldb/colddb homePath = volume:home/_internaldb/db thawedPath = $SPLUNK_DB/_internaldb/thaweddb This article suggests that it's possible to use volume based parameters with a single volume for hot/warm and cold buckets: When a volume containing warm buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to cold. When a volume containing cold buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to frozen. If a volume contains both warm and cold buckets (which will happen if an index's homePath and coldPath are both set to the same volume), the oldest bucket will be rolled to frozen. But the article does not include an example of how to specify these parameters in this scenario. It's also not clear to me from the indexes.conf documentation. I should add that 14 GB for $SPLUNK_DB (/opt/splunk/var/lib/splunk) is an approximation. The size of $SPLUNK_HOME (/opt/splunk) is actually 19.6 GB. I just want to say 14 GB is available for $SPLUNK_DB to account for the other stuff under /opt/splunk (e.g., the Splunk installation, configuration files, and some overhead for the minimum free space that Splunk requires in order to work). I wish it was possible to provide tighter control over the index space in this situation, but I don't think that is possible without putting $SPLUNK_DB in a separate dedicated file system by itself. ... View more

What is the best way to have Splunk run a shell command on a Splunk server on demand? I found a way to do this via scripted input. A bit cumbersome for my needs, but it works. Are there any better ways to do this? Further explanation is warranted to understand why I'm asking... Our Splunk servers were configured to run Splunk as root (on Linux). I don't know why they were set up this way, but my guess is that this was needed as an easy way to provide Splunk with access to certain files that it normally might not have access to. That works OK, but the problem comes in whenever we need to run commands on Splunk data folders (e.g., du, rm, etc.). Those of us who administer Splunk do not have root access since we are not part of the group that administers the servers themselves. However, we are given access to run specific sudo commands that typically meet our needs. For example, to run the du command for the volume where our Splunk indexes are located, I need to: 1) Shut down Splunk. 2) Change the ownership of all files to our own user account (since many of them will have been set to root). 3) Run the du command. 4) Restart Splunk. We currently have sudo commands for all of these except for du. I could certainly request to have du added to the list of sudo commands, but maintaining the sudo commands has been a headache as well, so I sought an alternative that might eliminate the need for many types of sudo commands. I found that by creating a script that provides scripted input, I was able to run whatever commands I wanted by Splunk itself, since Splunk was running as root. This is a little kludgy, since it's a use of scripted input that was never intended. For example, you need to provide a schedule for the script to run, even though I just want to run it on demand. Can anyone think of a better alternative for what I'm trying to do? The method I found is certainly workable, but I figured I would solicit other ideas. I see that it's possible to call a Python or Perl script via the search inteface and commands.conf file. I don't know Python or Perl, however. I guess I could try to figure out how to create a Python or Perl wrapper to regular bash script. I see a runshellscript.py in the bin folder for Search, but apparently it's only designed to run in an alert context. Thanks. ... View more

I've spent hours studying the documentation and articles outside of splunkbase about configuring indexing, and I'm still confused, and our indexing isn't working as expected. This shouldn't be that difficult. Hot + warm + cold usage is way beyond what I have configured for maxTotalDataSizeMB for the main index, but the volume for hot + warm is only at about 69% utilization, whereas the volume for cold is at 100%. Why did the cold volume fill up? I.e., why isn't cold going to frozen soon enough? I'm thinking now that it might be that I have only taken the main index into account. That's probably because it seems that most of the documentation and articles just talk about maxTotalDataSizeMB and other indexes.conf settings in reference to main. We have maxTotalDataSizeMB set to 160000, which is sufficiently low for main (hot+warm volume size is 100000, cold volume size is 100000). However, maxTotalDataSizeMB is set to the default value of 500000 for the other indexes (history, summary, etc.), which is way beyond the size of the two volumes combined. Don't I need to take those into account as well? That is, don't I need to keep the total of the maxTotalDataSizeMB values for all indexes below our total volume size for this to work? The splunk documentation isn't at all clear about this. I may file a support case for this, but I figured I'd try my luck on the forum first. ... View more