What's the best way to track down props.conf problems?

[Lowell]

I'm looking for tips & tricks for tracking down props-related configuration issues. These kind of things can be a pain to track down and generally use up a bunch of time getting to the root cause, so I would like to consolidate some common troubleshooting techniques to refer people to.

Often times these kind of problems are hard to describe beyond "splunk appears to be ignoring my settings"; and most often the truth is that it's not a splunk problem but a configuration problem...

If there are lots of ideas, perhaps the most useful stuff can be put on a community wiki page

Most common mistakes:

• Knowing when you have to restart splunkd?
• Config entry typos (Such as saying TRANSFORM instead of TRANSFORMS) These can be difficult to spot.
• Permissions issues. This is especially difficult if you have props.conf settings in one app and then setup an inputs.conf in a different app. Unless your props entries are exported, you probably will not get the results you are looking for.)
• Incorrectly named config files. (If you put your config entries is prop.conf (instead of props.conf), splunk will ignore them. Nearly all of the config files are plural.)

[yahuja_splunk]

I had the same issue, and my custom apps directory didnt have the metadata folder, that is why my custom props.conf wasn't working.
The metadata folder should have one file called local.meta and the contents of the file are

[]
access = read : [ * ], write : [ admin ]
export = system

Hope it helps!

Happy Splunking!

[gkanapathy]

Some people have found this useful: http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings%3F

[Lowell]

Wow. That is a great page, I don' think I had seen that page before. Thank's for pointing it out. (For the sake of a consolidated answer, I've added this link under "Additional resources" on the answer above). Thanks!

[brianirwin]

Lowell, this is fantastic, I had been working some issues around props.conf but it would have taken me months to resolve indexing a few records at a time. With these steps I had great success and have ironed out the majority of my issues.

Brian

[Lowell]

Glad to hear it.

[Lowell]

Update for Splunk 5/6:

I should probably rewrite this answer completely. Splunk has made some great changes that simplify troubleshooting this kind of problem since the early 4.x releases (which is when this answer was first written.) I don't have time for a full rewrite, but here are some highlights that I would expand upon if/when I get around do to a more thorough re-write...

1. Use the Data Preview tool whenever you can. (That pretty much replaces the | file /path/to/sample/file approach which is noted below.) With the data preview tool you can test your props.conf settings interactively and immediately see if your settings are correct or not. This is a bit more tricky when working with transformers. I've run into a few weird corner cases with the tool where it doesn't accurately represent the indexing process; but that's rare (and last time I had that problem was a few versions back). In general, you can rely on this tool. It will save you tons of time and frustration when on-boarding new data.
2. The | extract reload=T isn't generally necessary anymore. Since searches are kicked off in their own processes all the config file are re-read from the file system when the search process is launched. So the most up-to-date .conf files are normally available by just re-running your search. However, this command is still useful when reloading index-time props and transforms search settings.
3. Splunk checks for config errors at startup. Splunk added a feature (back in the mid 4.x releases, I think) that reports basic configuration errors. These are display during startup by default. You can get this list manually by running splunk btool check. This isn't foolproof but can catch simple typos that could cause lots of frustration. Sometimes these messages can be bogus, especially if you are using an undocumented feature. (This tool checks for config names based on entries in the README/*.props.spec files)
4. Valid lots of regexes at once. Run the splunk btool validate-regex command. If you see any output, then you have issues with one or more of your regexes. I haven't tested this thoroughly, but it finds errors with regexes in EXTRACT in props.conf and REGEX in transforms.conf (as of Splunk 6.1.1). I did not find a bogus regex in a SEDCMD or in an invalid prop stanza name based on an invalid regex. Splunk allows you to use regexes many places, and I'm guessing that for now only the most common stuff is checked. So I'd suggest using this to double check your work, but don't get lazy and expect that it will capture all your mistakes ;-). Similarly, splunk btool validate-strptime can be used to check for invalid times in the TIME_FORMAT entry in props.conf files, and possibly other places too. Unfortunately I haven't been able to find any documentation on either of these commands.

---

Splunk provides a few utilities that can greatly speed up the process of tracking down props-related config problems. The first two can even be used to check configuration changes without restarting splunkd. (Although, you need to restart splunkd once you get your configuration issues resolved.)

Splunk "test sourcetype" utility:

This utility will show you all of the relevant props config entries for a given log file. This let's you quickly test your source to sourcetype matching rules are working properly. (Or, it test that any rules:: or delayedrule:: are functioning properly.) It will also reveal simple issues of file permission. I've often found this useful in detecting overlapping source rules or issues with stanza priorities.

Example:

splunk test sourcetype /var/log/your_log_file

Update: The output format of this command is all on one line and difficult to read, IMHO. So I normally use this sed command to make it more readable:

splunk test sourcetype /var/log/your_log_file | sed -re "s/-> '([^']*)'/ = \1\n/g;s/\{/\{\n /;s/}/}\n/"

Splunk's "btool" utility

This utility can be used to dump a nested configuration for a specific config file or a specific stanza within a config file. You can optionally specify which application you would like to test this from. (Normally you will want to specify whichever application your have your inputs.conf setup to use.)

$SPLUNK_HOME/bin/splunk cmd btool props list <sourcetype>

Splunk | file search command

The file search command in splunk lets you view a local log file as though it were indexed. Not all props settings are honored during this process, but you can often use this to test your setup before adding a new input and catch minor (and sometimes major) problems with timestamp recognition, and event breaking logic before loading anything into your index.

Keep in mind that any settings changes do require a splunkd restart in order to take effect here. (While it's a pain to do so, it's still faster than indexing events improperly and then having to dump your indexes and reload, or try to find and delete incorrectly indexed events.)

Example: (Type this into the search bar)

| file /var/log/your_log_file

If you want to due additional searching on your log file, you can do so with a post-search command, like so:

| file /var/log/your_log_file | search my-search-term NOT unwanted-search-term ...

Splunk's | extract reload=T search command:

This trick is useful once you have your events indexed correctly and would like to setup field extractions. This is helpful if you edit your props.conf or transforms.conf files by hand. You should be able to stick it in your search and have your config changes be reloaded which is much faster and less problematic that restarting splunk. (Oh, and don't forget to remove this command before you save your search--you only want to use this interactively on an as-needed basis)

---

Additional resources:

• Splunk Docs:
  • Troubleshooting configurations - More in-depth examples using btool.
  • Configuration file precedence - Understanding Splunk's layered configurations.
  • Attribute precedence within a single props.conf file - Discussion on using priority and matching based on souce:: vs host:: vs sourcetype.
• Splunk Wiki:
  • Where do I configure my Splunk settings? - This is a great resource that provides insight into how splunk processes your events, how event processing changes for light vs heavy forwarders and which config entries are used in different processing phases. (A big thanks to Gerald for point out this link. It's extremely helpful.)

[hsesterhenn]

Hi,

great article.

Just found out that

splunk btool check

will do the same checks as a

splunk start

Example: (Typo SOULD_LINEMERGE instead of SHOULD_LINEMERGE)

Invalid key in stanza [myst:test] in
/opt/splk/splunk-624/etc/apps/aa_test/local/props.conf, line 2:
SOULD_LINEMERGE  (value:  true)

HTH,

Holger

[mjpieters]

The validate-regex command sounds just the ticket, but in 6.0.4 it doesn't appear to exist. The only reference to the tool that I can find is your post. Presumably it is new in 6.1 then?

[abonuccelli_spl]

/opt/SPLUNK/6.0/splunk/bin $ ./splunk help test
The 'test' and 'train' commands have been deprecated.
:-(

[gkanapathy]

Probably worth noting the --debug and --app options on btool. --app in particular will probably help catch many problems.

[duartet]

-debug and -app

[Lowell]

Chris, Your right, the "extract reload=T" is a very helpful thing to know about. (I guess I was originally just thinking about indexing issues, but I agree that it makes sense to mention it here.) I've added a note about it.

[chris]

You might want to add that "| extract reload=T" will make changes in props.conf active. It is explained in the beginning of props.conf.spec file so people who edit props.conf will probably know. But sometimes I forget

[Lowell]

Glad your found it useful. (Feel free to vote it up!)

[balbano]

This just made my day... thanks Lowell!!!

Brian