Splunk Search

How do I monitor changes to config files?

Explorer

Hi,

Brand new user of Splunk here. I'm currently evaluating Splunk Enterprise. I need a bit of help understanding why Splunk won't let me monitor a file from Microsoft IIS called "web.config". I can see the contents of a file called xxxyyyzzz.log from the same server using a forwarder, so there is not a configuration issue on the client itself.

My guess it has something to do with the source type, but what? One would think that a file such as web.config would be such a common file that source type "automatic" would be able to work? This file never changes of course, except for when we upgrade the system that this file controls. It would save me lots of time if I could see what settings the update destroys for us.

The file I want to see doesn't even show up in "Sources" when trying to search, even though it is entered in the exact same way as the .log file that does show up. What am I doing wrong?

Tags (2)
SplunkTrust

If you have multiple people touching the configuration files then you should really look into some type of version control. An example would include putting all your configs in GIT then committing changes to that and having a consensus on approving the changes before merging them into Splunk.

Using GIT will reduce the number of errors you allow into production and hold people accountable for their changes

Explorer

That is a good idea, but not what I want to do in this case. My problem is not multiple people touching configuration files, it is that the application itself when it gets updated adds lines that are wrong, or just flat out deletes data. This would be nice to be able to see in splunk.

SplunkTrust

Version control will solve this problem.. You're taking the wrong approach trying to monitor in Splunk

Explorer

Dude you're so unhelpful, I really wish this was stackoverflow so your answer could be voted down.

Enforcing your business logic onto others rather than helping them do what they are asking is really annoying, especially for others which have found this post with a similar quandary. I want to do something very similar to what OP was asking, but your perpetual just use git is so unhelpful and adds nothing to the thread.

Explorer

Hi,
just so I am understanding you correctly, I am as I said very new to Splunk: You are suggesting to not use Splunk at all in this particular case? Or is it some app that gives the functionality of version control?

SplunkTrust

What I'm saying is that you should version your changes to configuration files. This will help scale the deployment and keeps track of all changes

New Member

If you have the config file in version control, you could use those tools to produce a periodic comparison of what was last committed versus what is currently live. Append that comparison to a log, and have the Universal Forwarder watch that log.

You can then set up an alert to tell you when there were any changes, and both identify undesired changes and changes that need to be committed to source control.

Motivator

Additionally, did you configure your forwarder to collect "web.config" file as you did with "xxxyyyzzz.log"?

http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Whysourcetypesmatter

http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Listofpretrainedsourcetypes

Explorer

Hi,

yes, from what I can see they are configured exactly the same. I've tried looking at those links before I posted here, it does not shed any light on how this is supposed to be configured, at least not to a rookie Splunker like me. I take it that I will have to create a brand new sourcetype in the props.conf file on the host with the file I want to monitor?

Motivator

Ah, gotcha. Can you provide sample data so that I can give you basic props.conf structure.

If your file looks similar to below web.config (.NET) file, then this is how your inputs.conf and props.conf should be. If you can explain a little more about your architecture, then I can tell you what all places should these configurations go.

inputs.conf:

## Non-windows
[monitor:///path_to_file/web.config]
index=myidx
sourcetype=web_config

## Windows
[monitor://C:\path\to\file\web.config]
index=myidx
sourcetype=web_config


props.conf:

[web_config]
LINE_BREAKER = ([\r\n]+)\<\?xml\sversion\=
SHOULD_LINEMERGE = false
KV_MODE = xml


Sample data:

 <?xml version="1.0"?>
<configuration xmlns="http://schemas.microsoft.com/.NetConfiguration/v2.0">
<appSettings/>
<connectionStrings/>
<system.web>
<compilation debug="false"/>
<authentication mode="Windows"/>
<customErrors mode="RemoteOnly" defaultRedirect="GenericErrorPage.htm">
<error statusCode="403" redirect="NoAccess.htm"/>
<error statusCode="404" redirect="FileNotFound.htm"/>
</customErrors>
</system.web>
</configuration>

Explorer

Thanks for taking the time to help. Below is a portion of the start of the file, and some part that contain the data that is interesting to monitor:

The start of web.config:

<section name="system.identityModel" type="System.IdentityModel.Configuration.SystemIdentityModelSection, System.IdentityModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />
<section name="system.identityModel.services" type="System.IdentityModel.Services.Configuration.SystemIdentityModelServicesSection, System.IdentityModel.Services, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />

<!--  DYNAMIC DEBUG COMPILATION
Set compilation debug="true" to enable ASPX debugging.  Otherwise, setting this value to
false will improve runtime performance of this application.
Set compilation debug="true" to insert debugging symbols (.pdb information)
into the compiled page. Because this creates a larger file that executes
more slowly, you should set this value to true only when debugging and to
debugging ASP.NET files.
-->
<compilation defaultLanguage="c#" debug="false" targetFramework="4.5.2">
<assemblies>
<add assembly="Microsoft.VisualC, Version=10.0.0.0, Culture=neutral, PublicKeyToken=B03F5F7F11D50A3A" />
<add assembly="System.Data.OracleClient, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />
<add assembly="System.DirectoryServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B03F5F7F11D50A3A" />
<add assembly="System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />
<add assembly="System.Design, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B03F5F7F11D50A3A" />
<add assembly="System.Runtime.Serialization.Formatters.Soap, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B03F5F7F11D50A3A" />
</assemblies>
</compilation>


The data I am interested in monitoring in the file:

  <endpoint address="net.tcp://localhost/SECRET/Services/SecurityServices/AuthorizationService.svc" binding="netTcpBinding" bindingConfiguration="NetTcpBinding_IAuthorizationService" contract="AuthorizationService.IAuthorizationService" name="AuthorizationService" />
<endpoint address="https://secret.domain.com/Services/PServices/ESSBasic.svc" binding="wsHttpBinding" bindingConfiguration="secureWsHttpBinding" contract="ESSBasicService.IESSBasic" name="BasicHttpBinding_IESSBasic" />
<endpoint address="https://secret.domain.com/Services/PServices/ESSFlex.svc" binding="wsHttpBinding" bindingConfiguration="secureWsHttpBinding" contract="ESSFlexService.IESSFlex" name="BasicHttpBinding_IESSFlex" />
</client>

<clear />


My infrastructure regarding Splunk is basically just a single standalone server that will be collecting data from other servers in the same domain using forwarders that are installed on the hosts. Simplest possible installation in other words.

Motivator

Try this props.conf:

[web_config]
LINE_BREAKER = ([\r\n]+)\s?\<section\sname\=
SHOULD_LINEMERGE = false
KV_MODE = xml
DATETIME_CONFIG = NONE # "NONE" will leave the event time set to whatever time was selected by the input layer, since there is no timestamp in the events


Note: It is difficult to ingest only some part of the data in file. I am not saying it is impossible but if having few extra lines doesn't hurt you, then ingest the whole file.

Place this on both UF and your standalone server and restart splunk. Let me know how it goes.

Explorer

Thanks,
will try to do this during the week! I noticed there are a lot of props.conf on the server, which one is the one I am supposed to edit?
The documentation say you should edit the file in \$SPLUNK_HOME/etc/system/local/

I don't have any props.conf in that location, am I supposed to create a new file at that location? Or should I make a copy of the one in "C:\Program Files\Splunk\etc\system\default" and add the things you suggested in that file?

Motivator

Both ways works. Easiest is to create a new props.conf and copy the configs over, save and restart splunk.

Explorer

Hi again,
Tried your suggestions, probably I am doing something wrong because I get no data collected. Here is what I have done:

1. On both the Splunk server and the UF, created the file "props.conf" in C:\Program Files\SplunkUniversalForwarder\etc\system\local\ Contents of props.conf:

LINE_BREAKER = ([\r\n]+)\s?\

Explorer

Seems my answer got cut off. Anyway, I put your suggestion for the props.conf and the inputs.conf in the files and restarted splunk. Still wont collect anything sadly.

Motivator

Have a look at your input status to see if UF is reading the file, use this rest api http://<your_UF_hostname>:8089/services/admin/inputstatus.

Ultra Champion

Maybe you'll describe step by step what you have done? It's pretty difficult to guess ; -)

Explorer

Ok,