Splunk Search

How do I monitor changes to config files?

AndreasMartenss
Explorer

Hi,

Brand new user of Splunk here. I'm currently evaluating Splunk Enterprise. I need a bit of help understanding why Splunk won't let me monitor a file from Microsoft IIS called "web.config". I can see the contents of a file called xxxyyyzzz.log from the same server using a forwarder, so there is not a configuration issue on the client itself.

My guess it has something to do with the source type, but what? One would think that a file such as web.config would be such a common file that source type "automatic" would be able to work? This file never changes of course, except for when we upgrade the system that this file controls. It would save me lots of time if I could see what settings the update destroys for us.

The file I want to see doesn't even show up in "Sources" when trying to search, even though it is entered in the exact same way as the .log file that does show up. What am I doing wrong?

skoelpin
SplunkTrust
SplunkTrust

If you have multiple people touching the configuration files then you should really look into some type of version control. An example would include putting all your configs in GIT then committing changes to that and having a consensus on approving the changes before merging them into Splunk.

Using GIT will reduce the number of errors you allow into production and hold people accountable for their changes

0 Karma

AndreasMartenss
Explorer

That is a good idea, but not what I want to do in this case. My problem is not multiple people touching configuration files, it is that the application itself when it gets updated adds lines that are wrong, or just flat out deletes data. This would be nice to be able to see in splunk.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Version control will solve this problem.. You're taking the wrong approach trying to monitor in Splunk

jeremyfer
Explorer

Dude you're so unhelpful, I really wish this was stackoverflow so your answer could be voted down.

Enforcing your business logic onto others rather than helping them do what they are asking is really annoying, especially for others which have found this post with a similar quandary. I want to do something very similar to what OP was asking, but your perpetual just use git is so unhelpful and adds nothing to the thread.

AndreasMartenss
Explorer

Hi,
just so I am understanding you correctly, I am as I said very new to Splunk: You are suggesting to not use Splunk at all in this particular case? Or is it some app that gives the functionality of version control?

0 Karma

skoelpin
SplunkTrust
SplunkTrust

What I'm saying is that you should version your changes to configuration files. This will help scale the deployment and keeps track of all changes

0 Karma

LearninStuff
Explorer

If you have the config file in version control, you could use those tools to produce a periodic comparison of what was last committed versus what is currently live. Append that comparison to a log, and have the Universal Forwarder watch that log.

You can then set up an alert to tell you when there were any changes, and both identify undesired changes and changes that need to be committed to source control.

0 Karma

sudosplunk
Motivator

Additionally, did you configure your forwarder to collect "web.config" file as you did with "xxxyyyzzz.log"?

In case you haven't already, please refer to below links to learn more about source types.

http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Whysourcetypesmatter

http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Listofpretrainedsourcetypes

0 Karma

AndreasMartenss
Explorer

Hi,

yes, from what I can see they are configured exactly the same. I've tried looking at those links before I posted here, it does not shed any light on how this is supposed to be configured, at least not to a rookie Splunker like me. I take it that I will have to create a brand new sourcetype in the props.conf file on the host with the file I want to monitor?

0 Karma

sudosplunk
Motivator

Ah, gotcha. Can you provide sample data so that I can give you basic props.conf structure.

If your file looks similar to below web.config (.NET) file, then this is how your inputs.conf and props.conf should be. If you can explain a little more about your architecture, then I can tell you what all places should these configurations go.

inputs.conf:

## Non-windows
[monitor:///path_to_file/web.config]
index=myidx
sourcetype=web_config

## Windows
[monitor://C:\path\to\file\web.config]
index=myidx
sourcetype=web_config

props.conf:

[web_config]
LINE_BREAKER = ([\r\n]+)\<\?xml\sversion\=
SHOULD_LINEMERGE = false
KV_MODE = xml

Sample data:

 <?xml version="1.0"?>
    <configuration xmlns="http://schemas.microsoft.com/.NetConfiguration/v2.0">
        <appSettings/>
        <connectionStrings/>
        <system.web>
            <compilation debug="false"/>
            <authentication mode="Windows"/>
            <customErrors mode="RemoteOnly" defaultRedirect="GenericErrorPage.htm">
                <error statusCode="403" redirect="NoAccess.htm"/>
                <error statusCode="404" redirect="FileNotFound.htm"/>
            </customErrors>
        </system.web>
    </configuration>
0 Karma

AndreasMartenss
Explorer

Thanks for taking the time to help. Below is a portion of the start of the file, and some part that contain the data that is interesting to monitor:

The start of web.config:

<section name="system.identityModel" type="System.IdentityModel.Configuration.SystemIdentityModelSection, System.IdentityModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />
<section name="system.identityModel.services" type="System.IdentityModel.Services.Configuration.SystemIdentityModelServicesSection, System.IdentityModel.Services, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />


<!--  DYNAMIC DEBUG COMPILATION
      Set compilation debug="true" to enable ASPX debugging.  Otherwise, setting this value to
      false will improve runtime performance of this application. 
      Set compilation debug="true" to insert debugging symbols (.pdb information)
      into the compiled page. Because this creates a larger file that executes
      more slowly, you should set this value to true only when debugging and to
      false at all other times. For more information, refer to the documentation about
      debugging ASP.NET files.
-->
<compilation defaultLanguage="c#" debug="false" targetFramework="4.5.2">
  <assemblies>
    <add assembly="Microsoft.VisualC, Version=10.0.0.0, Culture=neutral, PublicKeyToken=B03F5F7F11D50A3A" />
    <add assembly="System.Data.OracleClient, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />
    <add assembly="System.DirectoryServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B03F5F7F11D50A3A" />
    <add assembly="System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />
    <add assembly="System.Design, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B03F5F7F11D50A3A" />
    <add assembly="System.Runtime.Serialization.Formatters.Soap, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B03F5F7F11D50A3A" />
  </assemblies>
</compilation>

The data I am interested in monitoring in the file:

  <endpoint address="net.tcp://localhost/SECRET/Services/SecurityServices/AuthorizationService.svc" binding="netTcpBinding" bindingConfiguration="NetTcpBinding_IAuthorizationService" contract="AuthorizationService.IAuthorizationService" name="AuthorizationService" />
  <endpoint address="https://secret.domain.com/Services/PServices/ESSBasic.svc" binding="wsHttpBinding" bindingConfiguration="secureWsHttpBinding" contract="ESSBasicService.IESSBasic" name="BasicHttpBinding_IESSBasic" />
  <endpoint address="https://secret.domain.com/Services/PServices/ESSFlex.svc" binding="wsHttpBinding" bindingConfiguration="secureWsHttpBinding" contract="ESSFlexService.IESSFlex" name="BasicHttpBinding_IESSFlex" />
</client>


<clear />
<add key="ConfigInNeptune" value="True" />
<add key="DBAPISection" value="RE_ODBC" />
<add key="DBPath" value="D:\Visma\Programs\Ciceron\" />
<add key="NeptuneConnectionName" value="SECRET" />
<add key="NeptuneSSO" value="False" />
<add key="PathToTravel" value="" />
<add key="TurbyteRastVarning" value="True" />
<add key="ActionLog" value="True" />
<add key="SkickaEmailFrMeddBemRes" value="True" />
<add key="VarningGammalAvvTjg" value="True" />
<add key="ShowRetro" value="True" />
<add key="ShowDoctorInterface" value="True" />
<add key="CheckOverDraft" value="True" />
<add key="ArbtBemFranvorsak" value="False" />
<add key="AlltidKostAvdrag" value="False" />
<add key="CheckCompensatoryBalance" value="-1" />
<add key="ProxyPath" value="" />
<add key="CheckAbsenceHighErrorMsgLevel" value="False" />
<add key="OblArbStalleVidKonto" value="False" />
<add key="WindowsIdentityFoundationEnabled" value="True" />
<add key="VismaWindowOnly" value="False" />
<add key="AMRunInIframePath" value="" />
<add key="AMDashBoardPath" value="/SECRET/Dashboard/Home/Get/1" />
<add key="AMSaldoTileMaxOkomp" value="50" />
<add key="AMSaldoTileShowSem" value="True" />
<add key="AMSaldoTileMaxOtid" value="200" />
<add key="AMFlexTileUseButtons" value="False" />
<add key="ESSBasicServiceLink" value="http://localhost/PWeb.Services/ESSBasic.svc/Web/" />
<add key="ValidationSettings:UnobtrusiveValidationMode" value="None" />
<add key="WIFLogoutLink" value="" />
<add key="TravelShowHigherCompensationWarning" value="False" />

My infrastructure regarding Splunk is basically just a single standalone server that will be collecting data from other servers in the same domain using forwarders that are installed on the hosts. Simplest possible installation in other words.

0 Karma

sudosplunk
Motivator

Try this props.conf:

[web_config]
LINE_BREAKER = ([\r\n]+)\s?\<section\sname\=
SHOULD_LINEMERGE = false
KV_MODE = xml
DATETIME_CONFIG = NONE # "NONE" will leave the event time set to whatever time was selected by the input layer, since there is no timestamp in the events

Note: It is difficult to ingest only some part of the data in file. I am not saying it is impossible but if having few extra lines doesn't hurt you, then ingest the whole file.

Place this on both UF and your standalone server and restart splunk. Let me know how it goes.

0 Karma

AndreasMartenss
Explorer

Thanks,
will try to do this during the week! I noticed there are a lot of props.conf on the server, which one is the one I am supposed to edit?
The documentation say you should edit the file in $SPLUNK_HOME/etc/system/local/

I don't have any props.conf in that location, am I supposed to create a new file at that location? Or should I make a copy of the one in "C:\Program Files\Splunk\etc\system\default" and add the things you suggested in that file?

0 Karma

sudosplunk
Motivator

Both ways works. Easiest is to create a new props.conf and copy the configs over, save and restart splunk.

0 Karma

AndreasMartenss
Explorer

Hi again,
Tried your suggestions, probably I am doing something wrong because I get no data collected. Here is what I have done:

  1. On both the Splunk server and the UF, created the file "props.conf" in C:\Program Files\SplunkUniversalForwarder\etc\system\local\ Contents of props.conf:

LINE_BREAKER = ([\r\n]+)\s?\

0 Karma

AndreasMartenss
Explorer

Seems my answer got cut off. Anyway, I put your suggestion for the props.conf and the inputs.conf in the files and restarted splunk. Still wont collect anything sadly.

0 Karma

sudosplunk
Motivator

Have a look at your input status to see if UF is reading the file, use this rest api http://<your_UF_hostname>:8089/services/admin/inputstatus.
Additionally, what does your outputs.conf say?

0 Karma

ddrillic
Ultra Champion

Maybe you'll describe step by step what you have done? It's pretty difficult to guess ; -)

AndreasMartenss
Explorer

Hi, thanks for the answer.
Ok,
In splunk - Add data
Forward data from Splunk forwarder - Selected my host and serverclass
Files & Directories - Enter the path to the file I want to index, using local path on the server that has the file.

next next done.

This works if I put in the path of a file with the extension .log, but if I enter a file with the extension .config, nothing shows up anywhere.

0 Karma
Get Updates on the Splunk Community!

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...