Getting Data In

What props.conf change do I need to prevent incorrect line breaking of XML data on embedded timestamps?

dolejh76
Communicator

I have not made any props.conf changes before, so looking for some help please / thanks.

I am pulling tomcat logs and Splunk is splitting up XML into multiple files because of the embedded times stamps within the XML. Can you tell me the best way to stop this? Time stamps within the XML should be ignored. Specifically in the example - ignore timestamp after <ClientDt> and <ServerDt>

Thanks for the help.

--- example snip of log ---

2015-08-07 11:54:13 [DEBUG] (com.somedomain.pbj.impl.dao.CommonBaseDao.java:195) - Engine URL: http://someserver:8080/WebServiceEngine/services/WSEngine/invoke
2015-08-07 11:54:13 [DEBUG] (com.somesomain.pbj.impl.dao.CommonBaseDao.java:174) - #### Response Xml ####
<?xml version="1.0" encoding="UTF-8"?><ACORD xmlns="http://www.some.org/standards/some/some/xml/somemore">
    <SignonRs>
        <Status>
            <StatusCd>0</StatusCd>
            <StatusDesc>Annonymous public user accepted.</StatusDesc>
        </Status>
        <CustId>
            <SPName>PBL</SPName>
            <CustLoginId>PUBLIC</CustLoginId>
        </CustId>
        <ClientDt>2015-08-07T11:54:13</ClientDt>
        <CustLangPref>ENG</CustLangPref>
        <ClientApp>
            <Org>internet</Org>
            <Name>COMML</Name>
            <Version>1.6</Version>
            <Env>qua</Env>
        </ClientApp>
        <ServerDt>2015-08-07T11:54:13</ServerDt>
        <Language>ENG</Language>
    </SignonRs>
    <PageRs>
        <RqUID>4DF230E0-1945-60D8-B80F-AE1BA2576488</RqUID>
        <TransactionResponseDt>2015-08-07T11:54:13</TransactionResponseDt>
        <MsgStatus>
            <MsgStatusCd>Success</MsgStatusCd>
        </MsgStatus>
        <CommlLocationGetRs>
            <MsgStatus>
                <MsgStatusCd>Success</MsgStatusCd>
            </MsgStatus>
            <Location>
                <ItemIdInfo>
                    <AgencyId>1</AgencyId>
                </ItemIdInfo>
                <SubLocation>
                    <ItemIdInfo>
                        <AgencyId>1</AgencyId>
                        <UnitNumber>1</UnitNumber>
                    </ItemIdInfo>
                    <SubLocationName>some location</SubLocationName>
                    <Addr>
                        <Addr1>some address</Addr1>
                        <City>some city</City>
                        <StateProvCd>some state</StateProvCd>
                        <PostalCode>some zip</PostalCode>
                    </Addr>
                    <com.somedomain>10252</com.somedomain>
                    <com.somedomain>1</com.somedomain>
                </SubLocation>
            </Location>
        </CommlLocationGetRs>
    </PageRs>
</some>
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Following configuration (on Indexer/Heavy forwarder) should just work fine.

[yourcustomsourcetype]
BREAK_ONLY_BEFORE=^\d{4}-\d{2}-\d{2}
MAX_TIMESTAMP_LOOKAHEAD=150
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%Y-%m-%d %H:%M:%S
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Are there any current settings in props.conf for this sourcetype? The default is to split events at line breaks, but that's not happening in your case. Perhaps someone else has made changes to the file?

---
If this reply helps you, Karma would be appreciated.
0 Karma

dolejh76
Communicator

no I am the only one that has touched this - I don't have any profs.conf yet - just the default and ones that are included in the deployed apps (splunk for *nix in this case)

0 Karma

woodcock
Esteemed Legend

Use this:

TIME_PREFIX = ^

This tells it to only use timestamps that are the very first thing in a line.

dolejh76
Communicator

Trying this - I will let you know how it goes - thanks

0 Karma

dolejh76
Communicator

I created /opt/splunkforwarder/etc/system/local/props.conf and added the 2 lines below. Java is my sourcetype for that index - so assumed that I needed it. This didn't work...

[java]
TIME_PREFIX=^

There is a props.conf in the Splunk_TA_nix app possible it is messing it up? - but it is pointing to index=os...

To expand a little...

We use the splunk_ta_nix app for os events, etc

We also have tomcat logs / debug going to /syslog/tomcat - so I added to inputs.conf

"add monitor /syslogs/tomcat/ -index java -sourcetype java"

0 Karma

woodcock
Esteemed Legend

This needs to go to all of your Indexers and all Splunk instances there need to be restarted before this will take effect.

0 Karma

dolejh76
Communicator

Indexer and forwarder service restarted - didn't help thanks

0 Karma

woodcock
Esteemed Legend

OK, let's back up all the way. What files have you created (include full path) and what servers have a copy? You should not have modified any files (you should create your own app-specific, minimalistic versions) but if you did, list the same thing for those, too. We have to be missing something basic because this absolutely should be working.

0 Karma

dolejh76
Communicator

typed response once but it doesn't look like it posted so hopefully this isn't a duplicate...

Indexer is splunk-svr
On this server via gui I added index "java"
On $SPLUNK_HOME/etc/system/local/props.conf - I add the following
[java]
TIME_PREFIX=^

on the app server - I added the UF
via the gui I used the following
opt/splunkforwarder/bin/splunk add monitor /syslog/tomcat -index java -sourcetype java

added /opt/splunkforwarder/etc/system/local/props.conf and added
[java]
TIME_PREFIX=^

You said this wouldn't work on app server but I could leave - I have not removed it.

That is all I had done and it was not working.

Splunk support said try this and it stopped the line breaks in the XML BUT also stopped it from breaking where it should - so this isn't working either - works to much...

on indexer
$SPLUNK_HOME/etc/system/local/props.conf
[java]
TIME_FORMAT=%Y-%m-%d %H:%M:%S
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=19
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}
NO_BINARY_CHECK=true

Below is a snip of a log entry that it did not break at all - this is just a snip - if goes on for about 50 lines. It is not breaking at the date in the middle of the XML, but it doesn't look good at all - a lot of breaks missing etc.

With these logs - there are two date formats at the beginning of the line that it should break on.

2015-08-13 09:58:30 and 08-13-2015 09:58:30


08-13-2015 09:58:30 DEBUG com.domain.web.service.engine.WSProvider - Thread 179 hit endpoint at Thu Aug 13 09:58:30 EDT 2015
08-13-2015 09:58:30 DEBUG com.domain.web.service.engine.WSProvider - Incoming request recieved for service: ACORD
08-13-2015 09:58:30 DEBUG com.domain.web.service.factories.request.RequestFactory - !!!!!!!!!!!!!!! Validating SAX Parser retrieved from stack!
08-13-2015 09:58:30 DEBUG com.domain.web.service.factories.request.RequestFactory - !!!!!!!!!!!!!!! Validating SAX Parser returned to stack!
08-13-2015 09:58:30 DEBUG com.domain.WebServiceBeans.beans.custom.token.ServiceToken - Adding request to token
08-13-2015 09:58:30 DEBUG com.domain.WebServiceBeans.beans.custom.token.ServiceToken - Adding to token ACORDRequestBean

0 Karma

woodcock
Esteemed Legend

Change this:

BREAK_ONLY_BEFORE=\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}

To this:

BREAK_ONLY_BEFORE=\d{2,4}-\d{2}-\d{2,4}\s+\d{2}:\d{2}:\d{2}
0 Karma

dolejh76
Communicator

Thansk for your help on this.

I have spunk-oma which is the indexer -
created index via gui - "java"
on $SPLUNK_HOME/etc/system/local/props.conf I added the following

[java]
TIME_PREFIX = ^

That is it for indexer.

On app server - UF was installed. From deployment server it pulls splunk_TA_nix and other configs to point to indexer.

From cli on this server I ran this
opt/splunkforwarder/bin/splunk add monitor /syslog/tomcat/ -index java -sourcetype java

That was all I did before I opened request for help. Since then I have tried the following -

added props.conf on UF to /opt/splunkforwarder/etc/system/local
[java]
TIME_PREFIX = ^

This didn't work but you said I could leave it so I have not removed.

Also tried on indexer $SPLUNK_HOME/etc/system/local/props.conf - didn't work as well
[java]
TIME_PREFIX = ^

Splunk support said use this on indexer $SPLUNK_HOME/etc/system/local/props.conf
[java]
TIME_FORMAT=%Y-%m-%d %H:%M:%S
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=19
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}
NO_BINARY_CHECK=true

I did this and it stopped breaking on the XML parts but also stopped breaking on other ones that it should be breaking on. That being said - I know that it is use this props.conf, but the above is to much.

Here is an example of what I am seeing now - this is a small part of ONE log entry that it did not break up. I have added **linebreak** where is should have broken


08-13-2015 09:13:20 DEBUG com.domain.web.service.audit.MessageLogger - MessageLogger: removed TaxId node!!! **linebreak **
08-13-2015 09:13:20 INFO com.domain.web.service.audit.MessageLogger - ********** Incoming XML for logging ********** **linebreak **
08-13-2015 09:13:20 INFO com.domain.web.service.audit.MessageLogger -

  <SignonTransport>
     <SignonRoleCd>Agent</SignonRoleCd>
     <CustId>
        <SPName>com.xml</SPName>
        <CustPermId>domain</CustPermId>
        <CustLoginId>domain</CustLoginId>
     </CustId>
  </SignonTransport>
  <ClientDt>2014-01-15T09:01:05</ClientDt>
  <CustLangPref>ENG</CustLangPref>
  <ClientApp>
     <Org>Internet</Org>
     <Name>com.mqm</Name>
     <Version>1.6</Version>
  </ClientApp>


  <RqUID>00000000-0000-0000-0000-000000000000</RqUID>
  <PersAutoPolicyQuoteInqRq>
     <RqUID>00000000-0000-0000-0000-000000000000</RqUID>
     <TransactionRequestDt>2015-01-08T09:30:20</TransactionRequestDt>
     <CurCd>USD</CurCd>
     <InsuredOrPrincipal>
        <GeneralPartyInfo>
0 Karma

woodcock
Esteemed Legend

Also, you should create a new props.conf and put it in the same directory as the new inputs.conf file you created for tomcat logs (just to keep things orderly).

0 Karma

dolejh76
Communicator

So props.conf should be on the indexer and not on the forwarder?

0 Karma

dolejh76
Communicator

still doing the same thing - is there a way to see what props.conf hit the log when it was indexed to find out why it is splitting them?>

It split them at the line ---- 2015-08-10T16:13:45 ---- seen below - thanks!!

2015-08-10 16:13:45 [DEBUG] (com.domain.pbj.impl.dao.CommonBaseDao.java:195) - Engine URL: http://domain:8080/WebServiceEngine/services/WSEngine/invoke
2015-08-10 16:13:46 [DEBUG] (com.domain.pbj.impl.dao.CommonBaseDao.java:174) - #### Response Xml ####
<SignonRs>
    <Status>
        <StatusCd>0</StatusCd>
        <StatusDesc>Annonymous public user accepted.</StatusDesc>
    </Status>
    <CustId>
        <SPName>PBL</SPName>
        <CustLoginId>PUBLIC</CustLoginId>
    </CustId>
    <ClientDt>2015-08-10T16:13:45</ClientDt>
    <CustLangPref>ENG</CustLangPref>
    <ClientApp>
        <Org>internet</Org>
        <Name>COMML</Name>
        <Version>1.6</Version>
        <Env>qua</Env>
    </ClientApp>
    <ServerDt>2015-08-10T16:13:46</ServerDt>
    <Language>ENG</Language>
</SignonRs>
0 Karma

woodcock
Esteemed Legend

There are 3 (or 4) events for this data sample when there should only be 2, right?
The second log was split into (at least) 2, The first one ended after ClientDt and a new one got created at that timestamp. Did it (incorrectly) break again after ServerDt, too?

0 Karma

dolejh76
Communicator

Yeah it breaks anywhere that there is a time stamps -

ClientDt

ServerDt

etc...

0 Karma

woodcock
Esteemed Legend

And you put the change on the indexers and you restarted the Splunk app instances on each Indexer, right? That's all there is too it, unless your events do not have sourcetype of java(note that Java is NOT the same as java)...???

0 Karma

dolejh76
Communicator

Yeah that was a type on the forum - used lowercase for all - thanks

0 Karma

dolejh76
Communicator

Did restart the services as well - didn't help thanks.

JD

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...