I have not made any props.conf changes before, so looking for some help please / thanks.
I am pulling tomcat logs and Splunk is splitting up XML into multiple files because of the embedded times stamps within the XML. Can you tell me the best way to stop this? Time stamps within the XML should be ignored. Specifically in the example - ignore timestamp after <ClientDt> and <ServerDt>
Thanks for the help.
--- example snip of log ---
2015-08-07 11:54:13 [DEBUG] (com.somedomain.pbj.impl.dao.CommonBaseDao.java:195) - Engine URL: http://someserver:8080/WebServiceEngine/services/WSEngine/invoke
2015-08-07 11:54:13 [DEBUG] (com.somesomain.pbj.impl.dao.CommonBaseDao.java:174) - #### Response Xml ####
<?xml version="1.0" encoding="UTF-8"?><ACORD xmlns="http://www.some.org/standards/some/some/xml/somemore">
<SignonRs>
<Status>
<StatusCd>0</StatusCd>
<StatusDesc>Annonymous public user accepted.</StatusDesc>
</Status>
<CustId>
<SPName>PBL</SPName>
<CustLoginId>PUBLIC</CustLoginId>
</CustId>
<ClientDt>2015-08-07T11:54:13</ClientDt>
<CustLangPref>ENG</CustLangPref>
<ClientApp>
<Org>internet</Org>
<Name>COMML</Name>
<Version>1.6</Version>
<Env>qua</Env>
</ClientApp>
<ServerDt>2015-08-07T11:54:13</ServerDt>
<Language>ENG</Language>
</SignonRs>
<PageRs>
<RqUID>4DF230E0-1945-60D8-B80F-AE1BA2576488</RqUID>
<TransactionResponseDt>2015-08-07T11:54:13</TransactionResponseDt>
<MsgStatus>
<MsgStatusCd>Success</MsgStatusCd>
</MsgStatus>
<CommlLocationGetRs>
<MsgStatus>
<MsgStatusCd>Success</MsgStatusCd>
</MsgStatus>
<Location>
<ItemIdInfo>
<AgencyId>1</AgencyId>
</ItemIdInfo>
<SubLocation>
<ItemIdInfo>
<AgencyId>1</AgencyId>
<UnitNumber>1</UnitNumber>
</ItemIdInfo>
<SubLocationName>some location</SubLocationName>
<Addr>
<Addr1>some address</Addr1>
<City>some city</City>
<StateProvCd>some state</StateProvCd>
<PostalCode>some zip</PostalCode>
</Addr>
<com.somedomain>10252</com.somedomain>
<com.somedomain>1</com.somedomain>
</SubLocation>
</Location>
</CommlLocationGetRs>
</PageRs>
</some>
Following configuration (on Indexer/Heavy forwarder) should just work fine.
[yourcustomsourcetype]
BREAK_ONLY_BEFORE=^\d{4}-\d{2}-\d{2}
MAX_TIMESTAMP_LOOKAHEAD=150
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%Y-%m-%d %H:%M:%S
Are there any current settings in props.conf for this sourcetype? The default is to split events at line breaks, but that's not happening in your case. Perhaps someone else has made changes to the file?
no I am the only one that has touched this - I don't have any profs.conf yet - just the default and ones that are included in the deployed apps (splunk for *nix in this case)
Use this:
TIME_PREFIX = ^
This tells it to only use timestamps that are the very first thing in a line.
Trying this - I will let you know how it goes - thanks
I created /opt/splunkforwarder/etc/system/local/props.conf and added the 2 lines below. Java is my sourcetype for that index - so assumed that I needed it. This didn't work...
[java]
TIME_PREFIX=^
There is a props.conf in the Splunk_TA_nix app possible it is messing it up? - but it is pointing to index=os...
To expand a little...
We use the splunk_ta_nix app for os events, etc
We also have tomcat logs / debug going to /syslog/tomcat - so I added to inputs.conf
"add monitor /syslogs/tomcat/ -index java -sourcetype java"
This needs to go to all of your Indexers and all Splunk instances there need to be restarted before this will take effect.
Indexer and forwarder service restarted - didn't help thanks
OK, let's back up all the way. What files have you created (include full path) and what servers have a copy? You should not have modified any files (you should create your own app-specific, minimalistic versions) but if you did, list the same thing for those, too. We have to be missing something basic because this absolutely should be working.
typed response once but it doesn't look like it posted so hopefully this isn't a duplicate...
Indexer is splunk-svr
On this server via gui I added index "java"
On $SPLUNK_HOME/etc/system/local/props.conf - I add the following
[java]
TIME_PREFIX=^
on the app server - I added the UF
via the gui I used the following
opt/splunkforwarder/bin/splunk add monitor /syslog/tomcat -index java -sourcetype java
added /opt/splunkforwarder/etc/system/local/props.conf and added
[java]
TIME_PREFIX=^
You said this wouldn't work on app server but I could leave - I have not removed it.
That is all I had done and it was not working.
Splunk support said try this and it stopped the line breaks in the XML BUT also stopped it from breaking where it should - so this isn't working either - works to much...
on indexer
$SPLUNK_HOME/etc/system/local/props.conf
[java]
TIME_FORMAT=%Y-%m-%d %H:%M:%S
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=19
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}
NO_BINARY_CHECK=true
Below is a snip of a log entry that it did not break at all - this is just a snip - if goes on for about 50 lines. It is not breaking at the date in the middle of the XML, but it doesn't look good at all - a lot of breaks missing etc.
With these logs - there are two date formats at the beginning of the line that it should break on.
2015-08-13 09:58:30 and 08-13-2015 09:58:30
08-13-2015 09:58:30 DEBUG com.domain.web.service.engine.WSProvider - Thread 179 hit endpoint at Thu Aug 13 09:58:30 EDT 2015
08-13-2015 09:58:30 DEBUG com.domain.web.service.engine.WSProvider - Incoming request recieved for service: ACORD
08-13-2015 09:58:30 DEBUG com.domain.web.service.factories.request.RequestFactory - !!!!!!!!!!!!!!! Validating SAX Parser retrieved from stack!
08-13-2015 09:58:30 DEBUG com.domain.web.service.factories.request.RequestFactory - !!!!!!!!!!!!!!! Validating SAX Parser returned to stack!
08-13-2015 09:58:30 DEBUG com.domain.WebServiceBeans.beans.custom.token.ServiceToken - Adding request to token
08-13-2015 09:58:30 DEBUG com.domain.WebServiceBeans.beans.custom.token.ServiceToken - Adding to token ACORDRequestBean
Change this:
BREAK_ONLY_BEFORE=\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}
To this:
BREAK_ONLY_BEFORE=\d{2,4}-\d{2}-\d{2,4}\s+\d{2}:\d{2}:\d{2}
Thansk for your help on this.
I have spunk-oma which is the indexer -
created index via gui - "java"
on $SPLUNK_HOME/etc/system/local/props.conf I added the following
[java]
TIME_PREFIX = ^
That is it for indexer.
On app server - UF was installed. From deployment server it pulls splunk_TA_nix and other configs to point to indexer.
From cli on this server I ran this
opt/splunkforwarder/bin/splunk add monitor /syslog/tomcat/ -index java -sourcetype java
That was all I did before I opened request for help. Since then I have tried the following -
added props.conf on UF to /opt/splunkforwarder/etc/system/local
[java]
TIME_PREFIX = ^
This didn't work but you said I could leave it so I have not removed.
Also tried on indexer $SPLUNK_HOME/etc/system/local/props.conf - didn't work as well
[java]
TIME_PREFIX = ^
Splunk support said use this on indexer $SPLUNK_HOME/etc/system/local/props.conf
[java]
TIME_FORMAT=%Y-%m-%d %H:%M:%S
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=19
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}
NO_BINARY_CHECK=true
I did this and it stopped breaking on the XML parts but also stopped breaking on other ones that it should be breaking on. That being said - I know that it is use this props.conf, but the above is to much.
Here is an example of what I am seeing now - this is a small part of ONE log entry that it did not break up. I have added **linebreak** where is should have broken
08-13-2015 09:13:20 DEBUG com.domain.web.service.audit.MessageLogger - MessageLogger: removed TaxId node!!! **linebreak **
08-13-2015 09:13:20 INFO com.domain.web.service.audit.MessageLogger - ********** Incoming XML for logging ********** **linebreak **
08-13-2015 09:13:20 INFO com.domain.web.service.audit.MessageLogger -
<SignonTransport>
<SignonRoleCd>Agent</SignonRoleCd>
<CustId>
<SPName>com.xml</SPName>
<CustPermId>domain</CustPermId>
<CustLoginId>domain</CustLoginId>
</CustId>
</SignonTransport>
<ClientDt>2014-01-15T09:01:05</ClientDt>
<CustLangPref>ENG</CustLangPref>
<ClientApp>
<Org>Internet</Org>
<Name>com.mqm</Name>
<Version>1.6</Version>
</ClientApp>
<RqUID>00000000-0000-0000-0000-000000000000</RqUID>
<PersAutoPolicyQuoteInqRq>
<RqUID>00000000-0000-0000-0000-000000000000</RqUID>
<TransactionRequestDt>2015-01-08T09:30:20</TransactionRequestDt>
<CurCd>USD</CurCd>
<InsuredOrPrincipal>
<GeneralPartyInfo>
Also, you should create a new props.conf
and put it in the same directory as the new inputs.conf
file you created for tomcat logs (just to keep things orderly).
So props.conf should be on the indexer and not on the forwarder?
still doing the same thing - is there a way to see what props.conf hit the log when it was indexed to find out why it is splitting them?>
It split them at the line ---- 2015-08-10T16:13:45 ---- seen below - thanks!!
2015-08-10 16:13:45 [DEBUG] (com.domain.pbj.impl.dao.CommonBaseDao.java:195) - Engine URL: http://domain:8080/WebServiceEngine/services/WSEngine/invoke
2015-08-10 16:13:46 [DEBUG] (com.domain.pbj.impl.dao.CommonBaseDao.java:174) - #### Response Xml ####
<SignonRs>
<Status>
<StatusCd>0</StatusCd>
<StatusDesc>Annonymous public user accepted.</StatusDesc>
</Status>
<CustId>
<SPName>PBL</SPName>
<CustLoginId>PUBLIC</CustLoginId>
</CustId>
<ClientDt>2015-08-10T16:13:45</ClientDt>
<CustLangPref>ENG</CustLangPref>
<ClientApp>
<Org>internet</Org>
<Name>COMML</Name>
<Version>1.6</Version>
<Env>qua</Env>
</ClientApp>
<ServerDt>2015-08-10T16:13:46</ServerDt>
<Language>ENG</Language>
</SignonRs>
There are 3 (or 4) events for this data sample when there should only be 2, right?
The second log was split into (at least) 2, The first one ended after ClientDt
and a new one got created at that timestamp. Did it (incorrectly) break again after ServerDt
, too?
Yeah it breaks anywhere that there is a time stamps -
ClientDt
ServerDt
etc...
And you put the change on the indexers and you restarted the Splunk app instances on each Indexer, right? That's all there is too it, unless your events do not have sourcetype
of java
(note that Java
is NOT the same as java
)...???
Yeah that was a type on the forum - used lowercase for all - thanks
Did restart the services as well - didn't help thanks.
JD