How to write a monitor stanza in inputs.conf to monitor a file in splunk ?

[Hemnaath]

Hi All, Can anyone guide us on how to create an input stanza to monitor a files through splunk. Need to monitor logs from application servers, that are running in both windows and Unix machine.

Logs to be monitored
Unix server:
/opt/IBM/middleware/user_projects/domains/Test/servers/TIM_server*/logs/TIM_server*.out*
/opt/IBM/middleware/user_projects/domains/Test/servers/TIM_server*/logs/TIM_server*-diag*.log

Windows server :
D:\ServerLog\ServerDaily-*.log

inputs.conf detail
[monitor:///opt/IBM/middleware/user_projects/domains/Test/servers/ITM_server*/logs/.*]
index=app
sourcetype=IBM:AUT:TAM
disabled = 0

inputs.conf detail for windows machine
[monitor://D:\ServerLog\ServerDaily-*.log]
index=app
sourcetype=IBM:AUT:TAM
disabled = 0

Kindly guide me whether the above stanza are defined correctly to monitor the required logs from UNIX server & windows server. If not, guide me with the correct stanza to be configured and also can we configure both windows/ UNIX monitor stanza in a single inputs.conf file.

Thanks in advance

[dineshraj9 · Accepted solution]

Hi Hemnaath,

This should work -

[monitor:///opt/IBM/middleware/user_projects/domains/Test/servers/ITM_server*/logs/*(.out|.log)*]
index=app
sourcetype=IBM:AUT:TAM
blacklist = (\.(tar|gz|bz2|tar.gz|tgz|tbz|tbz2|zip|z)$)

[Hemnaath]

thanks dineshraj, but can you please explain us how it works? and also can we configure Windows and Unix stanza in same inputs.conf files.

thanks in advance

[dineshraj9]

I would suggest create 2 seperate inputs file for Unix and Windows servers and have 2 set of stanzas in serverclass.conf(one for Unix and one for Windows). We don't want Splunk to monitor windows path on Unix servers or vice-versa.

The monitor path supports wildcard as well as regular expression. So here you are reading any log file name that contains ".out" or ".log" in it and in blacklist you are filtering out files with certain extensions.

More on inputs.conf here - http://docs.splunk.com/Documentation/Splunk/6.6.0/Admin/Inputsconf

[Hemnaath]

thanks dineshraj, its much needed help. This is the first time I got a request to monitor the set of files. Similarly we have to monitor the below logs detail in splunk for the same severs. Can I configure the stanza like you had mentioned in above comments in the same inputs.conf stanza.

Log details to be monitored :
/opt/IBM/middleware/user_projects/domains/Test/servers/TAM_server*/logs/TAM_server*.out*

/opt/IBM/middleware/user_projects/domains/Test/servers/TAM_server*/logs/TAM_server*-diag*.log

/opt/IBM/middleware/user_projects/domains/Test/servers/cl_server*/logs/cl_server*.out*

/opt/IBM/middleware/user_projects/domains/Test/servers/cl_server*/logs/cl_server*-diag*.log

Inputs.conf stanza

[monitor:///opt/IBM/middleware/user_projects/domains/Test/servers/TAM_server*/logs/(.out|.log)]
index=app
sourcetype=IBM:AUT:TAM
blacklist = (.(tar|gz|bz2|tar.gz|tgz|tbz|tbz2|zip|z)$)

[monitor:///opt/IBM/middleware/user_projects/domains/Test/servers/cl_server*/logs/(.out|.log)]
index=app
sourcetype=IBM:AUT:TAM
blacklist = (.(tar|gz|bz2|tar.gz|tgz|tbz|tbz2|zip|z)$)

But it is necessary to configure the blacklist stanza ?.

thanks in advance

[dineshraj9]

Blacklists are not mandatory, but when using wildcards will help you filter unwanted data.

The monitors look good. Just ensure that no change to sourcetype is required for the new set of logs from cl_server.

[Hemnaath]

thanks dineshraj, regarding the two set of serverclass can i define like this
For unix :
[serverClass:Test-TAM]
whitelist.0 = testtam*

[serverClass:Test-TAM:app:Test-TAM]
restartSplunkWeb = 0
restartSplunkd = 1
stateOnClient = enabled

For windows:
[serverClass:Test-TAM2]
whitelist.0 = testtIM*

[serverClass:Test-TAM2:app:Test-TAM2]
restartSplunkWeb = 0
restartSplunkd = 1
stateOnClient = enabled

thanks in advance.

[dineshraj9]

Yes, this looks fine!!

[Hemnaath]

Hi Dineshraj, After configuring /pushing the above stanza from DP to the remote systems we could see the data getting into splunk and unable to perform the search. But currently we face another issue, data pulled from the remote machine contain some large list of unwanted URL's followed by at.

Details :

at java.lang.reflect.Method.invoke(Method.java:606)
at IBM.idm.common.login.SignInBean.handleWeblogicAuthn(SignInBean.java:133)
at IBM.idm.common.login.SignInBean.doLogin(SignInBean.java:99)
at sun.reflect.GeneratedMethodAccessor5210.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.el.parser.AstValue.invoke(AstValue.java:187)
at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:297)

Kindly guide me in how to remove this URL from the logs.

[Hemnaath]

Hi Dineshraj, Could you please guide me on this issue, needs to remove above details from the events.

thanks in advance.

[dineshraj9]

These look like error logs and stack trace. If you don't want these events, add blacklist to filter out these logs or specify exact logs you want in monitor stanza.

Like if you want only info logs then specify this way -

[monitor:///opt/IBM/middleware/user_projects/domains/Test/servers/cl_server*/logs/(info.out|info.log)]

Or add in blacklist the logs you want to ignore -

blacklist = ((.+error\.log.+|.+systemerr\.out.+)|\.(tar|gz|bz2|tar.gz|tgz|tbz|tbz2|zip|z)$)

[Hemnaath]

thanks dineshraj for your timely help on this, but actually we need the events but not the content starting with "at" from the events.

Details of events

5/16/17
8:57:04.674 AM

[2017-05-16T08:57:04.674-04:00] [TIM_server1] [ERROR] [] [db2.tam.platform.kernel.impl] [tid: [ACTIVE].ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: xxx_TIMPASSSYNC] [ecid: d87cb14ef99e9513:-133033ab:15c0857bfd3:-8000-000000000005cb77,0] [APP: TIM#11.1.2.0.0] [J2EE_APP.name: spml-dsml] [J2EE_MODULE.name: spmlws] [WEBSERVICE.name: TIMProvisioning] [WEBSERVICE_PORT.name: TIMProvisioningPort] Kernel Information: {0}[[
db2.tam.platform.kernel.ValidationFailedException: IAM-3030006:The following password policy rules were not met:Password must not be one of 8 previous passwords.
:
at db2.tam.passwordmgmt.eventhandlers.UserPasswordValidationHandler.validate(UserPasswordValidationHandler.java:96)
at db2.tam.platform.kernel.impl.TIMEvent.executeHandlers(TIMEvent.java:204)
at db2.tam.platform.kernel.impl.MonitoredTIMEvent.invokeExecuteHandler(MonitoredTIMEvent.java:99)
at db2.tam.platform.kernel.impl.MonitoredTIMEvent.executeHandlers(MonitoredTIMEvent.java:69)
at db2.tam.platform.kernel.impl.TIMEvent.execute(TIMEvent.java:157)
at db2.tam.platform.kernel.impl.ProcessImpl.executeStage(ProcessImpl.java:223)
at db2.tam.platform.kernel.impl.TIMProcess.doStageExecution(TIMProcess.java:38)
at db2.tam.platform.kernel.impl.ProcessImpl.execute(ProcessImpl.java:182)
at db2.tam.platform.kernel.impl.MonitoredTIMProcess.execute(MonitoredTIMProcess.java:33)
at db2.tam.platform.kernel.impl.Utils.manageSyncProcessing(Utils.java:73)

Kindly guide me on this please.

[Hemnaath]

Hi Dineshraj, can you guide me on how to remove a particular values in the event.

thanks in advance.

[dineshraj9]

You need configure your line breaking properly, so that Splunk doesn't detect each line as an event and detects the whole event.

Example sourcetype in props.conf -

[IBM:AUT:TAM]
TIME_PREFIX =^\[
MAX_TIMESTAMP_LOOKAHEAD = 29
TIME_FORMAT = %FT%T.%3N
LINE_BREAKER = ([\n\r]+)(?=\[\d{4}(\-\d{1,2}){2})
SHOULD_LINEMERGE = False

Please accept an answer once you have a solution and upvote any other solution that helps.

[Hemnaath]

Hi Dineshraj, Good Evening , we are still seeing the huge log details getting into splunk.
Please guide us on how to remove the word starting with "at" from the events list.

Recent Log details :

[2017-05-24T09:21:31.473-04:00] [itm_server1] [ERROR] [] [com.xxxxx.tam.itm.plugins.eventhandlers] [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: xxxxx] [ecid: ce05a10ae6311cf9:15640c05:15c2c928226:-8000-00000000000b19be,1:123289:21] [APP: itm#11.1.2.0.0] [J2EE_APP.name: itm_11.1.2.0.0] [J2EE_MODULE.name: workflowservice] [WEBSERVICE.name: CallbackService] [WEBSERVICE_PORT.name: CallbackServicePort] Exception Occurred.[[
ibm.tam.identity.exception.AccessDeniedException: tam-3054101:The logged-in user itminternal does not have viewSearchEntity permission on Role xxxxx Inactive itm
Users entity.:itminternal:viewSearchEntity:Role:xxxxx Inactive itm Users
at ibm.tam.identity.rolemgmt.impl.RoleManagerCommon.hasAccess(RoleManagerCommon.java:401)
at ibm.tam.identity.rolemgmt.impl.RoleManagerCommon.hasAccess(RoleManagerCommon.java:251)
at ibm.tam.identity.rolemgmt.impl.RoleManagerImpl.getDetails(RoleManagerImpl.java:531)
at ibm.tam.identity.rolemgmt.impl.RoleManagerImpl.getDetails(RoleManagerImpl.java:492)
at sun.reflect.GeneratedMethodAccessor4906.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)

[Hemnaath]

Hi Dineshraj, Good Morning. we have added props.conf file with the below mentioned stanza to remove the word "at" and the blank space from the event. Added customized app with the props.conf stanza and pushed to the indexer instances for parsing the events at index level.

Props.conf details :
[ibm:auth:identitymanagement]
SEDCMD-removeat =s/at .+//g

Though it worked on most of the events with the above criteria were removed, but still there are some more events which are having blank space into the events .
Could please guide me on how to get ride of the white space as show in the events below.

Log details.

5/25/17
1:57:49.522 AM

[2017-05-25T01:57:49.522-04:00] [itm_server1] [ERROR] [] [ibm.tam.platform.kernel.impl] [tid: [ACTIVE].ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: xxxxx] [ecid: 061bf930d1319f9b:-30072ed4:15c2c8bbbd3:-8000-00000000000bb488,0] [APP: itm#11.1.2.0.0] [J2EE_APP.name: spml-dsml] [J2EE_MODULE.name: spmlws] [WEBSERVICE.name: itmProvisioning] [WEBSERVICE_PORT.name: itmProvisioningPort] Kernel Information: {0}[[
ibm.tam.platform.kernel.ValidationFailedException: IAM-3030006:The following password policy rules were not met:Password must not be one of 8 previous passwords.
:

Caused by: ibm.tam.passwordmgmt.exception.InvalidPasswordException

thanks in advance

[dineshraj9]

Using this kind of regular expression can cause important data to be missed, as it will start with the first match for the text "at" and start removing the rest. Like below starting from "at" in the word platform :

[2017-05-25T01:57:49.522-04:00] [itm_server1] [ERROR] [] [ibm.tam.platform.kernel.impl] [tid

Try this it might help -

SEDCMD-removeat =s/[\r\n]+at\s+(.*[\r\n]*)*//g

As requested earlier, please accept an answer once you have a solution and upvote any other solution that helps.

[Hemnaath]

Hi Dineshraj Good Evening, thanks for your effort on this, i have implemented the below stanza and it removed the blank space from the events.

[ibm:auth:identitymanagement]
SEDCMD-removeat =s/at .+//g
SEDCMD-removespaces = s/\s+\s+[\r\n]//g