After stopping the forwarder on an AIX server, why are we not retrieving events from files after starting it up again?

[chriselst]

I had a forwarder on an AIX server sending a number of log files to my Splunk Indexer and all was working well.

Then debugging got turned on on the application producing the log files. My Splunk license got blown out of the Window so I had to stop the forwarder.

Since then, whenever I turn on the forwarder again Splunk, only creates an event for the first (multi) line in the logfiles, giving it a timestamp of the system time as there is no date or time against the first line in the logs. Also, it creates an event if the logfile rolls over, again taking the first line in it.

The log files are Maximo WebSphere UI logs.

The event that is being recorded is like this;

************ Start Display Current Environment ************

WebSphere Platform 6.1 [ND 6.1.0.47 cf471333.02]  running with process name ctgCell01\ctgNode01\XXXXXXXServer and process id 426116
Detailed IFix information: Please use the versionInfo command to view this information
Host Operating System is AIX, version 5.3
Java version = 1.5.0, Java Compiler = NONE, Java VM name = IBM J9 VM
was.install.root = /hostname/IBM/WebSphere/AppServer
user.install.root = /hostname/IBM/WebSphere/AppServer/profiles/ctgAppSrv01
Java Home = /hostname/IBM/WebSphere/AppServer/java/jre
ws.ext.dirs = /hostname/IBM/WebSphere/AppServer/java/lib:/hostname/IBM/WebSphere/AppServer/profiles/ctgAppSrv01/classes:/hostname/IBM/WebSphere/AppServer/classes:/hostname/IBM/WebSphere/AppServer/lib:/hostname/IBM/WebSphere/AppServer/installedChannels:/hostname/IBM/WebSphere/AppServer/lib/ext:/hostname/IBM/WebSphere/AppServer/web/help:/hostname/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
Classpath = /hostname/IBM/WebSphere/AppServer/profiles/ctgAppSrv01/properties:/hostname/IBM/WebSphere/AppServer/properties:/hostname/IBM/WebSphere/AppServer/lib/startup.jar:/hostname/IBM/WebSphere/AppServer/lib/bootstrap.jar:/hostname/IBM/WebSphere/AppServer/lib/j2ee.jar:/hostname/IBM/WebSphere/AppServer/lib/lmproxy.jar:/hostname/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/hostname/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/hostname/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/hostname/IBM/WebSphere/AppServer/java/lib/tools.jar
Java Library path = /hostname/IBM/WebSphere/AppServer/java/jre/bin:/hostname/IBM/WebSphere/AppServer/java/jre/bin:/hostname/IBM/WebSphere/AppServer/java/jre/bin/classic:/hostname/IBM/WebSphere/AppServer/java/jre/bin:/hostname/IBM/WebSphere/AppServer/bin:/hostname/IBM/WebSphere/AppServer/java/jre/bin/j9vm:/hostname/IBM/WebSphere/AppServer/java/jre/bin/j9vm:/hostname/IBM/WebSphere/AppServer/java/jre/bin//headless:/hostname/IBM/WebSphere/AppServer/java/jre/bin/j9vm:/usr/lib:/hostname/IBM/WebSphere/AppServer/lib/WMQ/java/lib

************* End Display Current Environment *************

Subsequent lines are like this, but not appearing in Splunk;

[13/11/15 08:24:01:218 GMT] 0000002e SystemOut     O 13 Nov 2015 08:24:01:218 [INFO] BMXAA6370I - Total number of users connected to the system: 0
[13/11/15 08:24:01:219 GMT] 0000002e SystemOut     O 13 Nov 2015 08:24:01:219 [INFO] BMXAA7019I - The total memory is 2147483648 and the memory available is 1897199760.

All I did was stop the forwarder, then restart it again a few days later.

[chriselst · Accepted solution]

I started the forwarder again today and it all seems to be working as it should. Infuriating, but as it is working there is no need for this to still be open, is there any way for me to close it or mark it in some way?