Well today I learned a valuable lesson in Java. What ever you do, NEVER and I mean NEVER close the System.out OutputStream. Especially if you have a JNI program that prints to the console!
What seems to happen is that when you close the output stream it closes the stdout file descriptor. If you have JNI code that is writing to stdout it will start overwriting random bits of memory inside the JVM… As you might imagine this will cause problems ;-)
In my case it was inserting the output of my printf() messages into random tcp packets. After several hours of debugging why my network messages were invalid after running the system for a while I finally tracked down the problem by using ethereal and enabling all the JVM logging that I could. What I noticed was that about the time everything started failing the JVM loaded the OutputStream.close() method. With this clue I was able to track down the code that was causing the problems. As it turned out deep inside one of the logging modules it was calling close() on the debug streams inside the class finalizer. The only problem is that when running in test mode the default log destination was… System.out!
I hope this knowledge will save at least one other person from the hours of painful debugging that this beast caused me :-)