broker dead block. Multicast Discovery Notifier thread doesn't stop
I want to simulate different channels by implementing connections and network of brokers in an unusual way.
I have 2 processes A and B on the same server. Each have their own embedded AMQ broker. B is a supervisor process which echanges signal messages.
A is connected on its broker (conn1), and A is optionnaly connected on the B's broker (conn2), if a previous message on conn1 (answer to a previous broadcast), tells him that B is on.
A's broker and B's broker are in the same network of brokers (networkConnector is instance-nc, associated with multicast protocol). Each has the same networkConnector "instance-nc" multicast configuration
A and B brokers have a transportConnector which is associated with instance-nc, multicast protocol.
A connector : port 61616
B connector : port 61600
B has an other connector 61602
If A is started without B, and A is gracefully stop. All is right.
I have a problem when
1 : B (supervisor) is started (B is connected on its broker : vm transport)
2 : A is started (A is connected on its broker : vm transport)
3 : network connections are automatically established by ActiveMQ (61616 and 61600 ports) for instance-nc
4 : conn2 (A->B:61602) is established by my program (tcp transport)
5 : B is gracefully killed (B internal connection is gracefully closed, and AMQ embedded broker is stopped)
6 : on A, ExceptionListener is aware that A->B:61602 is broken.
7 : Then, myprogram on A, I wish to leave
7-1 : conn1 is gracefully stopped and close
7-2 : internal broker is gracefully stopped
In this situation, on A, the instance-nc multicast Discovery thread and scheduler, doesn't want to stop. (see thread dump)
With trace on, I see that conn2 (A->B:61602) is closed when B is killed. I see that "instance-nc" networkConnector is closed when A's broker is stopped by my program
For your information, in the trace, you will see that there is a third connection (61620) towards a distinct AMQ stand-alone broker which is active and not killed during the operation. I think all is right with this "passive" connection.
In the trace, you will see at the end that I tried to close the broken connection and that "close" generates the last exception. But the thread stay active although embedded broker is down
Then, if I interrupt the blocked process (CTRL-C), the 2 AMQ threads die very quickly, and my program leaves.
Version : AMQ 5.3-snapshot
I joined the trace of A internal embedded Broker and the thread dump at the end of A process.