Fwd: Apache.NMS and Apache.NMS.XMS

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Fwd: Apache.NMS and Apache.NMS.XMS

sramet
Transfered as a public archive for transparency.

Refers to JIRA issue AMQNET-185.

---------- Forwarded message ----------
From: Stéphane Ramet <[hidden email]>
Date: Sat, Dec 19, 2015 at 1:51 AM
Subject: Re: Apache.NMS and Apache.NMS.XMS
To: Jim Gomes <[hidden email]>


Hi Jim,

busy period indeed.

As I found it difficult to get a fair estimate of the amount of new code, I thought joining the code to my reply would be the easiest way for you to get an appreciation.

I attached 4 files :
1/ "2015-12-18_Apache.NMS.XMS.trunk.src.main.7z" is the main XMS provider source code.
2/ "2015-12-18_Apache.NMS.XMS.trunk.src.test.7z" is the unit test source code.
3/ "2015-12-18_Apache.EMS+MSMQ.QueueBrowser.7z" are patches to the EMS and MSMQ providers for queue browsing capabilities.
4/ "2015-12-18_Apache.NMS.test.evidence.7z" is not really code but the current state of test results, both NUnit and additional tests.

Getting into more details :

1/ Main source code
Most of the efforts in the main code went into the initialization of the ConnectionFactory properties from the specified broker URI ("XMSEnum.cs", "XMSConvert.cs", "ConnectionFactory.cs"), plus support for administered object ("InitialContext.cs").
To document the URI attributes, and decouple their names from the object properties, I introduced an "UriAttribute" annotation and rewrote the "IntrospectionSupport.cs" to support that.

2/ Unit test code
The unit test part would probably need more in-depth discussions. I wanted to reuse most of the NMS unit test code, since this covers a lot of standard use cases. But I didn't manage to use it as is, for various reasons:

- in the NMS test code, most destinations get a random / guid-based name, which implies that the provider should be able to dynamically create destinations. It is not the case with XMS (neither the MSMQ provider as far as I could see, and probably some others since JMS doesn't support it). Therefore I needed a way to specify to the core NMS nunit code the queues and topics it needed to address. I decided to pass references to nodes in the configuration file. It still defaults to guid-based name generation if a null reference is passed.

- some patterns in the NMS test code are not supported in XMS. For instance, XMS does not support changing the ClientId after the Connection is created; the ClientId must be set in the Factory beforehand. Since Connection creation is done in the NMSTestSupport, which is the root for all test modules, the logic cannot be easily changed from one provider to the other. I decided to implement some kind of bridge pattern : I introduced an NMSTest root, which forwards method calls to the "external", overridable NMSTestSupport.

- to more easily adapt to my test environment, I introduced an environment variable pointing to my specific "xmsprovider-test.config" file. I removed the ReplaceEnvVar parts since I thought they were not needed any more, but they could still easily be reintroduced if needed.

- I had to adapt some parts of the NMS unit test code to cope with another limitation in XMS : it doesn't support both synchronous and asynchronous operations in the same session. Therefore, in test cases where messages are sent synchronously and received asynchronously, I introduced a second session for consuming messages. I believe this is still representative of real world implementations, where you would have separate sessions for producing and consuming messages.

The final architecture is schematically :

Apache.NMS.XMS.Test.*Test
  --(inherits from)-->
    Apache.NMS.Test.*Test
      --(inherits from)-->
        Apache.NMS.Test.NMSTest
           --(invokes)-->
             Apache.NMS.Test.NMSTestSupport
                --(where required, overriden by)-->
                  Apache.XMS.Test.XMSTestSupport

The Apache.NMS.XMS.Test.*Test is a "thin" object responsible for :
- instanciating the XMSTestSupport object and passing it upwards to the NMSTest through the constructors chain.
- invoking the relevant Apache.NMS.Test.*Test methods, passing queue or topic references needed for running the test cases.

I hope I didn't go too far with this one. As I mentioned earlier, I'm new to Open Source contribution. But as I understand it now, this would usually be debated through discussion streams beforehand.

Plus the XMS unit test code now includes potentially standard / not provider-specific code, which is not ideal.

I'll let you tell me how you would like this part to be processed.

3/ QueueBrowser
Nothing special to mention.

4/ Current test status
I couldn't get a hand on a running publish/subscribe service in my customer's environment. Therefore I couldn't write and run ad-hoc tests for topics as planned. And many NMS unit tests, as they rely on topics, fail as well.
But the code did pass the planned additional tests (queue browsing and message selection, creation of connection factory from LDAP-administered definitions), plus many unit tests.
It proved to be usable in basic and some more advanced scenarios.
Therefore I would consider it to be in beta / advanced beta state.


I'll let you get back to me regarding whether a code grant is needed, and whether it would be safe to keep the excerpts from IBM documentation.

I'll also let you tell me how I should "officially" provide the source package. Should I create a JIRA issue and attach the code as suggested in the "Submitting patches" section of the "Contributing" page ?

Best regards,

Stéphane

On Wed, Dec 16, 2015 at 5:08 AM, Jim Gomes <[hidden email]> wrote:
Hi Stephane,

My apologies for the delay in replying. It's a busy time of year for me.

I think basing it on the EMS code was a good idea. I'm not sure where the line is when a regular license assignment is sufficient, or when a code grant is required.  It's a judgment call, so I think I'll ask around as to what the best way to proceed would be.  When I originally wrote the EMS provider implementation, it was enough new code that they wanted a code grant.  Since you based your work on that, would you estimate how many lines of new code there is?  Since you developed this on your own time, and were careful to keep it separate, that should make things much simpler.

Regarding the IBM documentation reference, I'm not sure if it would need to be reworked, or whether it would fall under the Fair Use copyright exclusion.  I think the Apache legal department could help out with that.

I think it's OK to have the code for the 3 types of messaging servers included, even if you can't test them all.  There might be others who can take what you did and test it out and submit reports back.  The code doesn't have to be 100% functional.  This is a great start.

You suggested timeline for features and work sounds great.  I think we can get the base functionality imported and then moving forward as you have time.

Thanks again for your willingness to contribute!

Best,
Jim



On Wed, Dec 9, 2015 at 12:54 PM, Stéphane Ramet <[hidden email]> wrote:
Hi Jim,

Thank you for your reply.

Mainly, this is the amount of code I developed, based on and adapted from the EMS provider code :

.:
-rwxrwxrwa    4102 Nov 29 02:02 BytesMessage.cs
-rwxrwxrwa    1346 Dec  5 15:59 CommonAssemblyInfo.cs
-rwxrwxrwa   11859 Nov 30 18:16 Connection.cs
-rwxrwxrwa   51947 Nov 30 23:03 ConnectionFactory.cs
-rwxrwxrwa    2828 Nov 28 18:08 ConnectionMetaData.cs
-rwxrwxrwa   17262 Nov 29 23:32 Destination.cs
-rwxrwxrwa    6262 Nov 30 13:41 InitialContext.cs
-rwxrwxrwa   15370 Nov 30 22:51 MapMessage.cs
-rwxrwxrwa   17732 Nov 29 22:39 Message.cs
-rwxrwxrwa    5029 Nov 30 14:05 MessageConsumer.cs
-rwxrwxrwa    9775 Nov 30 18:00 MessageProducer.cs
-rwxrwxrwa    9196 Nov 30 14:11 MessageProperties.cs
-rwxrwxrwa    1927 Nov 29 22:12 ObjectMessage.cs
-rwxrwxrwa    2068 Nov 29 02:01 Queue.cs
-rwxrwxrwa    2696 Nov 28 18:57 QueueBrowser.cs
-rwxrwxrwa   12681 Nov 30 14:36 Session.cs
-rwxrwxrwa   11210 Nov 29 02:02 StreamMessage.cs
-rwxrwxrwa    1856 Nov 27 00:10 TemporaryQueue.cs
-rwxrwxrwa    1856 Nov 27 00:10 TemporaryTopic.cs
-rwxrwxrwa    2156 Nov 29 02:02 TextMessage.cs
-rwxrwxrwa    2006 Nov 29 02:01 Topic.cs
drwxrwxrwx       0 Nov 29 01:48 Util
             
./Util:      
-rwxrwxrwa    4919 Nov 26 15:42 Dispatcher.cs
-rwxrwxrwa    4676 Nov 29 21:48 ExceptionUtil.cs
-rwxrwxrwa   13622 Nov 29 01:48 IntrospectionSupport.cs
-rwxrwxrwa    1792 Nov 18 21:22 UriAttributeAttribute.cs
-rwxrwxrwa   40560 Nov 30 17:52 XMSConvert.cs
-rwxrwxrwa   36672 Nov 29 22:21 XMSEnum.cs

I'm not exactly sure to understand what a Code Grant means. I suppose it means that, once accepted, I won't be able to claim any intellectual property on the code and therefore won't be able to prevent anyone from freely using and modifying it ? If so, it won't be a problem.

On the legal aspects, since all this was produced on my spare time and not on company time, on my hardware and not on my company's or customer's hardware, I believe my company or customer can't claim any intellectual property on the code. Only testing of the compiled DLL's requires me to use some of my customer's resources, as if they had been downloading the DLL from any web site. I don't think it gives them any intellectual rights on the code either. Furthermore, I don't think any clause in my employment contract prevents me from contributing to open source communities. Therefore I think there shouldn't be any legal complications on my side. I'll double-check anyway.

Still on the legal aspects : in order to make things fast and easy, I referenced and copy-pasted parts of IBM's documentation in the code.

eg.
// http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.msc.doc/prp_jms_ibm_encoding.htm?lang=en
/// <summary>
/// How numerical data in the body of the message is represented when the XMS
/// client forwards the message to its intended destination.
/// </summary>
public enum Encoding
{
    /// <summary>
    /// Normal integer encoding.
    /// </summary>
    IntegerNormal = MQC.MQENC_INTEGER_NORMAL,
...

This possibly would need to be reworked. Would it ?

You certainly know that the IBM XMS client can address more than just WebSphere MQ queue managers ("WMQ"), but also WebSphere Application Server Service Integration Buses ("WPM") or IBM Integration Buses through WebSphere MQ Real-Time Transport ("RTT"). I implemented all properties for supporting the 3 types of messaging servers. But in my environment, I will only be able to test the "WMQ" part. How should the other two be handled (tagged ? commented out ? ...) until me or someone else gets a chance to test them ?

XMS also supports creation of ConnectionFactory and other objects from repositories of administered objects. I implemented the logic to do so, but similarly I'm not sure how much of this I will be able to test.

Last point : what timelines would you expect ? My customer's expectations are to be able to handle basic use cases as described in my previous mail. This has already been tested OK. My personal aim is to cover more WMQ functionalities by the end of the year (at least queue browsing, selectors and topics), possibly including some NUnit tests (I didn't write any yet). Is that OK with you ?

Best regards,

Stephane Ramet

On Wed, Dec 9, 2015 at 1:38 AM, Jim Gomes <[hidden email]> wrote:
Hi Stephane,

Thank you for the e-mail. The Apache.NMS.MSMQ provider is still officially supported, but hasn't had any active development recently. It should still compile with the latest Apache.NMS 1.7.1 (or 1.7.2) API, using the trunk code base. It hasn't been tagged or branched in a while, so you'll need to build your own.

For the Apache.NMS.EMS provider, it has been branched into 1.7.x, and it will compile against the 1.7.1 or 1.7.2 API. Tagged versions have been created for this, but they haven't been officially released yet.

Having an XMS provider would be great! That's a work item that I entered into JIRA a while ago as a potential project to work on. If you would like to contribute it back to the project, I can help with that process. If it is a significant chunk of code, we will probably need to go through the process of a Code Grant. This is a little more involved and would require the legal department to be involved. It's not difficult, but the Apache Foundation wants to make sure that intellectual property rights are clearly preserved.  How much new code is involved with the XMS provider you've been working on?  I hope you aren't put off by the possibility of a Code Grant. When I first started working with NMS, I went through the process, and it was fairly straightforward.

I look forward to working with you on this.

Best Regards,
Jim


On Tue, Dec 8, 2015 at 12:20 PM, Stéphane Ramet <[hidden email]> wrote:
Hello,

for one of my customers', I need to provide messaging capabilities to some .NET applications in an environment where integration platforms are currently moved from WebsphereMQ and MSMQ to Tibco.

I was considering Apache.NMS for decoupling the application code from the underlying transport, but browsing through the site I found quite a number of broken links (MSMQ and EMS provider downloads are compiled in version 1.1, which is not available online any more ; Sandcastle Styles are not available any more either).

Therefore I was wondering what the status for this project is, or at least for those providers. Is it still active ?

I downloaded the source code from SVN and am currently testing it. The MSMQ provider works fine in my first basic use cases (simple message production, transacted production, synchronous and asynchronous consumption). I should get access to a sand box environment this week to test the EMS provider.

I also developed an XMS provider and started testing it (same basic use cases with the same test code as for MSMQ).

I can possibly contribute a .NET 4 source code, reasonably tested with XMS 8.0.0. Please tell me if it is of any interest, and guide me on how to contribute on that issue, since I'm quite new to that community, and to open source in general.

Regards,

Stephane Ramet