update on WildFly NoSQL prototype integration...

classic Classic list List threaded Threaded
47 messages Options
123
Reply | Threaded
Open this post in threaded view
|

update on WildFly NoSQL prototype integration...

Scott Marlow
Hi,

Below is an update on the WildFly NoSQL integration project.  The goal
is for deployed applications to have access to NoSQL databases (via
Hibernate OGM or native APIs).  Items 1-4, should be finished in our
first pass, with as much of the others items as we can do as well.

1. connection management will deal with obtaining NoSQL connections for
application use.

  - borrow/share Hibernate OGM connection configuration setup code
  - authentication integration
  - support transport level security

2. CDI programming simplifications will make it easy to inject NoSQL
data into your application classes.

  - https://github.com/antoinesd/javaee-nosql is initial idea

3. You will easily get a native NoSQL connection from the specified
NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in
your application.

4. You will also be able to easily use Hibernate OGM with the defined
NoSQL profiles (exactly how is TBD but will be awesome :-).
  - Hibernate OGM static module is included.
  - need to align with OGM dependencies (e.g. Hibernate ORM + other
dependencies).
  - as mentioned above, OGM already has some connection setup code,
which might be good to share for WildFly + standalone NoSQL use.
  - once WildFly has a common NoSQLSource (not a DataSource) that OGM
can use, OGM will be enhanced to use it.

5. How best for the WildFly NoSQL subsystem to be optional?
  - Is it enough to not run the wildfly/testsuite/nosql tests by default?
  - Or do we need to start a separate https://github.com/wildfly/nosql 
project for the NoSQL subsystems?

6. transaction enlistment

7. compensating transactions

8. runtime application monitoring

9. How soon can we make an evaluation distribution available for use on
OpenStack/OpenShift?
  - Would be great if we could do some load testing with all NoSQL
components.
  - Would be great if we could enable others to also test.

10. Are there any problems with our WildFly NoSQL subsystem injecting
MongoDatabase connections via:

    @Resource(lookup = "java:jboss/mongodb/test") MongoDatabase db;

  - No @Resource support expected for standalone Java, TBD is whether a
runtime library can be used.
  - Any problems expected on other EE application servers if this
approach becomes popular?

11. WIP topic branch is at
https://github.com/scottmarlow/wildfly/tree/nosql-dev9.  Note that every
once in a while, commits are squashed and pushed to nosql-devN+1.

12. Add proper unit tests
  - multi-threaded NoSQL access to show that works at all.
  - use NoSQL from different EE components (e.g. JAX-RS).
  - other use cases that represent how NoSQL could be used from WildFly.

Feedback/help is welcome!

Thanks,
Scott
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Jesper Pedersen-2
Hi,

On 05/04/2016 02:16 PM, Scott Marlow wrote:
> Below is an update on the WildFly NoSQL integration project.  The goal
> is for deployed applications to have access to NoSQL databases (via
> Hibernate OGM or native APIs).  Items 1-4, should be finished in our
> first pass, with as much of the others items as we can do as well.
>

Some general comments:

- Good start :)
- Maybe refactor from org.jboss.as to org.wildfly
- I think you can leave out the "subsystem" part of the package name
- Maybe use version identifiers in the package name since NoSQL APIs
change a lot
- Add some additional asserts to the test cases to verify input / output
from the stores

> 1. connection management will deal with obtaining NoSQL connections for
> application use.
>
>    - borrow/share Hibernate OGM connection configuration setup code
>    - authentication integration
>    - support transport level security
>

Connection management is a huge area to tackle. I think it is best to
start with what each NoSQL implementation offer.

The big question is if the connections should be truly managed which
would give the option to no-op dangerous API calls out. There aren't any
spec requirements for this, so it is really a matter of how the APIs
should be presented.

You may have to proxy some of the classes in order to pass the security
information into the stores. I think people would expect to use WildFly
security domains for this (SubjectFactory/Subject).

> 2. CDI programming simplifications will make it easy to inject NoSQL
> data into your application classes.
>
>    - https://github.com/antoinesd/javaee-nosql is initial idea
>
> 3. You will easily get a native NoSQL connection from the specified
> NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in
> your application.
>

I think this is key, as the APIs change almost per release.

> 4. You will also be able to easily use Hibernate OGM with the defined
> NoSQL profiles (exactly how is TBD but will be awesome :-).
>    - Hibernate OGM static module is included.
>    - need to align with OGM dependencies (e.g. Hibernate ORM + other
> dependencies).
>    - as mentioned above, OGM already has some connection setup code,
> which might be good to share for WildFly + standalone NoSQL use.
>    - once WildFly has a common NoSQLSource (not a DataSource) that OGM
> can use, OGM will be enhanced to use it.
>
> 5. How best for the WildFly NoSQL subsystem to be optional?
>    - Is it enough to not run the wildfly/testsuite/nosql tests by default?
>    - Or do we need to start a separate https://github.com/wildfly/nosql
> project for the NoSQL subsystems?
>
> 6. transaction enlistment
>

This will vary for each store, and there are many loopholes that you may
want to plug, f.ex. sharing a connection between 2 transactions.

For 1-phase, you will have to insert a LocalXAResource instance into the
transaction when the "connection" is obtained, and return it on the
boundary (afterCompletion). Same deal for 2-phase basically.

The LocalXAResource implementation will of course be different for each
store.

You may want a transaction option for the stores that supports this such
that people can choose the "level" of enlistment (ala jta="false").

One task I see is that people will have access to the transactional
methods in the API, and you don't want them to call these methods in
their apps unless the transaction setting allows this.

I think you can leave out the corner-cases in the 1st iteration, like
deferred enlistment (get connection, start transaction).

> 7. compensating transactions
>

Another huge area.

I would let the use-cases from WildFly/Swarm drive this.

A simple way to start is to have a SPI that apps can implement in case
the transaction is MARKED_FOR_ROLLBACK (hooks to an enlisted XAResource).

> 8. runtime application monitoring
>

As each store is different I would only look into what the subsystem can
expose, plus whatever the store can display.

> 9. How soon can we make an evaluation distribution available for use on
> OpenStack/OpenShift?
>    - Would be great if we could do some load testing with all NoSQL
> components.
>    - Would be great if we could enable others to also test.
>
> 10. Are there any problems with our WildFly NoSQL subsystem injecting
> MongoDatabase connections via:
>
>      @Resource(lookup = "java:jboss/mongodb/test") MongoDatabase db;
>
>    - No @Resource support expected for standalone Java, TBD is whether a
> runtime library can be used.
>    - Any problems expected on other EE application servers if this
> approach becomes popular?
>

This ties into connection management. I like what you have now for the
1st pass.

> 11. WIP topic branch is at
> https://github.com/scottmarlow/wildfly/tree/nosql-dev9.  Note that every
> once in a while, commits are squashed and pushed to nosql-devN+1.
>
> 12. Add proper unit tests
>    - multi-threaded NoSQL access to show that works at all.
>    - use NoSQL from different EE components (e.g. JAX-RS).
>    - other use cases that represent how NoSQL could be used from WildFly.
>

I think you can look at test cases for EE components that support @Resource.

Best regards,
  Jesper

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Tomaž Cerar-2

On Thu, May 5, 2016 at 7:48 PM, Jesper Pedersen <[hidden email]> wrote:
- Maybe refactor from org.jboss.as to org.wildfly
- I think you can leave out the "subsystem" part of the package name

not really, proper name of would be org.wildfly.extension.name-of-extension


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Stefano Maestri
In reply to this post by Jesper Pedersen-2


On Thu, May 5, 2016 at 7:48 PM, Jesper Pedersen <[hidden email]> wrote:
Hi,

On 05/04/2016 02:16 PM, Scott Marlow wrote:
> Below is an update on the WildFly NoSQL integration project.  The goal
> is for deployed applications to have access to NoSQL databases (via
> Hibernate OGM or native APIs).  Items 1-4, should be finished in our
> first pass, with as much of the others items as we can do as well.
>

Some general comments:

- Good start :)
- Maybe refactor from org.jboss.as to org.wildfly

+1
 
- Maybe use version identifiers in the package name since NoSQL APIs
change a lot

Well even if it is different we had bad experience on this w/ IJ and removed those version identifier from package name. I think package name should not have version ids, even if it could be acceptable in this first phase we should try to hide this kind of infos to users of OURS API even if APIs from NoSQL impl would be dependents from specific release.
 
- Add some additional asserts to the test cases to verify input / output
from the stores

> 1. connection management will deal with obtaining NoSQL connections for
> application use.
>
>    - borrow/share Hibernate OGM connection configuration setup code
>    - authentication integration
>    - support transport level security
>

Connection management is a huge area to tackle. I think it is best to
start with what each NoSQL implementation offer.

Yup it would be a good start, even if the other question is "why a user should use WF integration instead of direct use of various implementation?". One of the possible answer could be "to have a common connection management" for sure. So the (final) effort should be to have a common connection management.
 

The big question is if the connections should be truly managed which
would give the option to no-op dangerous API calls out. There aren't any
spec requirements for this, so it is really a matter of how the APIs
should be presented.

Yup exactly.
 

You may have to proxy some of the classes in order to pass the security
information into the stores. I think people would expect to use WildFly
security domains for this (SubjectFactory/Subject).

Yes it's another possible question to "why use WF integration?"
 

> 2. CDI programming simplifications will make it easy to inject NoSQL
> data into your application classes.
>
>    - https://github.com/antoinesd/javaee-nosql is initial idea
>
> 3. You will easily get a native NoSQL connection from the specified
> NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in
> your application.
>

I think this is key, as the APIs change almost per release.

Agree 100%. And it gives us also another requirement: should be easily to upgrade this Wildfly subsystem (or at least underlying NoSQL implementations) because NoSQL users would expect to use latest "cool" features of their NoSQL impl and don't want to wait for WildFLy (or EAP) release cycle.

 

> 4. You will also be able to easily use Hibernate OGM with the defined
> NoSQL profiles (exactly how is TBD but will be awesome :-).
>    - Hibernate OGM static module is included.
>    - need to align with OGM dependencies (e.g. Hibernate ORM + other
> dependencies).
>    - as mentioned above, OGM already has some connection setup code,
> which might be good to share for WildFly + standalone NoSQL use.
>    - once WildFly has a common NoSQLSource (not a DataSource) that OGM
> can use, OGM will be enhanced to use it.
>
> 5. How best for the WildFly NoSQL subsystem to be optional?
>    - Is it enough to not run the wildfly/testsuite/nosql tests by default?
>    - Or do we need to start a separate https://github.com/wildfly/nosql
> project for the NoSQL subsystems?
>
> 6. transaction enlistment
>

This will vary for each store, and there are many loopholes that you may
want to plug, f.ex. sharing a connection between 2 transactions.

For 1-phase, you will have to insert a LocalXAResource instance into the
transaction when the "connection" is obtained, and return it on the
boundary (afterCompletion). Same deal for 2-phase basically.

The LocalXAResource implementation will of course be different for each
store.

You may want a transaction option for the stores that supports this such
that people can choose the "level" of enlistment (ala jta="false").

One task I see is that people will have access to the transactional
methods in the API, and you don't want them to call these methods in
their apps unless the transaction setting allows this.

I think you can leave out the corner-cases in the 1st iteration, like
deferred enlistment (get connection, start transaction).

> 7. compensating transactions
>

Another huge area.

I would let the use-cases from WildFly/Swarm drive this.

A simple way to start is to have a SPI that apps can implement in case
the transaction is MARKED_FOR_ROLLBACK (hooks to an enlisted XAResource).

We implemented those in IJ (even if in much more standard JCA environment), and maybe we could share and eventually copy/paste some code.  

 

> 8. runtime application monitoring
>

As each store is different I would only look into what the subsystem can
expose, plus whatever the store can display.

Maybe a plugin approach could be taken as we did for datasource and RA subsystems reading statistics available during deployment and runtime and exposing them as pure runtime metrics in DMR.

Just my 2 cents.

Best regards
S.

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Gunnar Morling
In reply to this post by Scott Marlow
Hi Scott,

Thanks for driving this initiative!

Some comments/questions inline.

2016-05-04 20:16 GMT+02:00 Scott Marlow <[hidden email]>:
Hi,

Below is an update on the WildFly NoSQL integration project.  The goal
is for deployed applications to have access to NoSQL databases (via
Hibernate OGM or native APIs).  Items 1-4, should be finished in our
first pass, with as much of the others items as we can do as well.

1. connection management will deal with obtaining NoSQL connections for
application use.

  - borrow/share Hibernate OGM connection configuration setup code
  - authentication integration
  - support transport level security

Any ideas how this code sharing could look like?

Hibernate OGM should still be usable without WF; So maybe there should be a separate project/repo which defines an SPI to obtain/manage connections and implementations for different NoSQL stores?

I think it'd be beneficial to define some sort of connection URL syntax which can be used in NoSQL datasource definitions:

    nosql:mongodb://192.168.1.100:27017[,another-host:another-port]/some_db?customParam1=xyz&customParam2=abc&...

This common syntax might do for most cases, for others it might be needed to support specialized schemes handled by store-specific SPI implementations.

Regarding pooling, many NoSQL drivers do it in one way or another, so maybe leave that to drivers, at least in the first iteration?

2. CDI programming simplifications will make it easy to inject NoSQL
data into your application classes.

  - https://github.com/antoinesd/javaee-nosql is initial idea

+1 Sounds nice. 

3. You will easily get a native NoSQL connection from the specified
NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in
your application.

Would you take measures to e.g. prevent closing a connection in application code?

4. You will also be able to easily use Hibernate OGM with the defined
NoSQL profiles (exactly how is TBD but will be awesome :-).
  - Hibernate OGM static module is included.

WDYM by "static module"? The Hibernate OGM Core module?

How would people add the OGM backend module for the NoSQL store of their choice? I suppose you don't mean to add all the backends we have to WF by default (as this would increase size quite a bit), so would people still use the module ZIP we provide?
 
  - need to align with OGM dependencies (e.g. Hibernate ORM + other
dependencies).

Note we aim at the ORM version part of WF at a given time. But there are times where we need a newer ORM version for a new OGM release (when using new SPIs not part in the ORM version coming with the currently released WF), so there should be a way to achieve that.
 
  - as mentioned above, OGM already has some connection setup code,
which might be good to share for WildFly + standalone NoSQL use.

I'm not sure how re-usable this is. It's geared towards OGM's needs, I feel having a separate SPI dealing just with connections would be more useful.
 
  - once WildFly has a common NoSQLSource (not a DataSource) that OGM
can use, OGM will be enhanced to use it.

+1

5. How best for the WildFly NoSQL subsystem to be optional?
  - Is it enough to not run the wildfly/testsuite/nosql tests by default?
  - Or do we need to start a separate https://github.com/wildfly/nosql
project for the NoSQL subsystems?

Would there be one OGM sub-system? Or one per NoSQL store? And how would people add them (see above)? As said before in this thread, there should be a way to update modules of specific stores to get the latest and greatest.

6. transaction enlistment

7. compensating transactions

8. runtime application monitoring

9. How soon can we make an evaluation distribution available for use on
OpenStack/OpenShift?
  - Would be great if we could do some load testing with all NoSQL
components.
  - Would be great if we could enable others to also test.

10. Are there any problems with our WildFly NoSQL subsystem injecting
MongoDatabase connections via:

    @Resource(lookup = "java:jboss/mongodb/test") MongoDatabase db;

  - No @Resource support expected for standalone Java, TBD is whether a
runtime library can be used.
  - Any problems expected on other EE application servers if this
approach becomes popular?

11. WIP topic branch is at
https://github.com/scottmarlow/wildfly/tree/nosql-dev9.  Note that every
once in a while, commits are squashed and pushed to nosql-devN+1.

Any pointers where to start looking to see what's working already? 

12. Add proper unit tests
  - multi-threaded NoSQL access to show that works at all.
  - use NoSQL from different EE components (e.g. JAX-RS).
  - other use cases that represent how NoSQL could be used from WildFly.

Feedback/help is welcome!

Thanks,
Scott
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Tomaž Cerar-2


On 05/05/2016 06:42 PM, Tomaž Cerar wrote:

>
> On Thu, May 5, 2016 at 7:48 PM, Jesper Pedersen
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     - Maybe refactor from org.jboss.as <http://org.jboss.as> to org.wildfly
>     - I think you can leave out the "subsystem" part of the package name
>
>
> not really, proper name of would be org.wildfly.extension.name-of-extension
> for more see https://developer.jboss.org/wiki/WildFlyNamingPolicy

Thanks, I mostly followed the WildFlyNamingPolicy but didn't change the
maven artifacts yet (I think we are okay with the current artifact names
but if not, we can change them also).

>
>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Jesper Pedersen-2


On 05/05/2016 01:48 PM, Jesper Pedersen wrote:

> Hi,
>
> On 05/04/2016 02:16 PM, Scott Marlow wrote:
>> Below is an update on the WildFly NoSQL integration project.  The goal
>> is for deployed applications to have access to NoSQL databases (via
>> Hibernate OGM or native APIs).  Items 1-4, should be finished in our
>> first pass, with as much of the others items as we can do as well.
>>
>
> Some general comments:
>
> - Good start :)
> - Maybe refactor from org.jboss.as to org.wildfly
Done.

> - I think you can leave out the "subsystem" part of the package name
Done.

> - Maybe use version identifiers in the package name since NoSQL APIs
> change a lot
I'm not yet sure about versioning and about life cycle as well.

> - Add some additional asserts to the test cases to verify input / output
> from the stores

Will do.

>
>> 1. connection management will deal with obtaining NoSQL connections for
>> application use.
>>
>>    - borrow/share Hibernate OGM connection configuration setup code
>>    - authentication integration
>>    - support transport level security
>>
>
> Connection management is a huge area to tackle. I think it is best to
> start with what each NoSQL implementation offer.

It could be interesting to ping the various NoSQL communities to get
some dialog going on how to do platform level connection management.

>
> The big question is if the connections should be truly managed which
> would give the option to no-op dangerous API calls out. There aren't any
> spec requirements for this, so it is really a matter of how the APIs
> should be presented.
>
> You may have to proxy some of the classes in order to pass the security
> information into the stores. I think people would expect to use WildFly
> security domains for this (SubjectFactory/Subject).

Good point, users will want to use WildFly security.

>
>> 2. CDI programming simplifications will make it easy to inject NoSQL
>> data into your application classes.
>>
>>    - https://github.com/antoinesd/javaee-nosql is initial idea
>>
>> 3. You will easily get a native NoSQL connection from the specified
>> NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in
>> your application.
>>
>
> I think this is key, as the APIs change almost per release.
>
>> 4. You will also be able to easily use Hibernate OGM with the defined
>> NoSQL profiles (exactly how is TBD but will be awesome :-).
>>    - Hibernate OGM static module is included.
>>    - need to align with OGM dependencies (e.g. Hibernate ORM + other
>> dependencies).
>>    - as mentioned above, OGM already has some connection setup code,
>> which might be good to share for WildFly + standalone NoSQL use.
>>    - once WildFly has a common NoSQLSource (not a DataSource) that OGM
>> can use, OGM will be enhanced to use it.
>>
>> 5. How best for the WildFly NoSQL subsystem to be optional?
>>    - Is it enough to not run the wildfly/testsuite/nosql tests by default?
>>    - Or do we need to start a separate https://github.com/wildfly/nosql
>> project for the NoSQL subsystems?
>>
>> 6. transaction enlistment
>>
>
> This will vary for each store, and there are many loopholes that you may
> want to plug, f.ex. sharing a connection between 2 transactions.
>
> For 1-phase, you will have to insert a LocalXAResource instance into the
> transaction when the "connection" is obtained, and return it on the
> boundary (afterCompletion). Same deal for 2-phase basically.
>
> The LocalXAResource implementation will of course be different for each
> store.

Any links to existing IronJacamar code to share here?  I think that we
could prototype new transaction enlistment handling code, based on what
we currently have.

>
> You may want a transaction option for the stores that supports this such
> that people can choose the "level" of enlistment (ala jta="false").
>
> One task I see is that people will have access to the transactional
> methods in the API, and you don't want them to call these methods in
> their apps unless the transaction setting allows this.
>
> I think you can leave out the corner-cases in the 1st iteration, like
> deferred enlistment (get connection, start transaction).
>
>> 7. compensating transactions
>>
>
> Another huge area.
>
> I would let the use-cases from WildFly/Swarm drive this.
>
> A simple way to start is to have a SPI that apps can implement in case
> the transaction is MARKED_FOR_ROLLBACK (hooks to an enlisted XAResource).
>
>> 8. runtime application monitoring
>>
>
> As each store is different I would only look into what the subsystem can
> expose, plus whatever the store can display.
>
>> 9. How soon can we make an evaluation distribution available for use on
>> OpenStack/OpenShift?
>>    - Would be great if we could do some load testing with all NoSQL
>> components.
>>    - Would be great if we could enable others to also test.
>>
>> 10. Are there any problems with our WildFly NoSQL subsystem injecting
>> MongoDatabase connections via:
>>
>>      @Resource(lookup = "java:jboss/mongodb/test") MongoDatabase db;
>>
>>    - No @Resource support expected for standalone Java, TBD is whether a
>> runtime library can be used.
>>    - Any problems expected on other EE application servers if this
>> approach becomes popular?
>>
>
> This ties into connection management. I like what you have now for the
> 1st pass.
>
>> 11. WIP topic branch is at
>> https://github.com/scottmarlow/wildfly/tree/nosql-dev9.  Note that every
>> once in a while, commits are squashed and pushed to nosql-devN+1.
>>
>> 12. Add proper unit tests
>>    - multi-threaded NoSQL access to show that works at all.
>>    - use NoSQL from different EE components (e.g. JAX-RS).
>>    - other use cases that represent how NoSQL could be used from WildFly.
>>
>
> I think you can look at test cases for EE components that support @Resource.

Using @Inject may be another way, once we have
https://github.com/antoinesd/javaee-nosql going.  Will see which way
survives the prototype stage to what is merged into WildFly. :-)

>
> Best regards,
>   Jesper
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Emmanuel Bernard
In reply to this post by Scott Marlow
That reminds me that we wanted to make the connection to JDG as simple
as MongoDB and the like. (i.e. a URL scheme more or less).
I'm talking about JDG remote here even though local would be a nice to
have.

How hard would that be?

Emmanuel

On Wed 2016-05-04 14:16, Scott Marlow wrote:

> Hi,
>
> Below is an update on the WildFly NoSQL integration project.  The goal
> is for deployed applications to have access to NoSQL databases (via
> Hibernate OGM or native APIs).  Items 1-4, should be finished in our
> first pass, with as much of the others items as we can do as well.
>
> 1. connection management will deal with obtaining NoSQL connections for
> application use.
>
>   - borrow/share Hibernate OGM connection configuration setup code
>   - authentication integration
>   - support transport level security
>
> 2. CDI programming simplifications will make it easy to inject NoSQL
> data into your application classes.
>
>   - https://github.com/antoinesd/javaee-nosql is initial idea
>
> 3. You will easily get a native NoSQL connection from the specified
> NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in
> your application.
>
> 4. You will also be able to easily use Hibernate OGM with the defined
> NoSQL profiles (exactly how is TBD but will be awesome :-).
>   - Hibernate OGM static module is included.
>   - need to align with OGM dependencies (e.g. Hibernate ORM + other
> dependencies).
>   - as mentioned above, OGM already has some connection setup code,
> which might be good to share for WildFly + standalone NoSQL use.
>   - once WildFly has a common NoSQLSource (not a DataSource) that OGM
> can use, OGM will be enhanced to use it.
>
> 5. How best for the WildFly NoSQL subsystem to be optional?
>   - Is it enough to not run the wildfly/testsuite/nosql tests by default?
>   - Or do we need to start a separate https://github.com/wildfly/nosql 
> project for the NoSQL subsystems?
>
> 6. transaction enlistment
>
> 7. compensating transactions
>
> 8. runtime application monitoring
>
> 9. How soon can we make an evaluation distribution available for use on
> OpenStack/OpenShift?
>   - Would be great if we could do some load testing with all NoSQL
> components.
>   - Would be great if we could enable others to also test.
>
> 10. Are there any problems with our WildFly NoSQL subsystem injecting
> MongoDatabase connections via:
>
>     @Resource(lookup = "java:jboss/mongodb/test") MongoDatabase db;
>
>   - No @Resource support expected for standalone Java, TBD is whether a
> runtime library can be used.
>   - Any problems expected on other EE application servers if this
> approach becomes popular?
>
> 11. WIP topic branch is at
> https://github.com/scottmarlow/wildfly/tree/nosql-dev9.  Note that every
> once in a while, commits are squashed and pushed to nosql-devN+1.
>
> 12. Add proper unit tests
>   - multi-threaded NoSQL access to show that works at all.
>   - use NoSQL from different EE components (e.g. JAX-RS).
>   - other use cases that represent how NoSQL could be used from WildFly.
>
> Feedback/help is welcome!
>
> Thanks,
> Scott
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Stefano Maestri


On 05/10/2016 05:26 AM, Stefano Maestri wrote:

>
>
> On Thu, May 5, 2016 at 7:48 PM, Jesper
> Pedersen <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi,
>
>     On 05/04/2016 02:16 PM, Scott Marlow wrote:
>     > Below is an update on the WildFly NoSQL integration project.  The goal
>     > is for deployed applications to have access to NoSQL databases (via
>     > Hibernate OGM or native APIs).  Items 1-4, should be finished in our
>     > first pass, with as much of the others items as we can do as well.
>     >
>
>     Some general comments:
>
>     - Good start :)
>     - Maybe refactor from org.jboss.as <http://org.jboss.as/> to org.wildfly
>
>
> +1
>
>
>     - Maybe use version identifiers in the package name since NoSQL APIs
>     change a lot
>
>
> Well even if it is different we had bad experience on this w/ IJ and
> removed those version identifier from package name. I think package name
> should not have version ids, even if it could be acceptable in this
> first phase we should try to hide this kind of infos to users of OURS
> API even if APIs from NoSQL impl would be dependents from specific release.
>
>
>     - Add some additional asserts to the test cases to verify input / output
>     from the stores
>
>     > 1. connection management will deal with obtaining NoSQL connections for
>     > application use.
>     >
>     >    - borrow/share Hibernate OGM connection configuration setup code
>     >    - authentication integration
>     >    - support transport level security
>     >
>
>     Connection management is a huge area to tackle. I think it is best to
>     start with what each NoSQL implementation offer.
>
>
> Yup it would be a good start, even if the other question is "why a user
> should use WF integration instead of direct use of various
> implementation?". One of the possible answer could be "to have a common
> connection management" for sure. So the (final) effort should be to have
> a common connection management.

I wonder if any of the NoSQL database development communities would be
interested in establishing a common connection management client API.

>
>
>
>     The big question is if the connections should be truly managed which
>     would give the option to no-op dangerous API calls out. There aren't any
>     spec requirements for this, so it is really a matter of how the APIs
>     should be presented.
>
>
> Yup exactly.
>
>
>
>     You may have to proxy some of the classes in order to pass the security
>     information into the stores. I think people would expect to use WildFly
>     security domains for this (SubjectFactory/Subject).
>
>
> Yes it's another possible question to "why use WF integration?"
>
>
>
>     > 2. CDI programming simplifications will make it easy to inject NoSQL
>     > data into your application classes.
>     >
>     >    - https://github.com/antoinesd/javaee-nosql is initial idea
>     >
>     > 3. You will easily get a native NoSQL connection from the specified
>     > NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in
>     > your application.
>     >
>
>     I think this is key, as the APIs change almost per release.
>
>
> Agree 100%. And it gives us also another requirement: should be easily
> to upgrade this Wildfly subsystem (or at least underlying NoSQL
> implementations) because NoSQL users would expect to use latest "cool"
> features of their NoSQL impl and don't want to wait for WildFLy (or EAP)
> release cycle.
>
>
>
>
>     > 4. You will also be able to easily use Hibernate OGM with the defined
>     > NoSQL profiles (exactly how is TBD but will be awesome :-).
>     >    - Hibernate OGM static module is included.
>     >    - need to align with OGM dependencies (e.g. Hibernate ORM + other
>     > dependencies).
>     >    - as mentioned above, OGM already has some connection setup code,
>     > which might be good to share for WildFly + standalone NoSQL use.
>     >    - once WildFly has a common NoSQLSource (not a DataSource) that OGM
>     > can use, OGM will be enhanced to use it.
>     >
>     > 5. How best for the WildFly NoSQL subsystem to be optional?
>     >    - Is it enough to not run the wildfly/testsuite/nosql tests by default?
>     >    - Or do we need to start a separate https://github.com/wildfly/nosql
>     > project for the NoSQL subsystems?
>     >
>     > 6. transaction enlistment
>     >
>
>     This will vary for each store, and there are many loopholes that you may
>     want to plug, f.ex. sharing a connection between 2 transactions.
>
>     For 1-phase, you will have to insert a LocalXAResource instance into the
>     transaction when the "connection" is obtained, and return it on the
>     boundary (afterCompletion). Same deal for 2-phase basically.
>
>     The LocalXAResource implementation will of course be different for each
>     store.
>
>     You may want a transaction option for the stores that supports this such
>     that people can choose the "level" of enlistment (ala jta="false").
>
>     One task I see is that people will have access to the transactional
>     methods in the API, and you don't want them to call these methods in
>     their apps unless the transaction setting allows this.
>
>     I think you can leave out the corner-cases in the 1st iteration, like
>     deferred enlistment (get connection, start transaction).
>
>     > 7. compensating transactions
>     >
>
>     Another huge area.
>
>     I would let the use-cases from WildFly/Swarm drive this.
>
>     A simple way to start is to have a SPI that apps can implement in case
>     the transaction is MARKED_FOR_ROLLBACK (hooks to an enlisted
>     XAResource).
>
>
> We implemented those in IJ (even if in much more standard JCA
> environment), and maybe we could share and eventually copy/paste some
> code.

Check out "MongoDB integration test with compensating transactions"
https://github.com/scottmarlow/wildfly/commit/9bd67964a9259416add73cac47328cec5127d25c,
which is a good start.  We can work forward from what we have and ideas
from IJ.

>
>
>
>
>     > 8. runtime application monitoring
>     >
>
>     As each store is different I would only look into what the subsystem can
>     expose, plus whatever the store can display.
>
>
> Maybe a plugin approach could be taken as we did for datasource and RA
> subsystems reading statistics available during deployment and runtime
> and exposing them as pure runtime metrics in DMR.

Great idea and great examples.

>
> Just my 2 cents.
>
> Best regards
> S.
>
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Emmanuel Bernard


On 05/10/2016 10:01 AM, Emmanuel Bernard wrote:
> That reminds me that we wanted to make the connection to JDG as simple
> as MongoDB and the like. (i.e. a URL scheme more or less).
> I'm talking about JDG remote here even though local would be a nice to
> have.
>
> How hard would that be?

I think that org.wildfly.clustering.dispatcher.CommandDispatcher is for
invoking nodes on the WildFly cluster but I'm not sure of the API for
invoking JDG remote and whether that is planned to be part of WildFly
already, as part of our clustering effort.

Could/should JDG access fall into NoSQL integration?

>
> Emmanuel
>
> On Wed 2016-05-04 14:16, Scott Marlow wrote:
>> Hi,
>>
>> Below is an update on the WildFly NoSQL integration project.  The goal
>> is for deployed applications to have access to NoSQL databases (via
>> Hibernate OGM or native APIs).  Items 1-4, should be finished in our
>> first pass, with as much of the others items as we can do as well.
>>
>> 1. connection management will deal with obtaining NoSQL connections for
>> application use.
>>
>>   - borrow/share Hibernate OGM connection configuration setup code
>>   - authentication integration
>>   - support transport level security
>>
>> 2. CDI programming simplifications will make it easy to inject NoSQL
>> data into your application classes.
>>
>>   - https://github.com/antoinesd/javaee-nosql is initial idea
>>
>> 3. You will easily get a native NoSQL connection from the specified
>> NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in
>> your application.
>>
>> 4. You will also be able to easily use Hibernate OGM with the defined
>> NoSQL profiles (exactly how is TBD but will be awesome :-).
>>   - Hibernate OGM static module is included.
>>   - need to align with OGM dependencies (e.g. Hibernate ORM + other
>> dependencies).
>>   - as mentioned above, OGM already has some connection setup code,
>> which might be good to share for WildFly + standalone NoSQL use.
>>   - once WildFly has a common NoSQLSource (not a DataSource) that OGM
>> can use, OGM will be enhanced to use it.
>>
>> 5. How best for the WildFly NoSQL subsystem to be optional?
>>   - Is it enough to not run the wildfly/testsuite/nosql tests by default?
>>   - Or do we need to start a separate https://github.com/wildfly/nosql
>> project for the NoSQL subsystems?
>>
>> 6. transaction enlistment
>>
>> 7. compensating transactions
>>
>> 8. runtime application monitoring
>>
>> 9. How soon can we make an evaluation distribution available for use on
>> OpenStack/OpenShift?
>>   - Would be great if we could do some load testing with all NoSQL
>> components.
>>   - Would be great if we could enable others to also test.
>>
>> 10. Are there any problems with our WildFly NoSQL subsystem injecting
>> MongoDatabase connections via:
>>
>>     @Resource(lookup = "java:jboss/mongodb/test") MongoDatabase db;
>>
>>   - No @Resource support expected for standalone Java, TBD is whether a
>> runtime library can be used.
>>   - Any problems expected on other EE application servers if this
>> approach becomes popular?
>>
>> 11. WIP topic branch is at
>> https://github.com/scottmarlow/wildfly/tree/nosql-dev9.  Note that every
>> once in a while, commits are squashed and pushed to nosql-devN+1.
>>
>> 12. Add proper unit tests
>>   - multi-threaded NoSQL access to show that works at all.
>>   - use NoSQL from different EE components (e.g. JAX-RS).
>>   - other use cases that represent how NoSQL could be used from WildFly.
>>
>> Feedback/help is welcome!
>>
>> Thanks,
>> Scott
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Jesper Pedersen-2
In reply to this post by Scott Marlow
Hi,

On 05/10/2016 09:58 AM, Scott Marlow wrote:

> On 05/05/2016 01:48 PM, Jesper Pedersen wrote:
>>> 6. transaction enlistment
>>>
>>
>> This will vary for each store, and there are many loopholes that you may
>> want to plug, f.ex. sharing a connection between 2 transactions.
>>
>> For 1-phase, you will have to insert a LocalXAResource instance into the
>> transaction when the "connection" is obtained, and return it on the
>> boundary (afterCompletion). Same deal for 2-phase basically.
>>
>> The LocalXAResource implementation will of course be different for each
>> store.
>
> Any links to existing IronJacamar code to share here?  I think that we
> could prototype new transaction enlistment handling code, based on what
> we currently have.
>

The IronJacamar code is here for Narayana:

https://github.com/ironjacamar/ironjacamar/tree/master/core/src/main/java/org/ironjacamar/core/tx/narayana

However, that is based on the Java EE Connector Architecture
specification, so we have a *much* easier job.

Lets take an example from the NoSQL world using Neo4J:

http://neo4j.com/docs/api/java-driver/current/org/neo4j/driver/v1/Driver.html

...neo4j.LocalXAResource implements XAResource, org.jboss.tm.LastResource

  public void start(Xid xid, int flags) throws XAException {
    tx = session.beginTransaction();
  }

  public void commit(Xid xid, boolean onePhase) throws XAException {
    tx.success();
  }

  public void rollback(Xid xid) throws XAException {
    tx.failure();
  }

And the app, f.ex. in a SLSB method

public class MySLSB ...

   @Resource(mappedName="java:jboss/nosql/neo4j")
   private Driver neo4j;

   public void myMethod() {
#1   Session s = neo4j.createSession();
      s.run( "CREATE (n {name:'Bob'})" );

#2   Transaction tx = s.beginTransaction();
      tx.run( "CREATE (n {name:'Alice'})" );
      tx.run( "CREATE (n {name:'Tina'})" );

#3   tx.success();

#4   tx.close();
#5   s.close();
#6   neo4j.close();
   }

However, as you can see there are a number of hidden things going on
here if ".../neo4j" is deployed as "transaction=1phase".

#1: Here you have to intercept the call, create the LocalXAResource
instance and enlist it in the active transaction

#2: Here you have to a delegator instance that only executes the run() calls

#3: No-op call, happens upon EE transaction completion

#4: No-op call. happens upon EE transaction completion

#5: No-op call, happens in an enlisted Synchronization instance
(afterCompletion) (done in #1 too)

#6: No-op call, handled by subsystem

The value-add is that #3 is tied into the overall EE transaction.

If ".../neo4j" is deployed as "transaction=none", then all calls are on
the "real" Neo4J objects.

As you can see it isn't as simple as it may appear, and application flow
could be different from what the developer expects, as #3 could be a
real tx.failure() call in case of MARKED_FOR_ROLLBACK.

There are def pros/cons of doing tx enlistment for this area...

>>
>> You may want a transaction option for the stores that supports this such
>> that people can choose the "level" of enlistment (ala jta="false").
>>
>> One task I see is that people will have access to the transactional
>> methods in the API, and you don't want them to call these methods in
>> their apps unless the transaction setting allows this.
>>
>> I think you can leave out the corner-cases in the 1st iteration, like
>> deferred enlistment (get connection, start transaction).

Best regards,
  Jesper

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Emmanuel Bernard
In reply to this post by Scott Marlow

On 10 May 2016, at 16:45, Scott Marlow <[hidden email]> wrote:

That reminds me that we wanted to make the connection to JDG as simple
as MongoDB and the like. (i.e. a URL scheme more or less).
I'm talking about JDG remote here even though local would be a nice to
have.

How hard would that be?

I think that org.wildfly.clustering.dispatcher.CommandDispatcher is for invoking nodes on the WildFly cluster but I'm not sure of the API for invoking JDG remote and whether that is planned to be part of WildFly already, as part of our clustering effort.

Could/should JDG access fall into NoSQL integration?

Yes.

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Gunnar Morling


On 05/10/2016 06:34 AM, Gunnar Morling wrote:

> Hi Scott,
>
> Thanks for driving this initiative!
>
> Some comments/questions inline.
>
> 2016-05-04 20:16 GMT+02:00 Scott Marlow <[hidden email]
> <mailto:[hidden email]>>:
>
>     Hi,
>
>     Below is an update on the WildFly NoSQL integration project.  The goal
>     is for deployed applications to have access to NoSQL databases (via
>     Hibernate OGM or native APIs).  Items 1-4, should be finished in our
>     first pass, with as much of the others items as we can do as well.
>
>     1. connection management will deal with obtaining NoSQL connections for
>     application use.
>
>       - borrow/share Hibernate OGM connection configuration setup code
>       - authentication integration
>       - support transport level security
>
>
> Any ideas how this code sharing could look like?
>
> Hibernate OGM should still be usable without WF; So maybe there should
> be a separate project/repo which defines an SPI to obtain/manage
> connections and implementations for different NoSQL stores?

Excellent suggestion,  perhaps the SPI could be under
https://github.com/jboss, which is a common area for sharing.  Possible
locations for creating the per NoSQL store implementations could be
https://github.com/jboss or https://github.com/hibernate or
https://github.com/wildfly.

>
> I think it'd be beneficial to define some sort of connection URL syntax
> which can be used in NoSQL datasource definitions:
>
>
> nosql:mongodb://192.168.1.100:27017[,another-host:another-port]/some_db?customParam1=xyz&customParam2=abc&...

The WildFly NoSQL subsystem configuration, would likely need separate
fields for NoSQL datasource profiles/definition, instead of one long
URL.  Internally, the WildFly subsystem could build a connection URL, if
that becomes our common format.

>
> This common syntax might do for most cases, for others it might be
> needed to support specialized schemes handled by store-specific SPI
> implementations.

What are some examples that might need specialized schemes?  Would an
embedded or in-process Infinispan/Cassandra be considered one of the
specialized schemes because there are no remote connections?

>
> Regarding pooling, many NoSQL drivers do it in one way or another, so
> maybe leave that to drivers, at least in the first iteration?
>
>     2. CDI programming simplifications will make it easy to inject NoSQL
>     data into your application classes.
>
>       - https://github.com/antoinesd/javaee-nosql is initial idea
>
>
> +1 Sounds nice.
>
>
>     3. You will easily get a native NoSQL connection from the specified
>     NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in
>     your application.
>
>
> Would you take measures to e.g. prevent closing a connection in
> application code?

I think that this should be on the wish list, just not sure that we
should always introduce the overhead of wrapping/proxying NoSQL
connections to allow the check if the application.  Perhaps a "platform
level" listener in the native NoSQL client code, that allows the
platform to prevent applications from closing connections in some way.

>
>     4. You will also be able to easily use Hibernate OGM with the defined
>     NoSQL profiles (exactly how is TBD but will be awesome :-).
>       - Hibernate OGM static module is included.
>
>
> WDYM by "static module"? The Hibernate OGM Core module?

By "static module", I meant what we add to wildfly/modules/*.  Excellent
question about which OGM modules, I meant any OGM that depends on
Hibernate ORM, which I think is currently OGM Core.

>
> How would people add the OGM backend module for the NoSQL store of their
> choice? I suppose you don't mean to add all the backends we have to WF
> by default (as this would increase size quite a bit), so would people
> still use the module ZIP we provide?

If we use the OGM module ZIP, that should be extracted into a different
folder than wildfly/modules/system/layers, to avoid confusing the EAP
patch tool.  Having said that, I'm not sure how we will patch the OGM
modules, if they are not all included in WildFly.  Perhaps, we might
decide to include only certain ones.

Just for reference, my local
hibernate-ogm-modules-wildfly10-5.0.0-SNAPSHOT.zip is 53mb (including
OGM core).

>
>
>       - need to align with OGM dependencies (e.g. Hibernate ORM + other
>     dependencies).
>
>
> Note we aim at the ORM version part of WF at a given time. But there are
> times where we need a newer ORM version for a new OGM release (when
> using new SPIs not part in the ORM version coming with the currently
> released WF), so there should be a way to achieve that.

Are you thinking of the current Hibernate ORM master or the ORM 5.1
branch?  Or perhaps a new branch based on 5.1, that doesn't have all of
the ORM master branch changes?

>
>
>       - as mentioned above, OGM already has some connection setup code,
>     which might be good to share for WildFly + standalone NoSQL use.
>
>
> I'm not sure how re-usable this is. It's geared towards OGM's needs, I
> feel having a separate SPI dealing just with connections would be more
> useful.

+1

>
>
>       - once WildFly has a common NoSQLSource (not a DataSource) that OGM
>     can use, OGM will be enhanced to use it.
>
>
> +1
>
>     5. How best for the WildFly NoSQL subsystem to be optional?
>       - Is it enough to not run the wildfly/testsuite/nosql tests by
>     default?
>       - Or do we need to start a separate https://github.com/wildfly/nosql
>     project for the NoSQL subsystems?
>
>
> Would there be one OGM sub-system? Or one per NoSQL store?

I don't yet see a need for an OGM sub-system (I don't think OGM needs a
separate WildFly deployment processer).  I think that some of the OGM
deployment hacks will be done in Jipijapa (assuming we can change the
JPA deployer to allow that).  I'm not exactly sure what your asking here
with regard to question #5, about one OGM sub-system or one per NosQL
store.  Do you mean "static module" instead of sub-system?

I think that there will be one WildFly sub-system per NoSQL store but
that is based on the current prototype and the discussion so far.

> And how would
> people add them (see above)? As said before in this thread, there should
> be a way to update modules of specific stores to get the latest and
> greatest.

Not sure yet, maybe with-in minor releases it will be possible.

>
>     6. transaction enlistment
>
>     7. compensating transactions
>
>     8. runtime application monitoring
>
>     9. How soon can we make an evaluation distribution available for use on
>     OpenStack/OpenShift?
>       - Would be great if we could do some load testing with all NoSQL
>     components.
>       - Would be great if we could enable others to also test.
>
>     10. Are there any problems with our WildFly NoSQL subsystem injecting
>     MongoDatabase connections via:
>
>         @Resource(lookup = "java:jboss/mongodb/test") MongoDatabase db;
>
>       - No @Resource support expected for standalone Java, TBD is whether a
>     runtime library can be used.
>       - Any problems expected on other EE application servers if this
>     approach becomes popular?
>
>     11. WIP topic branch is at
>     https://github.com/scottmarlow/wildfly/tree/nosql-dev9.  Note that every
>     once in a while, commits are squashed and pushed to nosql-devN+1.
>
>
> Any pointers where to start looking to see what's working already?

https://github.com/scottmarlow/wildfly/tree/nosql-dev9 is still active. :)


>
>     12. Add proper unit tests
>       - multi-threaded NoSQL access to show that works at all.
>       - use NoSQL from different EE components (e.g. JAX-RS).
>       - other use cases that represent how NoSQL could be used from WildFly.
>
>     Feedback/help is welcome!
>
>     Thanks,
>     Scott
>     _______________________________________________
>     wildfly-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Scott Marlow

>>     > 7. compensating transactions
>>     >
>>
>>     Another huge area.
>>
>>     I would let the use-cases from WildFly/Swarm drive this.
>>
>>     A simple way to start is to have a SPI that apps can implement in case
>>     the transaction is MARKED_FOR_ROLLBACK (hooks to an enlisted
>>     XAResource).
>>
>>
>> We implemented those in IJ (even if in much more standard JCA
>> environment), and maybe we could share and eventually copy/paste some
>> code.
>
> Check out "MongoDB integration test with compensating transactions"
> https://github.com/scottmarlow/wildfly/commit/9bd67964a9259416add73cac47328cec5127d25c,
> which is a good start.  We can work forward from what we have and ideas
> from IJ.
>

Stefano, Could you please post github links to some of the IJ
compensating code?  It would be great to express the idea of what is
being accomplished in IJ (with regard to compensating transactions), as
well.

Thanks,
Scott
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Stefano Maestri


On Wed, May 11, 2016 at 4:48 PM, Scott Marlow <[hidden email]> wrote:

>>     > 7. compensating transactions
>>     >
>>
>>     Another huge area.
>>
>>     I would let the use-cases from WildFly/Swarm drive this.
>>
>>     A simple way to start is to have a SPI that apps can implement in case
>>     the transaction is MARKED_FOR_ROLLBACK (hooks to an enlisted
>>     XAResource).
>>
>>
>> We implemented those in IJ (even if in much more standard JCA
>> environment), and maybe we could share and eventually copy/paste some
>> code.
>
> Check out "MongoDB integration test with compensating transactions"
> https://github.com/scottmarlow/wildfly/commit/9bd67964a9259416add73cac47328cec5127d25c,
> which is a good start.  We can work forward from what we have and ideas
> from IJ.
>

Stefano, Could you please post github links to some of the IJ
compensating code?  It would be great to express the idea of what is
being accomplished in IJ (with regard to compensating transactions), as
well.



Hi Scott,
I take a mistake inserting my comment in previous mail. My comment was referring on your point 6 (for which Jesper already provided code)....sorry I'm not yet used to this Gmail collapse long quoted text "feature".....
Regarding point 7, I agree use-cases from WildFly/Swarm are a good start.

regards
S.

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Emmanuel Bernard
In reply to this post by Scott Marlow


On 11 mai 2016, at 16:02, Scott Marlow <[hidden email]> wrote:

>> Hibernate OGM should still be usable without WF; So maybe there should
>> be a separate project/repo which defines an SPI to obtain/manage
>> connections and implementations for different NoSQL stores?
>
> Excellent suggestion,  perhaps the SPI could be under
> https://github.com/jboss, which is a common area for sharing.  Possible
> locations for creating the per NoSQL store implementations could be
> https://github.com/jboss or https://github.com/hibernate or
> https://github.com/wildfly.

I'm starting to think that this might be way overkill. If we are creating a sub project just to share between 20 and 50 lines of code per provider and the overhead code to abstract property configuration to plus OGM and WF ones, we are losing more than gaining.

Thoughts ?

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Gunnar Morling
If OGM should be able to work with connections managed by the container, some sort of "neutral" SPI is needed IMO. Otherwise we'd have to create some sort of abstraction *in* OGM to ensure it still can be used in other environments than WF.

I think best would be to start with a PoC and see things are going to look like. Then we still can decide where it'd be best located or whether it's better to avoid re-use and accept some duplication.




2016-05-12 13:24 GMT+02:00 Emmanuel Bernard <[hidden email]>:


On 11 mai 2016, at 16:02, Scott Marlow <[hidden email]> wrote:

>> Hibernate OGM should still be usable without WF; So maybe there should
>> be a separate project/repo which defines an SPI to obtain/manage
>> connections and implementations for different NoSQL stores?
>
> Excellent suggestion,  perhaps the SPI could be under
> https://github.com/jboss, which is a common area for sharing.  Possible
> locations for creating the per NoSQL store implementations could be
> https://github.com/jboss or https://github.com/hibernate or
> https://github.com/wildfly.

I'm starting to think that this might be way overkill. If we are creating a sub project just to share between 20 and 50 lines of code per provider and the overhead code to abstract property configuration to plus OGM and WF ones, we are losing more than gaining.

Thoughts ?

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Sanne Grinovero
In reply to this post by Emmanuel Bernard
On 12 May 2016 at 12:24, Emmanuel Bernard <[hidden email]> wrote:

>
>
> On 11 mai 2016, at 16:02, Scott Marlow <[hidden email]> wrote:
>
>>> Hibernate OGM should still be usable without WF; So maybe there should
>>> be a separate project/repo which defines an SPI to obtain/manage
>>> connections and implementations for different NoSQL stores?
>>
>> Excellent suggestion,  perhaps the SPI could be under
>> https://github.com/jboss, which is a common area for sharing.  Possible
>> locations for creating the per NoSQL store implementations could be
>> https://github.com/jboss or https://github.com/hibernate or
>> https://github.com/wildfly.
>
> I'm starting to think that this might be way overkill. If we are creating a sub project just to share between 20 and 50 lines of code per provider and the overhead code to abstract property configuration to plus OGM and WF ones, we are losing more than gaining.
>
> Thoughts ?

I agree it's overkill, and have an alternative proposal:

Hibernate OGM should define an interface which is appropriate for its
own consumption; the Wildfly NoSQL subssystem can have its own
interface so to not depend on OGM, but they would be somewhat similar
for each given NoSQL technology we intend to support in this way.

Then JipiJapa can inject an adaptor into the OGM boostrap phase,
delegating from one to the other. So only the OGM specific JipiJapa
module would need to depend on both interfaces.
If this dependency is not desirable either, then I think we can live
with a non-typesafe generic provider of things.

Thanks,
Sanne

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Emmanuel Bernard


On 05/12/2016 07:24 AM, Emmanuel Bernard wrote:

>
>
> On 11 mai 2016, at 16:02, Scott Marlow <[hidden email]> wrote:
>
>>> Hibernate OGM should still be usable without WF; So maybe there should
>>> be a separate project/repo which defines an SPI to obtain/manage
>>> connections and implementations for different NoSQL stores?
>>
>> Excellent suggestion,  perhaps the SPI could be under
>> https://github.com/jboss, which is a common area for sharing.  Possible
>> locations for creating the per NoSQL store implementations could be
>> https://github.com/jboss or https://github.com/hibernate or
>> https://github.com/wildfly.
>
> I'm starting to think that this might be way overkill. If we are creating a sub project just to share between 20 and 50 lines of code per provider and the overhead code to abstract property configuration to plus OGM and WF ones, we are losing more than gaining.
>
> Thoughts ?
>

It could be overkill.  Mostly, I was noticing that OGM is already
abstracting some connection setup and my initial impression was that we
might like to do something similar in each WildFly NoSQL subsystem.
Then, I was thinking about whether to duplicate the code or not.  Since,
this is purely internal only, we can probably start with duplicating the
connection setup and later consider if we can (attempt) sharing some of
the connection setup.

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Gunnar Morling


On 05/12/2016 08:13 AM, Gunnar Morling wrote:
> If OGM should be able to work with connections managed by the container,
> some sort of "neutral" SPI is needed IMO. Otherwise we'd have to create
> some sort of abstraction *in* OGM to ensure it still can be used in
> other environments than WF.

This makes sense, thanks!

>
> I think best would be to start with a PoC and see things are going to
> look like. Then we still can decide where it'd be best located or
> whether it's better to avoid re-use and accept some duplication.

+1

>
>
>
>
> 2016-05-12 13:24 GMT+02:00 Emmanuel Bernard <[hidden email]
> <mailto:[hidden email]>>:
>
>
>
>     On 11 mai 2016, at 16:02, Scott Marlow <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>     >> Hibernate OGM should still be usable without WF; So maybe there should
>     >> be a separate project/repo which defines an SPI to obtain/manage
>     >> connections and implementations for different NoSQL stores?
>     >
>     > Excellent suggestion,  perhaps the SPI could be under
>     > https://github.com/jboss, which is a common area for sharing.  Possible
>     > locations for creating the per NoSQL store implementations could be
>     > https://github.com/jboss or https://github.com/hibernate or
>     > https://github.com/wildfly.
>
>     I'm starting to think that this might be way overkill. If we are
>     creating a sub project just to share between 20 and 50 lines of code
>     per provider and the overhead code to abstract property
>     configuration to plus OGM and WF ones, we are losing more than gaining.
>
>     Thoughts ?
>
>     _______________________________________________
>     wildfly-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
123