update on WildFly NoSQL prototype integration...

classic Classic list List threaded Threaded
47 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Davide D'Alto
> Hibernate OGM should define an interface which is appropriate for its
own consumption; the Wildfly NoSQL subssystem can have its own
interface so to not depend on OGM, but they would be somewhat similar
for each given NoSQL technology we intend to support in this way.

Then JipiJapa can inject an adaptor into the OGM boostrap phase,
delegating from one to the other. So only the OGM specific JipiJapa
module would need to depend on both interfaces.
If this dependency is not desirable either, then I think we can live
with a non-typesafe generic provider of things.

+1
This looks like a nice trade-off.
It will allow us to create a POC that can eventually evolve into a separate project.



On Thu, May 12, 2016 at 1:41 PM, Sanne Grinovero <[hidden email]> wrote:
On 12 May 2016 at 12:24, Emmanuel Bernard <[hidden email]> wrote:
>
>
> On 11 mai 2016, at 16:02, Scott Marlow <[hidden email]> wrote:
>
>>> Hibernate OGM should still be usable without WF; So maybe there should
>>> be a separate project/repo which defines an SPI to obtain/manage
>>> connections and implementations for different NoSQL stores?
>>
>> Excellent suggestion,  perhaps the SPI could be under
>> https://github.com/jboss, which is a common area for sharing.  Possible
>> locations for creating the per NoSQL store implementations could be
>> https://github.com/jboss or https://github.com/hibernate or
>> https://github.com/wildfly.
>
> I'm starting to think that this might be way overkill. If we are creating a sub project just to share between 20 and 50 lines of code per provider and the overhead code to abstract property configuration to plus OGM and WF ones, we are losing more than gaining.
>
> Thoughts ?

I agree it's overkill, and have an alternative proposal:

Hibernate OGM should define an interface which is appropriate for its
own consumption; the Wildfly NoSQL subssystem can have its own
interface so to not depend on OGM, but they would be somewhat similar
for each given NoSQL technology we intend to support in this way.

Then JipiJapa can inject an adaptor into the OGM boostrap phase,
delegating from one to the other. So only the OGM specific JipiJapa
module would need to depend on both interfaces.
If this dependency is not desirable either, then I think we can live
with a non-typesafe generic provider of things.

Thanks,
Sanne

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Sanne Grinovero


On 05/12/2016 08:41 AM, Sanne Grinovero wrote:

> On 12 May 2016 at 12:24, Emmanuel Bernard <[hidden email]> wrote:
>>
>>
>> On 11 mai 2016, at 16:02, Scott Marlow <[hidden email]> wrote:
>>
>>>> Hibernate OGM should still be usable without WF; So maybe there should
>>>> be a separate project/repo which defines an SPI to obtain/manage
>>>> connections and implementations for different NoSQL stores?
>>>
>>> Excellent suggestion,  perhaps the SPI could be under
>>> https://github.com/jboss, which is a common area for sharing.  Possible
>>> locations for creating the per NoSQL store implementations could be
>>> https://github.com/jboss or https://github.com/hibernate or
>>> https://github.com/wildfly.
>>
>> I'm starting to think that this might be way overkill. If we are creating a sub project just to share between 20 and 50 lines of code per provider and the overhead code to abstract property configuration to plus OGM and WF ones, we are losing more than gaining.
>>
>> Thoughts ?
>
> I agree it's overkill, and have an alternative proposal:
>
> Hibernate OGM should define an interface which is appropriate for its
> own consumption; the Wildfly NoSQL subssystem can have its own
> interface so to not depend on OGM, but they would be somewhat similar
> for each given NoSQL technology we intend to support in this way.

Just to clarify what we were discussing.  The idea for sharing
connection setup, would of involved extracting that code from OGM, into
a separate jar that OGM + WildFly NoSQL could depend on.

>
> Then JipiJapa can inject an adaptor into the OGM boostrap phase,
> delegating from one to the other. So only the OGM specific JipiJapa
> module would need to depend on both interfaces.
> If this dependency is not desirable either, then I think we can live
> with a non-typesafe generic provider of things.

I'm not exactly sure of what you mean here about JipiJapa injecting an
adaptor into the OGM bootstrap phase.  Are you talking about how we want
to have a JipiJapa adapter for OGM, that (magically) leaks native NoSQL
dependencies into application deployments, depending on which NoSQL
backend store is targeted?  That may be different than what we are
talking about here but that also needs a SPI, so the JipiJapa adapter
can ask the JPA PersistenceProvider what the backend store is (or
something like that).

[1] talks about some of the (OGM/ORM related) dependency issues that are
also important to solve.  I'm not really sure of the best way to address
the [1] dependency issues that involve WildFly Clustering, WildFly JPA
container, WildFly JipiJapa adapters for Hibernate ORM,
Hibernate-Infinispan, Infinispan.

I agree with Gunnar, that we should continue with POC coding but also
really love the feedback/discussion going on here.  I'd like to ping
some additional teams as well soon (Infinispan dev, native NoSQL store
teams, others that may need to extend the WildFly NoSQL infrastructure).

Scott

[1] https://github.com/jboss/jboss-nosql/issues/3
>
> Thanks,
> Sanne
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Sanne Grinovero
On 12 May 2016 at 14:50, Scott Marlow <[hidden email]> wrote:

>
>
> On 05/12/2016 08:41 AM, Sanne Grinovero wrote:
>>
>> On 12 May 2016 at 12:24, Emmanuel Bernard <[hidden email]> wrote:
>>>
>>>
>>>
>>> On 11 mai 2016, at 16:02, Scott Marlow <[hidden email]> wrote:
>>>
>>>>> Hibernate OGM should still be usable without WF; So maybe there should
>>>>> be a separate project/repo which defines an SPI to obtain/manage
>>>>> connections and implementations for different NoSQL stores?
>>>>
>>>>
>>>> Excellent suggestion,  perhaps the SPI could be under
>>>> https://github.com/jboss, which is a common area for sharing.  Possible
>>>> locations for creating the per NoSQL store implementations could be
>>>> https://github.com/jboss or https://github.com/hibernate or
>>>> https://github.com/wildfly.
>>>
>>>
>>> I'm starting to think that this might be way overkill. If we are creating
>>> a sub project just to share between 20 and 50 lines of code per provider and
>>> the overhead code to abstract property configuration to plus OGM and WF
>>> ones, we are losing more than gaining.
>>>
>>> Thoughts ?
>>
>>
>> I agree it's overkill, and have an alternative proposal:
>>
>> Hibernate OGM should define an interface which is appropriate for its
>> own consumption; the Wildfly NoSQL subssystem can have its own
>> interface so to not depend on OGM, but they would be somewhat similar
>> for each given NoSQL technology we intend to support in this way.
>
>
> Just to clarify what we were discussing.  The idea for sharing connection
> setup, would of involved extracting that code from OGM, into a separate jar
> that OGM + WildFly NoSQL could depend on.

My impression is that the code to setup these connections trivial, so
there is no much benefit in sharing code.
Hibernate and WildFly generally read configuration setting in
different ways (and conventions) so it might even be preferable to
avoid sharing anything.

I'd aim at injecting the WildFly representation of "NoSQL connection
provider" into Hibernate: in this case all what we need to agree on is
an interface.

Hibernate OGM will also be able to interpret its own configuration
properties as an alternative.
This conceptually maps to the existing case of Hibernate ORM, in which
two different approaches can be used:
 - use a Datasource (managed by the app server)
 - configure all JDBC settings explicitly, and optionally connection
pooling settings.

In the WildFly case, what OGM needs is for you to inject a "NoSQL datasource".


>> Then JipiJapa can inject an adaptor into the OGM boostrap phase,
>> delegating from one to the other. So only the OGM specific JipiJapa
>> module would need to depend on both interfaces.
>> If this dependency is not desirable either, then I think we can live
>> with a non-typesafe generic provider of things.
>
>
> I'm not exactly sure of what you mean here about JipiJapa injecting an
> adaptor into the OGM bootstrap phase.  Are you talking about how we want to
> have a JipiJapa adapter for OGM, that (magically) leaks native NoSQL
> dependencies into application deployments, depending on which NoSQL backend
> store is targeted?  That may be different than what we are talking about
> here but that also needs a SPI, so the JipiJapa adapter can ask the JPA
> PersistenceProvider what the backend store is (or something like that).

Ideally JipiJapa should work similarly to how it does with a JTA
DataSource: some identification string, like an URL or a JNDI name,
will be listed in the persistence.xml and this needs to match the
configuration of some NoSQL datasource.

By "adaptor" I was merely referring to the fact that whatever JipiJapa
will give to Hibernate OGM, this should implement an interface from
OGM so that this can be consumed.

>
> [1] talks about some of the (OGM/ORM related) dependency issues that are
> also important to solve.  I'm not really sure of the best way to address the
> [1] dependency issues that involve WildFly Clustering, WildFly JPA
> container, WildFly JipiJapa adapters for Hibernate ORM,
> Hibernate-Infinispan, Infinispan.
>
> I agree with Gunnar, that we should continue with POC coding but also really
> love the feedback/discussion going on here.  I'd like to ping some
> additional teams as well soon (Infinispan dev, native NoSQL store teams,
> others that may need to extend the WildFly NoSQL infrastructure).
>
> Scott
>
> [1] https://github.com/jboss/jboss-nosql/issues/3
>>
>>
>> Thanks,
>> Sanne
>>
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: update on WildFly NoSQL prototype integration...

Emmanuel Bernard
In reply to this post by Sanne Grinovero
On Thu 2016-05-12 13:41, Sanne Grinovero wrote:

> On 12 May 2016 at 12:24, Emmanuel Bernard <[hidden email]> wrote:
> >
> >
> > On 11 mai 2016, at 16:02, Scott Marlow <[hidden email]> wrote:
> >
> >>> Hibernate OGM should still be usable without WF; So maybe there should
> >>> be a separate project/repo which defines an SPI to obtain/manage
> >>> connections and implementations for different NoSQL stores?
> >>
> >> Excellent suggestion,  perhaps the SPI could be under
> >> https://github.com/jboss, which is a common area for sharing.  Possible
> >> locations for creating the per NoSQL store implementations could be
> >> https://github.com/jboss or https://github.com/hibernate or
> >> https://github.com/wildfly.
> >
> > I'm starting to think that this might be way overkill. If we are creating a sub project just to share between 20 and 50 lines of code per provider and the overhead code to abstract property configuration to plus OGM and WF ones, we are losing more than gaining.
> >
> > Thoughts ?
>
> I agree it's overkill, and have an alternative proposal:
>
> Hibernate OGM should define an interface which is appropriate for its
> own consumption; the Wildfly NoSQL subssystem can have its own
> interface so to not depend on OGM, but they would be somewhat similar
> for each given NoSQL technology we intend to support in this way.
>
> Then JipiJapa can inject an adaptor into the OGM boostrap phase,
> delegating from one to the other. So only the OGM specific JipiJapa
> module would need to depend on both interfaces.
> If this dependency is not desirable either, then I think we can live
> with a non-typesafe generic provider of things.

But in the end it's just a freaking
if (inWF==true) { //fetch from JNDI } else { ///do it yourself }

Why do you want two layers of abstractions?
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Jesper Pedersen-2
I think that we will move forward now and prototype the transaction
enlistment for ACID NoSQL databases.  This is important to nail down
now, as we would like forward application compatibility.  We need to
choose now, whether MySLSB.myMethod() performs the Neo4j database update
immediately when (#4) tx.close() is called or when the JTA transaction
completes.  By now, I mean that we need to decide before a release of
WildFly that contains the NoSQL integration is released, so that we know
that future releases will be compatible.

I don't think that transaction enlistment should be allowed for BASE
NoSQL databases, which is why I mentioned ACID above.  With BASE
databases, the application code is responsible for dealing with dirty
data.  As such, I think it would be misleading to enlist BASE operations
into a JTA transaction.  This means that BASE NoSQL database profiles
will not allow 1-phase/2-phase transaction attribute to be specified.

We started to write some design notes [1] for NoSQL integration.  We
will update the design notes after hearing more feedback on the above.

Scott

[1] https://developer.jboss.org/wiki/DesignNotesForNoSQLIntegrationOnWildFly

On 05/10/2016 11:06 AM, Jesper Pedersen wrote:

>> Any links to existing IronJacamar code to share here?  I think that we
>> could prototype new transaction enlistment handling code, based on what
>> we currently have.
>>
>
> The IronJacamar code is here for Narayana:
>
> https://github.com/ironjacamar/ironjacamar/tree/master/core/src/main/java/org/ironjacamar/core/tx/narayana
>
>
> However, that is based on the Java EE Connector Architecture
> specification, so we have a *much* easier job.
>
> Lets take an example from the NoSQL world using Neo4J:
>
> http://neo4j.com/docs/api/java-driver/current/org/neo4j/driver/v1/Driver.html
>
>
> ...neo4j.LocalXAResource implements XAResource, org.jboss.tm.LastResource
>
>  public void start(Xid xid, int flags) throws XAException {
>    tx = session.beginTransaction();
>  }
>
>  public void commit(Xid xid, boolean onePhase) throws XAException {
>    tx.success();
>  }
>
>  public void rollback(Xid xid) throws XAException {
>    tx.failure();
>  }
>
> And the app, f.ex. in a SLSB method
>
> public class MySLSB ...
>
>   @Resource(mappedName="java:jboss/nosql/neo4j")
>   private Driver neo4j;
>
>   public void myMethod() {
> #1   Session s = neo4j.createSession();
>      s.run( "CREATE (n {name:'Bob'})" );
>
> #2   Transaction tx = s.beginTransaction();
>      tx.run( "CREATE (n {name:'Alice'})" );
>      tx.run( "CREATE (n {name:'Tina'})" );
>
> #3   tx.success();
>
> #4   tx.close();
> #5   s.close();
> #6   neo4j.close();
>   }
>
> However, as you can see there are a number of hidden things going on
> here if ".../neo4j" is deployed as "transaction=1phase".
>
> #1: Here you have to intercept the call, create the LocalXAResource
> instance and enlist it in the active transaction
>
> #2: Here you have to a delegator instance that only executes the run()
> calls
>
> #3: No-op call, happens upon EE transaction completion
>
> #4: No-op call. happens upon EE transaction completion
>
> #5: No-op call, happens in an enlisted Synchronization instance
> (afterCompletion) (done in #1 too)
>
> #6: No-op call, handled by subsystem
>
> The value-add is that #3 is tied into the overall EE transaction.
>
> If ".../neo4j" is deployed as "transaction=none", then all calls are on
> the "real" Neo4J objects.
>
> As you can see it isn't as simple as it may appear, and application flow
> could be different from what the developer expects, as #3 could be a
> real tx.failure() call in case of MARKED_FOR_ROLLBACK.
>
> There are def pros/cons of doing tx enlistment for this area...
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Scott Marlow
The NoSQL prototype now has transaction enlistment for Neo4j (one phase
wrapper).

Test example link [1] shows a CMT test bean method and [2] shows a BMT
test bean method.

Currently, in a JTA transaction the Neo4j Session survives ending the
JTA transaction.  It might be better to close the Neo4j Session when the
JTA transaction ends, which would require the application to get a new
Session, this would ensure that applications don't forget to close the
session, with the down side that with BMT, application code would need
to call the Neo4j Driver.session(), to get a new Session instance after
each JTA transaction ends.  After a session is closed, there is no
guarantee that another session can be obtained, as the next
Driver.session() may time out if the internal pool of Driver sessions,
is empty and the configured time limit
(neo4j.driver.acquireSessionTimeout) is exceeded (defaults to 30
seconds).  So, there are costs to automatically closing Neo4j Sessions
at JTA transaction end time.  Although, you could say that closing the
Session, after each transaction, gives other threads a change to use the
Session that isn't really closed, but instead is really returned to the
internal Driver sessions pool.

By the way, the Neo4j Session class, is what applications use to access
the database.

Thanks,
Scott

[1]
https://github.com/scottmarlow/wildfly/blob/nosql-dev9/testsuite/nosql/src/test/java/org/jboss/as/test/compat/nosql/neo4j/StatefulTestBean.java#L85

[2]
https://github.com/scottmarlow/wildfly/blob/nosql-dev9/testsuite/nosql/src/test/java/org/jboss/as/test/compat/nosql/neo4j/BMTStatefulTestBean.java#L54


On 06/29/2016 09:50 AM, Scott Marlow wrote:

> I think that we will move forward now and prototype the transaction
> enlistment for ACID NoSQL databases.  This is important to nail down
> now, as we would like forward application compatibility.  We need to
> choose now, whether MySLSB.myMethod() performs the Neo4j database update
> immediately when (#4) tx.close() is called or when the JTA transaction
> completes.  By now, I mean that we need to decide before a release of
> WildFly that contains the NoSQL integration is released, so that we know
> that future releases will be compatible.
>
> I don't think that transaction enlistment should be allowed for BASE
> NoSQL databases, which is why I mentioned ACID above.  With BASE
> databases, the application code is responsible for dealing with dirty
> data.  As such, I think it would be misleading to enlist BASE operations
> into a JTA transaction.  This means that BASE NoSQL database profiles
> will not allow 1-phase/2-phase transaction attribute to be specified.
>
> We started to write some design notes [1] for NoSQL integration.  We
> will update the design notes after hearing more feedback on the above.
>
> Scott
>
> [1] https://developer.jboss.org/wiki/DesignNotesForNoSQLIntegrationOnWildFly
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Jesper Pedersen-2
Hi,

On 07/08/2016 12:24 PM, Scott Marlow wrote:
> The NoSQL prototype now has transaction enlistment for Neo4j (one phase
> wrapper).
>
> Test example link [1] shows a CMT test bean method

You should add a comment that tx.success() / tx.failure() is tied to the
out-come of the JTA transaction. And tx.close() is done in afterCompletion.

> and [2] shows a BMT test bean method.
>

This test case highlights the problem of checking for enlistment in 2
places, as the Session is obtained before the transaction is started,
and there is no way to get the session reference from the transaction
instance.

So, you can manage the transaction lifecycle, but not the session lifecycle.

> Currently, in a JTA transaction the Neo4j Session survives ending the
> JTA transaction.  It might be better to close the Neo4j Session when the
> JTA transaction ends, which would require the application to get a new
> Session, this would ensure that applications don't forget to close the
> session, with the down side that with BMT, application code would need
> to call the Neo4j Driver.session(), to get a new Session instance after
> each JTA transaction ends.  After a session is closed, there is no
> guarantee that another session can be obtained, as the next
> Driver.session() may time out if the internal pool of Driver sessions,
> is empty and the configured time limit
> (neo4j.driver.acquireSessionTimeout) is exceeded (defaults to 30
> seconds).  So, there are costs to automatically closing Neo4j Sessions
> at JTA transaction end time.  Although, you could say that closing the
> Session, after each transaction, gives other threads a change to use the
> Session that isn't really closed, but instead is really returned to the
> internal Driver sessions pool.

As you are enlisting the Neo4J resources into the transaction, I think
that you should define driver.session() as the enlistment barrier, and
therefore also close the session in afterCompletion.

That makes it more clear in what scenarios the resources are managed.

In JDBC you inject the DataSource instance, and obtain a Connection, so
it would be parallel to this case Driver vs. Session.

This means that people needs to be more careful in the BMT case to get
the Session within an active transaction. JCA has
CachedConnectionManager to verify these cases, so you could add a
similar concept.

Best regards,
  Jesper

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Scott Marlow


On 07/11/2016 01:38 PM, Jesper Pedersen wrote:

> Hi,
>
> On 07/08/2016 12:24 PM, Scott Marlow wrote:
>> The NoSQL prototype now has transaction enlistment for Neo4j (one phase
>> wrapper).
>>
>> Test example link [1] shows a CMT test bean method
>
> You should add a comment that tx.success() / tx.failure() is tied to the
> out-come of the JTA transaction. And tx.close() is done in afterCompletion.

Comment added, thanks for noticing the absence. :)

>
>> and [2] shows a BMT test bean method.
>>
>
> This test case highlights the problem of checking for enlistment in 2
> places, as the Session is obtained before the transaction is started,
> and there is no way to get the session reference from the transaction
> instance.
>
> So, you can manage the transaction lifecycle, but not the session
> lifecycle.

In [1], when we get the Neo4j Session, we enlist it within the JTA
transaction.  We also enlist the Neo4j transaction at the same time
(during the call to Driver.session() inside of a JTA transaction.)  For
the duration of the JTA transaction, attempts to get the
Session/Transaction for the named Neo4j profile, will return the already
JTA enlisted Session/Transaction.

>
>> Currently, in a JTA transaction the Neo4j Session survives ending the
>> JTA transaction.  It might be better to close the Neo4j Session when the
>> JTA transaction ends, which would require the application to get a new
>> Session, this would ensure that applications don't forget to close the
>> session, with the down side that with BMT, application code would need
>> to call the Neo4j Driver.session(), to get a new Session instance after
>> each JTA transaction ends.  After a session is closed, there is no
>> guarantee that another session can be obtained, as the next
>> Driver.session() may time out if the internal pool of Driver sessions,
>> is empty and the configured time limit
>> (neo4j.driver.acquireSessionTimeout) is exceeded (defaults to 30
>> seconds).  So, there are costs to automatically closing Neo4j Sessions
>> at JTA transaction end time.  Although, you could say that closing the
>> Session, after each transaction, gives other threads a change to use the
>> Session that isn't really closed, but instead is really returned to the
>> internal Driver sessions pool.
>
> As you are enlisting the Neo4J resources into the transaction, I think
> that you should define driver.session() as the enlistment barrier, and
> therefore also close the session in afterCompletion.

I pushed an update [1] that includes auto closing the session, when the
transaction ends.

I didn't add a Synchronization yet, to handle closing the session when
the transaction manager doesn't call XAResourceWrapper.commit/rollback.
I agree that will introduce an extra layer of safety to ensure that
resources are closed.

>
> That makes it more clear in what scenarios the resources are managed.
>
> In JDBC you inject the DataSource instance, and obtain a Connection, so
> it would be parallel to this case Driver vs. Session.
>
> This means that people needs to be more careful in the BMT case to get
> the Session within an active transaction. JCA has
> CachedConnectionManager to verify these cases, so you could add a
> similar concept.

Attempts to call Neo4j transaction/session "close", within the JTA
transaction are ignored.  How about calls to "close" after the JTA
transaction ends, should they also be ignored?  Is there a JCA
equivalent of ignoring "close" after the transaction ends?  I think that
Hibernate ORM is or was, closing DataSource's after the transaction
ends, so I assume that JCA ignores "close" after the JTA transaction ends.

Thanks again for the feedback!

Scott

[1]
https://github.com/scottmarlow/wildfly/commit/bfecbe881e6a16548b9712b3e52ff7acdd3142b4
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Jesper Pedersen-2
On 07/12/2016 11:36 AM, Scott Marlow wrote:

>> As you are enlisting the Neo4J resources into the transaction, I think
>> that you should define driver.session() as the enlistment barrier, and
>> therefore also close the session in afterCompletion.
>
> I pushed an update [1] that includes auto closing the session, when the
> transaction ends.
>
> I didn't add a Synchronization yet, to handle closing the session when
> the transaction manager doesn't call XAResourceWrapper.commit/rollback.
> I agree that will introduce an extra layer of safety to ensure that
> resources are closed.
>

I think you can use the XAResource instance for that at the moment.

>>
>> That makes it more clear in what scenarios the resources are managed.
>>
>> In JDBC you inject the DataSource instance, and obtain a Connection, so
>> it would be parallel to this case Driver vs. Session.
>>
>> This means that people needs to be more careful in the BMT case to get
>> the Session within an active transaction. JCA has
>> CachedConnectionManager to verify these cases, so you could add a
>> similar concept.
>
> Attempts to call Neo4j transaction/session "close", within the JTA
> transaction are ignored.  How about calls to "close" after the JTA
> transaction ends, should they also be ignored?

Hopefully the underlying implementation will ignore multiple close()
calls, and throw an exception on any other call.

You will have to test the behaviour of each NoSQL store.

>  Is there a JCA
> equivalent of ignoring "close" after the transaction ends?  I think that
> Hibernate ORM is or was, closing DataSource's after the transaction
> ends, so I assume that JCA ignores "close" after the JTA transaction ends.
>

You probably mean Connection.close() -- if all connections are closed in
afterCompletion then the connection is returned to the pool. Otherwise,
it waits until ref count becomes 0, e.g. c.close() after the transaction
has ended.

http://www.ironjacamar.org/doc/roadto12/txtracking.html

Best regards,
  Jesper

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Scott Marlow


On 07/13/2016 10:44 AM, Jesper Pedersen wrote:

> On 07/12/2016 11:36 AM, Scott Marlow wrote:
>>> As you are enlisting the Neo4J resources into the transaction, I think
>>> that you should define driver.session() as the enlistment barrier, and
>>> therefore also close the session in afterCompletion.
>>
>> I pushed an update [1] that includes auto closing the session, when the
>> transaction ends.
>>
>> I didn't add a Synchronization yet, to handle closing the session when
>> the transaction manager doesn't call XAResourceWrapper.commit/rollback.
>> I agree that will introduce an extra layer of safety to ensure that
>> resources are closed.
>>
>
> I think you can use the XAResource instance for that at the moment.

Yes, we are doing this currently.

>
>>>
>>> That makes it more clear in what scenarios the resources are managed.
>>>
>>> In JDBC you inject the DataSource instance, and obtain a Connection, so
>>> it would be parallel to this case Driver vs. Session.
>>>
>>> This means that people needs to be more careful in the BMT case to get
>>> the Session within an active transaction. JCA has
>>> CachedConnectionManager to verify these cases, so you could add a
>>> similar concept.
>>
>> Attempts to call Neo4j transaction/session "close", within the JTA
>> transaction are ignored.  How about calls to "close" after the JTA
>> transaction ends, should they also be ignored?
>
> Hopefully the underlying implementation will ignore multiple close()
> calls, and throw an exception on any other call.

isOpen is currently returning false instead of throwing an exception.
We are ignoring multiple close() calls and do throw an exception on any
other call.

>
> You will have to test the behaviour of each NoSQL store.
>
>>  Is there a JCA
>> equivalent of ignoring "close" after the transaction ends?  I think that
>> Hibernate ORM is or was, closing DataSource's after the transaction
>> ends, so I assume that JCA ignores "close" after the JTA transaction
>> ends.
>>
>
> You probably mean Connection.close() -- if all connections are closed in
> afterCompletion then the connection is returned to the pool. Otherwise,
> it waits until ref count becomes 0, e.g. c.close() after the transaction
> has ended.
>
> http://www.ironjacamar.org/doc/roadto12/txtracking.html
>
> Best regards,
>  Jesper
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Gunnar Morling
Hi Scott,

I had a short look at the tests you pointed to; in the one using BMT, why is it that the UserTransaction *and* the Neo4j TX API are used? I'd have expected it to be only one *or* the other?

In the one using CMT, there are also calls to the Neo4j TX API which apparently are ignored. What's the rationale behind that? Wouldn't it make more sense to raise an exception, similar to when EntityManager#getTransaction() is invoked for a JTA entity manager?

In the CMT case, did you consider to inject the Neo4j Session instead of Driver into the bean, again akin to injection of JTA entity managers?

All in all, I'm wondering whether things couldn't be handled analogous to the usage of entity managers:

* For a JTA-enabled NoSQL persistence unit, obtain the Session via injection (its lifecycle is controlled by the container and bound to the managed transaction), any manual interaction with the Neo4j TX API is forbidden, instead the container manages transactions or the user is (via UserTransaction)
* For a non-JTA-enabled NoSQL persistence unit, obtain the Driver via injection, and the user is in charge of Session lifecycle and transaction control

I hope that'd make sense for actually transactional stores such as Neo4j, for non-TX stores such as MongoDB thinks look different.

--Gunnar



2016-07-13 16:25 GMT+01:00 Scott Marlow <[hidden email]>:


On 07/13/2016 10:44 AM, Jesper Pedersen wrote:
> On 07/12/2016 11:36 AM, Scott Marlow wrote:
>>> As you are enlisting the Neo4J resources into the transaction, I think
>>> that you should define driver.session() as the enlistment barrier, and
>>> therefore also close the session in afterCompletion.
>>
>> I pushed an update [1] that includes auto closing the session, when the
>> transaction ends.
>>
>> I didn't add a Synchronization yet, to handle closing the session when
>> the transaction manager doesn't call XAResourceWrapper.commit/rollback.
>> I agree that will introduce an extra layer of safety to ensure that
>> resources are closed.
>>
>
> I think you can use the XAResource instance for that at the moment.

Yes, we are doing this currently.

>
>>>
>>> That makes it more clear in what scenarios the resources are managed.
>>>
>>> In JDBC you inject the DataSource instance, and obtain a Connection, so
>>> it would be parallel to this case Driver vs. Session.
>>>
>>> This means that people needs to be more careful in the BMT case to get
>>> the Session within an active transaction. JCA has
>>> CachedConnectionManager to verify these cases, so you could add a
>>> similar concept.
>>
>> Attempts to call Neo4j transaction/session "close", within the JTA
>> transaction are ignored.  How about calls to "close" after the JTA
>> transaction ends, should they also be ignored?
>
> Hopefully the underlying implementation will ignore multiple close()
> calls, and throw an exception on any other call.

isOpen is currently returning false instead of throwing an exception.
We are ignoring multiple close() calls and do throw an exception on any
other call.

>
> You will have to test the behaviour of each NoSQL store.
>
>>  Is there a JCA
>> equivalent of ignoring "close" after the transaction ends?  I think that
>> Hibernate ORM is or was, closing DataSource's after the transaction
>> ends, so I assume that JCA ignores "close" after the JTA transaction
>> ends.
>>
>
> You probably mean Connection.close() -- if all connections are closed in
> afterCompletion then the connection is returned to the pool. Otherwise,
> it waits until ref count becomes 0, e.g. c.close() after the transaction
> has ended.
>
> http://www.ironjacamar.org/doc/roadto12/txtracking.html
>
> Best regards,
>  Jesper
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Emmanuel Bernard
I haven't looked in detail but +1 to what Gunnar says. 
I would add that the WF Neo4j confuguration should have an option to either treat Neo4j as a JTA enlisted source or not. 
In the former, the behavior would be as Gunnar suggests. In the latter, direct Neo4j session creation and transaction would be used. 

On 13 juil. 2016, at 19:16, Gunnar Morling <[hidden email]> wrote:

Hi Scott,

I had a short look at the tests you pointed to; in the one using BMT, why is it that the UserTransaction *and* the Neo4j TX API are used? I'd have expected it to be only one *or* the other?

In the one using CMT, there are also calls to the Neo4j TX API which apparently are ignored. What's the rationale behind that? Wouldn't it make more sense to raise an exception, similar to when EntityManager#getTransaction() is invoked for a JTA entity manager?

In the CMT case, did you consider to inject the Neo4j Session instead of Driver into the bean, again akin to injection of JTA entity managers?

All in all, I'm wondering whether things couldn't be handled analogous to the usage of entity managers:

* For a JTA-enabled NoSQL persistence unit, obtain the Session via injection (its lifecycle is controlled by the container and bound to the managed transaction), any manual interaction with the Neo4j TX API is forbidden, instead the container manages transactions or the user is (via UserTransaction)
* For a non-JTA-enabled NoSQL persistence unit, obtain the Driver via injection, and the user is in charge of Session lifecycle and transaction control

I hope that'd make sense for actually transactional stores such as Neo4j, for non-TX stores such as MongoDB thinks look different.

--Gunnar



2016-07-13 16:25 GMT+01:00 Scott Marlow <[hidden email]>:


On 07/13/2016 10:44 AM, Jesper Pedersen wrote:
> On 07/12/2016 11:36 AM, Scott Marlow wrote:
>>> As you are enlisting the Neo4J resources into the transaction, I think
>>> that you should define driver.session() as the enlistment barrier, and
>>> therefore also close the session in afterCompletion.
>>
>> I pushed an update [1] that includes auto closing the session, when the
>> transaction ends.
>>
>> I didn't add a Synchronization yet, to handle closing the session when
>> the transaction manager doesn't call XAResourceWrapper.commit/rollback.
>> I agree that will introduce an extra layer of safety to ensure that
>> resources are closed.
>>
>
> I think you can use the XAResource instance for that at the moment.

Yes, we are doing this currently.

>
>>>
>>> That makes it more clear in what scenarios the resources are managed.
>>>
>>> In JDBC you inject the DataSource instance, and obtain a Connection, so
>>> it would be parallel to this case Driver vs. Session.
>>>
>>> This means that people needs to be more careful in the BMT case to get
>>> the Session within an active transaction. JCA has
>>> CachedConnectionManager to verify these cases, so you could add a
>>> similar concept.
>>
>> Attempts to call Neo4j transaction/session "close", within the JTA
>> transaction are ignored.  How about calls to "close" after the JTA
>> transaction ends, should they also be ignored?
>
> Hopefully the underlying implementation will ignore multiple close()
> calls, and throw an exception on any other call.

isOpen is currently returning false instead of throwing an exception.
We are ignoring multiple close() calls and do throw an exception on any
other call.

>
> You will have to test the behaviour of each NoSQL store.
>
>>  Is there a JCA
>> equivalent of ignoring "close" after the transaction ends?  I think that
>> Hibernate ORM is or was, closing DataSource's after the transaction
>> ends, so I assume that JCA ignores "close" after the JTA transaction
>> ends.
>>
>
> You probably mean Connection.close() -- if all connections are closed in
> afterCompletion then the connection is returned to the pool. Otherwise,
> it waits until ref count becomes 0, e.g. c.close() after the transaction
> has ended.
>
> http://www.ironjacamar.org/doc/roadto12/txtracking.html
>
> Best regards,
>  Jesper
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Gunnar Morling
Hi Gunnar/Emmanuel,

The Ne04j profile can either be "transaction=1pc" (JTA transaction
enlisted) or "transaction=none" (will not be enlisted into JTA
transaction).  I think this covers what Emmanuel suggested (I didn't get
Emmanuel's email yet for some unknown reason).

It sounds like we aren't yet connecting on what the WF Neo4j profile is
doing, so I'll respond below and we can continue from there. :)

On 07/13/2016 01:16 PM, Gunnar Morling wrote:
> Hi Scott,
>
> I had a short look at the tests you pointed to; in the one using BMT,
> why is it that the UserTransaction *and* the Neo4j TX API are used? I'd
> have expected it to be only one *or* the other?

With both container managed transactions and bean managed transactions
(BMT), a JTA transaction is in control of the transaction.  With BMT,
UserTransaction is used by the application code to start/end the
transaction.  In either case (BMT or CMT), the Neo4j TX API is
controlled by the JTA transaction.

If there isn't an active JTA transaction, the Ne04j TX API is in control
of the Neo4j transaction.  With one caveat, that the Ne04j transaction
should be ended by the application code, before entering into a JTA
transaction (since it won't be joined into the JTA transaction).

Users can also disable the JTA transaction enlistment by using a NoSQL
profile with "transaction=none", instead of "transaction=1pc".

>
> In the one using CMT, there are also calls to the Neo4j TX API which
> apparently are ignored.

With CMT + BMT, the JTA transaction is in control, when the JTA
transaction ends, the Neo4j transaction ends as well and not before
then.  If the application calls the Neo4j TX API, our TXProxy ignores
calls to TX.success/failure/close, other calls are passed through to the
underlying Neo4j Transaction object (e.g. run(String statement)), since
that is where the work actually occurs.

When the JTA transaction commits, the underlying Neo4j tx.success() +
tx.close() are called.  If the JTA transaction rolls back, the
underlying Neo4j tx.failure() + tx.close() are called.

> What's the rationale behind that?

To enlist the Neo4j transaction into the JTA transaction via a one phase
wrapper.

> Wouldn't it
> make more sense to raise an exception, similar to when
> EntityManager#getTransaction() is invoked for a JTA entity manager?

If the NoSQL profile is JTA enabled, it should be enlisted into the JTA
transaction, allowing the application to control all of the database
operations that occur within the JTA transaction.

We also have the equivalent of JPA RESOURCE_LOCAL, via NoSQL profiles
that are not JTA enabled ("transaction=none" in the profile definition).

>
> In the CMT case, did you consider to inject the Neo4j Session instead of
> Driver into the bean, again akin to injection of JTA entity managers?

It is possible that we could also look at TransactionScoped injection of
sessions, but not sure yet.  We started with (RequestScoped) Session
injection but commented that out [1] when we added JTA enlistment.
TransactionScoped injection of Sessions could be yet another way to
write CMT/BMT code.

>
> All in all, I'm wondering whether things couldn't be handled analogous
> to the usage of entity managers:
>
> * For a JTA-enabled NoSQL persistence unit, obtain the Session via
> injection (its lifecycle is controlled by the container and bound to the
> managed transaction), any manual interaction with the Neo4j TX API is
> forbidden, instead the container manages transactions or the user is
> (via UserTransaction)
> * For a non-JTA-enabled NoSQL persistence unit, obtain the Driver via
> injection, and the user is in charge of Session lifecycle and
> transaction control

Well, you haven't yet seen the above responses yet, I think the
TransactionScoped Session injection idea is good but also introduces a
different way to write CMT/BMT code for Ne04j.  I disagree on making the
current (WF NoSQL/Ne04j TX enlistment) approach illegal, as I don't yet
see a problem with the current approach.

Are there difficulties with Hibernate OGM that you anticipate with
either approach of JTA TX enlistment for Neo4j?

>
> I hope that'd make sense for actually transactional stores such as
> Neo4j, for non-TX stores such as MongoDB thinks look different.

Agreed, for MongoDB, we treat that as "transaction=none", meaning no JTA
transaction enlistment for MongoDB.

>
> --Gunnar
>

Scott

[1]
https://github.com/scottmarlow/wildfly/blob/nosql-dev9/nosql/neo4j/src/main/java/org/wildfly/extension/nosql/cdi/Neo4jExtension.java#L172
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Emmanuel Bernard
I'm lost.

What is the expected code a developer will write ?

If that' something along the lines of:

jta.beginTx() // declaratively or imperatively
neo4j.beginTx()
...
neo4j.commit()
jta.commit() //declaratively or imperatively

then I find it very user unfriendly to force the code to call both Tx
APIs. It should be one or the other.

On Wed 2016-07-13 15:05, Scott Marlow wrote:

> Hi Gunnar/Emmanuel,
>
> The Ne04j profile can either be "transaction=1pc" (JTA transaction enlisted)
> or "transaction=none" (will not be enlisted into JTA transaction).  I think
> this covers what Emmanuel suggested (I didn't get Emmanuel's email yet for
> some unknown reason).
>
> It sounds like we aren't yet connecting on what the WF Neo4j profile is
> doing, so I'll respond below and we can continue from there. :)
>
> On 07/13/2016 01:16 PM, Gunnar Morling wrote:
> > Hi Scott,
> >
> > I had a short look at the tests you pointed to; in the one using BMT,
> > why is it that the UserTransaction *and* the Neo4j TX API are used? I'd
> > have expected it to be only one *or* the other?
>
> With both container managed transactions and bean managed transactions
> (BMT), a JTA transaction is in control of the transaction.  With BMT,
> UserTransaction is used by the application code to start/end the
> transaction.  In either case (BMT or CMT), the Neo4j TX API is controlled by
> the JTA transaction.
>
> If there isn't an active JTA transaction, the Ne04j TX API is in control of
> the Neo4j transaction.  With one caveat, that the Ne04j transaction should
> be ended by the application code, before entering into a JTA transaction
> (since it won't be joined into the JTA transaction).
>
> Users can also disable the JTA transaction enlistment by using a NoSQL
> profile with "transaction=none", instead of "transaction=1pc".
>
> >
> > In the one using CMT, there are also calls to the Neo4j TX API which
> > apparently are ignored.
>
> With CMT + BMT, the JTA transaction is in control, when the JTA transaction
> ends, the Neo4j transaction ends as well and not before then.  If the
> application calls the Neo4j TX API, our TXProxy ignores calls to
> TX.success/failure/close, other calls are passed through to the underlying
> Neo4j Transaction object (e.g. run(String statement)), since that is where
> the work actually occurs.
>
> When the JTA transaction commits, the underlying Neo4j tx.success() +
> tx.close() are called.  If the JTA transaction rolls back, the underlying
> Neo4j tx.failure() + tx.close() are called.
>
> > What's the rationale behind that?
>
> To enlist the Neo4j transaction into the JTA transaction via a one phase
> wrapper.
>
> > Wouldn't it
> > make more sense to raise an exception, similar to when
> > EntityManager#getTransaction() is invoked for a JTA entity manager?
>
> If the NoSQL profile is JTA enabled, it should be enlisted into the JTA
> transaction, allowing the application to control all of the database
> operations that occur within the JTA transaction.
>
> We also have the equivalent of JPA RESOURCE_LOCAL, via NoSQL profiles that
> are not JTA enabled ("transaction=none" in the profile definition).
>
> >
> > In the CMT case, did you consider to inject the Neo4j Session instead of
> > Driver into the bean, again akin to injection of JTA entity managers?
>
> It is possible that we could also look at TransactionScoped injection of
> sessions, but not sure yet.  We started with (RequestScoped) Session
> injection but commented that out [1] when we added JTA enlistment.
> TransactionScoped injection of Sessions could be yet another way to write
> CMT/BMT code.
>
> >
> > All in all, I'm wondering whether things couldn't be handled analogous
> > to the usage of entity managers:
> >
> > * For a JTA-enabled NoSQL persistence unit, obtain the Session via
> > injection (its lifecycle is controlled by the container and bound to the
> > managed transaction), any manual interaction with the Neo4j TX API is
> > forbidden, instead the container manages transactions or the user is
> > (via UserTransaction)
> > * For a non-JTA-enabled NoSQL persistence unit, obtain the Driver via
> > injection, and the user is in charge of Session lifecycle and
> > transaction control
>
> Well, you haven't yet seen the above responses yet, I think the
> TransactionScoped Session injection idea is good but also introduces a
> different way to write CMT/BMT code for Ne04j.  I disagree on making the
> current (WF NoSQL/Ne04j TX enlistment) approach illegal, as I don't yet see
> a problem with the current approach.
>
> Are there difficulties with Hibernate OGM that you anticipate with either
> approach of JTA TX enlistment for Neo4j?
>
> >
> > I hope that'd make sense for actually transactional stores such as
> > Neo4j, for non-TX stores such as MongoDB thinks look different.
>
> Agreed, for MongoDB, we treat that as "transaction=none", meaning no JTA
> transaction enlistment for MongoDB.
>
> >
> > --Gunnar
> >
>
> Scott
>
> [1] https://github.com/scottmarlow/wildfly/blob/nosql-dev9/nosql/neo4j/src/main/java/org/wildfly/extension/nosql/cdi/Neo4jExtension.java#L172
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Scott Marlow
Thanks for the feedback.

On 07/18/2016 09:24 AM, Emmanuel Bernard wrote:

> I'm lost.
>
> What is the expected code a developer will write ?
>
> If that' something along the lines of:
>
> jta.beginTx() // declaratively or imperatively
> neo4j.beginTx()
> ...
> neo4j.commit()
> jta.commit() //declaratively or imperatively

They are not limited to writing the above code.  In addition, the
developer can also avoid calling neo4j.beginTx()/neo4j.commit().
Something like:

jta.beginTx() // declaratively or imperatively
session = driver.session() // internally does session.beginTransaction()
...
jta.commit() //declaratively or imperatively

The calls to neo4j.beginTx()/commit() are ignored.

The choice of ignoring those calls, is made against the alternative of
throwing an exception.  The reason for ignoring the neo4j transactional
calls, is to be more compatible with existing ne04j code that may be
using the neo4j transactional code, with the goal that the JTA
transaction controls the underlying ne04j transaction.  Neo4j
transactional code also runs the statements against the neo4j
transaction class, instead of the neo4j session class.

>
> then I find it very user unfriendly to force the code to call both Tx
> APIs. It should be one or the other.

Do you really want to rewrite your neo4j code to avoid the neo4j tx APIs
when enlisted into a JTA transaction?  Or should the neo4j tx API calls
be ignored when enlisted into a JTA transaction (as we are doing now)?

If we throw exceptions from Transaction.run(), developers may have to
rewrite more code for use with JTA.

If this is still confusing, we can have a call to discuss. :)

Scott
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Scott Marlow
>
> The choice of ignoring those calls, is made against the alternative of
> throwing an exception.  The reason for ignoring the neo4j transactional
> calls, is to be more compatible with existing ne04j code that may be
> using the neo4j transactional code, with the goal that the JTA
> transaction controls the underlying ne04j transaction.  Neo4j
> transactional code also runs the statements against the neo4j
> transaction class, instead of the neo4j session class.
>

In a JTA transaction, instead of ignoring calls to
org.neo4j.driver.v1.Transaction.failure(), we could call
setRollbackOnly() on the JTA transaction.  So, the existing neo4j code
can influence a JTA rollback.  IMO, this would improve our compatibility
with existing neo4j.

In our neo4j XAResource wrapper, we handle commit() by calling neo4j
Transaction.success().  We handle rollback() by calling neo4j
Transaction.failure().  For both cases, we call Transaction.close().

Scott
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Emmanuel Bernard
In reply to this post by Scott Marlow
On Mon 2016-07-18 12:20, Scott Marlow wrote:

> Thanks for the feedback.
>
> On 07/18/2016 09:24 AM, Emmanuel Bernard wrote:
> > I'm lost.
> >
> > What is the expected code a developer will write ?
> >
> > If that' something along the lines of:
> >
> > jta.beginTx() // declaratively or imperatively
> > neo4j.beginTx()
> > ...
> > neo4j.commit()
> > jta.commit() //declaratively or imperatively
>
> They are not limited to writing the above code.  In addition, the developer
> can also avoid calling neo4j.beginTx()/neo4j.commit(). Something like:
>
> jta.beginTx() // declaratively or imperatively
> session = driver.session() // internally does session.beginTransaction()
> ...
> jta.commit() //declaratively or imperatively
>
> The calls to neo4j.beginTx()/commit() are ignored.
>
> The choice of ignoring those calls, is made against the alternative of
> throwing an exception.  The reason for ignoring the neo4j transactional
> calls, is to be more compatible with existing ne04j code that may be using
> the neo4j transactional code, with the goal that the JTA transaction
> controls the underlying ne04j transaction.  Neo4j transactional code also
> runs the statements against the neo4j transaction class, instead of the
> neo4j session class.

You mean having existing code work with the new JTA transaction net?
I don't think you can safely support that. What if the code does

@Transactional
public void doThings() {
   session = drive.session();
   session.beginTransaction();
   try {
       // do things
   catch (Exception e) {
       // swallow exception because we will fix things
       session.rollback();
       session.beginTransaction();
       // do something else
       session.commit();
    }
}

Such an existing code won't work if JTA in involved and
session.beginTrnasaction() / commit() are ignored.

So only code written with the idea of the JTA net in mind will be safe.

> >
> > then I find it very user unfriendly to force the code to call both Tx
> > APIs. It should be one or the other.
>
> Do you really want to rewrite your neo4j code to avoid the neo4j tx APIs
> when enlisted into a JTA transaction?  Or should the neo4j tx API calls be
> ignored when enlisted into a JTA transaction (as we are doing now)?

See my above example of already existing code.

For new code that you want to run both in SE and Wildfly, that's an
interesting question but how would you do it in practice? A try catch
but rethrowing the exception in case JTA is involved and needs to be
notified of the tx rollback?
I'm interested to see concrete code because if doable, it will have lots
of constraints and be limited to very specific patterns of code I think.
But again, let's show each other code.

Emmanuel
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Emmanuel Bernard
In reply to this post by Scott Marlow
On Mon 2016-07-18 13:53, Scott Marlow wrote:

> >
> > The choice of ignoring those calls, is made against the alternative of
> > throwing an exception.  The reason for ignoring the neo4j transactional
> > calls, is to be more compatible with existing ne04j code that may be
> > using the neo4j transactional code, with the goal that the JTA
> > transaction controls the underlying ne04j transaction.  Neo4j
> > transactional code also runs the statements against the neo4j
> > transaction class, instead of the neo4j session class.
> >
>
> In a JTA transaction, instead of ignoring calls to
> org.neo4j.driver.v1.Transaction.failure(), we could call setRollbackOnly()
> on the JTA transaction.  So, the existing neo4j code can influence a JTA
> rollback.  IMO, this would improve our compatibility with existing neo4j.

Yes that is an improvement but I don't think it would help in the
example I showed in my previous email.
The big question is do we want to offer something that works sometimes,
or do we want to offer something that does not work but is explicit
about it?
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Scott Marlow
In reply to this post by Emmanuel Bernard


On 07/19/2016 04:17 AM, Emmanuel Bernard wrote:

> On Mon 2016-07-18 12:20, Scott Marlow wrote:
>> Thanks for the feedback.
>>
>> On 07/18/2016 09:24 AM, Emmanuel Bernard wrote:
>>> I'm lost.
>>>
>>> What is the expected code a developer will write ?
>>>
>>> If that' something along the lines of:
>>>
>>> jta.beginTx() // declaratively or imperatively
>>> neo4j.beginTx()
>>> ...
>>> neo4j.commit()
>>> jta.commit() //declaratively or imperatively
>>
>> They are not limited to writing the above code.  In addition, the developer
>> can also avoid calling neo4j.beginTx()/neo4j.commit(). Something like:
>>
>> jta.beginTx() // declaratively or imperatively
>> session = driver.session() // internally does session.beginTransaction()
>> ...
>> jta.commit() //declaratively or imperatively
>>
>> The calls to neo4j.beginTx()/commit() are ignored.
>>
>> The choice of ignoring those calls, is made against the alternative of
>> throwing an exception.  The reason for ignoring the neo4j transactional
>> calls, is to be more compatible with existing ne04j code that may be using
>> the neo4j transactional code, with the goal that the JTA transaction
>> controls the underlying ne04j transaction.  Neo4j transactional code also
>> runs the statements against the neo4j transaction class, instead of the
>> neo4j session class.
>
> You mean having existing code work with the new JTA transaction net?
> I don't think you can safely support that. What if the code does
>
> @Transactional
> public void doThings() {
>    session = drive.session();
>    session.beginTransaction();
// should be:
Transaction transaction = session.beginTransaction();
>    try {
>        // do things
>    catch (Exception e) {
>        // swallow exception because we will fix things
>        session.rollback();
// should be:
          transaction.failure();
>        session.beginTransaction();
          transaction = session.beginTransaction();
>        // do something else
>        session.commit();
>     }
> }
>
> Such an existing code won't work if JTA in involved and
> session.beginTrnasaction() / commit() are ignored.

With the current prototype, the above existing code won't work as
originally intended (with regard to what the original developer had in
mind) with JTA enlistment.

>
> So only code written with the idea of the JTA net in mind will be safe.

Clearly, any existing neo4j code that expects to fail the current neo4j
transaction, then start a new neo4j transaction to fix things, will not
work in the JTA net.  As the fixing of things in the JTA net, needs to
occur before marking the JTA transaction for rollback only *or* by
fixing things after the JTA transaction rolls back.

Are there any other holes that come to mind, with regard to existing
neo4j transaction code that won't work as original intended in the JTA net?

>
>>>
>>> then I find it very user unfriendly to force the code to call both Tx
>>> APIs. It should be one or the other.
>>
>> Do you really want to rewrite your neo4j code to avoid the neo4j tx APIs
>> when enlisted into a JTA transaction?  Or should the neo4j tx API calls be
>> ignored when enlisted into a JTA transaction (as we are doing now)?
>
> See my above example of already existing code.
>
> For new code that you want to run both in SE and Wildfly, that's an
> interesting question but how would you do it in practice? A try catch
> but rethrowing the exception in case JTA is involved and needs to be
> notified of the tx rollback?

The developer could rewrite their "workaround tx failure" code to avoid
further calls to the neo4j tx api, but this is kind of bad as they only
should do this when the neo4j transaction is enlisted into the JTA
transaction, but not when enlistment is disabled.  Some neo4j connection
profiles will be configured to allow JTA enlistment (transaction=1pc)
and some will be configured to not allow JTA enlistment
(transaction=none), so we can expect common application code will likely
be reused for both JTA + non-JTA.  Seems to me that SE is always the
same as the WildFly neo4j connection profile with "transaction=none", so
we have the same problem with SE + WildFly, trying to run the same user
code that is expected to work with JTA enlistment (neo4j connection
profile has "transaction=1pc").

I don't know of any CDI magic that helps, as if we add support for
inject of a javax.transaction.TransactionScoped neo4j session, that
works the same as a session obtained from Driver.session().

> I'm interested to see concrete code because if doable, it will have lots
> of constraints and be limited to very specific patterns of code I think.
> But again, let's show each other code.

That should help us explore the compatibility with existing code idea
and pick which approach to use (e.g. whether explicit neo4j transaction
use in a JTA enlisted transaction should throw an exception or ignore
the explicit transaction use as we are doing now in the prototype).

Should we also post on https://groups.google.com/forum/#!forum/neo4j to
ask for input on JTA enlistment?

>
> Emmanuel
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: (transaction enlistment for ACID but none for BASE...) update on WildFly NoSQL prototype integration...

Scott Marlow


On 07/19/2016 10:49 AM, Scott Marlow wrote:

>
>
> On 07/19/2016 04:17 AM, Emmanuel Bernard wrote:
>> On Mon 2016-07-18 12:20, Scott Marlow wrote:
>>> Thanks for the feedback.
>>>
>>> On 07/18/2016 09:24 AM, Emmanuel Bernard wrote:
>>>> I'm lost.
>>>>
>>>> What is the expected code a developer will write ?
>>>>
>>>> If that' something along the lines of:
>>>>
>>>> jta.beginTx() // declaratively or imperatively
>>>> neo4j.beginTx()
>>>> ...
>>>> neo4j.commit()
>>>> jta.commit() //declaratively or imperatively
>>>
>>> They are not limited to writing the above code.  In addition, the developer
>>> can also avoid calling neo4j.beginTx()/neo4j.commit(). Something like:
>>>
>>> jta.beginTx() // declaratively or imperatively
>>> session = driver.session() // internally does session.beginTransaction()
>>> ...
>>> jta.commit() //declaratively or imperatively
>>>
>>> The calls to neo4j.beginTx()/commit() are ignored.
>>>
>>> The choice of ignoring those calls, is made against the alternative of
>>> throwing an exception.  The reason for ignoring the neo4j transactional
>>> calls, is to be more compatible with existing ne04j code that may be using
>>> the neo4j transactional code, with the goal that the JTA transaction
>>> controls the underlying ne04j transaction.  Neo4j transactional code also
>>> runs the statements against the neo4j transaction class, instead of the
>>> neo4j session class.
>>
>> You mean having existing code work with the new JTA transaction net?
>> I don't think you can safely support that. What if the code does
>>
>> @Transactional
>> public void doThings() {
>>    session = drive.session();
>>    session.beginTransaction();
> // should be:
> Transaction transaction = session.beginTransaction();
>>    try {
>>        // do things
>>    catch (Exception e) {
>>        // swallow exception because we will fix things
>>        session.rollback();
> // should be:
>           transaction.failure();
>>        session.beginTransaction();
>           transaction = session.beginTransaction();
>>        // do something else
>>        session.commit();
>>     }
>> }
>>
>> Such an existing code won't work if JTA in involved and
>> session.beginTrnasaction() / commit() are ignored.
>
> With the current prototype, the above existing code won't work as
> originally intended (with regard to what the original developer had in
> mind) with JTA enlistment.
>
>>
>> So only code written with the idea of the JTA net in mind will be safe.
>
> Clearly, any existing neo4j code that expects to fail the current neo4j
> transaction, then start a new neo4j transaction to fix things, will not
> work in the JTA net.  As the fixing of things in the JTA net, needs to
> occur before marking the JTA transaction for rollback only *or* by
> fixing things after the JTA transaction rolls back.
>
> Are there any other holes that come to mind, with regard to existing
> neo4j transaction code that won't work as original intended in the JTA net?
>
>>
>>>>
>>>> then I find it very user unfriendly to force the code to call both Tx
>>>> APIs. It should be one or the other.
>>>
>>> Do you really want to rewrite your neo4j code to avoid the neo4j tx APIs
>>> when enlisted into a JTA transaction?  Or should the neo4j tx API calls be
>>> ignored when enlisted into a JTA transaction (as we are doing now)?
>>
>> See my above example of already existing code.
>>
>> For new code that you want to run both in SE and Wildfly, that's an
>> interesting question but how would you do it in practice? A try catch
>> but rethrowing the exception in case JTA is involved and needs to be
>> notified of the tx rollback?
>
> The developer could rewrite their "workaround tx failure" code to avoid
> further calls to the neo4j tx api, but this is kind of bad as they only
> should do this when the neo4j transaction is enlisted into the JTA
> transaction, but not when enlistment is disabled.  Some neo4j connection
> profiles will be configured to allow JTA enlistment (transaction=1pc)
> and some will be configured to not allow JTA enlistment
> (transaction=none), so we can expect common application code will likely
> be reused for both JTA + non-JTA.  Seems to me that SE is always the
> same as the WildFly neo4j connection profile with "transaction=none", so
> we have the same problem with SE + WildFly, trying to run the same user
> code that is expected to work with JTA enlistment (neo4j connection
> profile has "transaction=1pc").
>
> I don't know of any CDI magic that helps, as if we add support for
> inject of a javax.transaction.TransactionScoped neo4j session, that
> works the same as a session obtained from Driver.session().
>
>> I'm interested to see concrete code because if doable, it will have lots
>> of constraints and be limited to very specific patterns of code I think.
>> But again, let's show each other code.

I found some examples in the Neo4j Java driver unit tests.  I started a
gist [1] that lists a few of the test cases that I thought were
interesting for this discussion about "compatibility of existing Neo4j
tx handling code in the JTA net".

In actual applications, the use cases will not be so simple.  Instead,
we will see many different patterns mixed over several lines of
application library code, however, each unit case noted is presenting a
different example that would probably not run as is, when enlisted in a
JTA transaction.

This doesn't mean it is bad to enlist Neo4j transactions into a JTA
transaction, it just means that it is bad assume that a Neo4j tx works
the same when enlisted in a JTA transaction.

I think that a challenge here for us, is to deduce what our rules should
be when enlisting Neo4j transactions into a JTA transaction, with regard
to what the application can expect and shouldn't expect.

Scott

[1] https://gist.github.com/scottmarlow/2779d21653e8466d85e38c1d62d4bb1e

>
> That should help us explore the compatibility with existing code idea
> and pick which approach to use (e.g. whether explicit neo4j transaction
> use in a JTA enlisted transaction should throw an exception or ignore
> the explicit transaction use as we are doing now in the prototype).
>
> Should we also post on https://groups.google.com/forum/#!forum/neo4j to
> ask for input on JTA enlistment?
>
>>
>> Emmanuel
>>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
123