Quantcast

EJB Transactions Graceful Shutdown

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

EJB Transactions Graceful Shutdown

Flavia Rainone

Hi,

I'm creating this thread to discuss the remaining details of graceful shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe we could have the transactions subsystem responsible for keeping track of how many active transactions we have, instead of putting that code in EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at transactions subsystem as well? I'm thinking maybe not, because there are two points in the ejb subsystem we need to know if transactions suspension is over:

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is not over, we need to check if current invocation contains a reference to an active transaction

- at some point, we need to let control point notify that the ejb module is no longer available to ejb client after transaction suspension is over, i.e., we need to do that when suspend has been requested and there are no remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the transactions subsystem and what shouldn't. If the callback is done at transactions subsystem, we need a way of having ejb3 notified that it is done. If it is not done at transactions subsystem, ejb3 has to be notified of the active transactions going to zero, which seems a lot of overhead, so from this point of view maybe the callback should be in the transactions system after all.

Stuart and Gytis, any thoughts?
-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Richard Achmatowicz

Hi Flavia

Is there any overall design for this feature available to browse? For example, there has long been a problem (and still is) with remotely initiated transactions and the EJB client retry mechanism (both of them) which associates an XA resource with one node at the beginning of the transaction and then retries on another, completes on the other, then tries to commit on the original XA node which has shutdown. This is anything but clean.

Richard


On 02/12/16 11:56 AM, Flavia Rainone wrote:

Hi,

I'm creating this thread to discuss the remaining details of graceful shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe we could have the transactions subsystem responsible for keeping track of how many active transactions we have, instead of putting that code in EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at transactions subsystem as well? I'm thinking maybe not, because there are two points in the ejb subsystem we need to know if transactions suspension is over:

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is not over, we need to check if current invocation contains a reference to an active transaction

- at some point, we need to let control point notify that the ejb module is no longer available to ejb client after transaction suspension is over, i.e., we need to do that when suspend has been requested and there are no remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the transactions subsystem and what shouldn't. If the callback is done at transactions subsystem, we need a way of having ejb3 notified that it is done. If it is not done at transactions subsystem, ejb3 has to be notified of the active transactions going to zero, which seems a lot of overhead, so from this point of view maybe the callback should be in the transactions system after all.

Stuart and Gytis, any thoughts?
-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Richard Achmatowicz

Another consideration ... but this may be outside the scope of what you are working on.

I assume that by "graceful shutdown for ejb transactions" you mean a feature to allow ejb transactions, which have been initiated before a shutdown, to be given a reasonable chance to complete after shutdown has been initiated, and so avoiding aborting a transaction just because a shutdown was initiated at an inappropriate time. In other words, the feature is specific to the case of when the server is being shutdown cleanly/gracefully. It's worht mentioning that in addition to shutdown of the server, undeploying a deployment containing ejbs (in the middle of a transaction) is going to have a similar negative impact on ejb transactions as initiating a shutdown. Also, if the deployment is clustered, there are possibilities for retrying on another available node; but as I mentioned before, there are issues with that too.

Just to say that "graceful shutdown" has different meanings in different deployment contexts.

Richard
On 02/12/16 01:29 PM, Richard Achmatowicz wrote:

Hi Flavia

Is there any overall design for this feature available to browse? For example, there has long been a problem (and still is) with remotely initiated transactions and the EJB client retry mechanism (both of them) which associates an XA resource with one node at the beginning of the transaction and then retries on another, completes on the other, then tries to commit on the original XA node which has shutdown. This is anything but clean.

Richard


On 02/12/16 11:56 AM, Flavia Rainone wrote:

Hi,

I'm creating this thread to discuss the remaining details of graceful shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe we could have the transactions subsystem responsible for keeping track of how many active transactions we have, instead of putting that code in EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at transactions subsystem as well? I'm thinking maybe not, because there are two points in the ejb subsystem we need to know if transactions suspension is over:

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is not over, we need to check if current invocation contains a reference to an active transaction

- at some point, we need to let control point notify that the ejb module is no longer available to ejb client after transaction suspension is over, i.e., we need to do that when suspend has been requested and there are no remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the transactions subsystem and what shouldn't. If the callback is done at transactions subsystem, we need a way of having ejb3 notified that it is done. If it is not done at transactions subsystem, ejb3 has to be notified of the active transactions going to zero, which seems a lot of overhead, so from this point of view maybe the callback should be in the transactions system after all.

Stuart and Gytis, any thoughts?
-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Stuart Douglas-2
In reply to this post by Flavia Rainone
On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone <[hidden email]> wrote:

> Hi,
>
> I'm creating this thread to discuss the remaining details of graceful
> shutdown for ejb transactions.
>
> This is more or less what I've done so far:
>
> https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760
>
> While discussing this in the hip chat yesterday, Stuart mentioned that maybe
> we could have the transactions subsystem responsible for keeping track of
> how many active transactions we have, instead of putting that code in
> EjbRemoteTransactionsRepository.
>
> Stuart, does that include having the suspend callback being done at
> transactions subsystem as well? I'm thinking maybe not, because there are
> two points in the ejb subsystem we need to know if transactions suspension
> is over:
>

No, that still has to be handled at an EJB subsystem level.
Conceptually this is similar to what was done for the XTS subsytem, so
it should probably use a similar design. Ideally while the server is
in the running state the only graceful related code that is run is the
control point request tracking, however this may not be possible.

One other thing that came up on our hipchat discussion yesterday is TX
level graceful shutdown actually has some significant drawbacks, as
you cannot send out the module unavailability message until all the
transactions have been closed. This means that while we are waiting
for transactions to complete the node will still be part of a cluster,
and clients will send it requests that will be immediately rejected.

Stuart

> - at EjbSuspendInterceptor if it is over, no request is allowed, if it is
> not over, we need to check if current invocation contains a reference to an
> active transaction
>
> - at some point, we need to let control point notify that the ejb module is
> not longer available to ejb client after transaction suspension is over,
> i.e., we need to do that when suspend has been requested and there are no
> remaining active transactions available.
>
> On the other hand, it is hard to draw the line between what should be in the
> transactions subsystem and what shouldn't. If the callback is done at
> transactions subsystem, we need a way of having ejb3 notified that it is
> done. If it is not done at transactions subsystem, ejb3 has to be notified
> of the active transactions going to zero, which seems a lot of overhead, so
> from this point of view maybe the callback should be in the transactions
> system after all.
>
> Stuart and Gytis, any thoughts?
>
>
> --
> Flavia Rainone
> Principal Software Engineer
> JBoss EAP/WildFly Team
> M: (+55) 11 981-225-466
>
> Red Hat.
> Better technology.
> Faster innovation.
> Powered by community collaboration.
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Flavia Rainone
In reply to this post by Richard Achmatowicz

Hi Richard,

This is the corresponding Jira: https://issues.jboss.org/browse/WFLY-7207

There is a document for transactions graceful shutdown here: https://developer.jboss.org/wiki/DevAnalysisForTransactionsGracefulShutdown

In a nutshell, the EJB system will not complete suspension until all open transactions are concluded, and that includes XATransactions.

This issue you're mentioning involving EJB client retry plus remote transactions is possibly outside the scope of WFLY-7207. Do you have a Jira for that one? If I understand correctly, the problem is that the client shouldn't retry at the original XA node, because that node has shutdown, is that correct? In this case, I find it weird, because the ejb client is going to be notified that the client is no longer available on shutdown, thus preventing the client from invoking that node, as Stuart mentioned in his previous e-mail. We are going to have to workaround that mechanism, by delaying the client notification until all transactions are complete. As this behavior could be undesired, after all it would make invocations to the shutting down node possible while there are open transactions in that node, we are going to make it configurable, the user might not want graceful ejb transaction at all in this case.

Or maybe you mean that all remote invocations belonging to the same remote transaction should always be answered by the same cluster node?

Regarding your consideration, yes, you are correct, this feature is to give an oportunity for open transactions to complete before shutdown. I think Stuart has plans of extending the graceful shutdown feature to cover undeployments in the future.

R.

Flavia


On 02-12-2016 17:45, Richard Achmatowicz wrote:

Another consideration ... but this may be outside the scope of what you are working on.

I assume that by "graceful shutdown for ejb transactions" you mean a feature to allow ejb transactions, which have been initiated before a shutdown, to be given a reasonable chance to complete after shutdown has been initiated, and so avoiding aborting a transaction just because a shutdown was initiated at an inappropriate time. In other words, the feature is specific to the case of when the server is being shutdown cleanly/gracefully. It's worht mentioning that in addition to shutdown of the server, undeploying a deployment containing ejbs (in the middle of a transaction) is going to have a similar negative impact on ejb transactions as initiating a shutdown. Also, if the deployment is clustered, there are possibilities for retrying on another available node; but as I mentioned before, there are issues with that too.

Just to say that "graceful shutdown" has different meanings in different deployment contexts.

Richard
On 02/12/16 01:29 PM, Richard Achmatowicz wrote:

Hi Flavia

Is there any overall design for this feature available to browse? For example, there has long been a problem (and still is) with remotely initiated transactions and the EJB client retry mechanism (both of them) which associates an XA resource with one node at the beginning of the transaction and then retries on another, completes on the other, then tries to commit on the original XA node which has shutdown. This is anything but clean.

Richard


On 02/12/16 11:56 AM, Flavia Rainone wrote:

Hi,

I'm creating this thread to discuss the remaining details of graceful shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe we could have the transactions subsystem responsible for keeping track of how many active transactions we have, instead of putting that code in EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at transactions subsystem as well? I'm thinking maybe not, because there are two points in the ejb subsystem we need to know if transactions suspension is over:

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is not over, we need to check if current invocation contains a reference to an active transaction

- at some point, we need to let control point notify that the ejb module is no longer available to ejb client after transaction suspension is over, i.e., we need to do that when suspend has been requested and there are no remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the transactions subsystem and what shouldn't. If the callback is done at transactions subsystem, we need a way of having ejb3 notified that it is done. If it is not done at transactions subsystem, ejb3 has to be notified of the active transactions going to zero, which seems a lot of overhead, so from this point of view maybe the callback should be in the transactions system after all.

Stuart and Gytis, any thoughts?
-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev

-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Flavia Rainone
In reply to this post by Stuart Douglas-2
I think we can keep open transaction tracking only inside transactions subsystem while we are not shutting down, and then we can enroll for notification of open active transactions only on suspend if needed... IMHO that's as clean as we can get regarding shutdown code when the server is in running state.

I would go with some sort of ActiveTransactionListener, that would be notified of no more active transactions only if the listener is set?

Something along the lines below at ejb side:

ServerActivityCallback callback = null;

public void suspend(ServerActivityCallback callback) {
    if ( transactionSubsystem.getActiveTransactions() > 0) {
       transactionSubsystem.setActiveTransactionsListener(this);
    }
    else {
      callback.done(); // done suspending
   }
}

// listener method
public void noMoreActiveTransactions() {
    callback.done(); // done suspending
    // then we let control point notify clients that this node is no longer available
    ...
}

At transactions side:
ActiveTransactionListener listener = null;

private void incrementTxnCount() {
    ...
}

private void decrementTxnCount() {
   if (txnCountUpdater.decrementAndGet() == 0 && listener != null)
       listener.noMoreActiveTransactions();       
}

public int getActiveTransactions() {
   return txnCountUpdater.get();
}







On 04-12-2016 20:39, Stuart Douglas wrote:
On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone [hidden email] wrote:
Hi,

I'm creating this thread to discuss the remaining details of graceful
shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe
we could have the transactions subsystem responsible for keeping track of
how many active transactions we have, instead of putting that code in
EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at
transactions subsystem as well? I'm thinking maybe not, because there are
two points in the ejb subsystem we need to know if transactions suspension
is over:

No, that still has to be handled at an EJB subsystem level.
Conceptually this is similar to what was done for the XTS subsytem, so
it should probably use a similar design. Ideally while the server is
in the running state the only graceful related code that is run is the
control point request tracking, however this may not be possible.

One other thing that came up on our hipchat discussion yesterday is TX
level graceful shutdown actually has some significant drawbacks, as
you cannot send out the module unavailability message until all the
transactions have been closed. This means that while we are waiting
for transactions to complete the node will still be part of a cluster,
and clients will send it requests that will be immediately rejected.

Stuart

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is
not over, we need to check if current invocation contains a reference to an
active transaction

- at some point, we need to let control point notify that the ejb module is
not longer available to ejb client after transaction suspension is over,
i.e., we need to do that when suspend has been requested and there are no
remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the
transactions subsystem and what shouldn't. If the callback is done at
transactions subsystem, we need a way of having ejb3 notified that it is
done. If it is not done at transactions subsystem, ejb3 has to be notified
of the active transactions going to zero, which seems a lot of overhead, so
from this point of view maybe the callback should be in the transactions
system after all.

Stuart and Gytis, any thoughts?


--
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration.

-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Andrig Miller
I'm just wondering if we are making this graceful shutdown more complicated than necessary.

Why wouldn't we just cancel and force a rollback on any active transactions when shutting down?  Having experienced what a graceful shutdown can look like with a different architecture (BEA Tuxedo), I can tell you that it can take a very long time for the server to get to the point of shutting down, and appear to be hung by the administrator, depending on what was going on at the time the command was entered.

We used to get administrators killing Tuxedo while it was "gracefully" shutting down, and messing lots of stuff up.

Andy

On Mon, Dec 5, 2016 at 12:34 PM, Flavia Rainone <[hidden email]> wrote:
I think we can keep open transaction tracking only inside transactions subsystem while we are not shutting down, and then we can enroll for notification of open active transactions only on suspend if needed... IMHO that's as clean as we can get regarding shutdown code when the server is in running state.

I would go with some sort of ActiveTransactionListener, that would be notified of no more active transactions only if the listener is set?

Something along the lines below at ejb side:

ServerActivityCallback callback = null;

public void suspend(ServerActivityCallback callback) {
    if ( transactionSubsystem.getActiveTransactions() > 0) {
       transactionSubsystem.setActiveTransactionsListener(this);
    }
    else {
      callback.done(); // done suspending
   }
}

// listener method
public void noMoreActiveTransactions() {
    callback.done(); // done suspending
    // then we let control point notify clients that this node is no longer available
    ...
}

At transactions side:
ActiveTransactionListener listener = null;

private void incrementTxnCount() {
    ...
}

private void decrementTxnCount() {
   if (txnCountUpdater.decrementAndGet() == 0 && listener != null)
       listener.noMoreActiveTransactions();       
}

public int getActiveTransactions() {
   return txnCountUpdater.get();

}







On 04-12-2016 20:39, Stuart Douglas wrote:
On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone [hidden email] wrote:
Hi,

I'm creating this thread to discuss the remaining details of graceful
shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe
we could have the transactions subsystem responsible for keeping track of
how many active transactions we have, instead of putting that code in
EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at
transactions subsystem as well? I'm thinking maybe not, because there are
two points in the ejb subsystem we need to know if transactions suspension
is over:

No, that still has to be handled at an EJB subsystem level.
Conceptually this is similar to what was done for the XTS subsytem, so
it should probably use a similar design. Ideally while the server is
in the running state the only graceful related code that is run is the
control point request tracking, however this may not be possible.

One other thing that came up on our hipchat discussion yesterday is TX
level graceful shutdown actually has some significant drawbacks, as
you cannot send out the module unavailability message until all the
transactions have been closed. This means that while we are waiting
for transactions to complete the node will still be part of a cluster,
and clients will send it requests that will be immediately rejected.

Stuart

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is
not over, we need to check if current invocation contains a reference to an
active transaction

- at some point, we need to let control point notify that the ejb module is
not longer available to ejb client after transaction suspension is over,
i.e., we need to do that when suspend has been requested and there are no
remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the
transactions subsystem and what shouldn't. If the callback is done at
transactions subsystem, we need a way of having ejb3 notified that it is
done. If it is not done at transactions subsystem, ejb3 has to be notified
of the active transactions going to zero, which seems a lot of overhead, so
from this point of view maybe the callback should be in the transactions
system after all.

Stuart and Gytis, any thoughts?


--
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration.

-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



--
Andrig (Andy) T. Miller
Global Platform Director, Middleware
Red Hat, Inc.

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Stuart Douglas


On Tue, Dec 6, 2016 at 10:12 AM, Andrig Miller <[hidden email]> wrote:
I'm just wondering if we are making this graceful shutdown more complicated than necessary.

Why wouldn't we just cancel and force a rollback on any active transactions when shutting down?  Having experienced what a graceful shutdown can look like with a different architecture (BEA Tuxedo), I can tell you that it can take a very long time for the server to get to the point of shutting down, and appear to be hung by the administrator, depending on what was going on at the time the command was entered.


I think at the very least this has to be optional, so we can still have the old non transnational behavior (i.e. wait for requests to finish rather than transactions).

Stuart
 

We used to get administrators killing Tuxedo while it was "gracefully" shutting down, and messing lots of stuff up.

Andy

On Mon, Dec 5, 2016 at 12:34 PM, Flavia Rainone <[hidden email]> wrote:
I think we can keep open transaction tracking only inside transactions subsystem while we are not shutting down, and then we can enroll for notification of open active transactions only on suspend if needed... IMHO that's as clean as we can get regarding shutdown code when the server is in running state.

I would go with some sort of ActiveTransactionListener, that would be notified of no more active transactions only if the listener is set?

Something along the lines below at ejb side:

ServerActivityCallback callback = null;

public void suspend(ServerActivityCallback callback) {
    if ( transactionSubsystem.getActiveTransactions() > 0) {
       transactionSubsystem.setActiveTransactionsListener(this);
    }
    else {
      callback.done(); // done suspending
   }
}

// listener method
public void noMoreActiveTransactions() {
    callback.done(); // done suspending
    // then we let control point notify clients that this node is no longer available
    ...
}

At transactions side:
ActiveTransactionListener listener = null;

private void incrementTxnCount() {
    ...
}

private void decrementTxnCount() {
   if (txnCountUpdater.decrementAndGet() == 0 && listener != null)
       listener.noMoreActiveTransactions();       
}

public int getActiveTransactions() {
   return txnCountUpdater.get();

}







On 04-12-2016 20:39, Stuart Douglas wrote:
On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone [hidden email] wrote:
Hi,

I'm creating this thread to discuss the remaining details of graceful
shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe
we could have the transactions subsystem responsible for keeping track of
how many active transactions we have, instead of putting that code in
EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at
transactions subsystem as well? I'm thinking maybe not, because there are
two points in the ejb subsystem we need to know if transactions suspension
is over:

No, that still has to be handled at an EJB subsystem level.
Conceptually this is similar to what was done for the XTS subsytem, so
it should probably use a similar design. Ideally while the server is
in the running state the only graceful related code that is run is the
control point request tracking, however this may not be possible.

One other thing that came up on our hipchat discussion yesterday is TX
level graceful shutdown actually has some significant drawbacks, as
you cannot send out the module unavailability message until all the
transactions have been closed. This means that while we are waiting
for transactions to complete the node will still be part of a cluster,
and clients will send it requests that will be immediately rejected.

Stuart

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is
not over, we need to check if current invocation contains a reference to an
active transaction

- at some point, we need to let control point notify that the ejb module is
not longer available to ejb client after transaction suspension is over,
i.e., we need to do that when suspend has been requested and there are no
remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the
transactions subsystem and what shouldn't. If the callback is done at
transactions subsystem, we need a way of having ejb3 notified that it is
done. If it is not done at transactions subsystem, ejb3 has to be notified
of the active transactions going to zero, which seems a lot of overhead, so
from this point of view maybe the callback should be in the transactions
system after all.

Stuart and Gytis, any thoughts?


--
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration.

-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



--
Andrig (Andy) T. Miller
Global Platform Director, Middleware
Red Hat, Inc.

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Andrig Miller


On Mon, Dec 5, 2016 at 9:54 PM, Stuart Douglas <[hidden email]> wrote:


On Tue, Dec 6, 2016 at 10:12 AM, Andrig Miller <[hidden email]> wrote:
I'm just wondering if we are making this graceful shutdown more complicated than necessary.

Why wouldn't we just cancel and force a rollback on any active transactions when shutting down?  Having experienced what a graceful shutdown can look like with a different architecture (BEA Tuxedo), I can tell you that it can take a very long time for the server to get to the point of shutting down, and appear to be hung by the administrator, depending on what was going on at the time the command was entered.


I think at the very least this has to be optional, so we can still have the old non transnational behavior (i.e. wait for requests to finish rather than transactions).

​My personal opinion is that it should be the default behavior as well.  Most of the time, when our administrators would try to gracefully shutdown Tuxedo, then ended up killing it with a kill -9, because it took too long.  Of course, that caused all kinds of consistency problems and transaction recovery on the next startup.  Generally, it was a mess.

Andy
 

Stuart
 

We used to get administrators killing Tuxedo while it was "gracefully" shutting down, and messing lots of stuff up.

Andy

On Mon, Dec 5, 2016 at 12:34 PM, Flavia Rainone <[hidden email]> wrote:
I think we can keep open transaction tracking only inside transactions subsystem while we are not shutting down, and then we can enroll for notification of open active transactions only on suspend if needed... IMHO that's as clean as we can get regarding shutdown code when the server is in running state.

I would go with some sort of ActiveTransactionListener, that would be notified of no more active transactions only if the listener is set?

Something along the lines below at ejb side:

ServerActivityCallback callback = null;

public void suspend(ServerActivityCallback callback) {
    if ( transactionSubsystem.getActiveTransactions() > 0) {
       transactionSubsystem.setActiveTransactionsListener(this);
    }
    else {
      callback.done(); // done suspending
   }
}

// listener method
public void noMoreActiveTransactions() {
    callback.done(); // done suspending
    // then we let control point notify clients that this node is no longer available
    ...
}

At transactions side:
ActiveTransactionListener listener = null;

private void incrementTxnCount() {
    ...
}

private void decrementTxnCount() {
   if (txnCountUpdater.decrementAndGet() == 0 && listener != null)
       listener.noMoreActiveTransactions();       
}

public int getActiveTransactions() {
   return txnCountUpdater.get();

}







On 04-12-2016 20:39, Stuart Douglas wrote:
On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone [hidden email] wrote:
Hi,

I'm creating this thread to discuss the remaining details of graceful
shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe
we could have the transactions subsystem responsible for keeping track of
how many active transactions we have, instead of putting that code in
EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at
transactions subsystem as well? I'm thinking maybe not, because there are
two points in the ejb subsystem we need to know if transactions suspension
is over:

No, that still has to be handled at an EJB subsystem level.
Conceptually this is similar to what was done for the XTS subsytem, so
it should probably use a similar design. Ideally while the server is
in the running state the only graceful related code that is run is the
control point request tracking, however this may not be possible.

One other thing that came up on our hipchat discussion yesterday is TX
level graceful shutdown actually has some significant drawbacks, as
you cannot send out the module unavailability message until all the
transactions have been closed. This means that while we are waiting
for transactions to complete the node will still be part of a cluster,
and clients will send it requests that will be immediately rejected.

Stuart

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is
not over, we need to check if current invocation contains a reference to an
active transaction

- at some point, we need to let control point notify that the ejb module is
not longer available to ejb client after transaction suspension is over,
i.e., we need to do that when suspend has been requested and there are no
remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the
transactions subsystem and what shouldn't. If the callback is done at
transactions subsystem, we need a way of having ejb3 notified that it is
done. If it is not done at transactions subsystem, ejb3 has to be notified
of the active transactions going to zero, which seems a lot of overhead, so
from this point of view maybe the callback should be in the transactions
system after all.

Stuart and Gytis, any thoughts?


--
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration.

-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



--
Andrig (Andy) T. Miller
Global Platform Director, Middleware
Red Hat, Inc.

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev




--
Andrig (Andy) T. Miller
Global Platform Director, Middleware
Red Hat, Inc.

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Jim Tyrrell
Team,

How might we process the shutting down with a transaction log message/s saying it is shutting down or saying something like:

Started shutting down
100 pending transactions
10 shut down 90 pending
20 shut down 80 pending

or something like that.

Feedback to the user in this sounds important to ensure people don’t kill -9 the process.

Jim Tyrrell
Principal JBoss Solutions Architect
Public Sector Middleware Advocate
720-839-2251 mobile

On Dec 6, 2016, at 8:50 AM, Andrig Miller <[hidden email]> wrote:



On Mon, Dec 5, 2016 at 9:54 PM, Stuart Douglas <[hidden email]> wrote:


On Tue, Dec 6, 2016 at 10:12 AM, Andrig Miller <[hidden email]> wrote:
I'm just wondering if we are making this graceful shutdown more complicated than necessary.

Why wouldn't we just cancel and force a rollback on any active transactions when shutting down?  Having experienced what a graceful shutdown can look like with a different architecture (BEA Tuxedo), I can tell you that it can take a very long time for the server to get to the point of shutting down, and appear to be hung by the administrator, depending on what was going on at the time the command was entered.


I think at the very least this has to be optional, so we can still have the old non transnational behavior (i.e. wait for requests to finish rather than transactions). 

​My personal opinion is that it should be the default behavior as well.  Most of the time, when our administrators would try to gracefully shutdown Tuxedo, then ended up killing it with a kill -9, because it took too long.  Of course, that caused all kinds of consistency problems and transaction recovery on the next startup.  Generally, it was a mess.

Andy
 

Stuart
 

We used to get administrators killing Tuxedo while it was "gracefully" shutting down, and messing lots of stuff up.

Andy

On Mon, Dec 5, 2016 at 12:34 PM, Flavia Rainone <[hidden email]> wrote:
I think we can keep open transaction tracking only inside transactions subsystem while we are not shutting down, and then we can enroll for notification of open active transactions only on suspend if needed... IMHO that's as clean as we can get regarding shutdown code when the server is in running state.

I would go with some sort of ActiveTransactionListener, that would be notified of no more active transactions only if the listener is set?

Something along the lines below at ejb side:

ServerActivityCallback callback = null;

public void suspend(ServerActivityCallback callback) {
    if ( transactionSubsystem.getActiveTransactions() > 0) {
       transactionSubsystem.setActiveTransactionsListener(this);
    }
    else {
      callback.done(); // done suspending
   }
}

// listener method
public void noMoreActiveTransactions() {
    callback.done(); // done suspending
    // then we let control point notify clients that this node is no longer available
    ...
}

At transactions side:
ActiveTransactionListener listener = null;

private void incrementTxnCount() {
    ...
}

private void decrementTxnCount() {
   if (txnCountUpdater.decrementAndGet() == 0 && listener != null)
       listener.noMoreActiveTransactions();        
}

public int getActiveTransactions() {
   return txnCountUpdater.get();

}







On 04-12-2016 20:39, Stuart Douglas wrote:
On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone [hidden email] wrote:
Hi,

I'm creating this thread to discuss the remaining details of graceful
shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe
we could have the transactions subsystem responsible for keeping track of
how many active transactions we have, instead of putting that code in
EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at
transactions subsystem as well? I'm thinking maybe not, because there are
two points in the ejb subsystem we need to know if transactions suspension
is over:

No, that still has to be handled at an EJB subsystem level.
Conceptually this is similar to what was done for the XTS subsytem, so
it should probably use a similar design. Ideally while the server is
in the running state the only graceful related code that is run is the
control point request tracking, however this may not be possible.

One other thing that came up on our hipchat discussion yesterday is TX
level graceful shutdown actually has some significant drawbacks, as
you cannot send out the module unavailability message until all the
transactions have been closed. This means that while we are waiting
for transactions to complete the node will still be part of a cluster,
and clients will send it requests that will be immediately rejected.

Stuart

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is
not over, we need to check if current invocation contains a reference to an
active transaction

- at some point, we need to let control point notify that the ejb module is
not longer available to ejb client after transaction suspension is over,
i.e., we need to do that when suspend has been requested and there are no
remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the
transactions subsystem and what shouldn't. If the callback is done at
transactions subsystem, we need a way of having ejb3 notified that it is
done. If it is not done at transactions subsystem, ejb3 has to be notified
of the active transactions going to zero, which seems a lot of overhead, so
from this point of view maybe the callback should be in the transactions
system after all.

Stuart and Gytis, any thoughts?


--
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration.

-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev



-- 
Andrig (Andy) T. Miller
Global Platform Director, Middleware
Red Hat, Inc.

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev




-- 
Andrig (Andy) T. Miller
Global Platform Director, Middleware
Red Hat, Inc.
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Radim Hatlapatka

Definitely +1 for prompting in logs what is being happening and why shutdown takes longer period of time.

Also the graceful shutdown adds timeout which can be used to limit maximum number of seconds for finishing shutdown gracefully, after that time it should stop waiting for remaining transactions/requests to finish.

Note current default behaviour is to stop the server without using the graceful shutdown (timeout needs to be specified for actually using the graceful shutdown logic), for details see description of shutdown command timeout value.

Cheers.

Radim


On 12/07/2016 02:49 PM, Jim Tyrrell wrote:
Team,

How might we process the shutting down with a transaction log message/s saying it is shutting down or saying something like:

Started shutting down
100 pending transactions
10 shut down 90 pending
20 shut down 80 pending

or something like that.

Feedback to the user in this sounds important to ensure people don’t kill -9 the process.

Jim Tyrrell
Principal JBoss Solutions Architect
Public Sector Middleware Advocate
720-839-2251 mobile

On Dec 6, 2016, at 8:50 AM, Andrig Miller <[hidden email]> wrote:



On Mon, Dec 5, 2016 at 9:54 PM, Stuart Douglas <[hidden email]> wrote:


On Tue, Dec 6, 2016 at 10:12 AM, Andrig Miller <[hidden email]> wrote:
I'm just wondering if we are making this graceful shutdown more complicated than necessary.

Why wouldn't we just cancel and force a rollback on any active transactions when shutting down?  Having experienced what a graceful shutdown can look like with a different architecture (BEA Tuxedo), I can tell you that it can take a very long time for the server to get to the point of shutting down, and appear to be hung by the administrator, depending on what was going on at the time the command was entered.


I think at the very least this has to be optional, so we can still have the old non transnational behavior (i.e. wait for requests to finish rather than transactions). 

​My personal opinion is that it should be the default behavior as well.  Most of the time, when our administrators would try to gracefully shutdown Tuxedo, then ended up killing it with a kill -9, because it took too long.  Of course, that caused all kinds of consistency problems and transaction recovery on the next startup.  Generally, it was a mess.

Andy
 

Stuart
 

We used to get administrators killing Tuxedo while it was "gracefully" shutting down, and messing lots of stuff up.

Andy

On Mon, Dec 5, 2016 at 12:34 PM, Flavia Rainone <[hidden email]> wrote:
I think we can keep open transaction tracking only inside transactions subsystem while we are not shutting down, and then we can enroll for notification of open active transactions only on suspend if needed... IMHO that's as clean as we can get regarding shutdown code when the server is in running state.

I would go with some sort of ActiveTransactionListener, that would be notified of no more active transactions only if the listener is set?

Something along the lines below at ejb side:

ServerActivityCallback callback = null;

public void suspend(ServerActivityCallback callback) {
    if ( transactionSubsystem.getActiveTransactions() > 0) {
       transactionSubsystem.setActiveTransactionsListener(this);
    }
    else {
      callback.done(); // done suspending
   }
}

// listener method
public void noMoreActiveTransactions() {
    callback.done(); // done suspending
    // then we let control point notify clients that this node is no longer available
    ...
}

At transactions side:
ActiveTransactionListener listener = null;

private void incrementTxnCount() {
    ...
}

private void decrementTxnCount() {
   if (txnCountUpdater.decrementAndGet() == 0 && listener != null)
       listener.noMoreActiveTransactions();        
}

public int getActiveTransactions() {
   return txnCountUpdater.get();

}







On 04-12-2016 20:39, Stuart Douglas wrote:
On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone [hidden email] wrote:
Hi,

I'm creating this thread to discuss the remaining details of graceful
shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe
we could have the transactions subsystem responsible for keeping track of
how many active transactions we have, instead of putting that code in
EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at
transactions subsystem as well? I'm thinking maybe not, because there are
two points in the ejb subsystem we need to know if transactions suspension
is over:

No, that still has to be handled at an EJB subsystem level.
Conceptually this is similar to what was done for the XTS subsytem, so
it should probably use a similar design. Ideally while the server is
in the running state the only graceful related code that is run is the
control point request tracking, however this may not be possible.

One other thing that came up on our hipchat discussion yesterday is TX
level graceful shutdown actually has some significant drawbacks, as
you cannot send out the module unavailability message until all the
transactions have been closed. This means that while we are waiting
for transactions to complete the node will still be part of a cluster,
and clients will send it requests that will be immediately rejected.

Stuart

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is
not over, we need to check if current invocation contains a reference to an
active transaction

- at some point, we need to let control point notify that the ejb module is
not longer available to ejb client after transaction suspension is over,
i.e., we need to do that when suspend has been requested and there are no
remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the
transactions subsystem and what shouldn't. If the callback is done at
transactions subsystem, we need a way of having ejb3 notified that it is
done. If it is not done at transactions subsystem, ejb3 has to be notified
of the active transactions going to zero, which seems a lot of overhead, so
from this point of view maybe the callback should be in the transactions
system after all.

Stuart and Gytis, any thoughts?


--
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration.
-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 
_______________________________________________ wildfly-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/wildfly-dev
-- 
Andrig (Andy) T. Miller
Global Platform Director, Middleware
Red Hat, Inc.
_______________________________________________ wildfly-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/wildfly-dev
-- 
Andrig (Andy) T. Miller
Global Platform Director, Middleware
Red Hat, Inc.
_______________________________________________ wildfly-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Flavia Rainone

+1, I like the idea of logging the number of active pending transactions.


On 07-12-2016 12:16, Radim Hatlapatka wrote:

Definitely +1 for prompting in logs what is being happening and why shutdown takes longer period of time.

Also the graceful shutdown adds timeout which can be used to limit maximum number of seconds for finishing shutdown gracefully, after that time it should stop waiting for remaining transactions/requests to finish.

Note current default behaviour is to stop the server without using the graceful shutdown (timeout needs to be specified for actually using the graceful shutdown logic), for details see description of shutdown command timeout value.

Cheers.

Radim


On 12/07/2016 02:49 PM, Jim Tyrrell wrote:
Team,

How might we process the shutting down with a transaction log message/s saying it is shutting down or saying something like:

Started shutting down
100 pending transactions
10 shut down 90 pending
20 shut down 80 pending

or something like that.

Feedback to the user in this sounds important to ensure people don’t kill -9 the process.

Jim Tyrrell
Principal JBoss Solutions Architect
Public Sector Middleware Advocate
720-839-2251 mobile

On Dec 6, 2016, at 8:50 AM, Andrig Miller <[hidden email]> wrote:



On Mon, Dec 5, 2016 at 9:54 PM, Stuart Douglas <[hidden email]> wrote:


On Tue, Dec 6, 2016 at 10:12 AM, Andrig Miller <[hidden email]> wrote:
I'm just wondering if we are making this graceful shutdown more complicated than necessary.

Why wouldn't we just cancel and force a rollback on any active transactions when shutting down?  Having experienced what a graceful shutdown can look like with a different architecture (BEA Tuxedo), I can tell you that it can take a very long time for the server to get to the point of shutting down, and appear to be hung by the administrator, depending on what was going on at the time the command was entered.


I think at the very least this has to be optional, so we can still have the old non transnational behavior (i.e. wait for requests to finish rather than transactions). 

​My personal opinion is that it should be the default behavior as well.  Most of the time, when our administrators would try to gracefully shutdown Tuxedo, then ended up killing it with a kill -9, because it took too long.  Of course, that caused all kinds of consistency problems and transaction recovery on the next startup.  Generally, it was a mess.

Andy
 

Stuart
 

We used to get administrators killing Tuxedo while it was "gracefully" shutting down, and messing lots of stuff up.

Andy

On Mon, Dec 5, 2016 at 12:34 PM, Flavia Rainone <[hidden email]> wrote:
I think we can keep open transaction tracking only inside transactions subsystem while we are not shutting down, and then we can enroll for notification of open active transactions only on suspend if needed... IMHO that's as clean as we can get regarding shutdown code when the server is in running state.

I would go with some sort of ActiveTransactionListener, that would be notified of no more active transactions only if the listener is set?

Something along the lines below at ejb side:

ServerActivityCallback callback = null;

public void suspend(ServerActivityCallback callback) {
    if ( transactionSubsystem.getActiveTransactions() > 0) {
       transactionSubsystem.setActiveTransactionsListener(this);
    }
    else {
      callback.done(); // done suspending
   }
}

// listener method
public void noMoreActiveTransactions() {
    callback.done(); // done suspending
    // then we let control point notify clients that this node is no longer available
    ...
}

At transactions side:
ActiveTransactionListener listener = null;

private void incrementTxnCount() {
    ...
}

private void decrementTxnCount() {
   if (txnCountUpdater.decrementAndGet() == 0 && listener != null)
       listener.noMoreActiveTransactions();        
}

public int getActiveTransactions() {
   return txnCountUpdater.get();

}







On 04-12-2016 20:39, Stuart Douglas wrote:
On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone [hidden email] wrote:
Hi,

I'm creating this thread to discuss the remaining details of graceful
shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe
we could have the transactions subsystem responsible for keeping track of
how many active transactions we have, instead of putting that code in
EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at
transactions subsystem as well? I'm thinking maybe not, because there are
two points in the ejb subsystem we need to know if transactions suspension
is over:

No, that still has to be handled at an EJB subsystem level.
Conceptually this is similar to what was done for the XTS subsytem, so
it should probably use a similar design. Ideally while the server is
in the running state the only graceful related code that is run is the
control point request tracking, however this may not be possible.

One other thing that came up on our hipchat discussion yesterday is TX
level graceful shutdown actually has some significant drawbacks, as
you cannot send out the module unavailability message until all the
transactions have been closed. This means that while we are waiting
for transactions to complete the node will still be part of a cluster,
and clients will send it requests that will be immediately rejected.

Stuart

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is
not over, we need to check if current invocation contains a reference to an
active transaction

- at some point, we need to let control point notify that the ejb module is
not longer available to ejb client after transaction suspension is over,
i.e., we need to do that when suspend has been requested and there are no
remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the
transactions subsystem and what shouldn't. If the callback is done at
transactions subsystem, we need a way of having ejb3 notified that it is
done. If it is not done at transactions subsystem, ejb3 has to be notified
of the active transactions going to zero, which seems a lot of overhead, so
from this point of view maybe the callback should be in the transactions
system after all.

Stuart and Gytis, any thoughts?


--
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration.
-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 
_______________________________________________ wildfly-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/wildfly-dev
-- 
Andrig (Andy) T. Miller
Global Platform Director, Middleware
Red Hat, Inc.
_______________________________________________ wildfly-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/wildfly-dev
-- 
Andrig (Andy) T. Miller
Global Platform Director, Middleware
Red Hat, Inc.
_______________________________________________ wildfly-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Gytis Trikleris-2
In reply to this post by Flavia Rainone

Flavia,

it seems that it might be better for EJB subsystem to keep track of its own transaction. The reason being that all transactions are kept in the same map no matter what created them (EJB, CDI, or UserTransaction).

We could expose an API to get that map (or just a number of transactions in it) as well as a way to register callback. However, that will also include transactions created by other ways mentioned above.

Gytis


On 05/12/2016 20:34, Flavia Rainone wrote:
I think we can keep open transaction tracking only inside transactions subsystem while we are not shutting down, and then we can enroll for notification of open active transactions only on suspend if needed... IMHO that's as clean as we can get regarding shutdown code when the server is in running state.

I would go with some sort of ActiveTransactionListener, that would be notified of no more active transactions only if the listener is set?

Something along the lines below at ejb side:

ServerActivityCallback callback = null;

public void suspend(ServerActivityCallback callback) {
    if ( transactionSubsystem.getActiveTransactions() > 0) {
       transactionSubsystem.setActiveTransactionsListener(this);
    }
    else {
      callback.done(); // done suspending
   }
}

// listener method
public void noMoreActiveTransactions() {
    callback.done(); // done suspending
    // then we let control point notify clients that this node is no longer available
    ...
}

At transactions side:
ActiveTransactionListener listener = null;

private void incrementTxnCount() {
    ...
}

private void decrementTxnCount() {
   if (txnCountUpdater.decrementAndGet() == 0 && listener != null)
       listener.noMoreActiveTransactions();       
}

public int getActiveTransactions() {
   return txnCountUpdater.get();
}







On 04-12-2016 20:39, Stuart Douglas wrote:
On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone [hidden email] wrote:
Hi,

I'm creating this thread to discuss the remaining details of graceful
shutdown for ejb transactions.

This is more or less what I've done so far:

https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760

While discussing this in the hip chat yesterday, Stuart mentioned that maybe
we could have the transactions subsystem responsible for keeping track of
how many active transactions we have, instead of putting that code in
EjbRemoteTransactionsRepository.

Stuart, does that include having the suspend callback being done at
transactions subsystem as well? I'm thinking maybe not, because there are
two points in the ejb subsystem we need to know if transactions suspension
is over:

No, that still has to be handled at an EJB subsystem level.
Conceptually this is similar to what was done for the XTS subsytem, so
it should probably use a similar design. Ideally while the server is
in the running state the only graceful related code that is run is the
control point request tracking, however this may not be possible.

One other thing that came up on our hipchat discussion yesterday is TX
level graceful shutdown actually has some significant drawbacks, as
you cannot send out the module unavailability message until all the
transactions have been closed. This means that while we are waiting
for transactions to complete the node will still be part of a cluster,
and clients will send it requests that will be immediately rejected.

Stuart

- at EjbSuspendInterceptor if it is over, no request is allowed, if it is
not over, we need to check if current invocation contains a reference to an
active transaction

- at some point, we need to let control point notify that the ejb module is
not longer available to ejb client after transaction suspension is over,
i.e., we need to do that when suspend has been requested and there are no
remaining active transactions available.

On the other hand, it is hard to draw the line between what should be in the
transactions subsystem and what shouldn't. If the callback is done at
transactions subsystem, we need a way of having ejb3 notified that it is
done. If it is not done at transactions subsystem, ejb3 has to be notified
of the active transactions going to zero, which seems a lot of overhead, so
from this point of view maybe the callback should be in the transactions
system after all.

Stuart and Gytis, any thoughts?


--
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration.

-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EJB Transactions Graceful Shutdown

Flavia Rainone

I see, I'll keep the count in EJB system, then.

I'll complete my commit and make a PR, I'll let you know if I have any remaining questions in this thread.


On 12-12-2016 17:25, Gytis Trikleris wrote:
(EJB, CDI, or UserTransaction

-- 
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team 
M: (+55) 11 981-225-466

Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration. 

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Loading...