Graceful Shutdown (Revision 2)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Graceful Shutdown (Revision 2)

jtgreene
Administrator
*EDITED*

We've worked out the rough outline of how graceful shutdown will work in
AS7.

The process of graceful shutdown actually is reflected by a number of
states:

1. Running - all services acting normally
2. Suspending - services refuse new "permits" (see below), existing
permits are allowed to be retained (and threads running under such a
permit may still acquire new permits)
3. Suspended - no permits are present and none may be issued
4. Shutting Down - our existing server stop process / reload admin mode

The following transitions are allowed:

1. Running → Suspending: Transition occurs at user request (to suspend
or gracefully shut down).
2. Suspending → Suspended: Transition occurs when all permits are cleared.
3. Suspending → Running: Transition occurs at user request (to exit
suspend mode or cancel graceful shutdown before it completes).
4. Suspended → Running: Transition occurs at user request (to exit
suspend mode).
5. Suspended → Shutting Down: Transition occurs automatically (if a
graceful shutdown was requested) or at user request (if a shut down
request of any kind is entered in the Suspended state).
6. Running → Shutting Down: Transition occurs at user request (to shut
down the server "un-gracefully").
7. Suspending → Shutting Down (User aborts a graceful shutdown)

These "permits" are issued by the "Shutdown Manager", whose job is to
manage these states.  They are issued corresponding to the following events:

1. The invocation of an EJB method
2. The creation of a web session
3. A creation of a transaction
4. MessageEndpoint and WorkManager aquire permit allowing for release()
from a thirdparty to indicate connection close.

When a permit cannot be issued due to the server shutting down, a
standard exception message should be produced so that the user can see a
familiar error message regardless of what mechanism is used to access
the server.

--
- DML
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Graceful Shutdown (Revision 2)

Bill Burke
Are you sure explicity granting "permits" through a "Shutdown Manager"
the way to go?  Having to interact with a central service that grants
permits per-request is very worrisome.  The subsystem may already have a
better way to do graceful shutdown, or just cannot follow a permit pattern.

Is it not simpler and gives more flexibility to a subsystem to drive
this via events rather than explicit callbacks to a Shutdown Manager?

i.e.

class Suspending extends ShutdownEvent
{
   void iFinishedSuspending(); // the callback to the Shutdown Manager
}

GIves the subsystem flexibility on how best to implement a graceful
shutdown.


On 6/26/12 11:00 AM, Jason T. Greene wrote:

> *EDITED*
>
> We've worked out the rough outline of how graceful shutdown will work in
> AS7.
>
> The process of graceful shutdown actually is reflected by a number of
> states:
>
> 1. Running - all services acting normally
> 2. Suspending - services refuse new "permits" (see below), existing
> permits are allowed to be retained (and threads running under such a
> permit may still acquire new permits)
> 3. Suspended - no permits are present and none may be issued
> 4. Shutting Down - our existing server stop process / reload admin mode
>
> The following transitions are allowed:
>
> 1. Running → Suspending: Transition occurs at user request (to suspend
> or gracefully shut down).
> 2. Suspending → Suspended: Transition occurs when all permits are cleared.
> 3. Suspending → Running: Transition occurs at user request (to exit
> suspend mode or cancel graceful shutdown before it completes).
> 4. Suspended → Running: Transition occurs at user request (to exit
> suspend mode).
> 5. Suspended → Shutting Down: Transition occurs automatically (if a
> graceful shutdown was requested) or at user request (if a shut down
> request of any kind is entered in the Suspended state).
> 6. Running → Shutting Down: Transition occurs at user request (to shut
> down the server "un-gracefully").
> 7. Suspending → Shutting Down (User aborts a graceful shutdown)
>
> These "permits" are issued by the "Shutdown Manager", whose job is to
> manage these states.  They are issued corresponding to the following events:
>
> 1. The invocation of an EJB method
> 2. The creation of a web session
> 3. A creation of a transaction
> 4. MessageEndpoint and WorkManager aquire permit allowing for release()
> from a thirdparty to indicate connection close.
>
> When a permit cannot be issued due to the server shutting down, a
> standard exception message should be produced so that the user can see a
> familiar error message regardless of what mechanism is used to access
> the server.
>

--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com




_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Graceful Shutdown (Revision 2)

Carlo de Wolf
It definitely can not be driven by a simple (global) semaphore. If it is
a fair one, it'll be a throughput bottleneck. If it is an unfair one you
would get unacceptable round-trip time deviation.

Carlo

On 06/27/2012 01:13 AM, Bill Burke wrote:

> Are you sure explicity granting "permits" through a "Shutdown Manager"
> the way to go?  Having to interact with a central service that grants
> permits per-request is very worrisome.  The subsystem may already have a
> better way to do graceful shutdown, or just cannot follow a permit pattern.
>
> Is it not simpler and gives more flexibility to a subsystem to drive
> this via events rather than explicit callbacks to a Shutdown Manager?
>
> i.e.
>
> class Suspending extends ShutdownEvent
> {
>     void iFinishedSuspending(); // the callback to the Shutdown Manager
> }
>
> GIves the subsystem flexibility on how best to implement a graceful
> shutdown.
>
>
> On 6/26/12 11:00 AM, Jason T. Greene wrote:
>> *EDITED*
>>
>> We've worked out the rough outline of how graceful shutdown will work in
>> AS7.
>>
>> The process of graceful shutdown actually is reflected by a number of
>> states:
>>
>> 1. Running - all services acting normally
>> 2. Suspending - services refuse new "permits" (see below), existing
>> permits are allowed to be retained (and threads running under such a
>> permit may still acquire new permits)
>> 3. Suspended - no permits are present and none may be issued
>> 4. Shutting Down - our existing server stop process / reload admin mode
>>
>> The following transitions are allowed:
>>
>> 1. Running → Suspending: Transition occurs at user request (to suspend
>> or gracefully shut down).
>> 2. Suspending → Suspended: Transition occurs when all permits are cleared.
>> 3. Suspending → Running: Transition occurs at user request (to exit
>> suspend mode or cancel graceful shutdown before it completes).
>> 4. Suspended → Running: Transition occurs at user request (to exit
>> suspend mode).
>> 5. Suspended → Shutting Down: Transition occurs automatically (if a
>> graceful shutdown was requested) or at user request (if a shut down
>> request of any kind is entered in the Suspended state).
>> 6. Running → Shutting Down: Transition occurs at user request (to shut
>> down the server "un-gracefully").
>> 7. Suspending → Shutting Down (User aborts a graceful shutdown)
>>
>> These "permits" are issued by the "Shutdown Manager", whose job is to
>> manage these states.  They are issued corresponding to the following events:
>>
>> 1. The invocation of an EJB method
>> 2. The creation of a web session
>> 3. A creation of a transaction
>> 4. MessageEndpoint and WorkManager aquire permit allowing for release()
>> from a thirdparty to indicate connection close.
>>
>> When a permit cannot be issued due to the server shutting down, a
>> standard exception message should be produced so that the user can see a
>> familiar error message regardless of what mechanism is used to access
>> the server.
>>

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Graceful Shutdown (Revision 2)

Jason T. Greene
In reply to this post by Bill Burke
The problem is you have to allow existing activity to continue when the system is suspending. So if there is a web session, a transaction, an in-flight EJB, or a stateful session bean with an active session, those should all be allowed to continue until they are done. In other words if another bean is invoked from the web session it should work and the ejb layer can't be shut down. This requires global coordination.

On Jun 26, 2012, at 7:14 PM, Bill Burke <[hidden email]> wrote:

> Are you sure explicity granting "permits" through a "Shutdown Manager"
> the way to go?  Having to interact with a central service that grants
> permits per-request is very worrisome.  The subsystem may already have a
> better way to do graceful shutdown, or just cannot follow a permit pattern.
>
> Is it not simpler and gives more flexibility to a subsystem to drive
> this via events rather than explicit callbacks to a Shutdown Manager?
>
> i.e.
>
> class Suspending extends ShutdownEvent
> {
>   void iFinishedSuspending(); // the callback to the Shutdown Manager
> }
>
> GIves the subsystem flexibility on how best to implement a graceful
> shutdown.
>
>
> On 6/26/12 11:00 AM, Jason T. Greene wrote:
>> *EDITED*
>>
>> We've worked out the rough outline of how graceful shutdown will work in
>> AS7.
>>
>> The process of graceful shutdown actually is reflected by a number of
>> states:
>>
>> 1. Running - all services acting normally
>> 2. Suspending - services refuse new "permits" (see below), existing
>> permits are allowed to be retained (and threads running under such a
>> permit may still acquire new permits)
>> 3. Suspended - no permits are present and none may be issued
>> 4. Shutting Down - our existing server stop process / reload admin mode
>>
>> The following transitions are allowed:
>>
>> 1. Running → Suspending: Transition occurs at user request (to suspend
>> or gracefully shut down).
>> 2. Suspending → Suspended: Transition occurs when all permits are cleared.
>> 3. Suspending → Running: Transition occurs at user request (to exit
>> suspend mode or cancel graceful shutdown before it completes).
>> 4. Suspended → Running: Transition occurs at user request (to exit
>> suspend mode).
>> 5. Suspended → Shutting Down: Transition occurs automatically (if a
>> graceful shutdown was requested) or at user request (if a shut down
>> request of any kind is entered in the Suspended state).
>> 6. Running → Shutting Down: Transition occurs at user request (to shut
>> down the server "un-gracefully").
>> 7. Suspending → Shutting Down (User aborts a graceful shutdown)
>>
>> These "permits" are issued by the "Shutdown Manager", whose job is to
>> manage these states.  They are issued corresponding to the following events:
>>
>> 1. The invocation of an EJB method
>> 2. The creation of a web session
>> 3. A creation of a transaction
>> 4. MessageEndpoint and WorkManager aquire permit allowing for release()
>> from a thirdparty to indicate connection close.
>>
>> When a permit cannot be issued due to the server shutting down, a
>> standard exception message should be produced so that the user can see a
>> familiar error message regardless of what mechanism is used to access
>> the server.
>>
>
> --
> Bill Burke
> JBoss, a division of Red Hat
> http://bill.burkecentral.com
>
>
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Graceful Shutdown (Revision 2)

David Lloyd-2
In reply to this post by Bill Burke
On 6/26/12 7:13 PM, Bill Burke wrote:
> Are you sure explicity granting "permits" through a "Shutdown Manager"
> the way to go?

Yes, we are 100% sure.

> Having to interact with a central service that grants
> permits per-request is very worrisome.  The subsystem may already have a
> better way to do graceful shutdown, or just cannot follow a permit pattern.

No, we went into great detail to work this out.  But if you have a
specific case that you think warrants another look, feel free to raise it.

> Is it not simpler and gives more flexibility to a subsystem to drive
> this via events rather than explicit callbacks to a Shutdown Manager?

No, because the "suspended" state isn't really a state, and it is
certainly not detectable from each subsystem.

For example, there may be no remaining EJB sessions.  However, that does
not mean that EJB can refuse new session creation.  In particular, if
you are in a suspended state and deploy a singleton EJB which has a
@PostConstruct on it, and that interceptor causes a session to be
created, this is allowed as deployment must always gain a permit (else
it'd fail).

Note, people, that permits are a *conceptual* entity.  There is no need
to have permit objects, nor even a semaphore, nor even a non-blocking
counter.  I do know a thing or two about eliminating concurrency
bottlenecks.

> i.e.
>
> class Suspending extends ShutdownEvent
> {
>     void iFinishedSuspending(); // the callback to the Shutdown Manager
> }
>
> GIves the subsystem flexibility on how best to implement a graceful
> shutdown.
>
>
> On 6/26/12 11:00 AM, Jason T. Greene wrote:
>> *EDITED*
>>
>> We've worked out the rough outline of how graceful shutdown will work in
>> AS7.
>>
>> The process of graceful shutdown actually is reflected by a number of
>> states:
>>
>> 1. Running - all services acting normally
>> 2. Suspending - services refuse new "permits" (see below), existing
>> permits are allowed to be retained (and threads running under such a
>> permit may still acquire new permits)
>> 3. Suspended - no permits are present and none may be issued
>> 4. Shutting Down - our existing server stop process / reload admin mode
>>
>> The following transitions are allowed:
>>
>> 1. Running → Suspending: Transition occurs at user request (to suspend
>> or gracefully shut down).
>> 2. Suspending → Suspended: Transition occurs when all permits are cleared.
>> 3. Suspending → Running: Transition occurs at user request (to exit
>> suspend mode or cancel graceful shutdown before it completes).
>> 4. Suspended → Running: Transition occurs at user request (to exit
>> suspend mode).
>> 5. Suspended → Shutting Down: Transition occurs automatically (if a
>> graceful shutdown was requested) or at user request (if a shut down
>> request of any kind is entered in the Suspended state).
>> 6. Running → Shutting Down: Transition occurs at user request (to shut
>> down the server "un-gracefully").
>> 7. Suspending → Shutting Down (User aborts a graceful shutdown)
>>
>> These "permits" are issued by the "Shutdown Manager", whose job is to
>> manage these states.  They are issued corresponding to the following events:
>>
>> 1. The invocation of an EJB method
>> 2. The creation of a web session
>> 3. A creation of a transaction
>> 4. MessageEndpoint and WorkManager aquire permit allowing for release()
>> from a thirdparty to indicate connection close.
>>
>> When a permit cannot be issued due to the server shutting down, a
>> standard exception message should be produced so that the user can see a
>> familiar error message regardless of what mechanism is used to access
>> the server.
>>
>


--
- DML
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Graceful Shutdown (Revision 2)

Scott Marlow
On 06/27/2012 08:34 AM, David M. Lloyd wrote:

> On 6/26/12 7:13 PM, Bill Burke wrote:
>> Are you sure explicity granting "permits" through a "Shutdown Manager"
>> the way to go?
>
> Yes, we are 100% sure.
>
>> Having to interact with a central service that grants
>> permits per-request is very worrisome.  The subsystem may already have a
>> better way to do graceful shutdown, or just cannot follow a permit pattern.
>
> No, we went into great detail to work this out.  But if you have a
> specific case that you think warrants another look, feel free to raise it.
>
>> Is it not simpler and gives more flexibility to a subsystem to drive
>> this via events rather than explicit callbacks to a Shutdown Manager?
>
> No, because the "suspended" state isn't really a state, and it is
> certainly not detectable from each subsystem.
>
> For example, there may be no remaining EJB sessions.  However, that does
> not mean that EJB can refuse new session creation.  In particular, if
> you are in a suspended state and deploy a singleton EJB which has a
> @PostConstruct on it, and that interceptor causes a session to be
> created, this is allowed as deployment must always gain a permit (else
> it'd fail).

Does deploying an application during the "suspended" state, cancel the
shutdown (bringing the server back to a "running" state?)  You said
below that no permits are allowed in the "suspended" state, so a state
transition might be needed for deployment (I think).  Or is deployment
only allowed in "Suspending/Running" states?

>
> Note, people, that permits are a *conceptual* entity.  There is no need
> to have permit objects, nor even a semaphore, nor even a non-blocking
> counter.  I do know a thing or two about eliminating concurrency
> bottlenecks.
>
>> i.e.
>>
>> class Suspending extends ShutdownEvent
>> {
>>      void iFinishedSuspending(); // the callback to the Shutdown Manager
>> }
>>
>> GIves the subsystem flexibility on how best to implement a graceful
>> shutdown.
>>
>>
>> On 6/26/12 11:00 AM, Jason T. Greene wrote:
>>> *EDITED*
>>>
>>> We've worked out the rough outline of how graceful shutdown will work in
>>> AS7.
>>>
>>> The process of graceful shutdown actually is reflected by a number of
>>> states:
>>>
>>> 1. Running - all services acting normally
>>> 2. Suspending - services refuse new "permits" (see below), existing
>>> permits are allowed to be retained (and threads running under such a
>>> permit may still acquire new permits)
>>> 3. Suspended - no permits are present and none may be issued
>>> 4. Shutting Down - our existing server stop process / reload admin mode
>>>
>>> The following transitions are allowed:
>>>
>>> 1. Running → Suspending: Transition occurs at user request (to suspend
>>> or gracefully shut down).
>>> 2. Suspending → Suspended: Transition occurs when all permits are cleared.
>>> 3. Suspending → Running: Transition occurs at user request (to exit
>>> suspend mode or cancel graceful shutdown before it completes).
>>> 4. Suspended → Running: Transition occurs at user request (to exit
>>> suspend mode).
>>> 5. Suspended → Shutting Down: Transition occurs automatically (if a
>>> graceful shutdown was requested) or at user request (if a shut down
>>> request of any kind is entered in the Suspended state).
>>> 6. Running → Shutting Down: Transition occurs at user request (to shut
>>> down the server "un-gracefully").
>>> 7. Suspending → Shutting Down (User aborts a graceful shutdown)
>>>
>>> These "permits" are issued by the "Shutdown Manager", whose job is to
>>> manage these states.  They are issued corresponding to the following events:
>>>
>>> 1. The invocation of an EJB method

This sounds like an EJB component interceptor needs to be changed to
handle this part.

>>> 2. The creation of a web session

Would this be handled by an org.apache.catalina.Valve or some equivalent?

>>> 3. A creation of a transaction

What is the use case that requires this that is not already issuing a
permit at a higher (top level) container?

>>> 4. MessageEndpoint and WorkManager aquire permit allowing for release()
>>> from a thirdparty to indicate connection close.

What is the hook/API for plugging into these so the shutdown manager can
be invoked?

5.  Unmanaged Thread use

No one has mentioned naked or unmanaged threads.  Can naked threads also
call the Shutdown Manager?  If yes, I think the Shutdown Manager API
needs to be public so that custom threads could also participate in
graceful shutdown.

>>>
>>> When a permit cannot be issued due to the server shutting down, a
>>> standard exception message should be produced so that the user can see a
>>> familiar error message regardless of what mechanism is used to access
>>> the server.

Sounds like a good idea.  The Shutdown Manager should probably throw the
familiar exception.

>>>
>>
>
>

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev