Clustered invocation design

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Clustered invocation design

David Lloyd-2
There are at least two basic paths we can follow for clustered
invocation based on the current architecture.  The right choice is going
to depend primarily upon the expected use cases, which I am not in a
position to properly judge.

Option 1: Clustered Invocation Transport
----------------------------------------

In this option, we introduce a new "LAN" transport type for invocation
on the cluster.  The transport would use direct TCP connections or UDP
messages (or both, depending on request size) to convey the invocation.
  The characteristics of this option are as follows:

- Security: reliance on physical network security only (no TLS or
authentication)
- Latency is very low, even to new nodes
- Topology changes can be conveyed as separate asynchronous messages
- Invocations from external networks would happen through a proxy node,
with Remoting being bridged to the LAN, to perform security functions

Option 2: Load-balanced Remoting Connections
--------------------------------------------

In this option, we rely on the client to establish one or more Remoting
connection(s) to one or more of the nodes of the cluster.  Logic in the
client will be used to determine what connection(s) to use for what
clusters.  We have the option of automatically connecting as topology
changes or requiring the user to set up the connections in advance.
Note that automatic connection cannot work in the case of
user-interactive authentication.  Characteristics:

- Security: full authentication and TLS supported
- Latency is low once the connection is established, however there is
some overhead involved in authentication and security negotiation
- Topology changes should be asynchronous notifications
- Each connection has to be separately authenticated
- Automatically establishing connections is not presently supported, so
we'd need a bit of infrastructure for that.  Deal with user-interactive
authentication.  Deal with connection lifecycle management.  Deal with
configuration.  This will be a point of fragility

Summary
-------

For both options, we have to determine an appropriate load-balancing
strategy.  The choice of direction will affect how our clustering and
transaction interceptors function.  We also have to suss out the logic
around dealing with conflicting or wrongly-ordered topology updates;
hopefully our existing policies will continue to apply.

--
- DML
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

Paul Ferraro
On Tue, 2011-10-11 at 10:59 -0500, David M. Lloyd wrote:

> There are at least two basic paths we can follow for clustered
> invocation based on the current architecture.  The right choice is going
> to depend primarily upon the expected use cases, which I am not in a
> position to properly judge.
>
> Option 1: Clustered Invocation Transport
> ----------------------------------------
>
> In this option, we introduce a new "LAN" transport type for invocation
> on the cluster.  The transport would use direct TCP connections or UDP
> messages (or both, depending on request size) to convey the invocation.
>   The characteristics of this option are as follows:
>
> - Security: reliance on physical network security only (no TLS or
> authentication)
> - Latency is very low, even to new nodes
> - Topology changes can be conveyed as separate asynchronous messages
> - Invocations from external networks would happen through a proxy node,
> with Remoting being bridged to the LAN, to perform security functions
>
> Option 2: Load-balanced Remoting Connections
> --------------------------------------------
>
> In this option, we rely on the client to establish one or more Remoting
> connection(s) to one or more of the nodes of the cluster.  Logic in the
> client will be used to determine what connection(s) to use for what
> clusters.  We have the option of automatically connecting as topology
> changes or requiring the user to set up the connections in advance.
> Note that automatic connection cannot work in the case of
> user-interactive authentication.  Characteristics:
>
> - Security: full authentication and TLS supported
> - Latency is low once the connection is established, however there is
> some overhead involved in authentication and security negotiation
> - Topology changes should be asynchronous notifications
> - Each connection has to be separately authenticated
> - Automatically establishing connections is not presently supported, so
> we'd need a bit of infrastructure for that.  Deal with user-interactive
> authentication.  Deal with connection lifecycle management.  Deal with
> configuration.  This will be a point of fragility
>
> Summary
> -------
>
> For both options, we have to determine an appropriate load-balancing
> strategy.  The choice of direction will affect how our clustering and
> transaction interceptors function.  We also have to suss out the logic
> around dealing with conflicting or wrongly-ordered topology updates;
> hopefully our existing policies will continue to apply.

Do topology changes really need to be asynchronous notifications?  Can
we simply update cluster topology per invocation?

Maintaining an accurate cluster topology via asynchronous notifications
has the following benefits:
1. Topology changes between invocations won't require failover in the
event of a load balanced invocation (as opposed to a sticky one).
* Load balancing will potentially be more effective following topology
changes by leveraging new cluster members.
* Minimizes invocation payload (since we don't need to tack on cluster
topology to every invocation response).  We can optimize this somewhat
by sending a topology view ID with the invocation request, and only
including the topology in the response if the topology changed (i.e.
request view ID != current view ID).

The only disadvantage of which I can think is implementation complexity.
Topology update ordering is not an issue if we take the simpler
approach.  However, we can also make an assumption that topology changes
are not common - so it becomes a matter of whether or not to optimize
for frequent topology changes.

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

David Lloyd-2
On 10/13/2011 10:45 AM, Paul Ferraro wrote:

> On Tue, 2011-10-11 at 10:59 -0500, David M. Lloyd wrote:
>> There are at least two basic paths we can follow for clustered
>> invocation based on the current architecture.  The right choice is going
>> to depend primarily upon the expected use cases, which I am not in a
>> position to properly judge.
>>
>> Option 1: Clustered Invocation Transport
>> ----------------------------------------
>>
>> In this option, we introduce a new "LAN" transport type for invocation
>> on the cluster.  The transport would use direct TCP connections or UDP
>> messages (or both, depending on request size) to convey the invocation.
>>    The characteristics of this option are as follows:
>>
>> - Security: reliance on physical network security only (no TLS or
>> authentication)
>> - Latency is very low, even to new nodes
>> - Topology changes can be conveyed as separate asynchronous messages
>> - Invocations from external networks would happen through a proxy node,
>> with Remoting being bridged to the LAN, to perform security functions
>>
>> Option 2: Load-balanced Remoting Connections
>> --------------------------------------------
>>
>> In this option, we rely on the client to establish one or more Remoting
>> connection(s) to one or more of the nodes of the cluster.  Logic in the
>> client will be used to determine what connection(s) to use for what
>> clusters.  We have the option of automatically connecting as topology
>> changes or requiring the user to set up the connections in advance.
>> Note that automatic connection cannot work in the case of
>> user-interactive authentication.  Characteristics:
>>
>> - Security: full authentication and TLS supported
>> - Latency is low once the connection is established, however there is
>> some overhead involved in authentication and security negotiation
>> - Topology changes should be asynchronous notifications
>> - Each connection has to be separately authenticated
>> - Automatically establishing connections is not presently supported, so
>> we'd need a bit of infrastructure for that.  Deal with user-interactive
>> authentication.  Deal with connection lifecycle management.  Deal with
>> configuration.  This will be a point of fragility
>>
>> Summary
>> -------
>>
>> For both options, we have to determine an appropriate load-balancing
>> strategy.  The choice of direction will affect how our clustering and
>> transaction interceptors function.  We also have to suss out the logic
>> around dealing with conflicting or wrongly-ordered topology updates;
>> hopefully our existing policies will continue to apply.
>
> Do topology changes really need to be asynchronous notifications?  Can
> we simply update cluster topology per invocation?
>
> Maintaining an accurate cluster topology via asynchronous notifications
> has the following benefits:
> 1. Topology changes between invocations won't require failover in the
> event of a load balanced invocation (as opposed to a sticky one).
> * Load balancing will potentially be more effective following topology
> changes by leveraging new cluster members.
> * Minimizes invocation payload (since we don't need to tack on cluster
> topology to every invocation response).  We can optimize this somewhat
> by sending a topology view ID with the invocation request, and only
> including the topology in the response if the topology changed (i.e.
> request view ID != current view ID).
>
> The only disadvantage of which I can think is implementation complexity.
> Topology update ordering is not an issue if we take the simpler
> approach.  However, we can also make an assumption that topology changes
> are not common - so it becomes a matter of whether or not to optimize
> for frequent topology changes.

The problem is that (with R3 anyway) many threads may concurrently use a
single connection, and invocation replies can come in any order with
respect to the original invocation, so in effect even if we attach
topology information to the reply they're still essentially asynchronous
with the disadvantage that topology changes also bog down invocation
response times.

And if we did a non-persistent-connection-based transport, there's even
less of a guarantee because each reply could come in separate packets or
connections which can be arbitrary reordered at a network level.

In other words, topology update ordering is always an issue, even more
so when multiple nodes come into play.

Using a view ID is fine as long as all nodes in the cluster always agree
on what view ID is the latest (which afaik is essentially impossible to
guarantee).

But again all this ties back to the transport implementation.  R3
transport means persistent connections but we likely cannot
automatically bring up new connections to new nodes; custom transport
would mean no persistent connections but new nodes can be accessed
instantly.
--
- DML
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

Galder Zamarreño

On Oct 13, 2011, at 5:57 PM, David M. Lloyd wrote:

> On 10/13/2011 10:45 AM, Paul Ferraro wrote:
>> On Tue, 2011-10-11 at 10:59 -0500, David M. Lloyd wrote:
>>> There are at least two basic paths we can follow for clustered
>>> invocation based on the current architecture.  The right choice is going
>>> to depend primarily upon the expected use cases, which I am not in a
>>> position to properly judge.
>>>
>>> Option 1: Clustered Invocation Transport
>>> ----------------------------------------
>>>
>>> In this option, we introduce a new "LAN" transport type for invocation
>>> on the cluster.  The transport would use direct TCP connections or UDP
>>> messages (or both, depending on request size) to convey the invocation.
>>>   The characteristics of this option are as follows:
>>>
>>> - Security: reliance on physical network security only (no TLS or
>>> authentication)
>>> - Latency is very low, even to new nodes
>>> - Topology changes can be conveyed as separate asynchronous messages
>>> - Invocations from external networks would happen through a proxy node,
>>> with Remoting being bridged to the LAN, to perform security functions
>>>
>>> Option 2: Load-balanced Remoting Connections
>>> --------------------------------------------
>>>
>>> In this option, we rely on the client to establish one or more Remoting
>>> connection(s) to one or more of the nodes of the cluster.  Logic in the
>>> client will be used to determine what connection(s) to use for what
>>> clusters.  We have the option of automatically connecting as topology
>>> changes or requiring the user to set up the connections in advance.
>>> Note that automatic connection cannot work in the case of
>>> user-interactive authentication.  Characteristics:
>>>
>>> - Security: full authentication and TLS supported
>>> - Latency is low once the connection is established, however there is
>>> some overhead involved in authentication and security negotiation
>>> - Topology changes should be asynchronous notifications
>>> - Each connection has to be separately authenticated
>>> - Automatically establishing connections is not presently supported, so
>>> we'd need a bit of infrastructure for that.  Deal with user-interactive
>>> authentication.  Deal with connection lifecycle management.  Deal with
>>> configuration.  This will be a point of fragility
>>>
>>> Summary
>>> -------
>>>
>>> For both options, we have to determine an appropriate load-balancing
>>> strategy.  The choice of direction will affect how our clustering and
>>> transaction interceptors function.  We also have to suss out the logic
>>> around dealing with conflicting or wrongly-ordered topology updates;
>>> hopefully our existing policies will continue to apply.
>>
>> Do topology changes really need to be asynchronous notifications?  Can
>> we simply update cluster topology per invocation?
>>
>> Maintaining an accurate cluster topology via asynchronous notifications
>> has the following benefits:
>> 1. Topology changes between invocations won't require failover in the
>> event of a load balanced invocation (as opposed to a sticky one).
>> * Load balancing will potentially be more effective following topology
>> changes by leveraging new cluster members.
>> * Minimizes invocation payload (since we don't need to tack on cluster
>> topology to every invocation response).  We can optimize this somewhat
>> by sending a topology view ID with the invocation request, and only
>> including the topology in the response if the topology changed (i.e.
>> request view ID != current view ID).

As a side note, this is what Hot Rod does in Infinispan in order to detect stale views.

We might optimise this further in the future as indicated in https://issues.jboss.org/browse/ISPN-1403, but this optimisation is particular to the Infinispan use case.

Alternative optimisation avenues could be investigated for clustered invocations.

>>
>> The only disadvantage of which I can think is implementation complexity.
>> Topology update ordering is not an issue if we take the simpler
>> approach.  However, we can also make an assumption that topology changes
>> are not common - so it becomes a matter of whether or not to optimize
>> for frequent topology changes.
>
> The problem is that (with R3 anyway) many threads may concurrently use a
> single connection, and invocation replies can come in any order with
> respect to the original invocation, so in effect even if we attach
> topology information to the reply they're still essentially asynchronous
> with the disadvantage that topology changes also bog down invocation
> response times.
>
> And if we did a non-persistent-connection-based transport, there's even
> less of a guarantee because each reply could come in separate packets or
> connections which can be arbitrary reordered at a network level.
>
> In other words, topology update ordering is always an issue, even more
> so when multiple nodes come into play.
>
> Using a view ID is fine as long as all nodes in the cluster always agree
> on what view ID is the latest (which afaik is essentially impossible to
> guarantee).

Well, if they're running in a cluster, JGroups provides a guarantee via GMS that a common viewId is maintained, at least until a cluster partitition occurs.

When a partition occurs, several cluster islands can evolve their viewId independently.

>
> But again all this ties back to the transport implementation.  R3
> transport means persistent connections but we likely cannot
> automatically bring up new connections to new nodes; custom transport
> would mean no persistent connections but new nodes can be accessed
> instantly.
> --
> - DML
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

David Lloyd-2
On 10/14/2011 04:05 AM, Galder Zamarreño wrote:

>
> On Oct 13, 2011, at 5:57 PM, David M. Lloyd wrote:
>
>> On 10/13/2011 10:45 AM, Paul Ferraro wrote:
>>> On Tue, 2011-10-11 at 10:59 -0500, David M. Lloyd wrote:
>>>> There are at least two basic paths we can follow for clustered
>>>> invocation based on the current architecture.  The right choice is going
>>>> to depend primarily upon the expected use cases, which I am not in a
>>>> position to properly judge.
>>>>
>>>> Option 1: Clustered Invocation Transport
>>>> ----------------------------------------
>>>>
>>>> In this option, we introduce a new "LAN" transport type for invocation
>>>> on the cluster.  The transport would use direct TCP connections or UDP
>>>> messages (or both, depending on request size) to convey the invocation.
>>>>    The characteristics of this option are as follows:
>>>>
>>>> - Security: reliance on physical network security only (no TLS or
>>>> authentication)
>>>> - Latency is very low, even to new nodes
>>>> - Topology changes can be conveyed as separate asynchronous messages
>>>> - Invocations from external networks would happen through a proxy node,
>>>> with Remoting being bridged to the LAN, to perform security functions
>>>>
>>>> Option 2: Load-balanced Remoting Connections
>>>> --------------------------------------------
>>>>
>>>> In this option, we rely on the client to establish one or more Remoting
>>>> connection(s) to one or more of the nodes of the cluster.  Logic in the
>>>> client will be used to determine what connection(s) to use for what
>>>> clusters.  We have the option of automatically connecting as topology
>>>> changes or requiring the user to set up the connections in advance.
>>>> Note that automatic connection cannot work in the case of
>>>> user-interactive authentication.  Characteristics:
>>>>
>>>> - Security: full authentication and TLS supported
>>>> - Latency is low once the connection is established, however there is
>>>> some overhead involved in authentication and security negotiation
>>>> - Topology changes should be asynchronous notifications
>>>> - Each connection has to be separately authenticated
>>>> - Automatically establishing connections is not presently supported, so
>>>> we'd need a bit of infrastructure for that.  Deal with user-interactive
>>>> authentication.  Deal with connection lifecycle management.  Deal with
>>>> configuration.  This will be a point of fragility
>>>>
>>>> Summary
>>>> -------
>>>>
>>>> For both options, we have to determine an appropriate load-balancing
>>>> strategy.  The choice of direction will affect how our clustering and
>>>> transaction interceptors function.  We also have to suss out the logic
>>>> around dealing with conflicting or wrongly-ordered topology updates;
>>>> hopefully our existing policies will continue to apply.
>>>
>>> Do topology changes really need to be asynchronous notifications?  Can
>>> we simply update cluster topology per invocation?
>>>
>>> Maintaining an accurate cluster topology via asynchronous notifications
>>> has the following benefits:
>>> 1. Topology changes between invocations won't require failover in the
>>> event of a load balanced invocation (as opposed to a sticky one).
>>> * Load balancing will potentially be more effective following topology
>>> changes by leveraging new cluster members.
>>> * Minimizes invocation payload (since we don't need to tack on cluster
>>> topology to every invocation response).  We can optimize this somewhat
>>> by sending a topology view ID with the invocation request, and only
>>> including the topology in the response if the topology changed (i.e.
>>> request view ID != current view ID).
>
> As a side note, this is what Hot Rod does in Infinispan in order to detect stale views.
>
> We might optimise this further in the future as indicated in https://issues.jboss.org/browse/ISPN-1403, but this optimisation is particular to the Infinispan use case.
>
> Alternative optimisation avenues could be investigated for clustered invocations.
>
>>>
>>> The only disadvantage of which I can think is implementation complexity.
>>> Topology update ordering is not an issue if we take the simpler
>>> approach.  However, we can also make an assumption that topology changes
>>> are not common - so it becomes a matter of whether or not to optimize
>>> for frequent topology changes.
>>
>> The problem is that (with R3 anyway) many threads may concurrently use a
>> single connection, and invocation replies can come in any order with
>> respect to the original invocation, so in effect even if we attach
>> topology information to the reply they're still essentially asynchronous
>> with the disadvantage that topology changes also bog down invocation
>> response times.
>>
>> And if we did a non-persistent-connection-based transport, there's even
>> less of a guarantee because each reply could come in separate packets or
>> connections which can be arbitrary reordered at a network level.
>>
>> In other words, topology update ordering is always an issue, even more
>> so when multiple nodes come into play.
>>
>> Using a view ID is fine as long as all nodes in the cluster always agree
>> on what view ID is the latest (which afaik is essentially impossible to
>> guarantee).
>
> Well, if they're running in a cluster, JGroups provides a guarantee via GMS that a common viewId is maintained, at least until a cluster partitition occurs.
>
> When a partition occurs, several cluster islands can evolve their viewId independently.

Okay, so as long as our cluster client node is directly participating in
the JGroups group then we can rely on that I guess?

This still brings us no closer to a determination about what approach we
should take for the invocation protocol itself.
--
- DML
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

Jaikiran Pai
In reply to this post by David Lloyd-2
 From someone who has very limited knowledge of clustering internals -
the only "advantage" (for us) in going with #1 would be the ability to
setup easy connectivity? From what I understand, #2 would allow more
flexibility to the users (and a bit of complexity for us in managing the
connections and authentication) on setting up the cluster for the
invocations. Am I right?

-Jaikiran
On Tuesday 11 October 2011 09:29 PM, David M. Lloyd wrote:

> There are at least two basic paths we can follow for clustered
> invocation based on the current architecture.  The right choice is going
> to depend primarily upon the expected use cases, which I am not in a
> position to properly judge.
>
> Option 1: Clustered Invocation Transport
> ----------------------------------------
>
> In this option, we introduce a new "LAN" transport type for invocation
> on the cluster.  The transport would use direct TCP connections or UDP
> messages (or both, depending on request size) to convey the invocation.
>    The characteristics of this option are as follows:
>
> - Security: reliance on physical network security only (no TLS or
> authentication)
> - Latency is very low, even to new nodes
> - Topology changes can be conveyed as separate asynchronous messages
> - Invocations from external networks would happen through a proxy node,
> with Remoting being bridged to the LAN, to perform security functions
>
> Option 2: Load-balanced Remoting Connections
> --------------------------------------------
>
> In this option, we rely on the client to establish one or more Remoting
> connection(s) to one or more of the nodes of the cluster.  Logic in the
> client will be used to determine what connection(s) to use for what
> clusters.  We have the option of automatically connecting as topology
> changes or requiring the user to set up the connections in advance.
> Note that automatic connection cannot work in the case of
> user-interactive authentication.  Characteristics:
>
> - Security: full authentication and TLS supported
> - Latency is low once the connection is established, however there is
> some overhead involved in authentication and security negotiation
> - Topology changes should be asynchronous notifications
> - Each connection has to be separately authenticated
> - Automatically establishing connections is not presently supported, so
> we'd need a bit of infrastructure for that.  Deal with user-interactive
> authentication.  Deal with connection lifecycle management.  Deal with
> configuration.  This will be a point of fragility
>
> Summary
> -------
>
> For both options, we have to determine an appropriate load-balancing
> strategy.  The choice of direction will affect how our clustering and
> transaction interceptors function.  We also have to suss out the logic
> around dealing with conflicting or wrongly-ordered topology updates;
> hopefully our existing policies will continue to apply.
>

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

Jason T. Greene
I think 2 is the only realistic option since it supports distant remote clients. It's also similar to what we had before. Async topology changes shouldn't be an issue IMO. The server endpoint always has the option to redirect to the right location. As long as we have the requirement that the server endpoints be a jgroups cluster, then we can reuse the jgroups view IDs which have already formed a consensus.

Another interesting aspect is stateless load balancing policy and stateful session affinity. Since time is short my thoughts are start basic: round robin + sticky failover.

Sent from my iPad

On Oct 18, 2011, at 9:31 AM, Jaikiran Pai <[hidden email]> wrote:

> From someone who has very limited knowledge of clustering internals -
> the only "advantage" (for us) in going with #1 would be the ability to
> setup easy connectivity? From what I understand, #2 would allow more
> flexibility to the users (and a bit of complexity for us in managing the
> connections and authentication) on setting up the cluster for the
> invocations. Am I right?
>
> -Jaikiran
> On Tuesday 11 October 2011 09:29 PM, David M. Lloyd wrote:
>> There are at least two basic paths we can follow for clustered
>> invocation based on the current architecture.  The right choice is going
>> to depend primarily upon the expected use cases, which I am not in a
>> position to properly judge.
>>
>> Option 1: Clustered Invocation Transport
>> ----------------------------------------
>>
>> In this option, we introduce a new "LAN" transport type for invocation
>> on the cluster.  The transport would use direct TCP connections or UDP
>> messages (or both, depending on request size) to convey the invocation.
>>   The characteristics of this option are as follows:
>>
>> - Security: reliance on physical network security only (no TLS or
>> authentication)
>> - Latency is very low, even to new nodes
>> - Topology changes can be conveyed as separate asynchronous messages
>> - Invocations from external networks would happen through a proxy node,
>> with Remoting being bridged to the LAN, to perform security functions
>>
>> Option 2: Load-balanced Remoting Connections
>> --------------------------------------------
>>
>> In this option, we rely on the client to establish one or more Remoting
>> connection(s) to one or more of the nodes of the cluster.  Logic in the
>> client will be used to determine what connection(s) to use for what
>> clusters.  We have the option of automatically connecting as topology
>> changes or requiring the user to set up the connections in advance.
>> Note that automatic connection cannot work in the case of
>> user-interactive authentication.  Characteristics:
>>
>> - Security: full authentication and TLS supported
>> - Latency is low once the connection is established, however there is
>> some overhead involved in authentication and security negotiation
>> - Topology changes should be asynchronous notifications
>> - Each connection has to be separately authenticated
>> - Automatically establishing connections is not presently supported, so
>> we'd need a bit of infrastructure for that.  Deal with user-interactive
>> authentication.  Deal with connection lifecycle management.  Deal with
>> configuration.  This will be a point of fragility
>>
>> Summary
>> -------
>>
>> For both options, we have to determine an appropriate load-balancing
>> strategy.  The choice of direction will affect how our clustering and
>> transaction interceptors function.  We also have to suss out the logic
>> around dealing with conflicting or wrongly-ordered topology updates;
>> hopefully our existing policies will continue to apply.
>>
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

David Lloyd-2
Great.  We're now officially moving forward with #2.  I am not
addressing automatic connection management at this time; it will be the
user's responsibility to establish connections for now.  We can revisit
our options once we're up and running.

On 10/18/2011 08:07 PM, Jason Greene wrote:

> I think 2 is the only realistic option since it supports distant remote clients. It's also similar to what we had before. Async topology changes shouldn't be an issue IMO. The server endpoint always has the option to redirect to the right location. As long as we have the requirement that the server endpoints be a jgroups cluster, then we can reuse the jgroups view IDs which have already formed a consensus.
>
> Another interesting aspect is stateless load balancing policy and stateful session affinity. Since time is short my thoughts are start basic: round robin + sticky failover.
>
> Sent from my iPad
>
> On Oct 18, 2011, at 9:31 AM, Jaikiran Pai<[hidden email]>  wrote:
>
>>  From someone who has very limited knowledge of clustering internals -
>> the only "advantage" (for us) in going with #1 would be the ability to
>> setup easy connectivity? From what I understand, #2 would allow more
>> flexibility to the users (and a bit of complexity for us in managing the
>> connections and authentication) on setting up the cluster for the
>> invocations. Am I right?
>>
>> -Jaikiran
>> On Tuesday 11 October 2011 09:29 PM, David M. Lloyd wrote:
>>> There are at least two basic paths we can follow for clustered
>>> invocation based on the current architecture.  The right choice is going
>>> to depend primarily upon the expected use cases, which I am not in a
>>> position to properly judge.
>>>
>>> Option 1: Clustered Invocation Transport
>>> ----------------------------------------
>>>
>>> In this option, we introduce a new "LAN" transport type for invocation
>>> on the cluster.  The transport would use direct TCP connections or UDP
>>> messages (or both, depending on request size) to convey the invocation.
>>>    The characteristics of this option are as follows:
>>>
>>> - Security: reliance on physical network security only (no TLS or
>>> authentication)
>>> - Latency is very low, even to new nodes
>>> - Topology changes can be conveyed as separate asynchronous messages
>>> - Invocations from external networks would happen through a proxy node,
>>> with Remoting being bridged to the LAN, to perform security functions
>>>
>>> Option 2: Load-balanced Remoting Connections
>>> --------------------------------------------
>>>
>>> In this option, we rely on the client to establish one or more Remoting
>>> connection(s) to one or more of the nodes of the cluster.  Logic in the
>>> client will be used to determine what connection(s) to use for what
>>> clusters.  We have the option of automatically connecting as topology
>>> changes or requiring the user to set up the connections in advance.
>>> Note that automatic connection cannot work in the case of
>>> user-interactive authentication.  Characteristics:
>>>
>>> - Security: full authentication and TLS supported
>>> - Latency is low once the connection is established, however there is
>>> some overhead involved in authentication and security negotiation
>>> - Topology changes should be asynchronous notifications
>>> - Each connection has to be separately authenticated
>>> - Automatically establishing connections is not presently supported, so
>>> we'd need a bit of infrastructure for that.  Deal with user-interactive
>>> authentication.  Deal with connection lifecycle management.  Deal with
>>> configuration.  This will be a point of fragility
>>>
>>> Summary
>>> -------
>>>
>>> For both options, we have to determine an appropriate load-balancing
>>> strategy.  The choice of direction will affect how our clustering and
>>> transaction interceptors function.  We also have to suss out the logic
>>> around dealing with conflicting or wrongly-ordered topology updates;
>>> hopefully our existing policies will continue to apply.
>>>
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


--
- DML
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

Andrig Miller
One thing I would like to add to the discussion, which I was thinking about yesterday.

Our customer base has grown substantially, and the type of customer we are now engaging with have much larger deployments than in the past.  I continue to here complaints about those customers ability to move from one major EAP release to another.  We also have a specific customer that made us commit to having our JMS client remain compatible across major releases (EAP 5 JMS client can talk to an EAP 6 server, and an EAP 7 server too).  I know of another customer that has complained about remote EJB calls in the past, in this regard as well.  With these large deployments it becomes very difficult to a "big bang" upgrade, and they need to ability to have old clients talk to new servers.

So, with the design of this, it would be great to take into account this need.  I'm just thinking that if there ends up being a need for the wire protocol to change in the future, that we would need to still be able to support the older clients.

Andy


From: "David M. Lloyd" <[hidden email]>
To: [hidden email]
Sent: Tuesday, October 18, 2011 7:43:50 PM
Subject: Re: [jboss-as7-dev] Clustered invocation design

Great.  We're now officially moving forward with #2.  I am not
addressing automatic connection management at this time; it will be the
user's responsibility to establish connections for now.  We can revisit
our options once we're up and running.

On 10/18/2011 08:07 PM, Jason Greene wrote:

> I think 2 is the only realistic option since it supports distant remote clients. It's also similar to what we had before. Async topology changes shouldn't be an issue IMO. The server endpoint always has the option to redirect to the right location. As long as we have the requirement that the server endpoints be a jgroups cluster, then we can reuse the jgroups view IDs which have already formed a consensus.
>
> Another interesting aspect is stateless load balancing policy and stateful session affinity. Since time is short my thoughts are start basic: round robin + sticky failover.
>
> Sent from my iPad
>
> On Oct 18, 2011, at 9:31 AM, Jaikiran Pai<[hidden email]>  wrote:
>
>>  From someone who has very limited knowledge of clustering internals -
>> the only "advantage" (for us) in going with #1 would be the ability to
>> setup easy connectivity? From what I understand, #2 would allow more
>> flexibility to the users (and a bit of complexity for us in managing the
>> connections and authentication) on setting up the cluster for the
>> invocations. Am I right?
>>
>> -Jaikiran
>> On Tuesday 11 October 2011 09:29 PM, David M. Lloyd wrote:
>>> There are at least two basic paths we can follow for clustered
>>> invocation based on the current architecture.  The right choice is going
>>> to depend primarily upon the expected use cases, which I am not in a
>>> position to properly judge.
>>>
>>> Option 1: Clustered Invocation Transport
>>> ----------------------------------------
>>>
>>> In this option, we introduce a new "LAN" transport type for invocation
>>> on the cluster.  The transport would use direct TCP connections or UDP
>>> messages (or both, depending on request size) to convey the invocation.
>>>    The characteristics of this option are as follows:
>>>
>>> - Security: reliance on physical network security only (no TLS or
>>> authentication)
>>> - Latency is very low, even to new nodes
>>> - Topology changes can be conveyed as separate asynchronous messages
>>> - Invocations from external networks would happen through a proxy node,
>>> with Remoting being bridged to the LAN, to perform security functions
>>>
>>> Option 2: Load-balanced Remoting Connections
>>> --------------------------------------------
>>>
>>> In this option, we rely on the client to establish one or more Remoting
>>> connection(s) to one or more of the nodes of the cluster.  Logic in the
>>> client will be used to determine what connection(s) to use for what
>>> clusters.  We have the option of automatically connecting as topology
>>> changes or requiring the user to set up the connections in advance.
>>> Note that automatic connection cannot work in the case of
>>> user-interactive authentication.  Characteristics:
>>>
>>> - Security: full authentication and TLS supported
>>> - Latency is low once the connection is established, however there is
>>> some overhead involved in authentication and security negotiation
>>> - Topology changes should be asynchronous notifications
>>> - Each connection has to be separately authenticated
>>> - Automatically establishing connections is not presently supported, so
>>> we'd need a bit of infrastructure for that.  Deal with user-interactive
>>> authentication.  Deal with connection lifecycle management.  Deal with
>>> configuration.  This will be a point of fragility
>>>
>>> Summary
>>> -------
>>>
>>> For both options, we have to determine an appropriate load-balancing
>>> strategy.  The choice of direction will affect how our clustering and
>>> transaction interceptors function.  We also have to suss out the logic
>>> around dealing with conflicting or wrongly-ordered topology updates;
>>> hopefully our existing policies will continue to apply.
>>>
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


--
- DML
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

Scott Marlow
In reply to this post by David Lloyd-2
>
> Summary
> -------
>
> For both options, we have to determine an appropriate load-balancing
> strategy.  The choice of direction will affect how our clustering and
> transaction interceptors function.  We also have to suss out the logic
> around dealing with conflicting or wrongly-ordered topology updates;
> hopefully our existing policies will continue to apply.
>

I just wanted to mention, that when we talk about load-balancing
strategies/policies, that we might need some strategies that can handle
affinity for groups derived from extended persistence.  In other words,
this would be a policy that understands how to migrate a group of
stateful session beans to the same target node, during fail-over.

A simple example would be for two stateful session beans (SFSB1 +
SFSB2), that share an extended persistence context (XPC1).  When an
instance of SFSB1 fails-over to a new cluster node, the corresponding
SFSB2 instance should fail-over to the same target node.

Another example, would be the same as above, except SFSB2 has a
reference to SFSB3, which has a different XPC3.  XPC3 is also used by
SFSB4.  So, the load balancing policy would be need to have some
heuristic or maybe group-id that covers this case.

One reason for ensuring that the same node is chosen for groups derived
from use of extended persistence context, is so that applications can
continue to see dirty data that is associated with the XPC.

_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

David Lloyd-2
On 11/21/2011 02:40 PM, Scott Marlow wrote:

>>
>> Summary
>> -------
>>
>> For both options, we have to determine an appropriate load-balancing
>> strategy. The choice of direction will affect how our clustering and
>> transaction interceptors function. We also have to suss out the logic
>> around dealing with conflicting or wrongly-ordered topology updates;
>> hopefully our existing policies will continue to apply.
>>
>
> I just wanted to mention, that when we talk about load-balancing
> strategies/policies, that we might need some strategies that can handle
> affinity for groups derived from extended persistence. In other words,
> this would be a policy that understands how to migrate a group of
> stateful session beans to the same target node, during fail-over.
>
> A simple example would be for two stateful session beans (SFSB1 +
> SFSB2), that share an extended persistence context (XPC1). When an
> instance of SFSB1 fails-over to a new cluster node, the corresponding
> SFSB2 instance should fail-over to the same target node.
>
> Another example, would be the same as above, except SFSB2 has a
> reference to SFSB3, which has a different XPC3. XPC3 is also used by
> SFSB4. So, the load balancing policy would be need to have some
> heuristic or maybe group-id that covers this case.
>
> One reason for ensuring that the same node is chosen for groups derived
> from use of extended persistence context, is so that applications can
> continue to see dirty data that is associated with the XPC.

I'm going to move this out of the "clustered invocation" column and into
the "stateful session replication" category which is, to my
understanding, Paul's purview at the moment.


--
- DML
_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

Radoslaw Rodak
In reply to this post by Scott Marlow
> A simple example would be for two stateful session beans (SFSB1 +
> SFSB2), that share an extended persistence context (XPC1).  When an
> instance of SFSB1 fails-over to a new cluster node, the corresponding
> SFSB2 instance should fail-over to the same target node.

mabe just keeping it simple.
Each Node in cluster replicated state to exactly one other node in cluster, just everything which is defined as statefull.
to which node might be load balanced. So on faile over you have exact copy of the same state on other node.
So it doesn't mother which combinations...



Am 21.11.2011 um 21:40 schrieb Scott Marlow:

>>
>> Summary
>> -------
>>
>> For both options, we have to determine an appropriate load-balancing
>> strategy.  The choice of direction will affect how our clustering and
>> transaction interceptors function.  We also have to suss out the logic
>> around dealing with conflicting or wrongly-ordered topology updates;
>> hopefully our existing policies will continue to apply.
>>
>
> I just wanted to mention, that when we talk about load-balancing
> strategies/policies, that we might need some strategies that can handle
> affinity for groups derived from extended persistence.  In other words,
> this would be a policy that understands how to migrate a group of
> stateful session beans to the same target node, during fail-over.
>
> A simple example would be for two stateful session beans (SFSB1 +
> SFSB2), that share an extended persistence context (XPC1).  When an
> instance of SFSB1 fails-over to a new cluster node, the corresponding
> SFSB2 instance should fail-over to the same target node.
>
> Another example, would be the same as above, except SFSB2 has a
> reference to SFSB3, which has a different XPC3.  XPC3 is also used by
> SFSB4.  So, the load balancing policy would be need to have some
> heuristic or maybe group-id that covers this case.
>
> One reason for ensuring that the same node is chosen for groups derived
> from use of extended persistence context, is so that applications can
> continue to see dirty data that is associated with the XPC.
>
> _______________________________________________
> jboss-as7-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
Reply | Threaded
Open this post in threaded view
|

Re: Clustered invocation design

Scott Marlow
In reply to this post by David Lloyd-2
On 11/21/2011 03:50 PM, David M. Lloyd wrote:

> On 11/21/2011 02:40 PM, Scott Marlow wrote:
>>>
>>> Summary
>>> -------
>>>
>>> For both options, we have to determine an appropriate load-balancing
>>> strategy. The choice of direction will affect how our clustering and
>>> transaction interceptors function. We also have to suss out the logic
>>> around dealing with conflicting or wrongly-ordered topology updates;
>>> hopefully our existing policies will continue to apply.
>>>
>>
>> I just wanted to mention, that when we talk about load-balancing
>> strategies/policies, that we might need some strategies that can handle
>> affinity for groups derived from extended persistence. In other words,
>> this would be a policy that understands how to migrate a group of
>> stateful session beans to the same target node, during fail-over.
>>
>> A simple example would be for two stateful session beans (SFSB1 +
>> SFSB2), that share an extended persistence context (XPC1). When an
>> instance of SFSB1 fails-over to a new cluster node, the corresponding
>> SFSB2 instance should fail-over to the same target node.
>>
>> Another example, would be the same as above, except SFSB2 has a
>> reference to SFSB3, which has a different XPC3. XPC3 is also used by
>> SFSB4. So, the load balancing policy would be need to have some
>> heuristic or maybe group-id that covers this case.
>>
>> One reason for ensuring that the same node is chosen for groups derived
>> from use of extended persistence context, is so that applications can
>> continue to see dirty data that is associated with the XPC.
>
> I'm going to move this out of the "clustered invocation" column and into
> the "stateful session replication" category which is, to my
> understanding, Paul's purview at the moment.

Just to keep everything in the open, I think when we do cluster extended
persistence contexts, we will need a way to ensure that stateful session
beans that are related (direct or indirectly) through extended
persistence context inheritance, will need to fail-over as a group in
the cluster.  If we come up with a load balancing policy that covers
that magically, that would be cool.  I'll add more details on the "EJB
passivation/activation, clustering and JPA extended persistence context
handling..." thread...




_______________________________________________
jboss-as7-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev