Steamable content in management API responses

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Steamable content in management API responses

Brian Stansberry
tl;dr

I'm looking into supported streamed content in management API responses
and wanted to get feedback.

The admin console has a need for a streamed response, as we have a
requirement to let a user download a log file to store on local disk.
The browser can itself directly download a file using HTTP, but will not
let an app like the console make multiple DMR requests and itself append
data from the responses to a file on disk.

There are other likely uses cases for a similar thing, e.g. reading the
contents of the content (aka deployment) repository.

We already support attaching streams to requests; the proposal is to do
much the same with responses. For HTTP clients, if a request is for
something that attaches a stream to the response, the request URL can
include a query param instructing the server to pipe the stream for the
response instead of sending the standard JSON reponse value.

Long version:


Requirements:

1) Ability to send arbitrarily long amounts of data in management responses.

Currently everything in a management response has to be encodable in
DMR. DMR will allow you to send a byte[] as a response value, but for
memory reasons this isn't a practical approach for large responses.

2) Ability to send the data directly to an HTTP request, not wrapped in
a JSON wrapper. Even if we would include a huge value as a JSON encoded
byte[], a browser won't let an app like the console read those bytes out
of the JSON response and write them to disk. A direct response to an
HTTP GET/POST is needed.

3) Works for other remoting-based clients as well. This is a basic
requirement that HTTP and remoting-based clients can each do everything.
So, no HTTP only side features.

4) Requests go through the central management layer, so we have proper
security, audit logging etc. So, no special endpoints that bypass the
core management layer.

Proposal:

We already support attaching a stream to the request for native clients.
(This is how the CLI does deployments.) The client API lets the client
associate a stream with the request[1]. The stream has an index. The
operation that needs the stream takes a DMR param that tells it the
index of the stream. On the server side the handler uses that param
value to find the server-side representation of the stream. The remoting
layer of the remote management protocol in the background handles piping
the contents of the client side stream to the server.

The proposal is to simply mirror this in reverse. A server side
operation handler associates a stream with the response (via a call on
the OperationContext). The attached stream has an index. The normal DMR
response to the operation is the index of the stream. The client uses
that response value to find the attached stream. The remoting layer of
the remote management protocol in the background handles piping the
contents of the server side stream to the client.

This handles remoting based clients, including those using http upgrade.

For HTTP clients, the client can include a "useStreamAsResponse" query
param in the request URL. If this is present, the server side endpoint
will take the stream from the internal response and pipe it to the
client, instead of sending the normal JSON.

For example, a fictitious URL for a "stream-log-file" op against a
resource that represents the server.log file:

http://localhost:9090/management/subsystem/logging/log-file/server.log?operation=stream-log-file&useStreamAsResponse

In the corner case where a request results in more than one attached
response, the useStreamAsResponse could include an index.

useStreamAsResponse=1

Status:

It was reasonably straightforward to get this working, as a fairly
polished prototype:

https://github.com/bstansberry/wildfly-core/commits/resp-stream

The 2nd commit there just hacks in an attribute to the logging subsystem
root resource to expose the server log as a stream. There's no intent to
actually do it that specific way of course; it was just an easy way to
demonstrate.

With that built an HTTP GET will download the log file to your desktop.

http://localhost:9990/management/subsystem/logging?operation=attribute&name=server-log&useStreamAsResponse

The native interface works as well. The 2nd commit includes a test case
that confirms this. So the CLI  easily enough could add a high level
command for log reading too.

TODOs:

1) Domain mode: proxy the streams around the domain
(server->HC->DC->client). I think this should be quite easy.

2) A cleaner variant of the remoting-based management protocol for
handling the streams, one that doesn't require hacks to determine the
length of the stream in advance. Should be relatively straightforward.

3) Logic to clean up server side streams if the client doesn't properly
consume them.

4) Make sure there a no issues with the thread pools used to handle
management requests.

5) Make sure POST works well. My assumption is this is lower priority,
as the real use cases would likely use a GET.


[1]
https://github.com/wildfly/wildfly-core/blob/master/controller-client/src/main/java/org/jboss/as/controller/client/OperationBuilder.java

--
Brian Stansberry
Senior Principal Software Engineer
JBoss by Red Hat
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Steamable content in management API responses

Brian Stansberry
On 10/1/14, 9:52 AM, Brian Stansberry wrote:
<snip/>

>
> TODOs:
>

>
> 5) Make sure POST works well. My assumption is this is lower priority,
> as the real use cases would likely use a GET.
>

If you include the useStreamAsResponse as a query param in the URL, as
with GET, it works. If you do the proper POST thing and encode the query
param in the request body it doesn't. The DomainApiHandler tries to
parse the entire request body into a ModelNode and a leading
useStreamAsResponse& or trailing &useStreamAsResponse fails.

I'm fine with just requiring encoding the param in the URL, even though
it isn't proper. I don't plan on spending energy near term on parsing it
out of a POST request body.

If you're curious to try this with my resp-stream branch, this works:

curl --digest -L -D -
http://localhost:9990/management?useStreamAsResponse --header
"Content-Type: application/json" -u user:password -d
'{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'

This doesn't:

curl --digest -L -D - http://localhost:9990/management --header
"Content-Type: application/json" -u user:password -d
'{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
-d useStreamAsResponse


--
Brian Stansberry
Senior Principal Software Engineer
JBoss by Red Hat
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Steamable content in management API responses

David Lloyd-2
On 10/02/2014 05:46 PM, Brian Stansberry wrote:

> On 10/1/14, 9:52 AM, Brian Stansberry wrote:
> <snip/>
>
>>
>> TODOs:
>>
>
>>
>> 5) Make sure POST works well. My assumption is this is lower priority,
>> as the real use cases would likely use a GET.
>>
>
> If you include the useStreamAsResponse as a query param in the URL, as
> with GET, it works. If you do the proper POST thing and encode the query
> param in the request body it doesn't. The DomainApiHandler tries to
> parse the entire request body into a ModelNode and a leading
> useStreamAsResponse& or trailing &useStreamAsResponse fails.
>
> I'm fine with just requiring encoding the param in the URL, even though
> it isn't proper. I don't plan on spending energy near term on parsing it
> out of a POST request body.
>
> If you're curious to try this with my resp-stream branch, this works:
>
> curl --digest -L -D -
> http://localhost:9990/management?useStreamAsResponse --header
> "Content-Type: application/json" -u user:password -d
> '{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
>
> This doesn't:
>
> curl --digest -L -D - http://localhost:9990/management --header
> "Content-Type: application/json" -u user:password -d
> '{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
> -d useStreamAsResponse

Maybe because the content type isn't multipart/form-data?  I think if
you're going to send or receive content alongside the main payload,
you'll have to use a multipart request and/or response (this also
implies some accept-ish header stuff too IIRC).
--
- DML
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Steamable content in management API responses

Jason T. Greene
It's proper. It's  sending a content type of application/JSON not form data. So if you want the value other than the URL param, it would have to be in the DMR and read from the model node

>> On Oct 2, 2014, at 6:01 PM, David M. Lloyd <[hidden email]> wrote:
>>
>> On 10/02/2014 05:46 PM, Brian Stansberry wrote:
>> On 10/1/14, 9:52 AM, Brian Stansberry wrote:
>> <snip/>
>>
>>>
>>> TODOs:
>>
>>>
>>> 5) Make sure POST works well. My assumption is this is lower priority,
>>> as the real use cases would likely use a GET.
>>
>> If you include the useStreamAsResponse as a query param in the URL, as
>> with GET, it works. If you do the proper POST thing and encode the query
>> param in the request body it doesn't. The DomainApiHandler tries to
>> parse the entire request body into a ModelNode and a leading
>> useStreamAsResponse& or trailing &useStreamAsResponse fails.
>>
>> I'm fine with just requiring encoding the param in the URL, even though
>> it isn't proper. I don't plan on spending energy near term on parsing it
>> out of a POST request body.
>>
>> If you're curious to try this with my resp-stream branch, this works:
>>
>> curl --digest -L -D -
>> http://localhost:9990/management?useStreamAsResponse --header
>> "Content-Type: application/json" -u user:password -d
>> '{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
>>
>> This doesn't:
>>
>> curl --digest -L -D - http://localhost:9990/management --header
>> "Content-Type: application/json" -u user:password -d
>> '{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
>> -d useStreamAsResponse
>
> Maybe because the content type isn't multipart/form-data?  I think if
> you're going to send or receive content alongside the main payload,
> you'll have to use a multipart request and/or response (this also
> implies some accept-ish header stuff too IIRC).
> --
> - DML
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Steamable content in management API responses

Brian Stansberry
FYI, https://issues.jboss.org/browse/WFCORE-148 is the JIRA for this.

I didn't want to encode useStreamAsResponse in the DMR as an operation
header, as this isn't about how the core management layer executes the
op, just how the http server deals with sending the response.

But since it's Friday you guys inspired me to go ahead and handle
application/x-www-form-urlencoded

So I updated my branch and with the 3rd commit, a proper form with the
DMR as the value associated with a form key 'dmr' now works:

curl --digest -L -D - http://localhost:9990/management --header
"Content-Type: application/x-www-form-urlencoded" -u
bstansberry:admin.1234 -d
'dmr={"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
-d useStreamAsResponse

A caller sending dmr-encoded can even hint at that with the "Accept" header:

curl --digest -L -D - http://localhost:9990/management --header
"Content-Type: application/x-www-form-urlencoded" --header "Accept:
application/dmr-encoded" -u bstansberry:admin.1234 -d
'dmr=bwAAAAQACW9wZXJhdGlvbnMADnJlYWQtYXR0cmlidXRlAAdhZGRyZXNzbAAAAAFvAAAAAQAJc3Vic3lzdGVtcwAHbG9nZ2luZwAEbmFtZXMACnNlcnZlci1sb2cAC2pzb24ucHJldHR5SQAAAAE='
-d useStreamAsResponse

That hack of using 'Accept' and not just 'Content-Type' to trigger
dmr-encoded handling was already there, so don't blame me. ;)

On 10/2/14, 7:46 PM, Jason T. Greene wrote:

> It's proper. It's  sending a content type of application/JSON not form data. So if you want the value other than the URL param, it would have to be in the DMR and read from the model node
>
>>> On Oct 2, 2014, at 6:01 PM, David M. Lloyd <[hidden email]> wrote:
>>>
>>> On 10/02/2014 05:46 PM, Brian Stansberry wrote:
>>> On 10/1/14, 9:52 AM, Brian Stansberry wrote:
>>> <snip/>
>>>
>>>>
>>>> TODOs:
>>>
>>>>
>>>> 5) Make sure POST works well. My assumption is this is lower priority,
>>>> as the real use cases would likely use a GET.
>>>
>>> If you include the useStreamAsResponse as a query param in the URL, as
>>> with GET, it works. If you do the proper POST thing and encode the query
>>> param in the request body it doesn't. The DomainApiHandler tries to
>>> parse the entire request body into a ModelNode and a leading
>>> useStreamAsResponse& or trailing &useStreamAsResponse fails.
>>>
>>> I'm fine with just requiring encoding the param in the URL, even though
>>> it isn't proper. I don't plan on spending energy near term on parsing it
>>> out of a POST request body.
>>>
>>> If you're curious to try this with my resp-stream branch, this works:
>>>
>>> curl --digest -L -D -
>>> http://localhost:9990/management?useStreamAsResponse --header
>>> "Content-Type: application/json" -u user:password -d
>>> '{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
>>>
>>> This doesn't:
>>>
>>> curl --digest -L -D - http://localhost:9990/management --header
>>> "Content-Type: application/json" -u user:password -d
>>> '{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
>>> -d useStreamAsResponse
>>
>> Maybe because the content type isn't multipart/form-data?  I think if
>> you're going to send or receive content alongside the main payload,
>> you'll have to use a multipart request and/or response (this also
>> implies some accept-ish header stuff too IIRC).
>> --
>> - DML
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>


--
Brian Stansberry
Senior Principal Software Engineer
JBoss by Red Hat
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Steamable content in management API responses

Jason T. Greene
DMR as a form encoded value? I really don't think we should allow this. Form encoding is for web browsers typing in values and it will set off the extra security checks we have for cross origin posting (this is the kind of thing a JS hack would do to get your browser to send up arbitrary management ops to the server)

We probably should use an HTTP header if we don't want it in the payload, perhaps in combination with an Accept.


> On Oct 3, 2014, at 5:41 PM, Brian Stansberry <[hidden email]> wrote:
>
> FYI, https://issues.jboss.org/browse/WFCORE-148 is the JIRA for this.
>
> I didn't want to encode useStreamAsResponse in the DMR as an operation
> header, as this isn't about how the core management layer executes the
> op, just how the http server deals with sending the response.
>
> But since it's Friday you guys inspired me to go ahead and handle
> application/x-www-form-urlencoded
>
> So I updated my branch and with the 3rd commit, a proper form with the
> DMR as the value associated with a form key 'dmr' now works:
>
> curl --digest -L -D - http://localhost:9990/management --header
> "Content-Type: application/x-www-form-urlencoded" -u
> bstansberry:admin.1234 -d
> 'dmr={"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
> -d useStreamAsResponse
>
> A caller sending dmr-encoded can even hint at that with the "Accept" header:
>
> curl --digest -L -D - http://localhost:9990/management --header
> "Content-Type: application/x-www-form-urlencoded" --header "Accept:
> application/dmr-encoded" -u bstansberry:admin.1234 -d
> 'dmr=bwAAAAQACW9wZXJhdGlvbnMADnJlYWQtYXR0cmlidXRlAAdhZGRyZXNzbAAAAAFvAAAAAQAJc3Vic3lzdGVtcwAHbG9nZ2luZwAEbmFtZXMACnNlcnZlci1sb2cAC2pzb24ucHJldHR5SQAAAAE='
> -d useStreamAsResponse
>
> That hack of using 'Accept' and not just 'Content-Type' to trigger
> dmr-encoded handling was already there, so don't blame me. ;)
>
>> On 10/2/14, 7:46 PM, Jason T. Greene wrote:
>> It's proper. It's  sending a content type of application/JSON not form data. So if you want the value other than the URL param, it would have to be in the DMR and read from the model node
>>
>>>> On Oct 2, 2014, at 6:01 PM, David M. Lloyd <[hidden email]> wrote:
>>>>
>>>> On 10/02/2014 05:46 PM, Brian Stansberry wrote:
>>>> On 10/1/14, 9:52 AM, Brian Stansberry wrote:
>>>> <snip/>
>>>>
>>>>>
>>>>> TODOs:
>>>>
>>>>>
>>>>> 5) Make sure POST works well. My assumption is this is lower priority,
>>>>> as the real use cases would likely use a GET.
>>>>
>>>> If you include the useStreamAsResponse as a query param in the URL, as
>>>> with GET, it works. If you do the proper POST thing and encode the query
>>>> param in the request body it doesn't. The DomainApiHandler tries to
>>>> parse the entire request body into a ModelNode and a leading
>>>> useStreamAsResponse& or trailing &useStreamAsResponse fails.
>>>>
>>>> I'm fine with just requiring encoding the param in the URL, even though
>>>> it isn't proper. I don't plan on spending energy near term on parsing it
>>>> out of a POST request body.
>>>>
>>>> If you're curious to try this with my resp-stream branch, this works:
>>>>
>>>> curl --digest -L -D -
>>>> http://localhost:9990/management?useStreamAsResponse --header
>>>> "Content-Type: application/json" -u user:password -d
>>>> '{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
>>>>
>>>> This doesn't:
>>>>
>>>> curl --digest -L -D - http://localhost:9990/management --header
>>>> "Content-Type: application/json" -u user:password -d
>>>> '{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
>>>> -d useStreamAsResponse
>>>
>>> Maybe because the content type isn't multipart/form-data?  I think if
>>> you're going to send or receive content alongside the main payload,
>>> you'll have to use a multipart request and/or response (this also
>>> implies some accept-ish header stuff too IIRC).
>>> --
>>> - DML
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
> --
> Brian Stansberry
> Senior Principal Software Engineer
> JBoss by Red Hat
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: Steamable content in management API responses

Brian Stansberry
Ok, good to know. I'll just support a custom HTTP header. The query
param in the URL works too, so people can choose the approach they want.

On 10/4/14, 9:11 PM, Jason T. Greene wrote:

> DMR as a form encoded value? I really don't think we should allow this. Form encoding is for web browsers typing in values and it will set off the extra security checks we have for cross origin posting (this is the kind of thing a JS hack would do to get your browser to send up arbitrary management ops to the server)
>
> We probably should use an HTTP header if we don't want it in the payload, perhaps in combination with an Accept.
>
>
>> On Oct 3, 2014, at 5:41 PM, Brian Stansberry <[hidden email]> wrote:
>>
>> FYI, https://issues.jboss.org/browse/WFCORE-148 is the JIRA for this.
>>
>> I didn't want to encode useStreamAsResponse in the DMR as an operation
>> header, as this isn't about how the core management layer executes the
>> op, just how the http server deals with sending the response.
>>
>> But since it's Friday you guys inspired me to go ahead and handle
>> application/x-www-form-urlencoded
>>
>> So I updated my branch and with the 3rd commit, a proper form with the
>> DMR as the value associated with a form key 'dmr' now works:
>>
>> curl --digest -L -D - http://localhost:9990/management --header
>> "Content-Type: application/x-www-form-urlencoded" -u
>> bstansberry:admin.1234 -d
>> 'dmr={"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
>> -d useStreamAsResponse
>>
>> A caller sending dmr-encoded can even hint at that with the "Accept" header:
>>
>> curl --digest -L -D - http://localhost:9990/management --header
>> "Content-Type: application/x-www-form-urlencoded" --header "Accept:
>> application/dmr-encoded" -u bstansberry:admin.1234 -d
>> 'dmr=bwAAAAQACW9wZXJhdGlvbnMADnJlYWQtYXR0cmlidXRlAAdhZGRyZXNzbAAAAAFvAAAAAQAJc3Vic3lzdGVtcwAHbG9nZ2luZwAEbmFtZXMACnNlcnZlci1sb2cAC2pzb24ucHJldHR5SQAAAAE='
>> -d useStreamAsResponse
>>
>> That hack of using 'Accept' and not just 'Content-Type' to trigger
>> dmr-encoded handling was already there, so don't blame me. ;)
>>
>>> On 10/2/14, 7:46 PM, Jason T. Greene wrote:
>>> It's proper. It's  sending a content type of application/JSON not form data. So if you want the value other than the URL param, it would have to be in the DMR and read from the model node
>>>
>>>>> On Oct 2, 2014, at 6:01 PM, David M. Lloyd <[hidden email]> wrote:
>>>>>
>>>>> On 10/02/2014 05:46 PM, Brian Stansberry wrote:
>>>>> On 10/1/14, 9:52 AM, Brian Stansberry wrote:
>>>>> <snip/>
>>>>>
>>>>>>
>>>>>> TODOs:
>>>>>
>>>>>>
>>>>>> 5) Make sure POST works well. My assumption is this is lower priority,
>>>>>> as the real use cases would likely use a GET.
>>>>>
>>>>> If you include the useStreamAsResponse as a query param in the URL, as
>>>>> with GET, it works. If you do the proper POST thing and encode the query
>>>>> param in the request body it doesn't. The DomainApiHandler tries to
>>>>> parse the entire request body into a ModelNode and a leading
>>>>> useStreamAsResponse& or trailing &useStreamAsResponse fails.
>>>>>
>>>>> I'm fine with just requiring encoding the param in the URL, even though
>>>>> it isn't proper. I don't plan on spending energy near term on parsing it
>>>>> out of a POST request body.
>>>>>
>>>>> If you're curious to try this with my resp-stream branch, this works:
>>>>>
>>>>> curl --digest -L -D -
>>>>> http://localhost:9990/management?useStreamAsResponse --header
>>>>> "Content-Type: application/json" -u user:password -d
>>>>> '{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
>>>>>
>>>>> This doesn't:
>>>>>
>>>>> curl --digest -L -D - http://localhost:9990/management --header
>>>>> "Content-Type: application/json" -u user:password -d
>>>>> '{"operation":"read-attribute","address":[{"subsystem":"logging"}],"name":"server-log","json.pretty":1}'
>>>>> -d useStreamAsResponse
>>>>
>>>> Maybe because the content type isn't multipart/form-data?  I think if
>>>> you're going to send or receive content alongside the main payload,
>>>> you'll have to use a multipart request and/or response (this also
>>>> implies some accept-ish header stuff too IIRC).
>>>> --
>>>> - DML
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>>
>> --
>> Brian Stansberry
>> Senior Principal Software Engineer
>> JBoss by Red Hat
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev


--
Brian Stansberry
Senior Principal Software Engineer
JBoss by Red Hat
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev