This version (2017/05/27 13:44) is a draft.
Approvals: 0/1

[00:14:30] <testn> Hi there!

[08:36:48] * ChanServ sets mode: +o temporal_ [08:54:55] <cescoffier> ChicagoJohn: response → {} ? [10:23:17] * ChanServ sets mode: +o temporalfox

[15:08:47] <ranie> hey guys, anyone experiencing this issue when using an http client? java.lang.IllegalStateException: No response handler at io.vertx.core.http.impl.ClientConnection.handleResponse(ClientConnection.java:275) at io.vertx.core.http.impl.HttpClientImpl$ClientHandler.doMessageReceived(HttpClientImpl.java:869) at io.vertx.core.http.impl.HttpClientImpl$ClientHandler.doMessageReceived(HttpClientImpl.java:847) at io.vertx.core.http.impl.Ver

[15:49:12] <cescoffier> ranie : before calling end(), you need to attach a responseHandler on the request: client.get().responseHandler(response → {….}).end()

[15:57:38] <ranie> already did

[15:57:42] <ranie> found the problem

[15:57:53] <ranie> there was a 301 redirect to https

[16:41:52] <ChicagoJohn> anyone alive in here?

[16:42:36] <ChicagoJohn> ive got an implementation question

[16:44:47] <temporalfox> ChicagoJohn hi

[16:44:55] <temporalfox> maybe I can help you

[16:45:48] <ChicagoJohn> cool. so ive created an endpoint that accepts a file upload. that part is working.

[16:46:37] <ChicagoJohn> as the file comes in, each chunk slowly fills up a buffer. when that buffer is full, it opens a connection to another remote server and sends that buffer.

[16:46:52] <ChicagoJohn> i have all of this working sorta.

[16:47:43] <ChicagoJohn> the issue i am facing is that vertx completes the file upload before all of the additional connections sending out the buffers complete

[16:48:13] <ChicagoJohn> my guess is that i need to create a worker verticle for the secondary connections to be run within?

[16:50:41] <ChicagoJohn> i believe the problem lies in the fact that all new client requests live inside the routerContext.request().uploadHandler() callback. and once the routerContext.response().end() is called, it collapses all of the internal client requests and therfore the client responses have no where to return to

[17:01:29] <temporalfox> hum

[17:01:48] <ChicagoJohn> right….

[17:01:55] <temporalfox> I'm reading

[17:02:03] <ChicagoJohn> no worries

[17:02:21] <temporalfox> so you receive a file upload

[17:02:30] <temporalfox> and you wan t to transform the bytes to another server

[17:02:33] <temporalfox> using a connection

[17:02:56] <temporalfox> “to transfer”

[17:05:39] <ChicagoJohn> yes. my vertx app is merely a middle man at this point but its growing into a full app.

[17:07:22] <ChicagoJohn> I could always wait for the file upload to complete, and then build a scheduler to push the files to the other server later. with this method i could inspect the files if needed. but then i would need more storage space on my vertx side i guess

[17:07:57] <temporalfox> you should use a Pump for this

[17:08:06] <temporalfox> basically when you recieve the file upload

[17:08:09] <temporalfox> you pause() the request

[17:08:14] <temporalfox> then you connect to this other server

[17:08:19] <temporalfox> when you get the connection

[17:08:40] <temporalfox> you create a Pump between the HttpFileUpload buffer and the HttpClientRequest you connect to

[17:08:46] <temporalfox> and you resume() the initial request

[17:08:55] <temporalfox> look at Pump in vertx core doc

[17:09:34] <ChicagoJohn> will do. thanks

[17:09:49] <ChicagoJohn> you guys are all generally 'overseas' right? im in the US

[17:12:01] <temporalfox> we are on CET timezone

[17:14:53] <ChicagoJohn> yep. thats why this channel is dead when im at work. 8am here. thanks for the help ill jump on that pump stuff

[17:48:18] <temporalfox> not dead

[17:48:30] <temporalfox> I can be there sometimes in the evening

[17:48:39] <temporalfox> otherwise you have the vertx google group

[19:07:12] <ChicagoJohn> temporalfox, if you are still here i have a follow up question regarding pump

[19:24:37] <temporalfox> yes

[19:24:46] <temporalfox> I'm there for a couple of minutes ChicagoJohn

[19:26:00] <ChicagoJohn> so the server i have to send my data to places a limit on how much data it can receive.

[19:26:18] <ChicagoJohn> once that happens, i need to create a new request to a modified endpoint

[19:26:48] <ChicagoJohn> starting upload is /<uploadId>/<offset>/ which is id/0/

[19:27:20] <ChicagoJohn> after the first send, the new url is the same pattern but with an updated offset. which is why i was having to create so many requests in the first place

[19:27:58] <temporalfox> ChicagoJohn ok

[19:28:08] <temporalfox> then you would need to anticipate this

[19:28:23] <temporalfox> does the file uplaod sets a content-lenght header ?

[19:28:25] <ChicagoJohn> with the pump, i can only set one read stream and one write stream. can i recreate the pump per request using the original upload request, or would i need to build a new object to handle this

[19:28:29] <temporalfox> if yes you can anticipate it

[19:28:37] <temporalfox> you can stop the pump

[19:28:46] <temporalfox> and create a new one I htink

[19:29:06] <temporalfox> the pump itself is quite easy code

[19:29:16] <temporalfox> you could fork it and adapt it to your needs

[19:29:22] <temporalfox> if you can't make it work

[19:29:32] <temporalfox> but if you get the idea of how the pump works, it's the most important

[19:30:01] <temporalfox> perhaps one thing you could do is

[19:30:06] <temporalfox> create a WriteStream

[19:30:10] <temporalfox> implementation

[19:30:20] <temporalfox> that creates the many files

[19:30:29] <temporalfox> when a file is full

[19:30:31] <temporalfox> it stop the request

[19:30:35] <temporalfox> creates a new request

[19:30:45] <temporalfox> and until the request is created

[19:30:48] <temporalfox> you buffer the buffers

[19:30:58] <temporalfox> and you return true in writeQueueFull

[19:31:02] <temporalfox> when the request is created

[19:31:09] <temporalfox> you consume the buffers

[19:31:13] <temporalfox> and you call the drainHandler

[19:31:20] <temporalfox> so you can Pump this with the HttpServerRequest

[19:31:48] <temporalfox> also another altenative

[19:31:55] <temporalfox> is to buffer everything on the file system

[19:32:02] <temporalfox> but I think it depends on your use case

[19:49:23] <ChicagoJohn> my internet disconnected. did i miss your response?

[19:56:01] <ChicagoJohn> temporalfox just checking if you responded

[20:03:20] <testn> I have a problem with netSocket() where it leaks a connection https://github.com/eclipse/vert.x/issues/1396

[20:04:04] <testn> I actually prefer that Http{Client/Server}{Request/Response} provides a mechanism to terminate the request/response early

[20:04:39] <testn> we are trying to implement a good http proxy with vert.x but runs into a problem with large content either PUT/GET

[20:05:05] <testn> if the connection on one end aborts early, it's difficult to do that the same on the other side of the connection

[20:17:58] <temporalfox> ChicagoJohn you can look at the log for the response

[20:18:25] <temporalfox> testn I find netSocket() not reliable

[20:18:35] <temporalfox> what do you mean by early ?

[20:18:46] <temporalfox> and “terminate” :-)

[20:19:45] <temporalfox> testn what do you support ? HTTP/1.1 ? keep-alive and/or pipelining ?

[20:21:37] <testn> 1.1 with keepalive

[20:21:40] <testn> no pipelining

[20:22:04] <testn> i hate netSocket… i find that it rarely works

[20:22:17] <testn> I would prefer to have a reliable way to close a connection

[20:23:10] <testn> regarding my case, I have a proxy to front GET/PUT traffic but when under load, the GET request can slow down and the client may not wait for the response

[20:23:40] <testn> so it kills the connection… if I don't close the backend connection, it will keep streaming useless data to a blackhole

[20:24:00] <testn> and that kills the bandwidth on my end as user will only keep retrying and make the situation worse

[20:24:39] <testn> for the PUT request, I faced another challenge. If the connection is closed in the middle, I don't have a good way to abort a connection

[20:25:13] <testn> if I call end() early for chunked http request, it will assume that the request is completed successfully where it does not

[20:27:59] <temporalfox> for PUT by “close” you mean your client request or the request you make to the server ?

[20:28:21] <temporalfox> ah you mean your client and you want to end() the client request

[20:29:07] <temporalfox> so in both case you want to close the connection to the server ?

[20:29:43] <temporalfox> testn

[20:30:16] <testn> correct

[20:30:55] <temporalfox> and what is missing is a HttpClientRequest#close method

[20:31:18] <testn> no such method in 3.2.1

[20:31:28] <temporalfox> yes it does not exists

[20:31:57] <testn> no HttpClientResponse.close() for the client to close the stream earlier either

[20:32:07] <temporalfox> in 3.3 there is reset() method for HTTP/2

[20:32:10] <temporalfox> which achieve that

[20:32:20] <temporalfox> perhaps we can implement it also for HTTP/1.1 with a simple close

[20:32:30] <temporalfox> (instead of adding a new close() method)

[20:32:53] <testn> that'd be nice

[20:33:02] <testn> how close are we on 3.3?

[20:33:15] <temporalfox> it would be in June

[20:33:24] <temporalfox> maybe you can try to set a time out ?

[20:33:26] <testn> that's still far away

[20:33:28] <temporalfox> very small

[20:33:45] <temporalfox> not far away according to my perspective and schedule :-)

[20:33:48] <testn> timeout cannot be changed after the connection is made

[20:33:55] <temporalfox> ok

[20:34:44] <temporalfox> maybe you can use one HttpClient per connection and close the client as work around ?

[20:34:55] <temporalfox> (I don't think the overhead would be too much)

[20:34:55] <testn> righ now timeout i set it to 60 seconds to handle customers with low/unstable network

[20:35:31] <testn> the problem with that is that we cannot reuse the connection

[20:35:46] <testn> which is also bad in term of socket in use (especially with CLOSE_WAIT)

[20:36:12] <testn> I wish you guys can do a minor release much more often like what netty does

[20:36:46] <temporalfox> testn that's something we should do I agree

[20:37:17] <testn> all these little bugs here and there kinda put me off from using it production

[20:37:24] <temporalfox> testn that's a problem of HTTP : you cannot make a clean abort of a request, specially with keep-alive

[20:37:36] <temporalfox> http/2 supports stream reset()

[20:37:53] <testn> you can just close it… and thus the connection cannot be reused anymore

[20:37:57] <temporalfox> so the best is to close the connection

[20:38:14] <testn> the problem is more on the server side actually for keep alive

[20:38:26] <temporalfox> wdym ?

[20:38:38] <testn> basically, let's say that you make a request and the server responds and then try to close the connection.

[20:39:04] <testn> as soon as you read the response from the server, you will put the connection to be reused again only to know that the server has closed the connection

[20:39:22] <testn> we found that happened a lot for our busy service

[20:39:30] <temporalfox> but the client checks that the connection is not closed before using it

[20:39:41] <temporalfox> I agree that the connection might be closed when doing the connection

[20:39:42] <testn> so we work around that by sending “Connection: close” back to the client and let the client to close it

[20:39:55] <temporalfox> yes that's how an http/1.1 server should behave

[20:40:07] <temporalfox> a server that does not send Connection: close should not close the connection

[20:40:14] <testn> yup

[20:40:22] <temporalfox> I need to take care of my daughter :-)

[20:40:35] <temporalfox> can you open an issue for HttpClientRequest reset() for HTTP/1.1 at least ?

[20:40:59] <testn> for the server close, it should not close the connection either

[20:41:05] <temporalfox> right :-)

[20:41:36] <testn> for example, if the request is long or too large or too slow, I can send a response early to tell the client that i'm not interested in serving the request anymore

[20:42:22] <testn> if i close the connection, the client will get a socket reset from the sending side before it can read the response.

[20:43:07] <testn> regarding HttpClientRequest.reset() it is a weird problem to solve

[20:43:28] <testn> HttpClientRequest does not require to have an associated connection

[20:43:44] <testn> so when you call reset(), there can be either

[20:43:57] <testn> 1.it has not yet established a connection yet

[20:44:25] <testn> 2. the connection has already been reused by someone else (in the case that the response comes early)

[20:48:40] <testn> @temporalfox what do you think about https://github.com/eclipse/vert.x/issues/1369

[20:48:47] <testn> do you think it is a valid bug?

[21:36:59] <temporalfox> can you provide a reproducer testn ?

[21:37:48] <temporalfox> testn reset should only be called when the client request is associated with the ClientCOnnection otherwise throw an IllegalStateException

[21:38:12] <temporalfox> now there is sendHead(Handler<Void>) in HttpClientRequest

[21:38:29] <temporalfox> actually Handler<HttpVersion>

[21:38:56] <temporalfox> that is called when the client has connected

[21:39:08] <testn> i'd prefer an explicit callback for the connection

[21:39:15] <temporalfox> there is one actually now

[21:39:25] <temporalfox> but it only works for Http/2

[21:39:29] <testn> :(

[21:39:34] <temporalfox> HttpClientRequest connectionHandler(@Nullable Handler<HttpConnection> handler);

[21:39:41] <temporalfox> it could be extended I think to HTTP/1.1

[21:39:54] <temporalfox> but in this case there would be HttpConnection and Http2Connection

[21:39:58] <temporalfox> maybe

[21:40:11] <temporalfox> because HttpConnection has method that don't make sense for HTTP/1.1

[21:41:49] <temporalfox> or they would innoperate

[21:41:55] <temporalfox> but it would be confusing to explain

[21:47:28] <testn> what is the key differences between HttpConnection/HttpConnection2?

[21:47:38] <temporalfox> HttpConnection2 does not exists

[21:47:39] <testn> the method should be mostly interoperable isn't it?

[21:47:50] <testn> Http2Connection i mean

[21:48:06] <temporalfox> at the connection level you can change HTTP2 settings

[21:48:19] <temporalfox> send GO_AWAY frames

[21:48:22] <temporalfox> or handle them

[21:49:03] <temporalfox> these would only work for http/2

[21:54:22] <testn> I see

[21:55:03] <testn> for the connection close issue, I think I can probably work around it for now using reflection while waiting for 3.3

[21:56:20] <testn> what do you think of netsocket going forward? it seems to be very bug ridden

[22:08:54] <temporalfox> testn I think it's only usage today is for the CONNECT support

[22:09:06] <temporalfox> that's what it was designed for

[22:09:18] <temporalfox> given that with HTTP/2 netSOcket() returns a “fake” socket

[22:09:35] <temporalfox> that uses an HTTP/2 stream with Data frames

[22:09:54] <temporalfox> so it's more a pair of ReadStream/WriteStream

[22:10:23] <temporalfox> testn so you would like to have an HttpConnection that works for HTTP/1.1 in 3.3 ?

[22:19:15] <testn> i think it's much more compatible

[22:19:29] <testn> user don't need to treat them differently