Websocket API should support cumulative acknowledgement

[jeffgrunewald]

Is your feature request related to a problem? Please describe.
In order to support feature parity with other clients, the WebSocket should also be able to cumulatively acknowledge messages up to and including the supplied message ID.

Describe the solution you'd like
Ideally, the acknowledgement payload that currently only seems to support {"messageId":"CAoQKA=="} could accept an optional second field specifying individual acknowledgement or cumulative acknowledgement. The absence of the field could be assumed to equate to individual acknowledgement. As an example, the following two options could be accepted to allow for individual acknowledgement of a message:
{"messageId":"CAoQKA=="} or {"messageId":"CAoQKA==","ackType":"Individual"}
while the following would be required to acknowledge messages cumulatively:
{"messageId":"CAoQKA==","ackType":"Cumulative"}

Describe alternatives you've considered
Perhaps a topic-level setting would also work in which case only it would be required at topic configuration as with partitioned topics but I image that would require more work to support.

Additional context
In order to maximize throughput of messages across a WebSocket, particularly when the same socket connection is handling acknowledgement, it would be ideal to support cumulative acknowledgement to minimize the processing on the cluster side by minimizing the number of acknowledgement messages needed to keep data flowing to the consumer, while also minimizing the amount of bottlenecking that occurs within the socket client itself by not requiring a one-for-one ack message to received message.

Thanks very much!

[zuquan-song]

Does it like the TCP delayed acknowledgement?

[jeffgrunewald]

Does it like the TCP delayed acknowledgement?

I’m not sure how to accomplish that; can you give me an example of triggering a delayed ack of a block of messages that I could test?

[zuquan-song]

Just forget it. I think I misunderstood the meaning of the feature. But now I know what you mean.

[jeffgrunewald]

Thanks for clarifying. I am attempting to build and run pulsar locally for development mode but I receive a 404 whenever attempting to connect to the websocket to create a producer due to a missing "serviceUrl" param. I have set this value to my local host/port in the conf/websocket.conf file but the result is the same. Is there somewhere else this should be set or an undocumented requirement for connecting from the websocket client?

I have not experienced this issue when using the standard Pulsar docker image (apache/pulsar:latest) and running the command bin/pulsar standalone as the CMD directive.

[jiazhai]

@jeffgrunewald , Do you mean, you are running bin/pulsar standalone locally, but websocket client not able to connect? Since you are running success by docker, this localrun may related to host resolve? try to add advertise parameter after command, e.g. bin/pulsar standalone -a 127.0.0.1

[jeffgrunewald]

Your assumption is correct, but adding that didn't seem to produce any different behavior. I also tried the same with bin/pulsar standalone -a localhost because I believe MacOS can sometimes be difficult about resolving one or the other option. No change.
Just to be clear, I am attempting this off a clone of the tip of the master branch, not the latest release tarball. I do see in the logs the ConsumerStatsRecorderImpl Pulsar client config is listing the serviceUrl as "serviceUrl" : "pulsar://127.0.0.1:6650". Are the current websocket client connection instructions accurate for the latest unreleased version?

[jeffgrunewald]

For further context, I am writing a pulsar client library for the Elixir language based on the websocket api. My producer and consumer is connecting to the endpoint ws://localhost:8080/ws/v2/producer/persistent... where localhost is dynamically supplied as part of the client configuration and my dockerized pulsar for development purposes is mapping the default service port 8080 from the container to the host.

[jiazhai]

Thanks @jeffgrunewald for your feed back. Since the 404 error is not too related to the initial issue, I opened a new issue #5997, we could discuss it in 5997?

[hpvd]

just as crosslink to an other websocket performance relevant issue:
WebSocket broker: url (path and query) args (namespace, topic, etc...) exposed through message fields:
#8584

[hpvd]

@sijie this one likes to have the "component/websocket" label
edit: upps just saw the pull request #6680 already have.