Rework HTTP forward proxy implementation with WebSocket support (fixes #374) (#404)

* Rework HTTP forward proxy implementation with WebSocket support and comprehensive testing

This is a rework of #374 with a simplified approach that leverages copy_bidi for data transfer.

Refactors the HTTP proxy implementation to support modern HTTP features including WebSocket upgrades, chunked encoding, and various HTTP methods. The new implementation simplifies the data flow by writing HTTP request headers and delegating all bidirectional data transfer to the existing `copy_bidi` infrastructure.

- **Renamed**: `h11c.rs` → `http_proxy.rs` to reflect broader HTTP support
- **Simplified approach**: Write HTTP request headers, then let copy_bidi handle all data transfer
- **Protocol-agnostic**: Works with any HTTP feature (chunked encoding, WebSockets, etc.)
- **Removed complex manual parsing**: ~200 lines of HTTP handling → ~50 lines + copy_bidi delegation

- **Smart header detection**: Detects WebSocket upgrades via `Connection: upgrade` + `Upgrade: websocket`
- **Preserves upgrade headers**: WebSocket requests maintain original Connection/Upgrade headers
- **Connection management**: Regular HTTP requests get `Connection: close`, WebSockets preserve upgrade headers

- **Added Clone trait** to `HttpRequest` for header modification
- **Enhanced Context API**: Added HTTP request storage and improved stream management
- **URL dependency**: Added `url` crate for future HTTP parsing enhancements
- **Better error handling**: Improved stream lifecycle and error propagation

```rust
async fn on_connect(&self, ctx: &mut Context) {
    // 1. Smart Connection header management (close vs upgrade)
    // 2. Write HTTP request headers to server
    // 3. Return streams to context for copy_bidi to handle everything else
}
```

- **WebSocket upgrade test**: Verifies header preservation
- **Connection management test**: Confirms Connection: close behavior
- **Chunked encoding test**: Tests transparent chunked transfer support
- **HTTP method tests**: GET, POST with bodies and headers
- **Error handling test**: Connection failures and HTTP error responses
- **URI format test**: Absolute and relative URI support

- **Simpler implementation**: Leverages existing copy_bidi instead of reimplementing data transfer
- **Better WebSocket support**: Proper header detection and preservation
- **More reliable**: Fewer moving parts, better error handling
- **Comprehensive testing**: 7 integration tests covering all scenarios
- **Performance**: Uses optimized copy_bidi (splice on Linux, async everywhere)

| Feature | Before | After |
|---------|--------|-------|
| WebSocket Upgrades | ❌ Header conflicts | ✅ Full support |
| Chunked Encoding | ⚠️ Manual parsing | ✅ Transparent copy_bidi |
| Large Bodies | ⚠️ Content-Length limited | ✅ Streaming |
| Connection Lifecycle | ⚠️ Manual management | ✅ copy_bidi handles |

- `src/common/http_proxy.rs` (refactored from h11c.rs): New simplified implementation
- `src/common/http.rs`: Added Clone trait to HttpRequest
- `src/context.rs`: Enhanced with HTTP request storage
- `src/listeners/http_forward_tests.rs`: 739-line comprehensive test suite
- `Cargo.toml`: Added url dependency

```bash
cargo test http_forward_tests::tests
```

Tests cover WebSocket upgrades, connection management, chunked encoding, HTTP methods, error handling, and URI formats.

Fixes #374

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* Fix HTTP forward proxy implementation issues and enhance testing

- Revert take_client_stream() to return Result<IOBufStream> for proper error handling
- Rename functions for clarity: http_proxy_* → http_forward_proxy_*
- Extract common error response patterns into helper functions
- Add comprehensive integration tests with real HTTP servers
- Update documentation to reflect HTTP forward proxy capabilities
- Fix code formatting and address clippy suggestions

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* Add forceConnect option to HTTP connector

- Add configurable force_connect field to HttpConnectorConfig
- Modify http_forward_proxy_connect to accept force_connect parameter
- When enabled, forces all requests to use CONNECT tunneling
- When disabled (default), allows HTTP forward proxy for GET/POST/etc
- Add comprehensive tests for both force_connect modes
- Update CONFIG_GUIDE.md with detailed documentation
- Maintains backward compatibility with existing configurations

This option is useful for:
- Compatibility with upstream proxies that only support CONNECT
- Organizations requiring all traffic to be tunneled
- Fine-grained control over proxy behavior

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

---------

Co-authored-by: Claude <[email protected]>

Author: bearice

CLAUDE.md

@@ -52,8 +52,9 @@ The embedded Milu DSL enables:
 - Located in `milu/` workspace member with full parser and standard library
 
 ### Protocol Support Matrix
-- **Listeners**: HTTP/HTTPS (with mTLS), SOCKS v4/v4a/v5, QUIC, TPROXY (Linux), Reverse Proxy
-- **Connectors**: Direct TCP, HTTP CONNECT, SOCKS5, QUIC, Load Balancing
+- **Listeners**: HTTP/HTTPS (with mTLS), HTTP Forward Proxy, SOCKS v4/v4a/v5, QUIC, TPROXY (Linux), Reverse Proxy
+- **Connectors**: Direct TCP, HTTP CONNECT, HTTP Forward Proxy, SOCKS5, QUIC, Load Balancing
+- **HTTP Features**: Traditional CONNECT tunneling, HTTP Forward Proxy (GET/POST/PUT/DELETE/etc.), WebSocket upgrade support
 - **Security**: mTLS support across HTTP/SOCKS with certificate validation
 - **Performance**: Linux splice() syscall support, BBR congestion control for QUIC
 

CONFIG_GUIDE.md

@@ -367,6 +367,11 @@ The `http` connector forwards traffic to an upstream HTTP or HTTPS proxy.
     -   *Example*: `"192.168.100.1"`
 -   `port` (integer): The port number of the upstream proxy server.
     -   *Example*: `7081` (for HTTP) or `3333` (for HTTPS in the example)
+-   `forceConnect` (boolean, optional): Forces all requests to use HTTP CONNECT tunneling instead of HTTP forward proxy.
+    -   *Default value*: `false`.
+    -   When `false`, GET/POST/PUT/DELETE requests are forwarded directly as HTTP requests.
+    -   When `true`, all requests (including GET/POST/PUT/DELETE) are tunneled through HTTP CONNECT.
+    -   This option is useful for compatibility with upstream proxies that only support CONNECT method.
 -   `tls` (object, optional): If present, this section configures TLS for connecting to an HTTPS proxy.
     -   `insecure` (boolean):
         -   *Default value*: `false`.
@@ -386,12 +391,14 @@ connectors:
   - name: http
     server: 192.168.100.1
     port: 7081
+    # forceConnect: false # Default: use HTTP forward proxy when possible
 
-  # HTTPS Connector
+  # HTTPS Connector with CONNECT-only mode
   - name: https # 'type: http' is inferred
     type: http # Can be explicitly set
     server: 192.168.100.1
     port: 3333
+    forceConnect: true # Force all requests to use CONNECT tunneling
     tls:
       insecure: true # Example allows self-signed certs for the upstream proxy
       ca: ca.crt # Optional custom CA for upstream proxy

Cargo.lock

@@ -39,6 +39,7 @@ libc = "0.2"
 hickory-resolver = "0.25"
 chashmap-async = "0.1.0"
 lru = "0.16"
+url = "2.5.4"
 
 # features quic
 quinn = { version = "0.11", optional = true}
@@ -64,6 +65,8 @@ winapi = {version = "0.3.9", features = ["winsock2","in6addr","inaddr","ws2def",
 tokio-test = "0.4.2"
 test-log = { version = "0.2.18", default-features = false, features = ["trace"] }
 tempfile = "3.8"
+hyper = { version = "1.0", features = ["full"] }
+hyper-util = { version = "0.1", features = ["full"] }
 
 [build-dependencies]
 mime_guess = "2.0.4"

Cargo.toml

@@ -5,6 +5,7 @@ Proxy routing tool. It can translate from one protocol to another, or select des
 
 Protocol supported:
 - HTTP CONNECT with mTLS
+- HTTP Forward Proxy (GET, POST, PUT, DELETE, etc.) with WebSocket upgrade support
 - HTTP CONNECT over QUIC
 - SOCKS v4,v4a,v5 with mTLS (TCP CONNECT ONLY)
 - TPROXY on linux, used with iptables REDIRECT, that is where the name comes from: RED(irect)PROXY