How to Download HTML Content Using Session Cookie

[samirparikh]

I am trying to duplicate the curl command:

curl --cookie "session=5361...035a" https://example.com -o "output.html"

with Mojo::UserAgent in order to download some HTML from a URL that requires authentication. So far, I have the following:

use Mojo::UserAgent;
use Mojo::Cookie::Request;

my $ua  = Mojo::UserAgent->new;
my $url = "https://example.com";

$ua->cookie_jar->add(
  Mojo::Cookie::Request->new(
    name   => 'session',
    value  => '5361...035a',
    domain => '.example.com',
    path   => '/'
  )
);

my $res = $ua->get( $url )->result;
if    ($res->is_success)  { say $res->body }
elsif ($res->is_error)    { say $res->message }
elsif ($res->code == 301) { say $res->headers->location }
else                      { say 'Whatever...' }

I keep getting the error:

Can't locate object method "max_age" via package "Mojo::Cookie::Request" at /home/username/perl5/lib/perl5/Mojo/UserAgent/CookieJar.pm line 18.

I'm not sure why since I'm not calling the method max_age. Any idea as to what I am doing wrong? This is my first time using Mojolicious so please let me know if I'm way off track here.

Thanks in advance.

[ghandmann]

According to the Mojo::UserAgent::CookieJar example you should add a Mojo::Cookie::Response object and not a Mojo::Cookie::Request object.

M:C:Request does not have a max_age attribute, which would explain the error you are getting.

[samirparikh]

Thank you for your response. I changed Mojo::Cookie::Request to Mojo::Cookie::Response. I no longer get the error but $res->error is now 1 with $res->message returning Bad Request. The cookie I am trying to submit looks like the following, which was taken from Firefox's Developer Tools (Storage > Cookies):

session:"5361...35a"
Created:"Wed, 23 Nov 2022 16:41:55 GMT"
Domain:".example.com"
Expires / Max-Age:"Sat, 20 Nov 2032 16:41:55 GMT"
HostOnly:false
HttpOnly:true
Last Accessed:"Tue, 29 Nov 2022 01:03:07 GMT"
Path:"/"
SameSite:"None"
Secure:true
Size:135

Thanks again.

Edited to add: The new code now looks like this:

use Mojo::UserAgent;

my $ua  = Mojo::UserAgent->new;
my $url = "https://example.com";

$ua->cookie_jar->add(
  Mojo::Cookie::Response->new(
    name   => 'session',
    value  => '5361...035a',
    domain => '.example.com',
    path   => '/'
  )
);

my $res = $ua->get( $url )->result;
if    ($res->is_success)  { say $res->body }
elsif ($res->is_error)    { say $res->message }
elsif ($res->code == 301) { say $res->headers->location }
else                      { say 'Whatever...' }

[ghandmann]

That's hard to debug without having the actual values and servers, etc.

The mojo user agent will send only the matching cookies from the cookie jar to the server. Matching means, that the domain and requset path must match to a stored cookie in the jar. So at least the .examle.com domain in you code might interfere with a request to example.com.

In order to find out if the mojo user agent is sending a cookie at all, you can set the MOJO_CLIENT_DEBUG environment variable to 1 when executing your script. This will dump the raw http requests and responses on the terminal. There you can check if the cookie is sent to the server at all and if so, if it looks correct. Hint: In older versions of Mojo::UserAgents you need to set the variable MOJO_USERAGENT_DEBUG.

[samirparikh]

Looks like I accidentally responded using another Github account I have! Sorry about that!

Thanks, @ghandmann. It appears that changing the domain to example.com (without the dot) did the trick. Thank you also for the trick to setting the environment variable to help with debugging.