how to pass login data via the fabric -u linkedin.com | fabric -summarize

[pleabargain]

Specifically I want to get a summary of a linked in page
https://www.linkedin.com/search/results/all/?keywords=ILTM&origin=GLOBAL_SEARCH_HEADER&sid=o~k
I passed
fabric -u https://www.linkedin.com/search/results/all/?keywords=ILTM"&"origin=GLOBAL_SEARCH_HEADER"&"sid=o~k | fabric -p summarize

Note the "&"

but linkedin coughed out that I need to be logged in (which I am!) but LI needs more from me/fabric

Can someone share their login/password workaround for LinkedIn (and other sites?)

Thank you!

[bledburn]

The problem is that LinkedIn (like most major websites) uses login-based authentication with session cookies, CSRF tokens, and probably other security measures. You can automate the login process using curl, store the session cookies, and then pass those cookies to fabric -u when scraping the page.

Steps:
Log in to LinkedIn using curl and save the session cookies.

Use the saved session cookies to request the protected page.

Pass the retrieved page contents to fabric -u.

1. Log in and store cookies
   You'll need to send a POST request to LinkedIn's login endpoint with your email and password, while capturing cookies.

curl -c cookies.txt -X POST "https://www.linkedin.com/uas/login-submit"
-d "session_key=[email protected]"
-d "session_password=your_password"
-H "User-Agent: Mozilla/5.0"
-c cookies.txt → Saves cookies to cookies.txt

-X POST --> Sends a POST request to submit login credentials.

-d -->Sends email and password in the request body.

-H "User-Agent: ..." --> Some websites block curl unless you mimic a real browser.

Note: LinkedIn likely uses anti-bot protections (CSRF tokens, captchas, etc.), so this approach might not work directly. You'd then need to extract and pass a CSRF token in the request.

2. Fetch the protected page using stored cookies
   Once logged in, retrieve the page using the session stored in cookies.txt.

curl -b cookies.txt -L "https://www.linkedin.com/specific_page" -o page.html
-b cookies.txt --> Uses the saved login session.

-L --> Follows redirects.

-o page.html --> Saves the page content to a file.

3. Pass the page content to fabric -u, or locally you might be better using the --readability flag
   Now that you have the HTML content, pass it to fabric -u:

cat page.html | fabric -u #this won't work, you need to add xargs, the -u flag does not accept STDIN, so...
cat page.html | xargs fabric -u
Or, if Fabric supports reading from a file:

Potential Issues
LinkedIn may block automated logins. They use captchas, CSRF protection, and rate limits.

CSRF Token Requirement: LinkedIn probably requires a CSRF token in addition to the username and password. You would need to scrape it from the login page first.

Legal & Ethical Considerations: Scraping a site behind a login might violate LinkedIn’s Terms of Service.