The attention economy of late capitalism demands I spend time clicking on a browser window to do things, rather than automating the world like we thought we were going to have all worked out by now.
I need download a thing from one social network and post it to my blog or whatever
You don't need a web browser for that; just use their API or an automation service.
Of course, facebook probably doesn't rank this as highly and something manually uploaded, that you obediently stared at advertisements while writing, so it's up to you whether it's worth your time being a clickmonkey for them, for marketing idk.
I want to get data from some plain public HTML site with minimal pain
Scrapy does what you want.
Also there is a custom cloud service (scrapinghub) that will deploy it for you on a massive scale if you want.
Wait but I have to log in to get my data and I'm too lazy to configure that
Turns out you can automate your local Firefox to do this in an easy way, although not a scalable one, thanks Ian Bicking.
Oh dear you aren't trying to fake being on social media for weaponised mass opinion formation are you? Well, at least that pays, I hope.
At this point in history, where we are using billions of dollars of technological infrastructure to perform glorified ritual social behavior, I find I'd prefer just pick lice out of the pelts of my audience the old fashioned way. But maybe this is not an option for you? If so, here is some stuff I read before realising I wasn't being paid enough.
Chromeless, the headless chrome browser, seems to be a hip thing here. And it has various easy cloud-deployment options. (I don't know how cookies are handled.)
Selenium is a browser testing and automation tool that seems like it could be made to automate real work on the web. But how can one automate its deployment and a bunch of user credentials with some degree of security and yet the absolute minimum of thought or effort? I do not yet know. To be continued. If absolutely necessary.
- guru99's tutorials on this
- testing a facebook application using selenium
- Facebook login with selenium
- webdriver docs
SeLite automates browser navigation and testing. It extends Selenium. It
- improves Selenium (API, syntax and visual interface),
- enables reuse,
- supports reporting and interaction,[…]
SeLite enables DB-driven navigation with SQLite
You might also get some mileage out of the Firefox CLI, mozrepl.