The Living Thing / Notebooks :

Browse the internet for me

serfing the web, tilling the clickfarm

The attention economy of late capitalism demands I spend time clicking on a browser window to do things, rather than automating the world like we thought we were going to have all worked out by now.

I need download a thing from one social network and post it to my blog or whatever

You don’t need a web browser for that; just use their API or an automation service.

Of course, facebook probably doesn’t rank this as highly automatically, so it’s up to you whether it’s worth your time being a clickmonkey for them, for marketing idk.

I want to get data from some plain public HTML site with minimal pain

Scrapy does what you want.

Also there is a custom cloud service (scrapinghub) that will deploy it for you on a massive scale if you want.

Wait but I have to log in to get my data and I’m too lazy to configure that

Turns out you can automate your local Firefox to do this in an easy, although not scalable way, thanks Ian Bicking

Nah, but I actually need to do some arbitrary interaction with a javascript-heavy site and possibly even on a massive scale

Oh dear you aren’t trying to fake being on social media for weaponised mass opinion formation are you? Well, at least that pays.


nickjs is a javascript library to do this. If this is something you are doing for money, it might be worth your while paying phantombuster to automate hosting of this. See their explanatory blog post. (TODO check security guarantees)


Chromeless, the headless chrome browser, seems to be the hip thing here. And it has various easy cloud-deployment options. (I don’t know how cookies are handled.)


Selenium seems to work do this. But how can one automate its deployment and a bunch of user credentials with some degree of security and yet the absolute minimum of thought or effort? I do not yet know. To be continued. If absolutely necessary. (sidestep this with ) But to be honest, at this point, at this dizzying pinnacle of using billions of dollars to do glorified fake social behavior, I’d really prefer just pick lice out of the pelts of my audience the old fashioned way.

Anyway, here was some stuff I read before deciding it’s a gigantic time-waste unless someone pays you lots of money.

You might also get some mileage out of the Firefox CLI, mozrepl