The Living Thing / Notebooks :

Weaponised social media

Trolls, bots, lulz, infowars and other moods of the modern networked ape

Usefulness: 🔧
Novelty: 💡
Uncertainty: 🤪 🤪 🤪
Incompleteness: 🚧 🚧 🚧
Facebook selling elections
Facebook selling elections

Information warfare on the social information graph, for the purpose of human behaviour control, various notes to that theme.

The other side to trusted news; hacking the implicit reputation system of social media to suborn factual reporting, or to motivate people to behave to suit your goals.

Research in this area is notably terrible, possibly because our tools of causality on social graphs are weak and it is hard, or perhaps, because the tools that some of us have are really good but people with really good tools to control the public are not going to mention that.

How it could be done

Strategies in the wild

Sea-lioning is a common hack for trolls, and is a whole interesting essay in strategic conversation deraillment strategies. Here is one strategy against it, the FAQ off system for live FAQ.


How do you observational inference of these systems? Of course, standard survey modelling. There is some structure to exploit here, e.g. causalimpact? How about when the data is a mixture of time-series data and one-off results (e.g. polling before and election and the election itself)

Automatic trolling, infinite fake news

The controversial GPT-2 (Radford et al. 2019)

GPT-2 displays a broad set of capabilities, including the ability to generate conditional synthetic text samples of unprecedented quality, where we prime the model with an input and have it generate a lengthy continuation. In addition, GPT-2 outperforms other language models trained on specific domains (like Wikipedia, news, or books) without needing to use these domain-specific training datasets.

It takes 5 minutes to download this package and start generating decent fake news; Whether you gain anything voer the traditional manual method is an open question.

The controversial deepcom model enables automatic comment generation for your fake news. (Yang et al. 2019)

Assembling these into a twitter bot farm is left as an exercise for the student.

How it’s being done

Jonathan Stray, What tools do we have to combat disinformation?


Aistrope, Tim. 2016. “Social Media and Counterterrorism Strategy.” Australian Journal of International Affairs 70 (2): 121–38.

Arif, Ahmer, Leo Graiden Stewart, and Kate Starbird. 2018. “Acting the Part: Examining Information Operations Within #BlackLivesMatter Discourse.” Proc. ACM Hum.-Comput. Interact. 2 (CSCW): 20:1–20:27.

Bay, Morten. 2018. “Weaponizing the Haters: The Last Jedi and the Strategic Politicization of Pop Culture Through Social Media Manipulation.” First Monday 23 (11).

Benkler, Yochai, Rob Faris, and Harold Roberts. 2018. Network Propaganda: Manipulation, Disinformation, and Radicalization in American Politics. New York, NY: Oxford University Press.

Bradshaw, S., and P. Howard. 2017. “Troops, Trolls and Troublemakers: A Global Inventory of Organized Social Media Manipulation” 2017.12.

Brito, Kellyton, Natalia Paula, Manoel Fernandes, and Silvio Meira. 2019. “Social Media and Presidential Campaigns – Preliminary Results of the 2018 Brazilian Presidential Election.” In Proceedings of the 20th Annual International Conference on Digital Government Research, 332–41. Dg.o 2019. New York, NY, USA: ACM.

Broniatowski, David A., Amelia M. Jamison, SiHua Qi, Lulwah AlKulaib, Tao Chen, Adrian Benton, Sandra C. Quinn, and Mark Dredze. 2018. “Weaponized Health Communication: Twitter Bots and Russian Trolls Amplify the Vaccine Debate.” American Journal of Public Health 108 (10): 1378–84.

Bursztyn, Victor S, and Larry Birnbaum. 2019. “Thousands of Small, Constant Rallies: A Large-Scale Analysis of Partisan WhatsApp Groups,” 6.

Cadwalladr, Carole. 2017. “The Great British Brexit Robbery: How Our Democracy Was Hijacked.” The Guardian: Technology, May 7, 2017.

Callaway, Duncan S., M. E. J. Newman, Steven H. Strogatz, and Duncan J. Watts. 2000. “Network Robustness and Fragility: Percolation on Random Graphs.” Physical Review Letters 85 (25): 5468–71.

Coscia, Michele. 2017. “Popularity Spikes Hurt Future Chances for Viral Propagation of Protomemes.” Communications of the ACM 61 (1): 70–77.

Crooks, Andrew. n.d. “Bot Stamina: Examining the Influence and Staying Power of Bots in Online Social Networks.” Accessed August 15, 2019.

Dittmar, Jeremiah E., and Skipper Seabold. 2015. “Media, Markets, and Radical Ideas: Evidence from the Protestant Reformation.” Centre for Economic Performance Working Paper.

Evans, David S. 2017. “The Economics of Attention Markets.” SSRN Scholarly Paper ID 3044858. Rochester, NY: Social Science Research Network.

Farrell, Henry. n.d. “Analysis | Blame Fox, Not Facebook, for Fake News.” Washington Post: Monkey Cage Analysis Analysis Interpretation of the News Based on Evidence, Including Data, as Well as Anticipating How Events Might Unfold Based on Past Events. Accessed September 16, 2019.

Farrell, Henry, and Bruce Schneier. 2018. “Common-Knowledge Attacks on Democracy.” SSRN Scholarly Paper ID 3273111. Rochester, NY: Social Science Research Network.

Goel, Sharad, Ashton Anderson, Jake Hofman, and Duncan J. Watts. 2015. “The Structural Virality of Online Diffusion.” Management Science, July, 150722112809007.

Goel, Sharad, Jake M. Hofman, Sébastien Lahaie, David M. Pennock, and Duncan J. Watts. 2010. “Predicting Consumer Behavior with Web Search.” Proceedings of the National Academy of Sciences 107 (41): 17486–90.

Goel, Sharad, Winter Mason, and Duncan J. Watts. 2010. “Real and Perceived Attitude Agreement in Social Networks.” Journal of Personality and Social Psychology 99 (4): 611–21.

Goel, Sharad, Duncan J. Watts, and Daniel G. Goldstein. 2012. “The Structure of Online Diffusion Networks.” In Proceedings of the 13th ACM Conference on Electronic Commerce - EC ’12, 623. Valencia, Spain: ACM Press.

Gonzalez-Bailon, Sandra. 2009. “Opening the Black Box of Link Formation: Social Factors Underlying the Structure of the Web.” Social Networks 31 (4): 271–80.

Granovetter, Mark. 1983. “The Strength of Weak Ties: A Network Theory Revisited.” Sociological Theory 1 (1): 201–33.

Granovetter, Mark S. 1973. “The Strength of Weak Ties.” The American Journal of Sociology 78 (6): 1360–80.

Grossman, Gene, and Elhanan Helpman. 2019. “Electoral Competition with Fake News.” w26409. Cambridge, MA: National Bureau of Economic Research.

Howard, Philip N., and Bence Kollanyi. 2016. “Bots, #StrongerIn, and #Brexit: Computational Propaganda During the UK-EU Referendum.” Browser Download This Paper.

Hurd, T. R., and James P. Gleeson. 2012. “On Watts’ Cascade Model with Random Link Weights,” November.

Johnson, Hollyn M., and Colleen M. Seifert. 1994. “Sources of the Continued Influence Effect: When Misinformation in Memory Affects Later Inferences.” Learning, Memory 20 (6): 1420–36.

Klausen, Jytte. 2015. “Tweeting the Jihad: Social Media Networks of Western Foreign Fighters in Syria and Iraq.” Studies in Conflict & Terrorism 38 (1): 1–22.

Larson, Heidi J. 2018. “The Biggest Pandemic Risk? Viral Misinformation.” Nature 562 (October): 309–9.

Levy, Gilat, and Ronny Razin. 2019. “Echo Chambers and Their Effects on Economic and Political Outcomes.” Annual Review of Economics 11 (1): 303–28.

Lewis, Rebecca. n.d. “Broadcasting the Reactionary Right on YouTube,” 61.

Lin, Herbert, and Jaclyn Kerr. 2019. “On Cyber-Enabled Information Warfare and Information Operations.” SSRN Scholarly Paper ID 3015680. Rochester, NY: Social Science Research Network.

Machado, Caio, Beatriz Kira, Vidya Narayanan, Bence Kollanyi, and Philip Howard. 2019. “A Study of Misinformation in WhatsApp Groups with a Focus on the Brazilian Presidential Elections.” In Companion Proceedings of the 2019 World Wide Web Conference, 1013–9. WWW ’19. New York, NY, USA: ACM.

Mahmoodi, Ali, Dan Bang, Karsten Olsen, Yuanyuan Aimee Zhao, Zhenhao Shi, Kristina Broberg, Shervin Safavi, et al. 2015. “Equality Bias Impairs Collective Decision-Making Across Cultures.” Proceedings of the National Academy of Sciences 112 (12): 3835–40.

Martin, Gregory J., and Ali Yurukoglu. 2017. “Bias in Cable News: Persuasion and Polarization.” American Economic Review 107 (9): 2565–99.

Marwick, Alice, and Rebecca Lewis. 2017. “Media Manipulation and Disinformation Online.” Data & Society Research Institute.

Munn, Luke. 2019. “Alt-Right Pipeline: Individual Journeys to Extremism Online.” First Monday 24 (6).

Oliver, Eric, and Tom Wood. 2014. “Larger Than Life.” New Scientist 224 (3000): 36–37.

Oliver, J. Eric, and Thomas J. Wood. 2014. “Conspiracy Theories and the Paranoid Style(s) of Mass Opinion.” American Journal of Political Science 58 (4): 952–66.

Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. “Language Models Are Unsupervised Multitask Learners,” 24.

Redlawsk, David P., Andrew J. W. Civettini, and Karen M. Emmerson. 2010. “The Affective Tipping Point: Do Motivated Reasoners Ever ‘Get It’?” Political Psychology 31 (4): 563–93.

Ribeiro, Manoel Horta, Raphael Ottoni, Robert West, Virgílio A. F. Almeida, and Wagner Meira. 2019. “Auditing Radicalization Pathways on YouTube,” August.

Rieder, Bernhard, Ariadna Matamoros-Fernández, and Òscar Coromina. 2018. “From Ranking Algorithms to ‘Ranking Cultures’: Investigating the Modulation of Visibility in YouTube Search Results.” Convergence 24 (1): 50–68.

Roozenbeek, Jon, and Sander van der Linden. 2019. “Fake News Game Confers Psychological Resistance Against Online Misinformation.” Palgrave Communications 5 (1): 1–10.

Salganik, Matthew J., Peter Sheridan Dodds, and Duncan J. Watts. 2006. “Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market.” Science 311 (5762): 854–56.

Salganik, Matthew J., and Duncan J. Watts. 2008. “Leading the Herd Astray: An Experimental Study of Self-Fulfilling Prophecies in an Artificial Cultural Market.” Social Psychology Quarterly 74 (4): 338.

Schuchard, Ross, Andrew T. Crooks, Anthony Stefanidis, and Arie Croitoru. 2019. “Bot Stamina: Examining the Influence and Staying Power of Bots in Online Social Networks.” Applied Network Science 4 (1): 1–23.

Sharma, Amit, Jake M. Hofman, and Duncan J. Watts. 2015. “Estimating the Causal Impact of Recommendation Systems from Observational Data.” Proceedings of the Sixteenth ACM Conference on Economics and Computation - EC ’15, 453–70.

Staines, Cassie, and Will Moy. 2018. “Tackling Misinformation in an Open Society.”

Starbird, Kate. 2017. “Examining the Alternative Media Ecosystem Through the Production of Alternative Narratives of Mass Shooting Events on Twitter.” In Eleventh International AAAI Conference on Web and Social Media.

———. 2019. “Disinformation’s Spread: Bots, Trolls and All of Us.” Nature 571 (July): 449.

Stewart, Leo G, Ahmer Arif, and Kate Starbird. 2018. “Examining Trolls and Polarization with a Retweet Network,” 6.

Stewart, Leo Graiden, Ahmer Arif, A. Conrad Nied, Emma S. Spiro, and Kate Starbird. 2017. “Drawing the Lines of Contention: Networked Frame Contests Within #BlackLivesMatter Discourse.” Proceedings of the ACM on Human-Computer Interaction 1 (CSCW): 1–23.

Taylor, Emily, and Stacie Hoffmann. n.d. “Industry Responses to Computational Propaganda and Social Media Manipulation.” Oxford Information Labs. Accessed November 25, 2019.

Verwimp, Philip. 2005. “An Economic Profile of Peasant Perpetrators of Genocide: Micro-Level Evidence from Rwanda.” Journal of Development Economics 77 (2): 297–323.

Vosoughi, Soroush, Deb Roy, and Sinan Aral. 2018. “The Spread of True and False News Online.” Science 359 (6380): 1146–51.

Watts, Duncan J. 2014. “Common Sense and Sociological Explanations.” American Journal of Sociology 120 (2): 313–51.

Watts, Duncan J., and Peter Sheridan Dodds. 2007. “Influentials, Networks, and Public Opinion Formation.” Journal of Consumer Research 34 (4): 441–58.

Watts, Duncan J, and Steven H Strogatz. 1998. “Collective Dynamics of ’Small-World’ Networks.” Nature 393 (6684): 440–42.

Wilson, Tom, Kaitlyn Zhou, and Kate Starbird. 2018. “Assembling Strategic Narratives: Information Operations as Collaborative Work Within an Online Community.” Proc. ACM Hum.-Comput. Interact. 2 (CSCW): 183:1–183:26.

Winter, Aaron. 2019. “Online Hate: From the Far-Right to the ‘Alt-Right’ and from the Margins to the Mainstream.” In Online Othering: Exploring Digital Violence and Discrimination on the Web, edited by Karen Lumsden and Emily Harmer, 39–63. Palgrave Studies in Cybercrime and Cybersecurity. Cham: Springer International Publishing.

Yang, Ze, Can Xu, Wei Wu, and Zhoujun Li. 2019. “Read, Attend and Comment: A Deep Architecture for Automatic News Comment Generation,” October.