Toxic Behaviours in Esport: A Review of Data-Collection Methods Applied in Studying Toxic In-Gaming Behaviours

Highlights

• Much of the existing knowledge is based on self-reported accounts, i.e., surveys and interviews.
• Toxic behaviours that have been observed are mainly textual forms of toxic behaviours.

Introduction

Online multiplayer games – especially competitive games (esports) – have become notorious known for their “toxic gaming culture,” with toxic behaviours displayed by players (Consalvo, 2012; Kowert, 2020). As an academic concept, toxic behaviour has been criticised for its inherent vagueness and inconsistent use of research criteria when determining different types of behaviour (see, for example Kou, 2020; Kowert, 2020; Kwak et al., 2015). Nevertheless, despite little consensus on what constitute toxic behaviour, research across disciplines has consistently highlighted the prevalence of toxic behaviours in online games. Surveys conducted on prejudice and cyber-victimisation in online multiplayer games show that approximately 80% of players report witnessing prejudiced comments while playing online video games, with about 52–74 % experiencing direct victimisation (ADL, 2019; Ballard & Welch, 2017; Cary et al., 2020). Furthermore, about one in 10 players reports having depressive or suicidal thoughts due to exposure to harassment in online multiplayer games (ADL, 2019).

Thus, toxic behaviour and its negative consequences have directed research attention towards identifying key factors explaining why such behaviours emerge in online gaming contexts. A large body of work has established positive links between a range of personality traits and anti-social behaviours in online gaming contexts (Buckels et al., 2014; Lemercier-Dugarin et al., 2021; Tang et al., 2020; Tang & Fox, 2016). This research finds that personality traits like Machiavellianism, psychopathy, sadism, and gamer identification appear to positively correlate with toxic behaviours. Individual motivation of the perpetrator to engage in toxic behaviours, such as enjoyment, revenge, and thrill-seeking, has also been identified (Cook et al., 2018). As for the competitiveness of online gaming environments, researchers have identified a connection between group conflicts and aggression and the competitive nature of online gaming as a key factor in the emergence of toxic behaviours (Blackburn et al., 2014; Tan & Chen, 2022). Another extensive body of literature points towards anonymity and invisibility as the main driving forces of toxic behaviours (also known as the Online Disinhibition Effect), emphasising the dimension of perceived lack of restraints, accountability, and moral responsibility when navigating online (Beres et al., 2021; Kordyaka et al., 2020; Ruvalcaba et al., 2018; Souza et al., 2021; Suler, 2004). Lastly, normative belief systems and normalisation of toxic behaviours have been found as moderating aggressive toxic behaviours in online games (Beres et al., 2021; Hilvert-Bruce & Neill, 2020; Shores et al., 2014; Turkay et al., 2020).

It is beyond the scope of this article to assess or synthesise the empirical contributions of the research corpus. Rather, the objective of this article is to review the strengths and limitations of the data-collecting techniques applied in the research literature on toxic behaviours in online competitive multiplayer games. Throughout this work, methods are interpreted as data-collecting techniques. Different statistical or qualitative methods of analysis (e.g., content analysis, thematic analysis, phenomenological analysis methods) are not discussed. The article follows a systematic literature review procedure, reviewing 54 empirical articles on toxic behaviours, and outlines the three main data-collecting methods employed within the current research literature (self-reports, log data, and observations) and their various subtypes (e.g., in-gaming participant-observations, non-participatory observations, diary studies). Each methodological subtype is critically evaluated as to its analytical capacity for analysing toxic in-gaming behaviours in online competitive multiplayer games.

Finally, the reviewing process is summarised based on an assessment of the applicability of each research method for analysing toxic behaviours in online competitive multiplayer games. More specifically, each research method is judged against different epistemological research dimensions, such as its analytical capacity for interpreting meaning-making, establishing causal claims, or identifying health consequences related to toxic behaviours. This assessment follows the review procedures established by Philpot and colleagues (2019) in terms of evaluating established and emerging methodologies within a research field. Based on this assessment, it is argued that data-collecting techniques for capturing toxic behaviours are under-developed and adopted within the existing literature. To address this gap, future researchers are encouraged to employ the use of online video-based observations if interested in analysing toxic behaviours in online competitive multiplayer game contexts.

Methods

For this work, systematic mapping techniques for reviewing cross-disciplinary research as outlined by Curran et al. (2007) were applied in combination with recommendations set forth by Knopf (2006). As emphasised by Knopf (2006), tracing the methods applied and assessing their implications in terms of outlining a phenomenon enables the identification of overlooked issues relevant to advancing the field. Therefore, this article is organised as follows: 1) an explorative literature search to identify relevant search words, 2) systematic searches performed of international indexed bibliographic databases, 3) specification of inclusion and exclusion criteria, 4) filtering, screening, and sampling the research literature, 5) mapping the research literature corpus according to the data-collection techniques applied, and 6) reviewing and synthesising the strengths and limitations of various methodological approaches employed in studying toxic behaviours.

In December 2021, systematic literature searches were performed in the following databases: PsychInfo, Social Science Database, Sociological Abstracts; Arts & Humanities Database, International Bibliography of the Social Sciences (IBSS), Applied Social Sciences Index & Abstracts (ASSIA), Scopus, and Web of Science.

Owing to the conceptual vagueness and inconsistent definitions of toxic behaviours, a broad and comprehensive search strategy was applied in the indexed databases to generate and capture a large volume of relevant references. To generate a strong search string comprising relevant search words, a small sample of eight studies was initially read carefully with a focus on identifying various synonyms of toxic behaviours, as shown in Table 1. These eight studies were selected based on their level of citations and variety of examinations of different types of toxic behaviours in online gaming contexts (e.g., trolling, griefing). Based on this initial word- screening process, a comprehensive list of search words (see Table 2) was generated and applied in the performed database searches.

To be included, studies had to conform to the following inclusion criteria:

1. One or more data-collecting techniques were employed; hence, studies without empirical data were excluded from the sample.
2. Studies examined toxic in-gaming behaviours. Owing, however, to the concept’s vagueness and complexity, encompassing a variety of subcategories of toxic actions (as emphasised by Kou, 2020; Kowert, 2020), this inclusion criterion was intentionally broadly defined. To encapsulate studies examining the many different types of toxic in-gaming behaviours, studies were judged as conforming to this inclusion criteria if the author(s) classified behaviours as either toxic behaviours or as related subcategories, such as trolling, griefing, or deviant behaviour.
3. Studies examined toxic behaviours in an esport context. For this purpose, esport is operationalised to online multiplayer competitive gaming contexts, such as but not limited to: League of Legends, Counter Strike: Global Offensive, and Defense of the Ancient. Hence, studies failing to specify the gaming context or not conforming to the competitive multiplayer criteria were excluded from the sample.
4. Furthermore, the toxic behaviours had to occur within the game. As gaming situations compose specific contexts that frame the behaviours, studies addressing toxic behaviours unfolding outside of these in-gaming situations – for example swatting, scamming, and toxicity related to streaming – were excluded.
5. The studies were published in English.
6. Only peer-reviewed studies were included. Secondary research reporting, such as book chapters and editorials, were excluded. The performed searches were not limited to specific publication years.

These criteria were applied in the screening procedure to identify studies of relevance and dismiss studies of irrelevance. If the databases allowed it, the inclusion and exclusion criteria were operationalised and practicalised as “filters” for more efficient searches. Filters used were language, scholarly, and peer-reviewed research. The filtering spawned 9,501 references across the selected databases. These 9,501 references were subject to manual screening: i.e., reading through the generated list of titles and assessing whether a study accommodated the inclusion criteria or not. There then followed a screening of abstracts, then, finally, a complete reading of the studies. Based on the screening procedure, a total of 54 empirical studies on in-gaming toxic behaviours in esport or online competitive multiplayer games were identified and included in this literature review. The screening process is illustrated in Figure 1.

The research literature corpus was systematically coded using the qualitative data analysis software NVivo version 12 to extract and organise information from the individual research article. Methodological categories and subcategories were developed based on an initial reading and coding of the literature, allowing unexpected new applications of empirical methods to emerge throughout the coding process.

Results

Self-reports: Surveys and interviews

Table 3 summarises the data-collecting methods applied in the reviewed literature corpus. As shown, self-report methodologies constitute the most applied method for obtaining knowledge on toxic behaviours in online competitive multiplayer games. The two main types of self-reports are quantitative online surveys (e.g., victimisation surveys, experimental surveys) and qualitative interviews. Experimental online surveys have mainly explored player perceptions and beliefs regarding toxic behaviours (see Beres et al., 2021; Hilvert-Bruce & Neill, 2020). Few qualitative self-reports include focus groups or longitudinal diary data-collecting techniques (see Chen & Ong, 2018; Fox et al., 2018; Kordyaka et al., 2020).

Online victimisation surveys have focused on identifying shared traits by players most frequently victimised by toxic behaviours in online competitive multiplayer game contexts, such as gender and ethnicity (Ballard & Welch, 2017; Jiang & Yarosh, 2016; Ruvalcaba et al., 2018; Souza et al., 2021). Typically, victimised players are asked about the frequency of victimisation events, how they were victimised (e.g., sexual harassment, exclusion, hate speech), and their perception of the assumed motivation for the toxic behaviours (e.g., sexism, gaming rank/level, frustration, enjoyment). Online victimisation surveys have brought forth important insights on victimised players’ experiences with toxic behaviours, as well as contributed descriptions of behavioural properties involved in encounters with toxic behaviours.

Quantitative online surveys encompass large-scale datasets using standard protocols for measurement, yet these are typically limited to analyse toxic behaviours deductively from the questionnaires’ close-ended format and predefined categories, risking the possibility that unexpected behaviours will be overlooked. Complementing this stream of research, qualitative interviews – typically conducted as semi-structured interviews – offer greater in-depth explorative accounts of the experiences and behaviours enacted during toxic encounters. In particular, interviews have been employed to produce knowledge on victims’ coping strategies (Adinolf et al., 2018; Cote, 2017; Turkay et al., 2020). For example, analysing 37 in-depth semi- structured interviews with self-identified female gamers, Cote (2017) identified five main coping strategies (leaving online gaming, avoiding strangers, camouflaging gender, deploying skill and experience, and deliberately adopting aggressive personality traits) used by female gamers to deal with harassment during games.

While self-report data have brought forth detailed descriptions of behavioural properties and experiences with toxicity, these data sources are limited in their ability to capture mainly accounts of the behaviours and their social situations. As such, self-reports data encompass limitations as to provide detailed information on how the behaviours unfold (Jerolmack & Khan, 2014). What is more, self-reported retrospective accounts may further suffer from social desirability, error-recalling biases, and false memories when providing descriptions of encounters with toxicity. As already highlighted in the research literature, encounters with toxic behaviours are not rare events. On the contrary, players report frequent exposure to toxicity, as either victims, perpetrators, or bystanders. Furthermore, even a single encounter with toxicity may encompass multiple subtypes of toxic actions: e.g., both aggression and hate speech. Thus, differentiating between multiple events of toxic behaviours, as well as distinguishing the individual behavioural properties emerging in the situations, is likely a difficult challenge when using self-report data.

Online log data

In this stream of research, investigating and predicting the naturally occurring behaviours of individuals during toxic encounters constitutes by far the most prolific domain. Using large- scale datasets comprising actual behaviours (e.g., chatlogs, gameplay statistics, performances, reports), this stream of research has investigated the linguistic expressions displayed in encounters with toxic behaviours, such as the linguistic style differences between toxic and non-toxic players (Kwak & Blackburn, 2015), trolling interactions (Cook et al., 2019), and the influence of chat toxicity on team performances (Neto et al., 2017; Neto & Becker, 2018).

The application of log data has been successful in mapping out when toxic behaviours may occur, as well as establishing connections between players’ sociodemographic markers and their enactment of toxic behaviours. For example, studies analysing log data have consistently identified the pattern of the strong link between (the lack of) game success, such as the death of players or a perceived game loss, and the occurrence of toxicity (Cheng et al., 2019; Murnion et al., 2018; Märtens et al., 2015; Shen et al., 2020). As shown by Murnion et al. (2018), about 63% of all toxic messages proceed from the in-gaming death of a player. Moreover, experienced and skilful players tend to commit a higher degree of toxic behaviours, while simultaneously appearing more resilient towards in-gaming toxicity compared to players with low levels of experience (Murnion et al., 2018; Shen et al., 2020; Shores et al., 2014).

While this direction of research has contributed to the empirical advancement of capturing behaviours as they unfold within gameplay contexts, these behaviours tend to be limited to textual forms, as well as relatively decontextualised from the situations in which they occur. To this end, non-textual contextual information (e.g., avatar positions and movements) is to a large degree absent from the reviewed articles, and, as shown in Appendix (Table 1), only a tiny handful of toxic gameplay actions have been reported (e.g., intentional feeding, assisting the enemy team). In close connection to this issue, online log data suffer from selection bias, as only data created through automatic logging is available for interpretation. Whilst online logged data represent the objective behaviours of what players do in the game, they are limited to portraying merely a specific stream of visible technological mediated activity, thus neglecting non-logged yet potentially important contextual activities happening alongside it.

Observations: Online non-participatory observations and online participatory observations

Researchers have analysed observational data retrieved from online community-based forums and participatory-observations within online games with the aim of identifying a range of toxic behaviours and the contextual properties involved in players’ encounters with in-gaming toxicity.

Online non-participatory observations of official gaming sites deploy unobtrusive approaches for exploring visible traces of player descriptions of toxic behaviours. For instance, by analysing threads from the “r/leagueoflegends” subreddit, Kou (2020) identified five primary types of toxic behaviours (communicative aggression, cheating, hostage holding, mediocritising, and sabotaging) as well as five contextual factors that may lead to toxic behaviours (competitiveness, in-team conflict, perceived loss, powerlessness, and toxic behaviours). Similarly, Deslauriers et al. (2020), using observational data from the “r/deadbydaylight” subreddit, identified key aggravating factors involved in toxic behaviours: role identification, ambiguity in objective setting, individual gaming experience, task repetition, and the rigidity of norms. Furthermore, this research has also provided insights into the discrimination practices, such as racism and hate speech, involved in some encounters with toxic behaviours (Sengun, Salminen, Mawhorter, et al., 2019; Sengün et al., 2019; Wagener, 2018).

Nevertheless, online non-participatory observations of official gaming sites are limited primarily to reflecting community activity in the online gaming forums; hence, they naturally exclude insights from lurkers and non-participatory players, who may hold nuanced descriptions and experiences with toxic behaviours. Furthermore, although contributing valuable insights into player experiences and perceptions of toxic behaviours, observations of online forums tend to suffer from some of the same data-analytical capacities as self-reported data, as we merely access meaning-making and social negotiations related to toxic behaviours and not the actual situations and behaviours described.

Despite online participatory observations’ long history in online worlds research (see, for example, Boellstorff, 2008; Karhulahti, 2020; Nardi, 2010; Taylor, 2006), very little research has reported using such data-collecting techniques to investigate the naturally occurring behaviours of players during toxic encounters (Esmaeili & Woods, 2016; Monge & O’Brien, 2021). By immersing oneself in in-gaming situations, researchers may develop an embodied understanding of how encounters with toxic behaviours are felt and experienced, such as by victim and bystander, beyond the verbal accounts provided by participants. For example, by participating in over 20,000 battles in World of Tanks, Esmaeili and Woods (2016) established links between situational properties and behavioural patterns of toxicity, such as players performing blaming, revealing teammate locations, verbal abuse, and irrational behaviours in situations characterised by a “bad defeat” (situations with a decisive victory).

As in offline observational settings, however, the presence of a researcher may disturb the situations and behaviours they otherwise intend to study. While identity concealment (e.g., pseudo-anonymous nametags, avatars) is standard practice in most online games and may enable the researcher to remain unperceived, an immersed researcher cannot observe from a distance; rather, they are embedded in the immediate situation, taking on roles as bystanders, victims, or (but perhaps not as likely) antagonists, thus contaminating the situational setting they intend to study. Furthermore, because of the complex and chaotic nature of competitive multiplayer games, in-gaming participatory observations are challenged in their ability to observe (particularly in fine-grained detail) every behaviour enacted by every player. In addition, writing down detailed field notes in situ is almost impossible, owing to the fast- paced nature of most competitive games. Hence, documenting observations might be delayed, and, as emphasised by Philpot et al. (2019), in situ observations risk the same false memories, unconscious bias, and recollection failures usually attributed to self-report methods.

Online video-based analysis

A novel methodology to access direct observations of natural toxic behaviours is the use of online video-based observations. In online gaming cultures, the social practices of producing, consuming, and sharing gameplay videos constitute an integral part of the players’ natural environment, with video recordings typically publicly available on various online platforms and services. Hence, direct observations of gameplay actions captured, e.g., using screen recordings, allow researchers to access situated behaviours in naturally occurring situations and, furthermore, to systematically analyse the situational properties of interaction sequences in events of toxic behaviours in fine-grained detail. Researchers can observe the same events repeatedly, second-by-second, in slow motion, and cross-validate findings between researchers for high levels of reliability (Nassauer & Legewie, 2021; Pallante et al., 2022).

As online video-based analysis is still in its nascence, research on offline violence and bystander interventions will serve as case examples to exemplify some of the potential for using video-recorded direct observations to capture and study conflict situations as they evolve. For example, systematic video-based analysis has been used to analyse encounters with ticket-fining events on Danish buses, so as to investigate patterns of aggressive and non- aggressive behavioural actions, as well as the self-presentations strategies displayed (Friis et al., 2020; Friis & Lindegaard, 2022). Such micro-analysis of video-recorded workplace aggression proposes new avenue for informing preventive strategies and deescalating conflict situations. Furthermore, micro-analysis of surveillance recordings capturing actual public conflicts, such as street violence and bystander interventions, have allowed researchers to establish how perpetrators, victims, and bystanders behave during violent events. In particular, this stream of research has been attentive towards analysing bystander interventions, focusing on the likelihood of bystanders’ intervening in dangerous situations, the risk of intervening, and how bystander interventions are influenced by the development of the situation (Ejbye-Ernst et al., 2022; Liebst et al., 2018; Philpot et al., 2020). Researchers have demonstrated that, in real-life events of public violence, at least one bystander, but typically several, will intervene in nine out of 10 conflict situations (Philpot et al., 2020). As for the risk factors of bystander victimisation, video-based analysis show that victimisation does occur, but with a relatively low degree of severity (Liebst et al., 2018). Whilst online video-based analysis are relatively scarce, research on the situational dynamics of store robberies, with focus on the behavioural and emotional dynamics between perpetrators and clerks during actual robberies, has used footage uploaded to online video platforms like YouTube (Nassauer, 2018).

Nonetheless, although they capture direct observations of actual behaviours, video-based observations are not without methodological limitations. This method is particularly limited because of its dependence on visual access, thus constraining researchers in terms of what can be observed (e.g., due to camera angles or resolution quality) and whether a complete capture of an event has been recorded. As for online gaming, several interactions between players might be happing all at once yet may not be captured due to limitations in camera recording from just one player or camera angle. Furthermore, uploaded online video data – e.g., on YouTube or Twitch.tv. – risks biased data with incomplete, edited, selected, or over- accentuation of behaviours. Hence, some video data may be produced with an audience in mind, which risks threatening the naturalistic emergence of behaviours and interactions displayed. Following this, ethical challenges for using online available video data can be mitigated following the methods outlined by Legewie and Nassauer (2018), such as informed consent, anonymity, and ensuring no harm to the subjects.

Discussion

The objective of this article was to review the strengths and limitations of the methodological approaches applied in the existing research literature to study toxic behaviours in online competitive multiplayer games. The review process uncovered seven empirical methods employed in the research literature. Table 4 summarises the three main methods reported in the reviewed research literature (self-reports, log data analysis, and observations) and their subtypes, including an assessment of the analytical capacity of each subtype.

The application of self-reported accounts – quantitative as well as qualitative – have methodological advantages for relating the socio-psychological factors (e.g., personality traits, beliefs) and consequences of toxic behaviours (e.g., health consequences, coping strategies) to the toxic behaviours. Still, all self-reported approaches have the same limitations in terms of access to situational properties, actual behaviours, and interactions. Nevertheless, and compared to the other available methods, these methods stand out for providing researchers with unique access to analysing experiences, motivations, and social meaning related to encounters with toxic behaviours.

As for capturing large-scale datasets of textual manifestations of toxic behaviours and interactions, automatic logged data (e.g., chat logs) are preferred, as researchers can access unobtrusive observations of actual textual behaviours as they unfold within online gameplay. Such data sources are restrained, however, as they reflect exclusively objective behaviours automatically logged, thereby missing all other contextual information that shapes the interactional sequences and properties of these events. By contrast, direct observations through immersion in gameplay situations offers researchers access to the encounters with toxic behaviours as they unfold, thus allowing insights into the situational properties at play in the emergence of toxic behaviours. Nonetheless, direct observations through immersive participating observations encompass some methodological limitations in terms of their intrusive nature and challenges in collecting field notes in situ.

Although constituting a less methodological approach for capturing naturalistic behaviours of toxic encounters, online video-based behavioural analysis holds the potential to advance current behavioural research. This method encompasses, in particular, methodological benefits similar to those of direct observations, automatic logging activities, and in-game participation while avoiding the data-collection limitations of immersive strategies and acquiring more fine-grained contextual information compared to automatic log data. Hence, future researchers are encouraged to consider the availability of video data if interested in studying the interactional dynamics and distinctive behavioural patterns in encounters with toxic behaviours.

Limitations

This work is naturally limited to exclusively reviewing the data-collecting techniques applied in current research on toxic behaviours in online competitive multiplayer games. Obviously, such a review strategy encompasses several limitations. First, an assessment of the empirical contributions of the reviewed articles is outside the scope of this work; thus, this article does not provide a synthesis of toxic behaviours that may address and advance the current vagueness of toxic behaviours as an academic concept. Rather, this work takes an initial step towards considering alternative methodologies for studying toxic behaviours. Secondly, owing to the focus on competitive gaming contexts, it is possible that important contributions on toxic behaviours in non-competitive yet online multiplayer games are missing from the reviewed research corpus. Thus, research that may already be applying alternative methodologies for studying toxic behaviours in online games has not been included in this work. Thirdly, and in continuation, this work solely reviewed articles and conference papers, hence leaving out methodological contributions described in books and book chapters. Therefore, I encourage future researchers to further interpret the reviewed studies provided by this work, and to build on and discuss the application of different research methods to study areas of toxic gaming behaviours.

Finally, while this work was systematically undertaken following the research principles of reviewing and mapping procedures set forth by Curran et al. (2007) and Knopf (2006), as well as using coding software (NVivo), throughout the process, this work may be prone to human errors due to relying on a single coder. Multiple coders would have enhanced the trustworthiness and reliability of this work. Therefore, researchers are encouraged to employ multiple coders in future studies.

Conclusion

This article was based on a review of the applications of a range of empirical methods (self- reports, log data analysis, and observations) for analysing toxic behaviours in esport. Following systematic review procedures, it was demonstrated that knowledge of toxic behaviours is typically based on self-reported accounts (e.g., surveys and interviews), while less-established methodological available for capturing naturalistic behaviours of toxic encounters are rarely used. To address this gap, future researchers are encouraged to employ video observational techniques to study toxic behaviours. Despite of its methodological limitations, video-based behavioural analysis holds the potential to allow researchers access to the chronological structure and temporal properties of toxic behaviours, interactional sequences, and emotional expressions as they unfold.

Acknowledgments

Søren Kristiansen is thanked for his valuable feedback on earlier versions of this manuscript. Earlier versions of this work were presented in the research group CASTOR at the Department of Sociology and Social Work at Aalborg University, as well as in the NOS-HS funded workshops on esports in the Nordic countries organised by Anne Mette Thorhauge, Andreas Gregersen, Patrick Prax, and Veli-Matti Karhulahti, and the members of these networks are thanked for their feedback. Enormous thanks also go to the two anonymous reviewers and editors of International Journal of Esport. The publishing of this work was supported by NOS- HS exploratory workshop on Nordic Esports.

Declaration of interest statement

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

• Adinolf, S., & Turkay, S. (2018). Toxic Behaviors in Esports Games: Player Perceptions and Coping Strategies. I Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (s. 365–372). Association for Computing Machinery. https://doi.org/10.1145/3270316.3271545
• ADL. (2019). Free to Play? Hate, Harassment, and Positive Social Experiences in Online Games. https://www.adl.org/free-to-play
• Ballard, M. E., & Welch, K. M. (2017). Virtual warfare: Cyberbullying and cyber-victimization in MMOG play. Games and Culture: A Journal of Interactive Media, 12(5), 466–491. https://doi.org/10.1177/1555412015592473
• Beres, N. A., Frommel, J., Reid, E., Mandryk, R. L., & Klarkowski, M. (2021). Don’t You Know That You’re Toxic: Normalization of Toxicity in Online Gaming. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan. ACM New York, NY, USA. 1–15. https://doi.org/10.1145/3411764.3445157
• Blackburn, J., Kwak, H., & Assoc Comp Machinery. (2014). STFU NOOB! Predicting Crowdsourced Decisions on Toxic Behavior in Online Games (WOS:000455945100079). 877–887. https://doi.org/10.1145/2566486.2567987
• Boellstorff, T. (2008). Coming of Age in Second Life: An Anthropologist Explores the Virtually Human. Princeton University Press. https://press.princeton.edu/books/paperback/9780691168340/coming-of-age-in-second-life
• Buckels, E. E., Trapnell, P. D., & Paulhus, D. L. (2014). Trolls just want to have fun. Personality and Individual Differences, 67, 97–102. https://doi.org/10.1016/j.paid.2014.01.016
• Canossa, A., Salimov, D., Azadvar, A., Harteveld, C., & Yannakakis, G. (2021). For Honor, for Toxicity: Detecting Toxic Behavior through Gameplay. Proceedings of the ACM on Human- Computer Interaction, 5, 1–29. https://doi.org/10.1145/3474680
• Cary, L. A., Axt, J., & Chasteen, A. L. (2020). The interplay of individual differences, norms, and group identification in predicting prejudiced behavior in online video game interactions. Journal of Applied Social Psychology, 50(11), e12700. https://doi.org/10.1111/jasp.12700
• Chen, V., & Ong, J. (2018). The rationalization process of online game cheating behaviors. INFORMATION COMMUNICATION & SOCIETY, 21(2), 273–287. https://doi.org/10.1080/1369118X.2016.1271898
• Cheng, Z., Yang, Y., Tan, C., Cheng, D., Cheng, A., Zhuang, Y., & Assoc Comp Machinery. (2019). What Makes a Good Team? A Large-scale Study on the Effect of Team Composition in Honor of Kings. arxiv.org (WOS:000483508402073). 2666–2672. https://doi.org/10.1145/3308558.3313530
• Consalvo, M. (2012, november 11). Confronting Toxic Gamer Culture: A Challenge for Feminist Game Studies Scholars. Ada New Media. https://adanewmedia.org/2012/11/issue1-consalvo/
• Cook, C., Conijn, R., Schaafsma, J., & Antheunis, M. (2019). For Whom the Gamer Trolls: A Study of Trolling Interactions in the Online Gaming Context. Journal of Computer- Mediated Communication, 24(6).
• Cook, C., Schaafsma, J., & Antheunis, M. (2018). Under the bridge: An in-depth examination of online trolling in the gaming context. New Media & Society, 20(9), 3323–3340. https://doi.org/10.1177/1461444817748578
• Cornel, J., Pablo, C., Marzan, J., Mercado, V., Fabito, B., Rodriguez, R., Octaviano, M., Oco, N., De La Cruz, A., & IEEE. (2019). Cyberbullying Detection for Online Games Chat Logs using Deep Learning (WOS:000618530300087). 2019 IEEE 11TH INTERNATIONAL CONFERENCE ON HUMANOID, NANOTECHNOLOGY, INFORMATION TECHNOLOGY, COMMUNICATION AND CONTROL, ENVIRONMENT, AND MANAGEMENT (HNICEM).
• Cote, A. C. (2017). “I Can Defend Myself”: Women’s Strategies for Coping With Harassment While Gaming Online. Games and Culture, 12(2), 136–155. https://doi.org/10.1177/1555412015587603
• Curran, C., Burchardt, T., Knapp, M., McDaid, D., & Bingqin Li. (2007). Challenges in Multidisciplinary Systematic Reviewing: A Study on Social Exclusion and Mental Health Policy. Social Policy & Administration, 41(3), 289–312. https://doi.org/10.1111/j.1467- 9515.2007.00553.x
• Deslauriers, P., Bonenfant, M., & St-Martin, L. I. L. (2020). Assessing toxic behaviour in dead by daylight: Perceptions and factors of toxicity according to the game’s official subreddit contributors. Game Studies Online. http://gamestudies.org/2004/articles/deslauriers_iseutlafrancestmartin_bonenfant
• Ejbye-Ernst, P., Lindegaard, M. R., & Bernasco, W. (2022). Third Parties Mirror the Aggression of the Antagonists: A Video-Based Analysis of Third-Party Aggression in Interpersonal Conflicts. Journal of Interpersonal Violence, 37(17–18), NP15899–NP15924. https://doi.org/10.1177/08862605211023503
• Ekiciler, A., Ahioğlu, İ., Yıldırım, N., Ajas, İ., & Kaya, T. (2021). The Bullying Game: Sexism Based Toxic Language Analysis on Online Games Chat Logs by Text Mining. Journal of International Women's Studies 24(3). Bridgewater.
• Emmerich, K., Krekhov, A., & Krüger, J. (2020). “pls Uninstall”: On the interplay of the COVID-19 pandemic and toxic player behavior in competitive gaming. 224–228. Scopus. https://doi.org/10.1145/3383668.3419896. Conference: 2016 22nd International Conference on Virtual System & Multimedia (VSMM)
• Fox, J., Gilbert, M., & Tang, W. Y. (2018). Player experiences in a massively multiplayer online game: A diary study of performance, motivation, and social interaction. New Media & Society, 20(11), 4056–4073. https://doi.org/10.1177/1461444818767102
• Fox, J., & Tang, W. Y. (2017). Women’s experiences with general and sexual harassment in online video games: Rumination, organizational responsiveness, withdrawal, and coping strategies. New Media & Society, 19(8), 1290–1307. https://doi.org/10.1177/1461444816635778
• Friis, C. B., & Lindegaard, M. R. (2022). Citizen Self-Presentation and Aggression in Ticket Fining Events: A Video Observational Study. Deviant Behavior, 43(8), 897–911. https://doi.org/10.1080/01639625.2021.1921559
• Friis, C. B., Philpot, R., & Lindegaard, M. R. (2020). Ticket inspectors in action: Body-worn camera analysis of aggressive and nonaggressive passenger encounters. Psychology of Violence, 10(5), 483–492.
• Hilvert-Bruce, Z., & Neill, J. T. (2020). I’m just trolling: The role of normative beliefs in aggressive behaviour in online gaming. Computers in Human Behavior, 102, 303–311. https://doi.org/10.1016/j.chb.2019.09.003
• Irwin, S., Naweed, A., & Lastella, M. (2020). The mind games have already started: An in-depth examination of trash talking in Counter-Strike: Global Offensive esports using practice theory. Journal of Gaming and Virtual Worlds, 13(2), 173–194. https://doi.org/10.1386/jgvw_00035_1
• Jerolmack, C., & Khan, S. (2014). Talk Is Cheap: Ethnography and the Attitudinal Fallacy. Sociological Methods & Research, 43(2), 178–209. https://doi.org/10.1177/0049124114523396
• Jiang, J., & Yarosh, S. (2016). Why Do Teammates Hate Me? Cross-Cultural Tensions and Social Dynamics in Online Games (WOS:000468137600072). CSCW '16 Companion: Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion. 301–304. https://doi.org/10.1145/2818052.2869087
• Karhulahti, V.-M. (2020). Esport play: Anticipation, attachment, and addiction in psycholudic development. Bloomsbury Publishing USA.
• Knopf, J. W. (2006). Doing a Literature Review. PS: Political Science & Politics, 39(1), 127-132. doi:10.1017/S1049096506060264. Cambridge University Press.
• Komaç, G., & Çağıltay, K. (2019). An overview of trolling behavior in online spaces and gaming context. 1–4.
• Kordyaka, B., Jahn, K., & Niehaves, B. (2020). Towards a unified theory of toxic behavior in video games. Internet Research, 30(4), 1081–1102. https://doi.org/10.1108/INTR-08-2019- 0343. Conference: 2019 1st International Informatics and Software Engineering Conference (UBMYK)
• Kordyaka, B., Klesel, M., & Jahn, K. (2019). Perpetrators in League of Legends: Scale Development and Validation of Toxic Behavior (T. Bui, Red.; WOS:000625294902066; s. 2486–2495).Kou, Y. (2020). Toxic Behaviors in Team-Based Competitive Gaming: The Case of League of Legends. https://doi.org/10.1145/3410404.3414243 Conference: CHI PLAY'2020.
• Kou, Y. (2021). Punishment and Its Discontents: An Analysis of Permanent Ban in an Online Game Community. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2). Scopus. https://doi.org/10.1145/3476075
• Kou, Y., & Gui, X. (2014). Playing with strangers: Understanding temporary teams in league of legends. Proceedings of the first ACM SIGCHI annual symposium on Computer-human interaction in play, 161–169. https://doi.org/10.1145/2658537.2658538
• Kou, Y., & Gui, X. (2021). Flag and Flaggability in Automated Moderation: The Case of Reporting Toxic Behavior in an Online Game Community. 1–12.
• Kou, Y., Gui, X., (2017). The Rise and Fall of Moral Labor in an Online Game Community. Assoc Comp Machinery. (WOS:000455085000055). 223–226. https://doi.org/10.1145/3022198.3026312
• Kowert, R. (2020). Dark Participation in Games. Frontiers in Psychology, 11. https://doi.org/10.3389/fpsyg.2020.598947
• Kwak, H., & Blackburn, J. (2015). Linguistic Analysis of Toxic Behavior in an Online Video Game. I L. M. Aiello & D. McFarland (Red.), Social Informatics (s. 209–217). Springer International Publishing. https://doi.org/10.1007/978-3-319-15168-7_26
• Kwak, H., Blackburn, J., & Han, S. (2015). Exploring Cyberbullying and Other Toxic Behavior in Team Competition Online Games. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 3739–3748. https://doi.org/10.1145/2702123.2702529
• Lee, M., Johnson, D., Tanjitpiyanond, P., & Louis, W. R. (2021). It’s habit, not toxicity, driving hours spent in DOTA 2. Entertainment Computing, 100472.
• Lee, S. J., Jeong, E. J., & Jeon, J. H. (2019). Disruptive behaviors in online games: Effects of moral positioning, competitive motivation, and aggression in “League of Legends.” Social Behavior and Personality: An International Journal, 47(2), 1–9.
• Legewie, N., & Nassauer, A. (2018). YouTube, Google, Facebook: 21st Century Online Video Research and Research Ethics. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, Vol 19, No 3 (2018): Research Ethics in Qualitative Research. https://doi.org/10.17169/FQS-19.3.3130
• Lemercier-Dugarin, M., Romo, L., Tijus, C., & Zerhouni, O. (2021). “Who Are the Cyka Blyat?” How Empathy, Impulsivity, and Motivations to Play Predict Aggressive Behaviors in Multiplayer Online Games. Cyberpsychology, Behavior, and Social Networking, 24(1), 63– 69. https://doi.org/10.1089/cyber.2020.0041
• Li, Q., & Pustaka, A. (2017). When cyberbullies meet gamers: What do young adults think? Educational Research, 59(4), 426–443. https://doi.org/10.1080/00131881.2017.1369857
• Liebst, L. S., Heinskou, M. B., & Ejbye-Ernst, P. (2018). On the Actual Risk of Bystander Intervention: A Statistical Study Based on Naturally Occurring Violent Emergencies. Journal of Research in Crime and Delinquency, 55(1), 27–50. https://doi.org/10.1177/0022427817710776
• Mattinen, T., & Macey, J., (2018). Online Abuse and Age in Dota 2 . Mindtrek '18: Proceedings of the 22nd International Academic Mindtrek Conference (WOS:000460473700008). 69–78. https://doi.org/10.1145/3275116.3275149
• Monge, C., & O’Brien, T. (2021). Effects of individual toxic behavior on team performance in League of Legends. Media Psychology, 25(1), 82-105. https://doi.org/10.1080/15213269.2020.1868322
• Murnion, S., Buchanan, W., Smales, A., & Russell, G. (2018). Machine learning and semantic analysis of in-game chat for cyberbullying. COMPUTERS & SECURITY, 76, 197–213. https://doi.org/10.1016/j.cose.2018.02.016
• Märtens, M., Shen, S., Iosup, A., Kuipers, F., & IEEE. (2015). Toxicity Detection in Multiplayer Online Games (WOS:000380505500005). Conference: The 14th International workshop on network and systems support for games (Netgames).Croatia.
• Nardi, B. A. (2010). My life as a night elf priest: An anthropological account of world of warcraft. University of Michigan Press.
• Nassauer, A. (2018). How Robberies Succeed or Fail: Analyzing Crime Caught on CCTV. Journal of Research in Crime and Delinquency, 55(1), 125–154.
• Nassauer, A., & Legewie, N. M. (2021). Video Data Analysis: A Methodological Frame for a Novel Research Trend. Sociological Methods & Research, 50(1), 135–174. https://doi.org/10.1177/0049124118769093
• Neto, J., & Becker, K. (2018). Relating conversational topics and toxic behavior effects in a MOBA game. Entertainment Computing, 26, 10–29. https://doi.org/10.1016/j.entcom.2017.12.004
• Neto, J., Yokoyama, K., & Becker, K. (2017). Studying toxic behavior influence and player chat in an online video game (s. 33). Conference paper in: Proceedings of WI'17, Leipzig, Germany. https://doi.org/10.1145/3106426.3106452
• Pallante, V., Liebst, L. S., Ejbye-Ernst, P., Friis, C. B., & Lindegaard, M. R. (2022). Putting actual behavior back into the social sciences: A Plea for video-based interaction ethology. PsyArXiv. https://doi.org/10.31234/osf.io/7xy39
• Philpot, R., Liebst, L. S., Levine, M., Bernasco, W., & Lindegaard, M. R. (2020). Would I be helped? Cross-national CCTV footage shows that intervention is the norm in public conflicts. American Psychologist, 75(1), 66–75. https://doi.org/10.1037/amp0000469
• Philpot, R., Liebst, L. S., Møller, K. K., Lindegaard, M. R., & Levine, M. (2019). Capturing violence in the night-time economy: A review of established and emerging methodologies. Aggression and Violent Behavior, 46, 56–65. https://doi.org/10.1016/j.avb.2019.02.004
• Pujante Jr, N. T. (2021). Speech for Fun, Fury, and Freedom: Exploring Trash Talk in Gaming Stations. Asian Journal of Language, Literature and Culture Studies, 4(1), 1–11.
• Ruvalcaba, O., Shulze, J., Kim, A., Berzenski, S. R., & Otten, M. P. (2018). Women’s Experiences in eSports: Gendered Differences in Peer and Spectator Feedback During Competitive Video Game Play. Journal of Sport & Social Issues, 42(4), 295–311. https://doi.org/10.1177/0193723518773287
• Sengun, S., Salminen, J., Jung, S., Mawhorter, P., Jansen, B., & Assoc Comp Machinery. (2019). Analyzing Hate Speech Toward Players from the MENA in League of Legends (WOS:000482042101047). CHI EA ’19 EXTENDED ABSTRACTS: EXTENDED ABSTRACTS OF THE 2019 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS. https://doi.org/10.1145/3290607.3312924
• Sengun, S., Salminen, J., Mawhorter, P., Jung, S., Jansen, B., & ACM. (2019). Exploring the Relationship Between Game Content and Culture-based Toxicity A Case Study of League of Legands and MENA Players. In: Processings of ACM Conference on Hypertext and Social Media. (WOS:000629029900014). 87–95. https://doi.org/10.1145/3342220.3343652
• Shen, C., Sun, Q., Kim, T., Wolff, G., Ratan, R., & Williams, D. (2020). Viral vitriol: Predictors and contagion of online toxicity in World of Tanks. Computers in Human Behavior, 108, 9 pages. https://doi.org/10.1016/j.chb.2020.106343
• Shores, K. B., He, Y., Swanenburg, K. L., Kraut, R., & Riedl, J. (2014). The identification of deviance and its impact on retention in a multiplayer game. 1356–1365. Conference: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing
• Souza, A. C. C., Pegorini, J. I., Yada, N. T., Costa, L. M., & Souza, F. C. M. (2021). Exploring toxic behavior in multiplayer online games: Perceptions of different genders. Conference: SBC – Proceedings of SBGames 2021
• Stoop, W., Kunneman, F., Van den Bosch, & A., Miller, B., (2019). Detecting harassment in real time as conversations develop (WOS:000538480400003). 19–24. Assoc Computat Linguist.
• Suler, J. (2004). The Online Disinhibition Effect. CyberPsychology & Behavior, 7(3), 321–326. https://doi.org/10.1089/1094931041291295
• Tan, W., & Chen, L. (2022). That’s not my fault: Excuses given by players exhibiting in-game intra-team aggressive behavior in online games. COMPUTERS IN HUMAN BEHAVIOR, 127. https://doi.org/10.1016/j.chb.2021.107045
• Tang, W. Y., & Fox, J. (2016). Men’s harassment behavior in online video games: Personality traits and game factors. Aggressive Behavior, 42(6), 513–521. https://doi.org/10.1002/ab.21646
• Tang, W. Y., Reer, F., & Quandt, T. (2020). Investigating sexual harassment in online video games: How personality and context factors are related to toxic sexual behaviors against fellow players. Aggressive Behavior, 46(1), 127–135. https://doi.org/10.1002/ab.21873
• Taylor, T. L. (2006). Play between worlds: Exploring online game culture. MIT Press.
• Thompson, J. J., Leung, B. H. M., Blair, M. R., & Taboada, M. (2017). Sentiment analysis of player chat messaging in the video game StarCraft 2: Extending a lexicon-based model. Knowledge-Based Systems, 137, 149–162. https://doi.org/10.1016/j.knosys.2017.09.022
• Turkay, S., Formosa, J., Adinolf, S., Cuthbert, R., & Altizer, R., (2020). See No Evil, Hear No Evil, Speak No Evil: How Collegiate Esports Players Define, Experience and Cope with Toxicity (WOS:000695432500064). PROCEEDINGS OF THE 2020 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS (CHI’20), 13 pages. https://doi.org/10.1145/3313831.3376191
• Wagener, A. (2018). Russophobia in DotA 2 A critical discursive analysis of online discrimination. INTERNATIONAL REVIEW OF PRAGMATICS, 10(1), 57–75. https://doi.org/10.1163/18773109-01001003
• Weld, H., Huang, G., Lee, J., Zhang, T., Wang, K., Guo, X., Long, S., Poon, J., & Han, S. C. (2021). CONDA: a CONtextual Dual-Annotated dataset for in-game toxicity understanding and detection. arXiv preprint arXiv:2106.06213.
• Wright, M. F. (2019). Friends and console-gaming aggression: The role of friendship quality, anger, and revenge planning. Games and Culture: A Journal of Interactive Media, 14(6), 604–621. https://doi.org/10.1177/1555412017720554