[Note: If you landed here looking for Mastodon- or Fediverse-related resources, there is now a page dedicated to administering Mastodon instances, though very much a work in progress. This post will also be updated shortly, but as it stands, should serve as an introduction to many of the relevant issues.]
I recently responded to this broad, open-ended question on Quora: “What are some ways to prevent and/or deal with legal issues that arise from actions of users on user generated content websites?” Given that I spent five years at MySpace and eHarmony dealing with a panoply of those very issues, I decided to swing for the fence in answering that one. You can read the original answer on Quora, but I decided it was worth posting a beefed-up version here at Techlexica with some additional background covering the bases on fraud, abuse, and similar social media trust and safety challenges.
As background, in the late 2000s, cross-functional risk/abuse teams, now commonly known as Trust and Safety (T&S), grew out of necessity as social media and commerce exploded in popularity. As dominant platforms scaled by orders of magnitude to hundreds of millions or even billions of users, the challenges of the full range of human character and (mis)conduct — exacerbated by the relative anonymity and distance of the global Internet — made for unintended consequences and unprecedented threats at every turn. T&S lies at the intersection between legal, compliance, customer relations, community management/moderation, content review, editorial policy, privacy (“data protection” in EU parlance), and data security. In a nutshell, TSPA defines trust & safety as “the global community of professionals who develop and enforce principles and policies that define acceptable behavior and content online.” (More on conduct vs. content below.)
Site owners in a hurry may wish to skip the commentary and jump to the TL;DR list of practical tips below. As always, as my friends at the Social Media Club like to say, “If you get it, share it!”
Social media trust & safety: Policing the fastest-growing boomtowns in the virtual Wild West
Broadly speaking, from the site owner’s perspective, as the headline reads, if you build it, they will abuse it. That’s actually not quite true; if you build it, and they come — i.e., users visit in significant numbers and keep coming back, some will abuse it. Whether it’s spammers hawking Canadian pharmacy deals, pedophiles, identity thieves or Nigerian money transfer scammers, those wearing black hats are all too familiar with the strategy of going fishing — or phishing, as the case may be — where the fish are. (At eHarmony, one of our engineers joked that we should consider it a compliment of sorts the first time we got phished; stealing someone’s online dating password doesn’t exactly rank up there with getting access to their bank account.) Put differently, as Willie Sutton apocryphally said, he robbed banks “because that’s where the money is.”
We learned this the hard way at MySpace around 2005, when the site exploded in popularity among teens and young adults — at one point adding more than a quarter million new users per day. (Facebook was still limiting itself to the college market at the time.) It was an overnight cultural phenomenon among people aged roughly 12-25, to which everyone over 40 was completely oblivious, including parents, teachers, police and prosecutors. Tragically, it didn’t take long for the fishermen to figure out where the largest school of fish in history had gathered.
To be candid, an open, profile-based social media site like MySpace can be a predator’s dream. Drawing on the natural traits of exhibitionism, narcissism and voyeurism that blossom in adolescence, the site was brilliantly designed to empower young people to display their individuality online to the whole world as simply, flexibly and frictionlessly as possible, sharing every detail of their daily lives, friends, interests and passions.
Not surprisingly, young people jumped at the opportunity to do just that, to the tune of more than 1.5 billion page views per day — roughly 17,000 per second — with 2.3 million concurrent users. To grasp the size of that virtual crowd, picture a stadium the size of the Rose Bowl, packed to capacity — 25 times over. (At one point, our CTO, Aber Whitcomb, told me we had to unbox, install and provision eight or 10 new servers every day to keep up with growth. This was before AWS came along, of course.) To a high schooler, MySpace became the online equivalent of wearing band T-shirts to school the day after a concert, decorating a locker with pictures and bumper stickers, and plastering bedroom walls with posters of idols and sex symbols — but open to the entire world. Continue Reading