Tag Archives: safe harbor

Social Media Trust & Safety: “If you build it, they will abuse it”

“If You Build It, They Will Abuse It”: Social Media Trust & Safety, Risk, Fraud and Abuse

[Note: If you landed here looking for Mastodon- or Fediverse-related resources, there is now a page dedicated to administering Mastodon instances, though very much a work in progress. This post will also be updated shortly, but as it stands, should serve as an introduction to many of the relevant issues.]

I recently responded to this broad, open-ended question on Quora: “What are some ways to prevent and/or deal with legal issues that arise from actions of users on user generated content websites?” Given that I spent five years at MySpace and eHarmony dealing with a panoply of those very issues, I decided to swing for the fence in answering that one. You can read the original answer on Quora, but I decided it was worth posting a beefed-up version here at Techlexica with some additional background covering the bases on fraud, abuse, and similar social media trust and safety challenges.

As background, in the late 2000s, cross-functional risk/abuse teams, now commonly known as Trust and Safety (T&S), grew out of necessity as social media and commerce exploded in popularity. As dominant platforms scaled by orders of magnitude to hundreds of millions or even billions of users, the challenges of the full range of human character and (mis)conduct — exacerbated by the relative anonymity and distance of the global Internet — made for unintended consequences and unprecedented threats at every turn. T&S lies at the intersection between legal, compliance, customer relations, community management/moderation, content review, editorial policy, privacy (“data protection” in EU parlance), and data security. In a nutshell, TSPA defines trust & safety as “the global community of professionals who develop and enforce principles and policies that define acceptable behavior and content online.” (More on conduct vs. content below.)

Site owners in a hurry may wish to skip the commentary and jump to the TL;DR list of practical tips below.  As always, as my friends at the Social Media Club like to say, “If you get it, share it!

Social media trust & safety: Policing the fastest-growing boomtowns in the virtual Wild West

Broadly speaking, from the site owner’s perspective, as the headline reads, if you build it, they will abuse it. That’s actually not quite true; if you build it, and they come — i.e., users visit in significant numbers and keep coming back, some will abuse it. Quora answer by Antone Johnson about social media trust & safety Whether it’s spammers hawking Canadian pharmacy deals, pedophiles, identity thieves or Nigerian money transfer scammers, those wearing black hats are all too familiar with the strategy of going fishing — or phishing, as the case may be — where the fish are. (At eHarmony, one of our engineers joked that we should consider it a compliment of sorts the first time we got phished; stealing someone’s online dating password doesn’t exactly rank up there with getting access to their bank account.) Put differently, as Willie Sutton apocryphally said, he robbed banks “because that’s where the money is.”

We learned this the hard way at MySpace around 2005, when the site exploded in popularity among teens and young adults — at one point adding more than a quarter million new users per day. (Facebook was still limiting itself to the college market at the time.) It was an overnight cultural phenomenon among people aged roughly 12-25, to which everyone over 40 was completely oblivious, including parents, teachers, police and prosecutors. Tragically, it didn’t take long for the fishermen to figure out where the largest school of fish in history had gathered.

To be candid, an open, profile-based social media site like MySpace can be a predator’s dream. Drawing on the natural traits of exhibitionism, narcissism and voyeurism that blossom in adolescence, the site was brilliantly designed to empower young people to display their individuality online to the whole world as simply, flexibly and frictionlessly as possible, sharing every detail of their daily lives, friends, interests and passions.

MySpace Page - Antone Johnson

Not surprisingly, young people jumped at the opportunity to do just that, to the tune of more than 1.5 billion page views per day — roughly 17,000 per second — with 2.3 million concurrent users.  To grasp the size of that virtual crowd, picture a stadium the size of the Rose Bowl, packed to capacity — 25 times over.  (At one point, our CTO, Aber Whitcomb, told me we had to unbox, install and provision eight or 10 new servers every day to keep up with growth.  This was before AWS came along, of course.) To a high schooler, MySpace became the online equivalent of wearing band T-shirts to school the day after a concert, decorating a locker with pictures and bumper stickers, and plastering bedroom walls with posters of idols and sex symbols — but open to the entire world. Continue Reading

Crowdsourced Auto Design: Local Motors Audaciously Challenges the Status Quo

As part of Social Media Week, I visited the Autodesk Gallery for a panel discussion entitled “The Next Challenge For The Crowd: Quickly Build Real, Hard, Complex Shit That Matters (Like Cars).” With a title like that, I was not about to miss it. The Rally Fighter parked out front (around the corner from a Tesla roadster, natch) foreshadowed the fascinating discussion inside. Many of the ideas which had been swirling around in my head for the past week coalesced in this 90-minute conversation with Jay Rogers, CEO and co-founder of Local Motors; Marten Mickos, former CEO of MySQL; and Chris Anderson, Editor-in-Chief of Wired, moderated by Matt Johnston. Continue Reading

Bottom Line Law Group

Email: info@bottomlinelawgroup.com


2081 Center Street
Berkeley, CA 94704
+1 (415) 729-5405

1450 2nd Street, Suite 109
Santa Monica, CA 90401
+1 (310) 776-5484

Mastodon: @antone@sfba.social

Post: https://post.news/antone

Linktree: https://linktr.ee/antonejohnson

Twitter: @techlexica

LinkedIn: linkedin.com/in/antonejohnson


Verified by MonsterInsights