Two Ways of Looking at a Blue Bird: What’s Really Going on at Twitter?

“Twitter? Twitter? Twitter? Twitter?” “Um, Twitter’s sick. My best friend’s sister’s boyfriend’s brother’s girlfriend heard from this guy who knows this kid who’s going with the girl who saw Twitter pass out at 31 Flavors last night. I guess it’s pretty serious.” That’s basically where we are with the coverage. Meanwhile: “‘Twitter Is Dead,’ 300 Million People Post On Twitter.”

I like to think of myself as technically literate, moreso back in the day than I am now. Thanks to my Web 1.0 heritage, I can set up and run a website, including the backend. However, the systems that Twitter engineers are (or were) maintaining at scale are unimaginably complex to me. These systems — their design and implementation, who runs them, what they deliver to users — are at the heart of the Twitter story. Unfortunately, the first two aspects of Twitter are opaque just now, so finding out what’s really going on is very difficult (and on this story, Twitter the platform isn’t helping very much). So what’s really going on?

The most basic question: Is Twitter down? The answer as of this writing (12:03PM EDT, November 21, 2022) is no. Twitter is not down. Twitter’s APi page says all systems are green:

Will Twitter go down? Down for Everyone or Just Me? says it has, so doubtless it will:

Interestingly, the last five outages are all since Musk’s ascension to CEO, but that’s as far back as the data I can find goes (and Twitter’s record of being up has been no means perfect. I can say, however, that as an extremely online person who practically lives on Twitter, that I didn’t feel a thing).

In this post, I will look at two aspects of Twitter. First, I’ll look at Twitter’s software, and show why the World Cup will stress it. Then, I’ll look at the players in the ongoing Twitter drama, and the role of each in Twitter’s current drama. I will also add an Appendix on Mastodon, Twitter’s open source competitor.

Twitter Software

The Twitter software handles a lot of data. From the Twitter Engineering Blog, “Next generation data insights using natural language queries“:

At Twitter, we process approximately 400 billion events in real time and generate petabyte (PB)-scale data every day.

That’s a lot. In the face of this data flow, the essential technical challenge to keeping Twitter up — the baseline requirement for Twitter creating value — is spikes. From the same blog, “When seconds really do matter“:

On Twitter, the real world happens in sub-minutely intervals and activity on the platform follows. Whether it’s a retirement announcement from a member of a boy band, a government coup, a world wide soccer tournament goal, or simply the start of a new year, traffic patterns change quickly and unpredictably as these events unfold. These are the moments when the minutely model becomes insufficient. Let’s imagine that a celebrity couple announces the birth of a child at two seconds after the nearest minute — for this example, let’s say 2:00:02. With this announcement, a spike of traffic causes undue pressure on one of the many micro services[1] that powers Twitter causing service errors to some users.

The classical example of this sort of spike is the load on sewage systems when a million toilets simultaneously flush during a commercial break for the Superbowl. For Twitter, the 28 days of the World Cup pose an equivalent challenge. From the New York Times:

[The World Cup] is expected to bring a deluge of traffic to Twitter, which is the world’s fourth most visited website, according to Similarweb, a digital intelligence platform that tracks web traffic. Twitter gets 6.9 billion visits each month, slightly more than Instagram’s 6.4 billion, though far fewer than Google, YouTube or Facebook, according to Similarweb estimates.

In the past, Twitter has a “Command Center” to handle the spikes, but many have left. From The Verge:

Several members of Twitter’s “Command Center” team, a group of engineers that is on call 24/7 and acts as the clearing house for problems internally, also tweeted about their departures. “If they go down, there is no one to call when shit breaks,” said a person familiar with how the team operates.

“A person.” Most of the coverage on internal Twitter issues is single-sourced. Another single source from the Guardian:

Twitter stands a 50% chance of a major outage that could take the site offline during the World Cup, according to a recently departed employee with knowledge of how the company responds to large-scale events.

The former employee, who was granted anonymity because of the sensitivity of what was discussed, has knowledge of the workings of Twitter Command Centre, the platform’s team of troubleshooters who monitor the site for issues such as traffic spikes and data centre outages.

Between the lack of preparations and the lack of staffing, I think it’s going to be a rough World Cup for Twitter,” said the former employee.

He suggested that an incident of some kind – such as a service responding slowly or incorrectly – is almost a certainty during the 29-day [sic] competition in Qatar, estimating a 90% possibility of something going wrong that users would see.

Well, we’ll just have to wait and see. 28 days will be quite a stress test.

From handling spikes, we turn to Twitter’s basic data structure, the graph (or “social graph”). Here’s a very simple diagram of a very small graph:

More technically, once again from Twitter’s blog, “Graph ML at Twitter.” The circles in the image above are “nodes.” The lines between them are “edges”[2]:

Graphs are mathematical abstractions of complex systems of relations and interactions. A graph represents a network of objects (called nodes or vertices) with pairwise connections (edges). Graphs are ubiquitously used in fields as diverse as biology, quantum chemistry, and high-energy physics’. . Public conversations happening on our platform typically generate hundreds of millions of Tweets and Retweets every day. This makes Twitter perhaps one of the largest producers of graph-structured data in the world, second perhaps only to the Large Hadron Collider.

“Second perhaps only to the Large Hadron Collider.” Oof! Since Twitter as a business exists to monetize relationships between users as expressed through replies, Retweets, and Likes, the representational power of the Twitter graph is of the first importance. No doubt that’s why Musk may improve it. From Harvard Business School’s Andy Wu:

Doing that kind of switch is very, very dangerous. So we’re going to have to wait and see what’s going to happen here. My sense is that they are prioritizing the internal core of the technology of Twitter, the social graph.

So, what’s really going on with Twitter? In the short term, an enormous stress test of the software: The World Cup. In the long term, improving Twitter’s social graph.

Twitter Players

But what’s really going on? Let’s look at the players, and the struggles between them. The players fall into to buckets: The oligarchs, and the Professional Managerial Class (PMC). Let’s look at each in turn.

First, the oligarchs. In this section, the set of all oligarchs has only one member: Musk. Software luminary Grady Booch writes in the unrolled Twitter thread: “Twitter 2.0: a modern story about systems engineering“:

Take a legacy organization that has over the years built a not perfect but reasonably solid web-centric system at elastic global scale.

Let that engineering team operate under the demonstrably, poor leadership of @jack and the distinct lack of adult supervision by Twitter’s board for too many years.

Bring in a reluctant CEO with the charisma of Mussolini, the bluster of Trump, and the sensitive nature of a bull in a china shop.

Have that CEO finance the deal with a debt that would stun a team of ox, a debt that has to be serviced on top of that organization’s already negative cash flow.

On day one and for many more days that follow, have that CEO alienate his newly-acquired users, scare away his few advertisers, and decimate the development team.

Have that CEO demand strict allegiance to his capricious edicts, institute unsustainable engineering processes, and tear apart the existing infrastructure by ripping out parts he barely understands.

So here’s the thing:

Making a Twitter 2.0 is only partly a software engineering problem.

It’s a systems engineering problem that requires a firm and steady hand on all the other moving parts such as the business model, understanding the needs of its customers and other stakeholders, defining a compelling vision for the future.

However…

Elon is approaching Twitter 2.0 as if it’s just a “hardcore software engineering” problem.

That’s where he’s gone off the rails.

Twitter 2.0 is literally not rocket science, but Elon is treating it like it is. His focus on the hardcore side of it is understandable yet misguided, and represents a dangerous distraction and opportunity cost.

I have no idea how this will evolve in the coming weeks.

But I do know that it will be uncomfortable to watch, as the lives of Twitter’s staff are upended and as this important global resource is trashed.

This is as good a description of Musk’s situation as I have seen, and from a Master of the Great Runes, too. (In Musk’s defense, he’s clearly thinking about the business model, too; introducing payments to creators — Twitter has many, many artists, writers, performers, etc. — would be a major change to the business model, and a welcome one, too.)

Now, the PMC. For those who came in late, the definition from Barbara and John Ehrenreich (1977):

Salaried mental workers who do not own the means of production and whose major function in the social division of labor may be described broadly as the reproduction of capitalist culture and capitalist class relations.

Certainly programmers; certainly managers. First, the managers. Matt Stoller writes:

While I am no fan of Musk, I know that Twitter was horribly managed before his takeover.

Grady Booch (above) noted the “demonstrably, poor leadership of @jack and the distinct lack of adult supervision by Twitter’s board.” And from HBS’s Andy Wu:

Twitter was actually in very bad shape and didn’t quite have a future anyway. In terms of really tough problems, [Musk] is the kind of CEO you probably need to try out. Twitter is actually a very, very difficult business challenge that nobody else has been able to solve. So at this point, we might need to, like, swing the car around and see what happens.

This is a company that couldn’t deliver an Edit button to users for the entire period of its existence, even though users were clamoring for it. (Mastodon, Twitter’s open source competitor, has an edit button.)

Now the professionals. These can be divided into staff (inside Twitter) and the press (outside, reporting on it). As far as the staff goes, others echo Gooch’s grudging praise (“not perfect but reasonably solid). Andy Wu once more:

The people who built the Twitter product definitely built a pretty resilient infrastructure to survive through all of this. The technical part can operate without a lot of the people.

An example of such resilience: graceful degradation (like when a web page loads a barely formatted version of itself, but still gives you something to read:

The staff did some good engineering, there. (Other examples of the sort of unsung enginering it takes to keep a very large system up and running here, here, and here.) Of course, from the oligarch’s perspective, none of this translates directly into headcount[3],[4]. Musk may figure he can hire back what he needs:

Here, for example, is a very happy camper who stayed:

Let’s just hope the Peter Principle doesn’t apply here.

And now the second set of professionals, the press. The key point here is that the press have done what old-school journalists should never do: They’ve become part of the story by doing very little reporting, in “print” or on Twitter, instead engaging in an enormous dogpile of Elon Musk. (I find the press’s pearl-clutching over a set of software professionals moving on to green fields and pastures new, and their silent indifference to, say, railroad worker or nurses, really galling. I also find the implication that Musk, and I suppose Trump, are the only billionaires worthy of censure galling as well.) For now, I will simply note the economic incentives. From the very last sentence of this Times story:

Mr. Musk has increasingly downplayed the role of traditional media over the past few months, citing Twitter as one of the best platforms for the rise in “citizen journalism,” as he put it.

Oh. Can’t have that. Arthur wouldn’t like it.

Conclusion

As usual, Atrios is sensible. “The Bird App Will Be Gone And We Will All Be Free“:

Twitter had two main contributions to the world (well, the US, anyway): It killed the influence of Drudge….

Quite right. I remember when Drudge drove the news cycle. Having Twitter drive the news cycle, for all its many faults, is an enormous improvement.

… and suddenly a bunch of immensely self-important people were confronted with hundreds of people, often deservedly, calling them shitheads every day.

By calling them shitheads, I mean often calling them shitheads backed up with good arguments. And a lot of those arguments were from people of color and other “riff raff” such as people who went to lesser schools than Harvard.

Trivially, Twitter allows me to call Rochelle Walensky a eugenicist to her face (or her intern’s face), and have others see it. To me, that’s what’s really happening at Twitter. I’m very dubious that there’s another platform that enables that. Non-trivially, Twitter allowed aerosol scientists — many of whom are from [shudder] state schools — to call out the droplet dogmatists successfully (“backed up with good arguments”). That’s a great achievement in human terms, in medical terms, in scientific terms, in every way. We will see whether that’s the Twitter Musk preserves (as I know the censorship-mad PMC will not). We will also see how Twitter holds up under the World Cup stress test. Interesting times!

NOTES

[1] These are the microservices Musk calls “bloatware.” Individually, perhaps. In the aggregate, I’m not so sure. For example, if Twitter’s copyright strike system was a microservice, there’s an issue:

Social media users on Twitter can now upload entire movies in threads as the company’s automated copyright enforcement/ takedown system no longer seems to be functional, Forbes reported on Sunday.

A user uploaded the 2006 movie, The Fast and the Furious: Tokyo Drift in two-minute video clippings spread out in a thread of 49 tweets, which when viral on the platform. It wasn’t a one-off event that somehow just got through. Another user uploaded the movie Hackers in a similarly long thread on Saturday, which even got 14,000 likes, Business Insider reported.

Hollywood won’t like that. Nor will FIFA. But I have a suggestion:

Twitter just algorithmically pushed me to a pirated full movie thread when I swiped down, and that is extremely funny

— Faine Greenwood (@faineg) November 20, 2022

I bet the pirated movies wouldn’t go viral if the algo didn’t push them. Musk should turn off the algo! Give people what they want: A timeline of tweets from accounts that they follow, in reverse chronological order. This would remove an enormous amount of bloat.

[2] What I call a “yarn diagram” in conspiracy theory is a graph. The nodes are conspirators; the yarn is relationships. When I say that the yarn in a yarn diagram is “too tight,” I mean that the creator of the graph hasn’t been nuanced enough about relationships (for example, assuming orders are given when shared interest is enough).

[3] Or maybe it does. From former staffer Dan Luu:

Another reason to have in-house expertise in various areas is that they easily pay for themselves, which is a special case of the generic argument that large companies should be larger than most people expect because tiny percentage gains are worth a large amount in absolute dollars. If, in the lifetime of the specialist team like the kernel team, a single person found something that persistently reduced TCO [Total Cost of Ownership] by 0.5%, that would pay for the team in perpetuity, and Twitter’s kernel team has found many such changes. In addition to kernel patches that sometimes have that kind of impact, people will also find configuration issues, etc., that have that kind of impact.

[4] If Musk is right about headcount, and you play the ponies:

APPENDIX Mastodon

More than a few Twitter users have ventured out into Mastodon, Twitter’s open-source competitor. Naturally, they reported their findings on Twitter.

Mastodon is large by its own standards, but not by Twitter’s:

There is one Twitter; there are many Mastodon servers (hence jokes about garages). That means onboarding is hard. You have to pick a server, then apply for an account, then wait. Some servers have shut down applications because they can’t handle the load;

The Mastodon servers are collectively known as the “fediverse” (federated universe):

The fediverse is not simple:

Sometimes members of the fediverse fight:

And then there are the admins:

I’ve set up an account on Mastodon. Whether I move my tiny presence there is an open question…