Hacking. Disinformation. Surveillance. CYBER is Motherboard’s podcast about the dark underbelly and internet.
At the end of the day on a Friday in October 2014, just a few months after Amazon paid nearly $1 billion for the video game streaming website Twitch, an engineer stumbled upon what at the time was the platform’s first ever hack while doing tech support for a colleague who worked remotely.
“Oh fuck,”The former Twitch engineer recalls saying. “But I remember thinking that there was so much ‘I told you so’ here.”
The engineer, who requested anonymity to discuss confidential details about the incident, stated that he had found logs that indicated that hackers had gained access the account of his colleague. He stated that the hackers had left no trace evidence of their intrusion.
“This attack definitely had the characteristic of a minimally skilled adversary,”He told Motherboard.
The full story of Twitch’s 2014 hack has never been reported. This report is based upon interviews with seven former Twitch employees. Motherboard granted anonymity for sources because they are subject to confidentiality agreements that prohibit them from disclosing details of their work at the moment.
An intense investigation began after the suspicious logs were discovered. Nearly all Twitch employees were involved in the investigation. One former employee claimed that they worked 20 hours per day for two months. Another said that he worked a similar amount. “three weeks straight.”Other employees claimed they worked long hours and stayed at the hotel for several weeks. Five former Twitch employees were present at the company at that time. They said that Twitch did not have any dedicated cybersecurity engineers at the time. So developers and engineers from other companies were brought in to help.
Twitch had to rebuild a lot of its code infrastructure after the hack. Twitch eventually assumed that most of its servers had been compromised and decided to label them. They decided to label them instead. “dirty,”Three former employees who have worked with these servers said they would slowly migrate them to new servers.
“The hackers had such wide access before they were detected, we basically had to rebuild everything from scratch”
You can still find remnants of that hack today in Twitch code stolen and uploaded to the internet by hackers last Wednesday in another major data breach. This breach exposed the revenues of streamers as well as the source code of internal Twitch servers. Although Twitch has seen significant changes since 2014, former employees claim that the hack had knock-on consequences that can still be seen today.
Twitch did no other than to disclose the details of the breach or its extent.
Twitch’s users would not find out about the breach until six months later. The breach was first discovered by the company on March 23, 2015. On that date, the company published a blog post explaining the situation. “there may have been unauthorized access to some Twitch user account information,”However, he did not reveal how harmful the hack was for Twitch internally.
The 2014 incident would be codenamed inside the company. “Urgent Pizza,”This became a joke. Twitch leadership eventually printed tee-shirts with the name. According to one source and code leakage in the incident 2021, the company also decided that all future incidents should have food-themed codesnames.
“The event was called ‘urgent pizza’ because management had everyone do ridiculous amounts of overtime and ordered pizzas as incentive lol,”Motherboard was told by a former Twitch employee. “People who participated got t-shirts and ‘joke’ about having PTSD from the long hours and lack of understanding of the scope of the hack which necessitated the company-wide rebuild.”
Some of the data leaked last week contained trace of the 2014 hack as well as the company’s response. These were artifacts from past centuries etched in source codes. Some files that were leaked last week include strings such as “remove pizza script,” “a pizza thing,” “indicate that the server is ‘urgent-pizza clean,’” “move pizza to securelogin,” “dirty_status = True.”
According to a former employee, the hack surprised even though the company hadn’t invested in its security.
“Security efforts kept getting cancelled or deprioritized with the argument that ‘everyone loves Twitch; no one wants to hack us,'”They added.
Are you a Twitch employee? Do you have any information on the recent leak and breach? We’d love to hear from you. Using a non-work phone or computer, you can contact Lorenzo Franceschi-Bicchierai securely on Signal at +1 917 257 1382, OTR chat at [email protected], or email [email protected]
Twitch disclosed the hack in March 2015 to security engineers at Twitch (and Amazon) who had been there to help with the incident response. However they did not know if the hackers had actually broken into the system earlier than that.
“That was long enough for them to learn entirely how our whole system worked and the attacks they launched demonstrated that knowledge,”The former employee said.
“The hackers had such wide access before they were detected, we basically had to rebuild everything from scratch,”The former employee added. “Some of the response involved rewriting the login process because the hackers had figured out how to send a copy of every single user’s password to their servers. They also gained access to all of our source code and all of our databases.”
Two other former employees confirmed the severity of the data breach.
However, many Twitch employees did not fully understand how deep the hackers had penetrated the company’s network. It is unclear to them if the company ever fully comprehended the extent of the attack.
Several servers and services were internally classified as “Insider” for several months after the public announcement and discovery. “dirty,”As a way to remind developers and engineers to be cautious and ensure that they get cleaned up. Three former employees said that although they were still in use and available for use, engineers had placed restrictions on their use in the event they were compromised.
“The plan apparently was just to rebuild the entire infra[structure] from known-good code and deprecate the old ‘dirty’ environment. We still, years later, had a split between ‘dirty’ services (servers or other things that were running when the hack took place) and ‘clean’ services, which were fired up after,”One of the former employees stated that. “We celebrated office-wide the day we took down the last dirty service!”
“Twitch wasn’t aware of what [the hackers] had access to, or how long they had access,”Motherboard heard from one of the former employees who joined a few years after hacking. “Which is how we ended up with the dirty hosts: they were hosts that were probably fine, but they couldn’t definitively say one way or the other.”
“You’d think that would be a teachable moment but it was either something that newer (non-security) people largely didn’t know about, and older employees kind of giggled about,”He said.
Motherboard was able to get details from Twitch about the 2014 hack. They also shared information about how security has evolved over time. The hack that Twitch suffered years ago could be compared to the damage it sustained in the latest incident. This could indicate whether the company has learned its lesson from 2014 and has put better protections in place.
“This  incident was a very worst-case-scenario attack that was entirely preventable. But leadership’s desire to avoid considering the security problem was the root cause,”One of the former employees stated that. “And this more recent incident demonstrates that they didn’t learn anything from the incident in 2014.”
Twitch didn’t respond to questions regarding the 2014 hack and the more recent incident.
An anonymous poster on the online forum 4Chan published a post last week that they called “The Last Week in 4Chan”. “part one”A series of Twitch leaks.
“Their community is also a disgusting toxic cesspool, so to foster more disruption and competition in the online video streaming space, we have completely pwned them, and in part one, are releasing the source code from almost 6,000 internal Git repositories,”The post was written anonymously by a 4Chan user and has since been removed.
The leak contained hundreds of gigabytes worth of internal Twitch source codes and financial data from streamers. This leak is both embarrassing for Twitch as well as extremely damaging to streamers.
“It’s definitely troubling, to say the least,”Motherboard was informed by Amouranth, a well-known streamer. “As a streamer, Twitch is actively serving as your employer, and you want to believe that you can trust them to have security measures in place to prevent this kind of thing from happening.
Twitch has yet to disclose details of the data breach, as it’s still investigating. In a blog post, the company said that “An error in the Twitch server configuration change caused some data to be exposed to the internet. This data was then accessed by malicious third parties.”
“In the earlier days of Twitch the security team had a presence, but they seemed to be stretched to the limit”
When the former Twitch employees who spoke to Motherboard found out about the leak, some were surprised, and others weren’t.
“I was shocked to hear this. Amazon is a parent!” one of them said.
Other former employees, however, said that the damage of this new data breach appears to be less severe than the 2014 hack. And that it’s likely thanks to Twitch taking security more seriously since then.
“Twitch’s security team was present in the early days, but they seemed stretched beyond their limits. Twitch had a bad reputation for adding tons more employees to build new products and not growing the teams around them.” one source said. “They knew they had to take it seriously under the watchful eye of Amazon. It certainly improved over time. It has tightened access controls and increased security scrutiny of the internal tools, among other things.
Another voiced the same sentiment.
“Things were much worse by the time that I left, but I think even though this hack is much larger in terms of impact on technical side of business, the security posture is mature enough to make remediation a more gentle process.” they said. “It’s still pretty embarrassing, but the theories I’ve seen so far on what the means of access was make the blast radius a lot smaller. Because despite it meaning some serious audits of the security of pretty much all of Twitch’s code, it shouldn’t need to be a company-wide-overtime-for-months kind of situation.”
But in the end, Twitch is now investigating another data breach, six years after the worst hack of its history.
“The security team really did everything they could,” a former employee said. “So it’s frustrating to see it come to this.”
Matthew Gault contributed reporting.
Subscribe to our new cybersecurity podcast, CYBER. Subscribe to our new Twitch channel.