Twitter’s misguided barriers to academic research

Last month, Twitter’s chief executive, Jack Dorsey, told the Senate Intelligence Committee he believes “the people use Twitter as they would a public square and they often have the same expectations that they would have of any public space.”
But for all his vows of openness, the platform is making it increasingly difficult and expensive for academics to access tweets to research important questions from how information about diseases like Zika spread to how social movements like Black Lives Matter work to how social media can be used to promote democracy.
Here’s the problem: It’s not possible to use Twitter’s advanced search function to pull all tweets related to a certain subject within a date range. For example, when I tried searching recently for use of the #MeToo hashtag during the first three days of the movement, I got a subset of tweets, but not all of them, and there was no way for me to know how many tweets there had been in aggregate. Astonishingly, when I searched for use of the #MeToo hashtag on October 15, 2017 — the first day of the movement — I got zero results.
And Twitter’s policies have made it increasingly challenging and costly for academics like me to access historical tweets in other ways.
Justin Littman, a software developer at Stanford University Libraries, explained that researchers with advanced software skills used to be able to sign up to write their own software and get the data they needed from Twitter’s Application Programming Interface (API). In July, the company announced that researchers would have to first request access to the API for such projects order to prevent malicious use of the application. The company also limits how data sets obtained this way can be shared. Users who sign up to use the developer platform also have to agree to Twitter’s policies.
It’s great that Twitter is trying to prevent its data from being used for nefarious purposes, such as interfering in elections. But it’s disturbing that the company requires researchers using their API to agree to policies that include restrictions on studying topics like identity politics.
Academics can get access to any past tweets except deleted tweets via third parties — but only if they’re willing to pay. While companies like Crimson Hexagon, a business intelligence firm, provide access to archives of tweets, they target their services primarily to corporations for business intelligence or marketing. Littman said the only product that was focused on providing historical data to academic researchers was Sifter, which stopped operating on September 29.
Stu Shulman, the founder and chief executive of Texifter, the creator and operator of Sifter, said this was because Twitter rejected its application to renew its contract to use Twitter data for academic purposes and said that only use of Twitter data for businesses would be acceptable. Shulman also said Twitter made the pricing prohibitive. A spokesperson for Twitter told me it was the company that decided not to renew the agreement. Littman said that another challenge of getting data from third parties is that Twitter does not allow people using such outside services to download more than 50,000 tweets a day. Therefore, these services usually don’t let academics export tweets in bulk, which makes it much harder for those working with large data sets.
Researchers can also request data sets directly from Twitter — at a price. Shulman said a dozen different researchers told him that when they approached the company directly to request data, they were quoted prices starting at $1,250 — a much larger fee than his company used to typically charge for small projects, which is prohibitive for many academics.
It shouldn’t be up to Twitter or Facebook to decide whether professors are allowed to study subjects like identity politics. They should make access to their data available for free to academic researchers. The social networks should either make all historical posts available on their websites through advanced searches or, if they want companies to have to pay for such data, they should set up a separate way for academics to access past posts for free without having to justify their projects. Academics would never be blocked from accessing what happened in a public space. Twitter’s role is “helping to not only serve that public conversation so that everyone can benefit,” Dorsey said, “but also to increase the health of that conversation as well, and in order to do that, we need to be able to measure it.”
Twitter needs to stop thwarting the very people who can help it do that — and provide solutions to many other social problems at the same time.

—Bloomberg

Kara Alaimo is an assistant professor of public relations at Hofstra University and author of “Pitch, Tweet, or Engage on the Street: How to Practice Global Public
Relations and Strategic Communication” She previously served in the Obama administration