Pilot Study Completed, New Documents Published

Research and analysis has now been completed on the Phase 1 Pilot Study conducted in partnership with Sustainalytics, a leading independent research firm with extensive experience assessing the environmental, social, and governance (ESG) performance of global companies.

The purpose of the study was to test the latest iteration of our Phase 1 methodology on 12 companies. Based on what we’ve learned from the pilot, we will further revise the methodology and research process before implementing the full Phase 1 ranking for public launch in late 2015.  To that end, we have written the first draft of a pilot report. Over the next several weeks we will seek feedback on the draft from stakeholders and experts. After further revision we plan to publish it later in March.

We are also pleased to announce the publication of several new documents that help to explain the research and consultation process that we undertook to develop the methodology:

Links to those documents and other resources related to the methodology development, plus links to all versions of the methodology, can be found on this page.

New Partnership with Sustainalytics and Other Good News

Thanks to everybody who provided feedback on Version 2 of our Phase 1 methodology. After further revision we will conduct a pilot study this Fall, testing the methodology on up to 10 companies. In accordance with our work plan and timeline, we will then make final adjustments to the methodology and carry out a final round of stakeholder consultation before launching the full Phase 1 ranking of  internet and telecommunications companies in 2015.

Given the importance of rigor and quality control in comparative company research, in July 2014 we launched a partnership with Sustainalytics, a leading independent research firm with extensive experience assessing the environmental, social, and governance (ESG) performance of global companies. As the research partner of Ranking Digital Rights, Sustainalytics will be involved in the design and research for the 2014 pilot study, contributing its significant expertise in the ICT sector, human rights issues, research methodology and rankings design.With experienced staff around the world  and over 20 years of experience, Sustainalytics is a research provider and consultant to some of the largest global investors and financial institutions.  Sustainalytics is also the primary research partner  for the 2014 Access to Medicines Index.

Update (Aug 15): We are also thrilled to announce that we will be partnering with Indaba, developed by Global Integrity, to build the back-end research management system for our Phase 1 pilot this Fall.

Concurrently, with the help of University of Pennsylvania Annenberg School for Communication doctoral student Tim Libert, this summer we kicked off initial research on Phase 2 of the methodology (focusing on devices, networking equipment, and software). After further research and consultation over the next year the Phase 2 methodology will be incorporated into the ranking starting in 2016.

We are also preparing a white paper summarizing the results of our case study research, conducted with our research partners in the second half of 2013 and early 2014, which informed the development of our Phase 1 methodology. Nathalie Marechal, doctoral student at the University of Southern California’s Annenberg School, led the drafting of that white paper which will be published later this year. Marechal also represented RDR at the University of Toronto’s Citizen Lab Summer Institute. We are thankful to the Annenberg COMPASS program for supporting her summer fellowship with RDR.

In addition to the Knight News Challenge grant received in June, RDR’s work receintly received a further vote of confidence from the Ford Foundation with a substantial two-year grant. These foundations join our other funders to propel the project forward through 2015 and beyond.

Finally, we are pleased to announce the addition of a new full-time staff member, Priya Kumar, now in her third week of work as a Program Associate based at the New America Foundation in Washington, DC.  She joins Europe-based research coordinator and human rights specialist, Allon Bar, on what is now a three-person full-time team. At the same time we are sad to say goodbye to Hae-in Lim, our hard-working research assistant for the past year, who is returning to full-time graduate studies this month. Her inputs have been invaluable to the project’s progress thus far.

Progress report: From case studies to a draft methodology

In the five months since I last posted a public update, our research partners in the United States, Brazil, the UK, Germany, Hungary, Russia, Egypt, China, and India have been hard at work conducting case study research on the Phase 1 Draft Criteria. We are all grateful to  human rights specialist and research coordinator Allon Bar, and research coordinator Richard Danbury, for their indispensable work in coordinating and conducting research as well as conducting outreach with civil society and corporate stake-holders. (Click here to learn more about Allon and Richard.)

Based on the results and recommendations made by our case study researchers (whose work will be published later this spring), we are now working to produce a discussion draft of the full methodology that will eventually be applied to evaluate and rank up to 50 major Internet and telecommunications companies across the world.

We plan to publish that discussion draft online for public comment in early March, both on this website as well as on the websites of allied organizations. The draft will also be presented at workshops at conferences such as Rightscon in San Francisco and Cyber-dialogue in Toronto. We will arrange further opportunities to for online and in-person feedback by stakeholders (civil society groups, investors, and companies) on the methodology as well.

In April and May we will revise the methodology based on the feedback we have received. The revised version will be published in May alongside the edited case studies, plus other research materials that will provide context and background for those wanting to understand how the methodology was developed.

Before our methodology is  applied to dozens of the world’s most powerful ICT sector companies in a public ranking, it is important to prepare the field so that those we are seeking to serve (civil society advocates and investors) and influence (companies and policymakers) will be in a position to use and act upon the data that we aim to generate on an annual basis.

To that end, we continue to re-evaluate and revise our work plan and timeline for 2014 and 2015.  In the Summer and Fall of 2014 we are likely to conduct a pilot study applying the methodology we are now developing, with the aim of rolling out the full annual ranking in 2015.  The precise nature and scope of that pilot study will be shaped by the results of the methodology consultation phase that we are about to launch.

None of the above would be possible without the continued support of our funders. We continue to reach out to other potential funders and  partners whose support would be compatible with maintaining the integrity and independence of our work. Suggestions are always welcome!

September progress report

Much has happened since my last blog post of early July, announcing publication of our draft criteria and launch of case study research. Here is an overview:


We are deeply grateful to the MacArthur Foundation for a substantial 2-year investment in our work for 2014 and 2015. While the grant does not cover all project costs, MacArthur’s commitment lays the base to continue our work through the next two years in some way or another. The ultimate scope of the ranking, how much engagement we are able to conduct, and how well it is promoted among other things, depends on the extent to which we can raise other funds. Please also see our lists of 2013 funders and project partners without whose support and commitment we would never have gotten to this point.

Case Study Research and Methodology Development:

In July we launched case studies focused on Internet companies in India, Russia, China, and the U.S. In August and September we began the process of developing and launching case studies focused on telecommunications companies, including: a comparative case study looking at Deutsche Telekom in Germany and its subsidiary T-mobile Hungary; a comparative study of several telcos operating in Brazil, with an added examination of Telefonica based in Spain, whose subsidiaries include Vivo in Brazil; an examination of two Indian telcos Bharti Airtel and BSNL. A U.S. telco case study is also getting off the ground and we are examining whether further case studies are needed on other European telcos as part of this methodology-development stage. The purpose of the case study research is to test out the draft criteria on selected companies in several different types of jurisdictions, in order to gain a better understanding of how the draft criteria play out across a range of companies in a range of jurisdictions. Most importantly we hope to be able to answer to the following questions:

  • What criteria should apply to all companies everywhere regardless of size, maturity, or jurisdiction? (In other words what things do we believe that no company anywhere has an excuse not to be doing – even in hostile legal and political environments?)
  • What criteria can only reasonably be met after company has reached a certain size?
  • What criteria can only be met under certain legal/political/regulatory conditions?
  • How should we focus the methodology – what is most important and urgent to measure companies on and what is better addressed by government-focused advocacy? (The answer to this will be informed heavily by the answers to the first three questions.)
  • What can actually be meaningfully measured and compared and what simply cannot?
  • On what basis should we select the companies we will evaluate in Phase 1?
  • In particular countries and parts of the industry, what impact is such a ranking likely to have on company behavior?

It is too early to report on the final results or conclusions as research is still ongoing, but we can say that most researchers are telling us that in the next iteration, the criteria should be more streamlined and focused in order to be effective. That means we won’t be able to include everything on everybody’s laundry list of what an ideal company ought to be doing on all fronts; we will have to prioritize and focus on what is most urgent, important, and measurable.

We are also learning important things about the relationship between what companies say and commit to publicly, what they actually do in practice, and what their practices mean concretely for users’ free expression and privacy. While there is clearly a relationship between companies’ statements and public commitments and practice, that relationship is different in different contexts. Companies that make similar commitments can still have a wide range of actual practice that plays out with users in a range of different ways. We need to make sure our methodology reflects a nuanced understanding of this reality, and emphasizes the right things.

If all goes as planned we hope to have final drafts of the case study write-ups completed in December. The goal is to publish them in late January or early February, alongside a final draft of our proposed ranking methodology which will be the product of what we learned from our case study research. The methodology will then be subjected to a period of public consultation and stakeholder engagement in early 2014. Funds permitting we hope to then make a final revision and begin applying it in Q2 2014 with an aim to produce our first Phase 1 ranking report in Q4 2014. Please see the project timeline for more details.

Company engagement: 

The case study research includes company interviews. We have encountered a range of reactions from companies: from highly enthusiastic, to curious, to neutral but willing to talk, to negative, to hostile, to indifferent, to radio silence. On the positive end of the spectrum, we have had some truly enlightening and energizing meetings and calls. We have learned a tremendous amount from companies that have been willing to talk with us about the specifics of the draft criteria in the context of their own products, services, and operations. We continue to reach out to other companies with the message that if we can talk to them now during the case study research phase, the final draft methodology – and ultimately the ranking – is more likely to take their concerns and perspectives on board. We have developed an FAQ for companies.

Civil society engagement:

It is vital that our ranking should promote best practices by ICT companies that will better minimize digital threats to human rights defenders and journalists under threat. One of the things we intend to do this Fall is to revise our human rights risks scenarios (which haven’t been revised much since April) through consultation with groups working with human rights defenders and journalists, and to make sure that these scenarios play a strong role in the methodology development process.

Also note that we have been actively reaching out to all the organizations and projects that we know of whose work is remotely related to supporting the emergence of better standards and practices around digital rights, to maximize synergies and minimize duplication.

Investor engagement:

We are exploring whether and how we might work with an organization that provides research data to responsible investors on environmental, social, human rights, and governance factors.

Academic relationships:

Our case study research network includes academics from around the world. Our partnership with the University of Pennsylvania has yielded a course on “Human Rights, Corporate Responsibility and ICT” at Penn Law, two public events, and a workshop. If you are in or near Philadelphia on October 17th please join us for an event called “Scholarship After Snowden. This project is committed to supporting our academic partners – whose research contributions are so vital – in their efforts to beef up teaching and research on issues related to digital rights.


I remain the only full-time staff member of the project, so you can blame me for all of the project’s shortcomings. Click here to see the long list of people without whom the project would be nothing and nowhere. While we still cannot afford to offer a one-year full-time contract to a person with the kind of experience we need, we do plan to hire a 6-month consultant, ideally based in Europe, to support methodology development and stakeholder engagement. The job description is posted on the New America Foundation website.

Case study research now underway!

We have now published a research draft of our Phase 1 Criteria.  It would not have been possible without a wealth of excellent advice and input from a wide group of academics, investors, human rights advocates, technologists, experts in business and human rights, and experts in corporate rankings. This advice was given generously through an advisory mailing list, several expert consultation meetings, and a number of one-on-one meetings. Thanks to everybody who helped us reach this point.

This is just the end of the beginning, however. We are far from ready to actually start scoring or ranking companies. The draft criteria will now be tested on selected companies in selected jurisdictions by researchers over the Summer and Fall. Click here for more details about our case study research.

We welcome public input into this process. Please feel free to post your comments on the Criteria page, or if you do not want them shared publicly please send them to mackinnon AT newamerica DOT net.