Rutgers Faculty Timeline in the Fight against Academic Analytics
What is Academic Analytics? Academic Analytics is a subscription-based software package that makes the claim that it can assist faculty and universities by displaying objective data in ways that enhance the communication of quality research at higher education institutions. Rutgers has bought into this package, but Rutgers faculty are asking key questions about the assumptions and methodologies that are behind this product.
Dr. John Holmwood, "Death by Metrics," April 2015
Rutgers University faculty began our education in what Academic Analytics is doing by inviting a guest speaker on April 28, 2015, Dr. John Holmwood. In the United Kingdom, universities have been using this package for longer than in the United States. Professor John Holmwood spoke to Rutgers faculty on April 28, 2015. Dr. Holmwood is the President of the British Sociological Association and teaches at the University of Nottingham, UK. Read an article by Dr. John Holmwood published on the website of Global Dialogue: Newsletter for the International Sociological Association, October 30, 2013: Death by Metrics. Also read the Leiden Manifesto, an article written by Diana Hicks, Paul Houks, and others in Nature, published on April 23, 2015, in which they explain 10 principles to guide research evaluations that are much better than Academic Analytics. View the April 28, 2015, talk by John Holmwood on YouTube:
Faculty Graduate School Faculty Vote Against the Use of AA, May 10, 2016
On 10 May 2016, faculty of the Graduate School, New Brunswick voted 114 to 2 against the use of Academic Analytics. The resolution and press coverage are here: https://www.insidehighered.com/news/.... In response, NB Chancellor Richard Edwards promised to share the database with faculty. He failed to share the AA data with faculty.
October 2016 Rutgers Faculty Mobilization Intensives against Academic Analytics
October 3, 2016
The Union has been working to make your Academic Analytics (AA) ratings available to you but the Administration has still not opened the files - files to which the University has access. To assist the many of you who have asked to see your ratings of "scholarly productivity," along with the data used by AA to calculate your rating, the Union will file a Common Law/ Open Public Records Act (OPRA) information request.
If you wish to inspect your AA records, please follow this link to the request form. In the event that the Administration refuses this request, the Union intends to pursue the matter in court.
The embedded link above, goes to this direct link to the petition on iPetitions.com: http://www.ipetitions.com/petition/academic-analytics-information-request
To read the full text of the petition, "Academic Analytics Information Request and Permission to Sue," on the Rutgers AAUP-AFT webpage, visit this page: http://www.rutgersaaup.org/node/893
The Union pursues legal action against the Administration only rarely and with much forethought and consideration. In this case, we believe that there is a lot at stake. As is well known, AA data are inaccurate and flawed, and the case of the Art History department, discussed below, demonstrates this fact at Rutgers. Over the past months, the AAUP-AFT gained additional information about the disturbing ways in which the Administration is using secret data from Academic Analytics. In 2015, the Graduate School (New Brunswick) produced a chart ranking 72 PhD programs. The internal ranking followed AA’s ranking of those programs within their disciplines on the basis of the “scholarly productivity” of relevant faculty.
Over the summer, the Union made an OPRA request for the chart reflecting program rankings. Old Queens denied our request on the grounds that the document belonged to Academic Analytics and that the document was “advisory, consultative, and deliberative,” as well as “part of the decision-making process.” So we know that the Administration is using this chart for important, entirely opaque activities.
It is the position of the AAUP-AFT that faculty have a right to information about their own scholarly productivity ranking and the data used by Academic Analytics to compute that ranking. What happened with the Art History Department underscores the importance of transparency with respect to our rankings.
In 2014, administrators informed New Brunswick’s Department of Art History of its low ranking (#57 as it eventually appeared on the chart). Faculty questioned the validity of the ranking and obtained their individual “scholarly productivity” scores. (They did not have direct access to AA’s confidential methodology, also recently obtained by the Union). By examining their own records these art historians deduced the gaps in AA’s journal data base. Then they compiled a list of important Art History journals overlooked by AA. (They did not delve into the much more complicated process of filling in the vast gaps among AA’s measures themselves, for instance, AA’s failure to count the curatorships art historians frequently do.) Finally, they underwent an external review, whose findings went to the Committee on Academic Planning and Review (CAPR). That body viewed all the evidence and concluded that AA’s ranking of the Art History department was flat-out wrong.
Art History struggled for and achieved a good outcome because the faculty were able to pry loose secret information about their own “productivity” and reveal the serious flaws in the process used unfairly to rank their department. The University’s reliance on potentially faulty data with regard to any individual’s academic work is disturbing. The Administration’s reliance on the same flawed data to make decisions with regard to the distribution of resources throughout the University is an even graver threat.
Can the rest of us do what our Art History colleagues did? Yes, we can begin to restore the multiplex model of scholarship that preceded corporate blueprints, such as, Academic Analytics. But we need access to our AA files. As you recall, the faculty of the School of Arts and Sciences (NB) as well as the Graduate School (NB) passed resolutions, with overwhelming support, demanding that Old Queens release these data to interested faculty. The last resolution stipulated a deadline of September 1, 2016. Old Queens has released nothing, meanwhile denying more than 70 requests for the files under the OPRA.
On the advice the Union’s attorneys, we will pursue a more forceful legal avenue: a Common Law Information Request – which encompasses a broader array of public records than OPRA – along with an OPRA request. If necessary, the AAUP-AFT is prepared to file a lawsuit to obtain AA information to which faculty are legally entitled. I have attached that request. Please read it carefully. If you wish to sign on – which I hope many of you will – fill out the form. At this stage, I would suggest that only those of you with tenure join officially in the request.
The Union pursues legal action against the Administration only rarely and with much forethought and consideration. Your signing on as an individual will assist us in the goal of having your own AA data made accessible to you. If we are successful, AA files will eventually be provided to all faculty.
It is a long road to restore and preserve the University as a place of knowledge, innovation, learning, and public criticism in the broadest sense. I welcome your ideas and, above all, your involvement as the Union stands with you in support of transparency and your right to access information about your AA ranking.
David M. Hughes
President, Rutgers AAUP-AFT
Professor, Anthropology (SAS-New Brunswick)
The union is not simply concerned to correct flawed Academic Analytics data but to question the flawed assumptions underlying this company's approach to "measuring" faculty productivity. We are not intellectual "widgits"!
October 17, 2016
As you my know from my message two weeks ago (October 3 message above), the Union is now trying aggressively to pry your scores out of our Administration and out of the Academic Analytics corporation. Many of you have already signed the petition for access to your profiles. If you have not yet done so and wish to join in this demand, you may sign the petition.
Unfortunately, we cannot guarantee that the Administration will respond positively. For three years now, they kept these data on “scholarly productivity” entirely secret from the productive scholars, namely, from ourselves. For ten months, the Chancellor’s office has refused to negotiate with the Union, has denied numerous OPRA requests, and has dismissed two faculty votes against Academic Analytics. Such stonewalling is not necessary. The Administration of Georgetown University responded to faculty criticism of Academic Analytics by conducting a thorough review. Then, less than two weeks ago, Georgetown announced it would cancel its subscription to the database. Why did they do so? When the provost analyzed Academic Analytics he found rates of undercounting that ranged from 35% to 52% for articles and published conference proceedings. Perhaps Old Queens wishes to cover up exactly such embarrassing errors in its data and, therefore, in its decision-making.
Please sign the petition so that we can see what the university is hiding. If all else fails, the Union intends to pursue this matter in court.
David M. Hughes
President, Rutgers AAUP-AFT
Professor, Anthropology (SAS--New Brunswick)
Rutgers Administration Has Begun to Concede on Academic Analytics
October 26, 2016
By the beginning of last week, nearly two hundred faculty members had signed the “Common Law Information Request and Permission to Sue.” [link to petition]. They are demanding to see the secret personnel records compiled by Academic Analytics (AA) and used by our Chancellor’s Office. At the end of last week, Chancellor Edwards announced his concession to that demand. (His message is appended below.) This step represents a victory - after more than a year and a half of persistent activism by the Union, by the New Brunswick Faculty Council, by the NB graduate faculty, and by the SAS faculty. But we still have work to do to address the use of metrics in the University.
As the Chancellor notes in his message, we will now be able to gain access to our AA profiles through chairs or other specially-trained individuals. This process seems unnecessarily burdensome to chairs and cumbersome to everyone, but it may work to provide access previously denied. We are therefore going to suspend filing the Common Law Information Request. Should those records not appear shortly, we will, indeed, deliver the document and its nearly 200 signatures to Court. Please do let us know if and when you obtain your AA profiles. And we urge that you NOT attempt to correct errors in the AA data. That company does not pay us to improve its product — one that is deeply flawed and inappropriate for academia.
In another apparent concession to faculty pressure, Chancellor Edwards has agreed to reconsider the use of Academic Analytics altogether. At least, the new Chancellor's Advisory Committee on Metrics and Analytics may choose to recommend that Rutgers follow the path laid by Georgetown University – and cancel its subscription to AA. The Committee will also address the much wider issue of metrics, ranking, and their abuse in Academia. The Union will certainly convey expertise and information to the members of this committee, whose names are below. I would invite all of you to do the same.
Regrettably Chancellor Edwards’s named no faculty from arts or literature departments to his committee. As AA mis-measures those fields in ways that are particularly egregious, we suggest to the Chancellor that he rectify the oversight.
Finally, we will be recommending to the Administration that it suspend its use of Academic Analytics while this important committee deliberates. Surely, the concerns and outrage regarding this database – now acknowledged by Old Queens – warrant a pause and a step back.
David M. Hughes
Professor of Anthropology (SAS-New Brunswick)
President, Rutgers AAUP-AFT faculty union
FROM CHANCELLOR RICHARD EDWARDS, OCT. 20, 2016
I am writing to let you know that I have established a Chancellor’s Advisory Committee on Metrics and Analytics. As I am sure you know, virtually all higher education institutions these days are using various metrics to measure a whole range of types of performance, from student recruitment to financial performance to faculty productivity, and virtually everything in between that can be measured. Such metrics may include a range of “dashboard” indicators, as well as other ways of categorizing information. One of several types of measures is the type of information collated by companies like Academic Analytics, but there are others. Of course, the whole idea of measurement of performance is not new to higher education—we have been giving grades to our students for a long time. Given the growth of use of measures and metrics, I want the new committee members to become well-versed on what we are doing here at Rutgers University–New Brunswick and to advise me on how best to use these measures.
Given the increasing attention to metrics and analytics, I have asked a group of faculty and a few administrators to serve on the new Advisory Committee. The membership consists of faculty who represent a broad range of disciplines and schools. It will be chaired by Professor Mark Miller from the School of Environmental and Biological Sciences. Below is the list of members.
Of course, I am aware that some faculty members are concerned whether their publications and funding profiles as compiled by Academic Analytics are accurate. Therefore, as I told the New Brunswick Faculty Council I would do, I am establishing a process whereby any individual faculty member is welcome to check her or his file at Academic Analytics. If the faculty member then believes there are discrepancies, Academic Analytics has identified a member of its staff who can be contacted by the faculty member to consult about the discrepancies and make corrections if necessary. To review her or his Academic Analytics file, the faculty member will go through her or his department chair, who will be able to access the Academic Analytics files (in the case of non-departmentalized schools the dean will identify the individual to perform this function). To obtain access, the department chairs or other authorized individuals must complete a brief orientation on Academic Analytics. Department chairs should contact Jerry Kukor, Dean of the Graduate School, for information about obtaining access from Academic Analytics.
I expect the first meeting of the new Chancellor’s Advisory Committee on Metrics and Analytics to occur within the next couple of weeks. I thank all of those who have agreed to serve.
Members, Chancellor's Advisory Committee on Metrics and Analytics
(1) Chair: Mark Miller, Department of Environmental Sciences and Institute of Earth, Ocean, and Atmospheric Sciences, School of Environmental and Biological Sciences
(2) Doyle White, Department of Mechanical and Aerospace Engineering, School of Engineering
(3) Sanjib Bhuyan, Department of Agricultural, Food, and Resource Economics, School of Environmental and Biological Sciences
(4) Florence Hamrick, Graduate School of Education
(5) Richard Padgett, Department of Biology, School of Arts and Sciences
(6) Jerry Kukor, Dean, Graduate School–New Brunswick
(7) Cathy Potter, Dean, School of Social Work
(8) Robert Heffernan, Office of Institutional Research and Strategic Planning
(9) Krisellen Maloney, Rutgers University Libraries
(10) Mia Bay, Department of History, School of Arts and Sciences
(11) Simon Thomas, Department of Mathematics, School of Arts and Sciences
(12) Eugene White, Department of Economics, School of Arts and Sciences
(13) Staffed by: Paul Hammond, Assistant Vice Chancellor, Office of the New Brunswick Chancellor
Richard L. Edwards, Ph.D.
Chancellor, Rutgers—New Brunswick