Your cart is empty.

Scholarly Podcasting Open Peer Review

Introduction

By Siobhan McMenemy, Senior Editor

The peer review of the first season of the Secret Feminist Agenda podcast, created by Hannah McGregor, was completed in May 2018. The reviewers were asked to respond to a set of questions designed to engage with issues related to the various aims of our research into the ways and means of producing, editing, reviewing, and broadcasting scholarly podcasts under a university press imprint. Our questions and the reviewers’ comments have served two purposes: To provide Hannah McGregor with suggestions for improving her podcast in form and content from two scholars who are themselves engaged in scholarly activities relevant to our work on the genre of the scholarly podcast; and to help us to hone a set of peer review questions for the review of subsequent seasons of Hannah’s podcast and other, future podcasts. We encouraged the reviewers to reflect on whether they considered the peer review questions themselves to be relevant and helpful to their task as peer reviewers and to our responsibilities as editors, producers, and publishers of the podcast. 

The first set of reviewers’ reports raised interesting questions about the appropriateness and practicalities of peer reviewing podcasts, the philosophical matter of what constitutes scholarship, and the evaluative practices of the tenure and promotion review process and where born-digital scholarship fits into these performance assessments. A number of questions in this first round of review were meant to encourage further consideration of the relationship between conventional forms of scholarly publishing (e.g. journal articles, book chapters, monographs, edited collections) and the publication of a scholarly podcast series. We appreciated the reasons for the reviewers’ frustration with our repeated insistence on creating points of comparison between print and digital forms where the comparisons were awkward at best. Part of the aim of such comparisons was to highlight the commitment of university presses to provide authors—and podcasters—with valuable community engagement with their work with an eye to strengthening the work for publication, regardless of genre or format. Likewise, we know that the rigour of peer review and the value of the university press imprint contribute to institutional support for a scholar’s work. It is our hope that this peer review exchange, and those that will follow the completion of the second and third seasons of the podcast, will allow us to demonstrate the success of new forms of peer review and to encourage meaningful discussions with scholars and university administrators of the significant value of unconventional scholarly methods and creative forms of publication. Having such familiar points of comparison, however awkward they may be, will be helpful in the course of advocating on behalf of creative and unconventional forms of scholarship.

The desire of scholars and university administrators to disseminate their scholarship beyond the campuses to reach broader audiences isn’t new, but the widespread enthusiasm for forms of audio entertainment, including audiobooks and podcasts, presents an obvious opportunity for those committed to the production and dissemination of scholarship to broaden our vision of scholarly publishing to accommodate new scholarly genres. A podcast that undergoes a publisher’s peer review, which shares the rigour of the manuscript review process and results in the support of colleagues in the relevant field, and that is, additionally, given the professional editorial support, production assistance, and promotional attention of a university press, ought, in turn, to be recognised as making a meaningful contribution to an academic’s publishing record.

We are aware that the peer review process for podcasts needs to break with conventional manuscript reviewing. The first set of review questions, the reviewers’ reports and the podcaster’s response, along with the subsequent iterations of this ongoing experiment in peer reviewing the podcast, will contribute to our thinking about the changing nature of scholarly peer review, editing, and publishing.

The questions for the peer reviewers of the second season of Secret Feminist Agenda are now available. As with the first season, we will post the reviews of the second season, along with Hannah’s response to them, on our website. Please do visit us again to read them in due course.

SSHRC Logo

This research was supported by the Social Sciences and Humanities Research Council of Canada. 

 

Podcasts

 

 

Season 1

Questions Used to Create the Peer Evaluation

A. Podcast Form

  1. It is the contention of this pilot podcast project that the form of the podcast presents scholars with the opportunity to experiment with new modes and means of scholarly communication. Do you agree with this contention? In what ways does this particular podcast series demonstrate the potential of the medium for scholarly dissemination? Are there specific ways in which you think the series could be improved in order to take advantage of the form as a way to engage listeners in new lines of scholarly inquiry?
  2. The podcast currently includes a first season of 15 episodes. Please identify the episodes you listened to for the sake of this report. What, in your view, is the overarching structure of this 15-episode series? Does the podcast as a whole have a discernible scholarly intent informing its approach to its subject(s)? If so, what about the podcast defines it as scholarly? What are the strengths and weaknesses of the overall form of this first series?
  3. What are the strengths and weaknesses of the structure of the individual episodes? Is the length of the episodes in keeping with the aims and scope of the episodes?
  4. Is the apparatus associated with the podcast (including links posted with each episode and related social media sites) scholarly? If not, how might it be strengthened to meet scholarly standards?
  5. One of the specific aims of this pilot podcast is to engage the form of the podcast as an object of research production. What are the strengths and weaknesses of the series as an expression of scholarly research engagement?

B. Podcast Content

  1. What, if anything, is the principle scholarly concern, or central argument, of the podcast series? What is the guiding scholarly methodology of the podcast series? Is it clear? If not, how might its methods be made more evident without sacrificing the tone of the podcast?
  2. To what audience(s) is the podcast directed? To what discipline(s) in particular, if any?
  3. Each podcast episode includes a reflective introductory piece and a feature conversation with guest(s). Each episode is accompanied by textual apparatus (e.g. episode notes; external links; references), as well as the podcaster’s associated website and social media platforms. Do you have any suggestions for the improvement of the podcast related to these various components with respect to their style, structure, format, features?
  4. In light of the aims of this pilot scholarly podcast series, which of the episodes you have considered (please identify them) were the strongest and which were the weakest? Why?

C. General

  1. Does the podcast series, as it stands, make a significant contribution to its field(s)?
  2. What are the competing and comparable podcasts in the field(s), and how does this one relate to them?
  3. What are the competing and comparable books in the field(s), and how does the podcast relate to them?
  4. If you were to assess the potential of a peer-reviewed podcast series to contribute new scholarship to a particular field, to which scholarly form would a podcast series be comparable: A monograph, an edited collection, a special issue of a journal article, a journal article, none of these? Why?
  5. Because this pilot scholarly podcast represents, equally, an experiment with the role of peer review in the production of such a podcast series, would you please comment on the following aspects of the peer review:
  • In light of your response to question #4, how many podcast episodes should a peer reviewer be asked to assess? Does your answer relate more to the number of hours’ listening or to your ability to answer fully the questions asked regarding the content and form of the podcast episodes and season?
  • Are the questions, as formed, suitably thorough, in your view, to provide a podcaster and the Press editor with useful and detailed feedback to ensure the scholarly quality of a podcast series? Are there questions you think should be asked of peer reviewers of podcasts that were not? If so, what are they?
  • In light of your response to question #4, if a podcast series intends to be open-ended, with no scheduled end to its production, how many rounds of peer review, and with what frequency, do you feel it would be necessary to ensure the ongoing quality of the podcast?
  • Does the open peer review process add value—editorial, production, other—to the podcast series? Could the same, or greater, value be provided to the podcaster and Press with a blind review?

The podcaster will respond to the reports and there may also be a (limited) subsequent engagement between reviewers and podcaster online as well. We invite you to follow the unfolding of this open peer review. If you have any questions about this pilot project, please feel free to write to me at: smcmenemy@wlu.ca

 

Season 2

A. General Content

  1. What are the overarching objectives of this podcast series? Does the second season present a uniformity of purpose? Does it cohere? How?
  2. Does the podcast contribute to scholarship by presenting new research or by offering a new understanding of, or approach to, familiar material?
  3. In what ways does the podcast relate to scholarship in its field(s)? Is the scholarship sound? Do you have suggestions for improving the podcast’s engagement with existing scholarship?
  4. Are there competing or comparable podcasts in the field(s) and how does this series compare to them? Are there books or other digital research projects that might be considered comparable to this podcast series? If so, how does the podcast compare?

B. Audience

  1. To what audience(s) is the podcast directed? Will it serve only specialists in the field(s), or will scholars, broadly, find the podcast of interest?
  2. Is the podcast likely to appeal to an audience of non-academic listeners?
  3. Would you have use of this podcast in a course? If so, how might you use it with students? In what courses might you use it?
  4. In what other ways might a scholarly audience make use of this podcast?

C. Organisation, Episode Content, and Style

  1. Is the organisation of the podcast’s second season appropriate? Do you have suggestions for improving its organisation?
  2. What are the strengths and weaknesses of the overall form and style of the second season?
  3. Of the episodes you listened to—please identify them by number for reference—which were the strongest and which were the weakest? Why?
  4. The second season includes episodes that alternate between the podcaster’s commentaries (“minisodes”) and the interviews with guests. What are the strengths and weaknesses of this structure to the podcast series?
  5. The podcast includes conventional text apparatus associated with each episode (e.g. show notes). Do you have any suggestions for improving this material? (Please also note that the podcaster also maintains a website for the podcast and a Twitter feed.)
  6. Are the lengths of the episodes appropriate? Would the podcast episodes benefit from being shortened or lengthened? If so, why?
  7. Do you have any suggestions for the improvement of the podcast, related either to its organisation, content, or style?

D. Overall Evaluation

  1. In what ways does this particular podcast series demonstrate the potential of the medium for scholarly dissemination? Are there specific ways in which you think the series could be improved in order to take advantage of the form as a way to engage listeners in new lines of scholarly inquiry?
  2. Does the podcast series, as it stands, make a significant contribution to its field(s)? How would you rate its importance and scholarship?

We are grateful to the Social Sciences and Humanities Research Council (SSHRC) for the support of this collaborative project through its Insight Development Grant programme.