Your cart is empty.

Scholarly Podcasting Open Peer Review

Season 1 Questions          Season 2 Questions          Season 3 Questions

Introduction

By Siobhan McMenemy, Senior Editor

The peer review of the first season of the Secret Feminist Agenda podcast, created by Hannah McGregor, was completed in May 2018. The reviewers were asked to respond to a set of questions designed to engage with issues related to the various aims of our research into the ways and means of producing, editing, reviewing, and broadcasting scholarly podcasts under a university press imprint. Our questions and the reviewers’ comments have served two purposes: To provide Hannah McGregor with suggestions for improving her podcast in form and content from two scholars who are themselves engaged in scholarly activities relevant to our work on the genre of the scholarly podcast; and to help us to hone a set of peer review questions for the review of subsequent seasons of Hannah’s podcast and other, future podcasts. We encouraged the reviewers to reflect on whether they considered the peer review questions themselves to be relevant and helpful to their task as peer reviewers and to our responsibilities as editors, producers, and publishers of the podcast. 

The first set of reviewers’ reports raised interesting questions about the appropriateness and practicalities of peer reviewing podcasts, the philosophical matter of what constitutes scholarship, and the evaluative practices of the tenure and promotion review process and where born-digital scholarship fits into these performance assessments. A number of questions in this first round of review were meant to encourage further consideration of the relationship between conventional forms of scholarly publishing (e.g. journal articles, book chapters, monographs, edited collections) and the publication of a scholarly podcast series. We appreciated the reasons for the reviewers’ frustration with our repeated insistence on creating points of comparison between print and digital forms where the comparisons were awkward at best. Part of the aim of such comparisons was to highlight the commitment of university presses to provide authors—and podcasters—with valuable community engagement with their work with an eye to strengthening the work for publication, regardless of genre or format. Likewise, we know that the rigour of peer review and the value of the university press imprint contribute to institutional support for a scholar’s work. It is our hope that this peer review exchange, and those that will follow the completion of the second and third seasons of the podcast, will allow us to demonstrate the success of new forms of peer review and to encourage meaningful discussions with scholars and university administrators of the significant value of unconventional scholarly methods and creative forms of publication. Having such familiar points of comparison, however awkward they may be, will be helpful in the course of advocating on behalf of creative and unconventional forms of scholarship.

The desire of scholars and university administrators to disseminate their scholarship beyond the campuses to reach broader audiences isn’t new, but the widespread enthusiasm for forms of audio entertainment, including audiobooks and podcasts, presents an obvious opportunity for those committed to the production and dissemination of scholarship to broaden our vision of scholarly publishing to accommodate new scholarly genres. A podcast that undergoes a publisher’s peer review, which shares the rigour of the manuscript review process and results in the support of colleagues in the relevant field, and that is, additionally, given the professional editorial support, production assistance, and promotional attention of a university press, ought, in turn, to be recognised as making a meaningful contribution to an academic’s publishing record.

We are aware that the peer review process for podcasts needs to break with conventional manuscript reviewing. The first set of review questions, the reviewers’ reports and the podcaster’s response, along with the subsequent iterations of this ongoing experiment in peer reviewing the podcast, will contribute to our thinking about the changing nature of scholarly peer review, editing, and publishing.

(Update for Season 2, January 7, 2019)
The peer review of the second season of Secret Feminist Agenda is now available online. As Hannah’s response indicates, the first two rounds of peer review have raised a number of important questions—theoretical and practical—to consider over the coming months and into the future of scholarly podcast peer reviewing and publication.

(Update for Season 3, September 12, 2019)
The peer review of the third season of Hannah McGregor’s Secret Feminist Agenda represents the capstone of this experiment in creating, editing, and peer reviewing a scholarly podcast. In scholarly publishing, where work is often reviewed over multiple incarnations, there is a convention of seeking the engagement of scholars for subsequent rounds of peer review who have reported on previous versions of a work. The aim of such consistency is to establish a helpful degree of familiarity with the work over time and to bring a depth of engagement with the work that should enrich and enhance the review process and produce a stronger publication. This practice also affords the reviewers the opportunity to see—or hear—for themselves the changes the researcher-author-podcaster has brought to her work in response to reviewers’ earlier comments and suggestions.

With these aims in mind, I requested the involvement of two reviewers for the assessment of the third season of the podcast, each representing the review process of one of the previous seasons. The peer review questions for season three build on the earlier sets of questions and the reviewers’ engagement with them.

In response to comments from listeners, scholars, and colleagues, Hannah and I decided to push our experiment further for the sake of this final round of review. I asked that the two reviewers engage collaboratively in their assessment of season three. I invited the reviewers to listen to a series of episodes, to consider the review questions while doing so, and then to respond in conversation with one another. I asked them to record their conversation for the record, with the understanding that Hannah and I would subsequently respond in conversation to this final round of review as collaborators on this project, also recording our conversation for the record.

The purpose of this particular undertaking is to experiment further with peer review, to make explicit the degree to which peer review, when working at its best, is indeed an iterative and collaborative venture. We were also pleased to bring to the final round of peer review of this podcast a format and a spirit that echo those of Secret Feminist Agenda itself.

The audio peer review and response are available. In due course, we will upload a transcription of each as well.

This third round of peer review marks the conclusion of WLU Press’s formal peer review of Secret Feminist Agenda, after which I will oversee the Press’s remaining internal stages of assessment of the podcast in consideration of its suitability for publication under the WLU Press imprint.

SSHRC Logo

This research was supported by the Social Sciences and Humanities Research Council of Canada. 

 

Podcasts

 

Questions Used to Create the Peer Evaluation

Season 1

A. Podcast Form

  1. It is the contention of this pilot podcast project that the form of the podcast presents scholars with the opportunity to experiment with new modes and means of scholarly communication. Do you agree with this contention? In what ways does this particular podcast series demonstrate the potential of the medium for scholarly dissemination? Are there specific ways in which you think the series could be improved in order to take advantage of the form as a way to engage listeners in new lines of scholarly inquiry?
  2. The podcast currently includes a first season of 15 episodes. Please identify the episodes you listened to for the sake of this report. What, in your view, is the overarching structure of this 15-episode series? Does the podcast as a whole have a discernible scholarly intent informing its approach to its subject(s)? If so, what about the podcast defines it as scholarly? What are the strengths and weaknesses of the overall form of this first series?
  3. What are the strengths and weaknesses of the structure of the individual episodes? Is the length of the episodes in keeping with the aims and scope of the episodes?
  4. Is the apparatus associated with the podcast (including links posted with each episode and related social media sites) scholarly? If not, how might it be strengthened to meet scholarly standards?
  5. One of the specific aims of this pilot podcast is to engage the form of the podcast as an object of research production. What are the strengths and weaknesses of the series as an expression of scholarly research engagement?

B. Podcast Content

  1. What, if anything, is the principle scholarly concern, or central argument, of the podcast series? What is the guiding scholarly methodology of the podcast series? Is it clear? If not, how might its methods be made more evident without sacrificing the tone of the podcast?
  2. To what audience(s) is the podcast directed? To what discipline(s) in particular, if any?
  3. Each podcast episode includes a reflective introductory piece and a feature conversation with guest(s). Each episode is accompanied by textual apparatus (e.g. episode notes; external links; references), as well as the podcaster’s associated website and social media platforms. Do you have any suggestions for the improvement of the podcast related to these various components with respect to their style, structure, format, features?
  4. In light of the aims of this pilot scholarly podcast series, which of the episodes you have considered (please identify them) were the strongest and which were the weakest? Why?

C. General

  1. Does the podcast series, as it stands, make a significant contribution to its field(s)?
  2. What are the competing and comparable podcasts in the field(s), and how does this one relate to them?
  3. What are the competing and comparable books in the field(s), and how does the podcast relate to them?
  4. If you were to assess the potential of a peer-reviewed podcast series to contribute new scholarship to a particular field, to which scholarly form would a podcast series be comparable: A monograph, an edited collection, a special issue of a journal article, a journal article, none of these? Why?
  5. Because this pilot scholarly podcast represents, equally, an experiment with the role of peer review in the production of such a podcast series, would you please comment on the following aspects of the peer review:
  • In light of your response to question #4, how many podcast episodes should a peer reviewer be asked to assess? Does your answer relate more to the number of hours’ listening or to your ability to answer fully the questions asked regarding the content and form of the podcast episodes and season?
  • Are the questions, as formed, suitably thorough, in your view, to provide a podcaster and the Press editor with useful and detailed feedback to ensure the scholarly quality of a podcast series? Are there questions you think should be asked of peer reviewers of podcasts that were not? If so, what are they?
  • In light of your response to question #4, if a podcast series intends to be open-ended, with no scheduled end to its production, how many rounds of peer review, and with what frequency, do you feel it would be necessary to ensure the ongoing quality of the podcast?
  • Does the open peer review process add value—editorial, production, other—to the podcast series? Could the same, or greater, value be provided to the podcaster and Press with a blind review?

The podcaster will respond to the reports and there may also be a (limited) subsequent engagement between reviewers and podcaster online as well. We invite you to follow the unfolding of this open peer review. If you have any questions about this pilot project, please feel free to write to me at: smcmenemy@wlu.ca

 

Season 2

A. General Content

  1. What are the overarching objectives of this podcast series? Does the second season present a uniformity of purpose? Does it cohere? How?
  2. Does the podcast contribute to scholarship by presenting new research or by offering a new understanding of, or approach to, familiar material?
  3. In what ways does the podcast relate to scholarship in its field(s)? Is the scholarship sound? Do you have suggestions for improving the podcast’s engagement with existing scholarship?
  4. Are there competing or comparable podcasts in the field(s) and how does this series compare to them? Are there books or other digital research projects that might be considered comparable to this podcast series? If so, how does the podcast compare?

B. Audience

  1. To what audience(s) is the podcast directed? Will it serve only specialists in the field(s), or will scholars, broadly, find the podcast of interest?
  2. Is the podcast likely to appeal to an audience of non-academic listeners?
  3. Would you have use of this podcast in a course? If so, how might you use it with students? In what courses might you use it?
  4. In what other ways might a scholarly audience make use of this podcast?

C. Organisation, Episode Content, and Style

  1. Is the organisation of the podcast’s second season appropriate? Do you have suggestions for improving its organisation?
  2. What are the strengths and weaknesses of the overall form and style of the second season?
  3. Of the episodes you listened to—please identify them by number for reference—which were the strongest and which were the weakest? Why?
  4. The second season includes episodes that alternate between the podcaster’s commentaries (“minisodes”) and the interviews with guests. What are the strengths and weaknesses of this structure to the podcast series?
  5. The podcast includes conventional text apparatus associated with each episode (e.g. show notes). Do you have any suggestions for improving this material? (Please also note that the podcaster also maintains a website for the podcast and a Twitter feed.)
  6. Are the lengths of the episodes appropriate? Would the podcast episodes benefit from being shortened or lengthened? If so, why?
  7. Do you have any suggestions for the improvement of the podcast, related either to its organisation, content, or style?

D. Overall Evaluation

  1. In what ways does this particular podcast series demonstrate the potential of the medium for scholarly dissemination? Are there specific ways in which you think the series could be improved in order to take advantage of the form as a way to engage listeners in new lines of scholarly inquiry?
  2. Does the podcast series, as it stands, make a significant contribution to its field(s)? How would you rate its importance and scholarship?

 

Season 3

A. General Content

  1. What are the overarching objectives of this podcast series? Does the third season present a uniformity of purpose? Based on your assessments of more than one season of the podcast, do you think the podcast, as a whole, has established a level of coherence conventionally expected of published scholarship?
  2. Does the podcast contribute to scholarship by presenting new research or by offering a new understanding of, or approach to, familiar material?
  3. In what ways does the podcast relate to scholarship in its field(s)? Is the scholarship sound? Do you have suggestions for improving the podcast’s engagement with existing scholarship?
  4. Are there competing or comparable podcasts in the field(s) and how does this series compare to them? Are there books or other digital research projects that might be considered comparable to this podcast series? If so, how does the podcast compare?

B. Audience

  1. To what audience(s) is the podcast directed? Will it serve only specialists in the field(s), or will scholars, broadly, find the podcast of interest?
  2. Would you use this podcast in a course? If so, how might you use it with students? In what courses might you use it?

C. Organisation and Episode Content

  1. Is the organisation of the podcast’s third season appropriate? Do you have suggestions for improving its organisation in any future seasons?
  2. Of the episodes you listened to—please identify them by number for reference—which were the strongest and which were the weakest? Why?
  3. The third season includes episodes that built thematic arcs between ‘minisodes’ and interview episodes. Were these thematic arcs successful? Why, or why not?
  4. Do you have any suggestions for the improvement of the podcast, related either to its organisation or content?

D. Format and Style

  1. Is the audio format of the podcast appropriate for the scholarly aims of the project? Does the audio format contribute to the scholarship? If so, how? If not, why not?
  2. Does the podcaster use the audio format effectively? Is the quality of the audio production of the podcast publishable?
  3. What are your impressions of the development of the podcast’s form and style in the third season in relation to the season you previously reviewed for the Press?
  4. The podcast includes conventional text apparatus associated with each episode (e.g. show notes). Do you find this related material suitable to the aims and scope of the scholarly podcast? (Please also note that the podcaster also maintains a website for the podcast and a Twitter feed.)
  5. Are the lengths of the episodes appropriate? Would the podcast episodes benefit from being shortened or lengthened? If so, why?

E. Peer Review Process Evaluation

  1. Based on your experience with the peer review process of this podcast over multiple seasons, are there general or specific questions you consider important to the evaluation of scholarly podcasts that you would hope to see incorporated into future such peer reviews? If so, what are they and why are they important?
  2. Based on your experience with this—and other—peer reviews of born-digital scholarship, do you feel this podcast peer review has met existing standards of evaluation of such work? If not, why not? How might it be improved?

F. Overall Evaluation

  1. Does the podcast series, as it stands, make a significant contribution to its field(s)? Do you recommend publication of the podcast under the imprint of WLU Press?

 

We are grateful to the Social Sciences and Humanities Research Council (SSHRC) for the support of this collaborative project through its Insight Development Grant programme.