Virtual Peer Review?

Jan 18 2011 Published by under Uncategorized

Faithful readership, all four of you (it would be five, but mom's "google machine" has crashed) I wanted to survey your opinions on an interesting topic.  I read in the a recent issue of Science, that NSF is looking at virtual peer review of grant proposals through use of Second Life.  Seems like an interesting way to "get everyone in the same room" but not be in the same room to hash out these proposals in a virtual panel.  So I'm hoping PLS can chime in since he has been in on NSF grant evaluation panels.

The NIH is starting to take some interest in this since they are becoming more reliant on videoconferencing.  I can see that the infrastructure for videoconferencing could be somewhat expensive but this Second Life-based review system would be less expensive.  However I don't like this because the NSF is using someone else's infrastructure and you have to wonder about access issues, accounts getting hacked, DDoS attacks, etc.  Also, I think part of the review panel is getting folks out of the office, shoving them in a room together and plowing through these proposals.  Will reviewers be stay focused in this virtual review environment when their office phone is ringing, the damn trainees are beating on the door to get some form signed, and chair really wants you to show up at the faculty meeting this afternoon because after all, you are here are you not?

Those are some of the downsides, lets check out the positives.  You don't have to travel, which may be important to those with families or who hate sitting in airport lounges waiting for their delayed flights.  It would be a huge cost savings, NSF isn't having to lodge and feed all these folks.  This might increase the number of folks who would participate in an evaluation panel that would not normally?

If this ever makes its way to the NIH, I could see:
-CPP's avatar stumbling around drunk with a bottle of Jameson that he might chuck at a reviewer who gets off topic.
-Dr. Isis, changing her virtual footwear throughout the study section.
-Pascal avatar blogging on her iPad because she bored of some old graybeard rambling on about something off topic.
-Some torrid affair of tenure-tracked professors potentially involving cybersex on top of the conference table while a mortified Drugmonkey looks on.  Pascal would be capturing video but her Flip Mino is still busted.

Thoughts?

20 responses so far

  • Dorothea says:

    So, as a digital-preservation type, my question is how much of a paper/pixel trail comes out of these sessions. If the answer is "not much if any; that's part of why they're in person to begin with," then I hope the NIH is taking that into account as they design the system. ("What happens in a grant-review meeting STAYS in a grant-review meeting.")

    • Good point, you can record audio conversations in Second Life yourself or NIH can add in a "ghost man" in the room whose only function is to record conversations. I also think you can move far away from others so you can have more private conversations. This definitely could put some people on their toes if what gets said in the room doesn't stay in the room, or at least gets recorded for posterity.

  • Karen says:

    Word, Dorothea. But also, really -- Second Life? I mean, they're super-nice people, but who on earth uses this? Why not just meet on Xbox Live and kill zombies together or something? This cannot possibly be the best cheap way to meet virtually.

    • Karen, you and I are holding a study section and we are only reviewing zombie plans. I do use my Xbox live connection to chat with friends all over the US and world. It only costs me $50 for the year for the best long distance plan ever. And you can add up to 8 people to a voice chat party too I think.

  • Bashir says:

    Second Life? Really? That's what they think the best mechanism for this would be? Doesn't Skpye or Google offer some services that would suffice?

  • drugmonkey says:

    "mortified". exactly. by the notion of "second life" being used for reviews. what a freaking travesty of a clusterbork that would be.....

  • drugmonkey says:

    I think the CSR is on the wrong track with all this asynchronous online discussion and video discussion crapola. It is hard *enough* to get the non-assigned reviewers to concentrate on the discussion as is....yet sometimes their engaged input is highly critical to the disposition of the review.

  • proflikesubstance says:

    IMHO, this would significantly reduce the quality of the discussion. On top of that, what stick would NSF then have for people to get their summary statements written? In person they don't let you out of the room, but if you could just hang up and turn your attention elsewhere, that evaporates.

    In addition to the issues outlined above, who they fuck over the age 0f 18 interacts well with avatars? Do you really think Prof graybeard isn't going to find some way to take down they whole internet if they try and set up a second life account? This would be an epic fail all the way around.

    They brought some of this up at the panel I was on in the fall and there was universal agreement that not having it in person would lead to less input on all the proposals.

    • PLS do all the NSF grant evaluation panels take place in Arlington or are they regional and if not would you prefer a more regional approach so not everyone has to fly to Virginia?

      • proflikesubstance says:

        They are all in DC, as far as I know. And regional would be tough because people come from all over. You can't really decide that for a given panel you are going to restrict the geography of the participants to a certain area. POs have enough trouble finding willing participants without that issue adding to the problems.

  • Odyssey says:

    I'm with PlS on this. We were also surveyed on my last NSF panel and there was unanimous agreement that any kind of virtual panel set up would be a bad thing.

    As far as the conference call/online type review panels go, I've served on a couple for the AHA. They SUCK. There simply isn't the same level of discussion that occurs in person, and many of the nuances of face to face interactions are simply lost. I'm strongly of the opinion that reviews, and therefore PI's, suffer when panels are held in this format. And I know from interactions with AHA staff that they are have been having an increasingly hard time finding people willing to serve on review panels now they've largely moved to this format. Coincidence? I think not.

  • Drugmonkey says:

    In other words, Scarpa is high as a kite with this particular obsession of his.

  • I think the CSR is on the wrong track with all this asynchronous online discussion and video discussion crapola.

    Dunno about video, but I served on an asynchronous on-line discussion panel, and it worked really well. With a relatively small number of grants and small number of participants, my impression (and that of other members whom I spoke with afterwards) was that the review was very complete and fair. It was just like commenting on a blog! (For realz!)

  • DrugMonkey says:

    "just like commenting on a blog" is not an endorsement of this insanity in my book.

  • BugDoc says:

    Having done both in-person study sections and the asynchronous online discussion (AOD), I think the AOD has a lot of offer. To me, the best part of attending study section in person was the chance to hang out with friends and colleagues for dinner. The study section discussion itself was interesting, but not really an efficient use of time. AOD was terrific - limited the temptation for blowhards to go on and on just to hear themselves talk, resulted in better crafted and concise comments (vs. off the cuff remarks) but still encouraged lots of discussion. Thus, each proposal actually got longer and more careful consideration in my opinion. I also found that people on the study section who were not reviewers of a given proposal were more likely to weigh in on the discussion, since they had time to skim through the proposal and maybe look up a few points. Not sure Second Life is the way to go though.

  • drugmonkey says:

    As an established blowhard, it is difficult for me to understand these people who actually have an opinion, but are somehow inhibited from expressing it in the context of a study section meeting.

  • BugDoc says:

    We have a different definition of blowhard. I'm talking about the cases where everybody's scores are similar so the chair says, "hey, let's keep it short since we're all agreed", then Dr. Blowhard goes on for 20 minutes about what we all agree on. Or Reviewer #3 says, "I agree with what reviewers #1 and #2 said" but then goes on to reiterate their points in gory detail for an additional 15 min. Yawn.

  • drugmonkey says:

    Aren't you making some kind assumptions there, BugDoc?

Leave a Reply