13.0536 evaluating WWW materials

From: Humanist Discussion Group (willard@lists.village.virginia.edu)
Date: Thu Apr 13 2000 - 14:19:39 CUT

  • Next message: Humanist Discussion Group: "13.0539 divers queries & requests"

                  Humanist Discussion Group, Vol. 13, No. 536.
          Centre for Computing in the Humanities, King's College London

      [1] From: Matt Kirschenbaum <mgk@pop.uky.edu> (14)
            Subject: Re: 13.0531 evaluating WWW materials

      [2] From: cbf@socrates.berkeley.edu (83)
            Subject: Re: 13.0531 evaluating WWW materials

      [3] From: "P. T. Rourke" <ptrourke@mediaone.net> (45)
            Subject: Evaluating WWW Materials

            Date: Sat, 08 Apr 2000 14:08:11 -0400
            From: Matt Kirschenbaum <mgk@pop.uky.edu>
            Subject: Re: 13.0531 evaluating WWW materials

    On the original question, the MLA's Draft Guidelines on Evaluating Work with
    Digital Media in the Modern Languages may be helpful:


    But I think the issue could also be considered not from the standpoint of
    whether non-peer-reviewed materials should or should not count as "research,"
    but rather what constitutes peer-review in the first place. In Eric Johnson's

    > As far as documentation of use and of acceptance by the scholarly
    >community, the faculty member might list the sites that contain links to
    >the material -- along with any annotation (for example, "The best
    >web-based corpus of words of historical Spanish texts can be found at
    >http:// ...").

    "The best web-based corpus . . . " Is this not a form of peer-review? Is
    peer-review necessarily limited to the mechanisms in place on the editorial
    board of a journal or university press? Matt

            Date: Sat, 8 Apr 2000 11:15:42 -0700 (PDT)
            From: cbf@socrates.berkeley.edu
            Subject: Re: 13.0531 evaluating WWW materials

    UC Berkeley's Academic Senate adopted language last year that essentially
    eliminates the difference in format among faculty research publications:
    web, print, CD-ROM will all be considered equally. What did _not_ change
    was the evaluation requirement, essentially peer-reviewed vs.

    I would argue that the web, in particular, allows us to get beyond that
    dichotomy because it provides, potentially, the possibility of actually
    measuring the impact of a particular web site on its discipline.
    Unfortunately, the number of hits is as yet a fairly imprecise measure and
    subject to manipulation ("Hey, I'm up for tenure; assign all your students
    to look at my web site."). A better measure would be links _into_ a web
    site. As Willard pointed out to me the other day, this is how the
    Google.com portal works.

    In some respects, this is like the traditional citation index that
    measures the number of times a given study is cited in other studies.

    What peer revieww does, essentially, is certify that three knowledgeable
    members of a given discipline have read the piece of work and certified
    that it's worthwhile. Considering the amazing amount of fairly useless
    work that is published in peer-reviewed journal, such certification is a
    fairly slender reed upon which to render academic judgments; but so far
    it's the best we have.

    [Of course, in the ideal case, peer review often improves the quality of
    the piece of work reviewed.]

    There is at least one initiative--the name escapes me at the moment--that
    is attempting to divorce peer review from publication, sort of a Good
    Housekeeping Seal of Approval that could be attached even to a
    self-publication. It would be a Good Thing if the members of the
    humanities computing community could devise such a system for ourselves
    and our colleagues.

    Charles Faulhaber The Bancroft Library UC Berkeley, CA 94720-6000
    (510) 642-3782 FAX (510) 642-7589 cfaulhab@library.berkeley.edu

            Date: Sat, 8 Apr 2000 20:46:52 -0400
            From: "P. T. Rourke" <ptrourke@mediaone.net>
            Subject: Evaluating WWW Materials

    Sounds interesting as far as you go.

    > We are in the process of revising the promotion and evaluation
    > procedures in all of the departments at Illinois State University, and I
    > have been asked by a committee to get input from individuals at other
    > institutions concerning how non-peer-reviewed, web-based materials
    > could/should be evaluated by the institution.

    > Some of the best sources of online information on a variety of subjects
    > have been created by faculty. There ought to be some means of giving the
    > creators credit toward academic advancement.

    Not from academic experience, but publishing experience, I'd suggest that
    it's far too early to come up with a firm policy for web-based materials; it
    would be better to just add some wiggle room. Ultimately a web-based
    resource should be evaluated by "published" reviews rather than blind peer
    review. The problem is that there aren't a lot of reviews available of
    electronic resources. When they are available, one hopes that authors will
    do their readers and evaluators the favor of linking to them.

    So for now evaluation should be on a case-by-case basis. If the resource is
    well-known and well used, the author should be able to get letters of
    reference from users with some degree of expertise that can stand in for
    published reviews until such time as those become available. A major
    project should have the same cachet as a book or a series editorship; a
    minor but well-appreciated project should have about the cachet of a
    published paper of the same weight. And ultimately members of the same
    department should have some sense of the scholarly quality of the work
    without reference to other's reviews, unless you're talking about a
    situation in which e.g. you have one Asian expert or Africanist in a History
    department dominated by European historians.

    > As you note, non-peer-reviewed materials on the web might be considered
    > as a contribution to scholarship or teaching, but there are strong
    > objections to doing so since such materials are very different from
    > traditional scholarship and teaching. Perhaps the creation of online
    > materials might be considered as service (a common third category of
    > evaluation for faculty).

    But sometimes services (if you mean what I think you mean, committees and
    professional society activities, e.g.) don't have a scholarly component at
    all, while a good web resource will. Perhaps a fourth category is waiting?

    Patrick T. Rourke

                           Humanist Discussion Group
           Information at <http://www.kcl.ac.uk/humanities/cch/humanist/>

    This archive was generated by hypermail 2b29 : Thu Apr 13 2000 - 14:36:50 CUT