Kathy Sierra includes some publishing rants in When process goes bad

Kathy Sierra, author of the brilliantly conceived "Head First" books (but if you are reading my blog, you certainly knew that already), hits on a couple of fairly common "we can't do that here" publishing process requests in Creating Passionate Users: When process goes bad. She mentions in one that she's NOT talking about O'Reilly or Osborne.

1 of the ideas she mentions - reducing technical errors by using numerous peer reviewers - is something I've been trying to understand a practical way to implement here without making it a burden for the editorial staff. And without requiring new IT infrastructure (i.e. no, I can't install Sharepoint Portal Server or some other technical solution to help with this). Anyone (Kathy included) want to clue me in on how to do this? (BTW, back in the late 90s, I was looking for a way to make this work at my previous book publishing employer. I don't think we're alone in not having mastered this yet).

Incidentally, we have done this on at least a couple of books here. In those cases, the authors have taken on the burden of collecting and sorting through the multiple sets of comments. In cases where there's no time-sensistive deadline to get a book (the topic isn't tied to a specific software or specification version) out and the authors are willing to put this extra work into it, this is workable. But in our experiements, it did add a lot of time and work for the authors.

But my favorite bit of her whole post is this:

"Sometimes the employee or user is the only one who DOES "get it". Sometimes it's the lower-level (or at least more user-facing) employee who really knows how damaging a company's policies can be, or where the points of leverage really are. Sometimes it's the user who has a basis of comparison -- who hasn't bought into the company's worldview so long that they can't see any other reality."

I think that's one of the big points of Naked Conversations - of why blogs are such an empowering tool for companies to take advantage of the passion and knowledge of "lower-level" employees and users.

Comments

[Kathy Sierra]

Hi Jim!
(Thanks for the kind words : )

We stumbled on the review process we use now by accident, but it's been so successful that we've kept it and convinced *both* our publishers to let us use it. Our particular implementation is unique to Java, but I reckon it can be applied to different domains pretty simply -- here's how our current version works:

1) We solicit volunteers from online forums, user groups, or previous readers who *want* advance access to the content in the book.

2) We set up a private discussion forum (although this could be done on any private email list, etc.)

3) We post the PDF chapters as we complete them.

4) The reviewers are notified through an email that a chapter is ready for review.

5) They download the chapter, and start posting their comments to the thread for that chapter. Each person makes a single post with all their comments, using a simple editing format (WAS: blah blah blah CHANGE TO: (or SHOULD BE:) foo foo foo).

6) The part that makes it work, though is this -- we have one *manager* of the process, who is paid (what a single well-paid tech reviewer might make), and although he does not actually do any technical reviewing himself, he compiles all of the reviewer comments into a single document for us. That's been the biggest key. He is also responsible for all the coordination -- assigning passwords to the reviewers, sending out emails, etc.

7) When the review is complete, the publisher then *donates* additional money on behalf of the other volunteer reviewers to Javaranch for a new server, or something. (Obviously this donation could be made to something else the reviewers agree on). The reviewers are told from the beginning that no single reviewer is paid, but that the group as a whole works together as volunteers -- it is a win for them because they get advance access to the material (so it helps to have people who consider this a benefit), and the result of their effort leads to a donation for a cause they support (in our case, Javaranch). And... there is less pressure/obligation on any individual reviewer since they aren't technically being paid to do it.

8) If reviewers do not give us any feedback, they are eventually dropped from the review process.

9) If reviewers give us a LOT of good feedback, they get their name and photo in the book.

10) When we're not overwhelmed, we also make special edition reviewer t-shirts for the ones who do a lot, and of course we send them a book when it's done. (And the manager gets multiple books)

The key is finding the manager. Finding the volunteers has not been difficult.

The biggest potential problem is that because no single reviewer is responsible for giving a good review, you might get poor quality. None of the unpaid reviewers is likely to take the time to compile every bit of code, etc.

The fix for this is to have a high enough number of reviewers, that you get a "wisdom of crowds" effect, and virtually everything is uncovered by at least one person. We started out with only four or five people, but we've found that having around twelve people (only some of them will have the time to do a detailed review) works well. If a couple of key people have to drop out for vacations or something, we recruit someone else to step in. We always have a waiting list.

FYI -- the guy who manages *our* reviews is always happy to talk to other publishers about doing the same thing. : ) He had been a member of the review team for one of our books, and then took over coordinating all the feedback.

I cannot tell you how much this has saved our butts. Each of our books has come out with far less errata than the previous book. The reviewers are happy to be part of it -- and have a sense of ownership -- and the quality is greatly improved. We won't do it any other way now.

One key component is to have content that people really want to review -- and are appreciative of the opportunity to get advance access.

Feel free to ask me any other questions about it, or for contact info for the guy who has been managing it (Johannes de Jong).

[Jim Minatel]

Kathy: This is great. Step 6 (getting the review manager) seems like an essential step I'd been missing. Step 7 sounds like a great motivator too, but just thinking about the paperwork/accounting explanation involved to make that happen hurts. Maybe if I donate my own blood I could get the requisition approved. That will limit the # of projects I can do this on though.

Seriously, thanks for sharing this. I'm guilty sometimes of being too secretive and protective of my little process tweaks. I think we would all benefit if we all learned to share more like Kathy.

[Kathy Sierra]

I hear you about the paperwork and explanations, but it's basically just cutting two pre-budgeted checks -- one to the manager, and one on behalf of the entire review team (again, in our case, Javaranch). It's the same budget regardless of how many reviewers ultimately end up on the team.
For us, O'Reilly and Osborne do virtually nothing except write the checks. We handle it all, coordinating with our review manager. Only when the chapters have gone through this process do they go to the publisher for copy editing.
I think in the end it's been a big win for everyone -- most especially the readers. It's really a wisdom-of-crowds approach... with that many sets of eyes, a lot more is caught -- even when each individual reviewing it has less incentive to be thorough.
Cheers

[shel Israel]

No wonder, Naked Conversations is moving up in the blog rankings. All our links seem to be coming from you. Just to make it a Marchall McLuhan moment--yes. An essential point of Naked Conversations, is that blogs harness the passion and knowledge of the middle ranks of an organization. More than that, this level employee is often seen as more credible, than the polished and coached big brass.