May 24, 2010
To Crowdsourcing Friends, Foes & Fanatics: Just How Loyal Is Your Community?

Depending upon who you ask, crowdsourcing is either evil, revolutionary, or a next gen of internships.

But one thing that ALL crowdsourcing companies like to preach is how loyal
and trustworthy and professional their
community is. I know because I’ve read it in 100 different sites. Hell,
I’ve written it a 100 different times here at uTest. So why do
crowdsourcing companies insist upon telling the world how loyal and
earnest their community is?  Maybe it’s to assuage the fears of
prospective customers about entrusting their logo design, app
development, content production or marketing to a community of
strangers. Maybe it’s because if marketers say it enough times, we hope
it’ll come true.

The more pessimistic view is that people — cloaked in the anonymity
of the web — often act in greedy, selfish, mean-spirited ways (this
perspective didn’t make it into the crowdsourcing brochure, by the way).
Such dark behavior is well-documented and takes the form of flame wars
on message boards, bullying via social media and online fraud.

So which is it — are people good-natured and honest?  Or are they
money-hungry malcontents who will do anything to get ahead, as long as
they don’t get caught?  Obviously it depends on the people, but I
learned the truth about our community this week — and it was a lesson we
learned the hard way.

A little background:  At uTest, we pay our testers twice per month
via PayPal or Payoneer. And at this point in our growth, each pay cycle
involves a non-trivial amount of cash — pretty deep into the five-figure
range.  Now, it’s not easy or flattering to admit this, but in our most
recent pay cycle, we experienced a glitch that caused us to pay our
testers twice. That’s right folks, it was double payday
here at uTest

We discovered the error moments later when several testers emailed us
to point it out to us. We immediately split our efforts up into two
camps:  one group trying to identify the issue and make sure it never
happened again; the other trying to figure out how to get our money
back.  I was in the second group.

After exhausting our options with our payment vendors (who were
responsive and sympathetic, but unable to help), we came to the
realization that we would have to simply explain to our testers what
happened and ask them to refund the duplicated payments via a
not-so-simple five-step process. Now, I love our community and I think
the world of them, but let’s just say I was skeptical. I mean, c’mon,
it’s our mistake and
it’s free money.

So what happened?

  • At 2am on Sunday morning, we sent out an email to all the affected
    testers… and hoped for the best
  • Within 24 hours, more than two-thirds of the money had been refunded
  • Within 72 hours, more than 90% was refunded
  • And within 96 hours, more than 98% had made it’s way home
  • I expect that within a week, less than 0.5% will remain outstanding
  • We’re not talking about a few dollars here and there… one U.S.
    tester refunded nearly $2,000. Several others from the U.S., Canada and
    Western Europe issued refunds in the $1,500 range. We had testers in
    developing nations who refunded nearly $1,000 — in places where $1,000
    is more than one month’s salary for a QA engineer.

So in addition to proactively bringing this gaffe to our attention,
our community went out of its way to refund this money promptly and in
full. Sure, I’m still pissed and embarrassed that it happened. But it
did, and the takeaways have been surprising and worthwhile:

  1. Communities beat crowds. Crowdsourcing companies
    should stop trying to build crowds or mobs. Instead, invest in building
    a community. We pour a lot of time, energy and money into our
    community. From our tester forums to free training programs to Bug
    Battle competitions to town hall meetings to our uTester of the year
    awards.  And we don’t do it because it’s always fun or easy or even
    public. We do it because we believe that communities are
    better than crowds or unruly mobs.
  2. Surprise, surprise, surprise. Even after nearly 7
    years of building community-driven businesses, I can still be genuinely
    surprised by the loyalty and integrity of a large group of people who
    have never met one another in person.  I wonder how else our community
    would surprise me if I keep a more open mind.
  3. Deep, trusting relationships are a necessity.  We
    treat the men and women in our community like respected professionals.
    We work hard to give them a lot of different ways to engage with us on
    their own terms. And they have always treated us well in return. But
    when push came to shove and there were real dollars on the line, the
    true impact of the relationships between us and our community became
    abundantly clear.
  4. My new favorite story. The next time a prospective
    customer asks me “who are these 25,000 testers and how do I know I can
    trust them with my super-secret app?”, I know exactly what story I’ll
    tell them in response.

Now, my fellow crowdsourcing enthusiasts (on both sides of the
aisle), let me ask you this:  when push comes to shove, just how loyal
is your community?  While I don’t recommend this particular path as a
way of proving it, I would love to hear other stories about what happens
when a community is transparent and accountable to one another. Or, if
you have them, tell us your nightmarish tales of community deception and

Matt Johnston is the VP of Marketing & Community at uTest.  This blog post was originally published on May 20, 2010.  You can find this blog post, as well as additional content on the uTest blog located here.