

Silly Rabbits: Google is for Spam not for Search - atularora
http://smoothspan.wordpress.com/2011/01/10/silly-rabbits-google-is-for-spam-not-for-search/

======
gamble
Spot-on. Google used to talk a good game about combatting search spam, but
once Demand Media and the ilk called their bluff Google chose to fold and
opened the doors to an industrial-scale search spam industry. They may have
stumbled into it, but now they're in a bind because taking on the spammers
would have an immediate and noticeable hit on their revenue - but in the long
term, surrendering to spam is going to undermine their reputation.

~~~
jacquesm
> but in the long term, surrendering to spam is going to undermine their
> reputation.

That's already happened. I am pretty sure that this is not intentional though.

What would sway me is if say 'bing' would have zero spam and google would have
a whole pile of it.

Is there any research confirming or denying that?

~~~
gamble
Not sure about bing, but DuckDuckGo is basically Yahoo with the search spam
filtered out. On heavily spammed searches DDG has become noticeably better,
IMO.

~~~
parfe
Yahoo is powered by Bing. What do you mean that DDG is basically yahoo?

~~~
gamble
DDG is built on Yahoo BOSS, (Build Your Own Search Service) Yahoo's white-
label search engine API. DDG adds spam filtering and some custom indexing on
top, but the bulk of their data comes from Yahoo's index.

AFAIK BOSS is still Yahoo technology, not bing. There may be a plan to migrate
BOSS to bing.

------
spiffworks
Does anybody really believe that Google engineers are stupid enough to think
that it is good for them to serve spam to their users? This cockamamie
conspiracy theory is getting really old.

~~~
larrik
Perhaps it isn't up to the engineers. If Google's outward appearances seem to
be deteriorating so profoundly, then it may be a reflection of something
happening internally.

Has anyone from inside Google said anything about this yet?

~~~
spiffworks
Okay, then do you think that Larry Page, Sergey Brin and Eric Schmidt are
stupid enough?

> Has anyone from inside Google said anything about this yet?

Matt Cutts and moultano are here all the time, saying that they're working on
a solution. But for some reason, it is better to make up shit and then believe
it than to actually listen to the people who talk.

~~~
rorrr
The problem has been out for a long time. Mahalo.com is a good example. Google
will lose a lot of money if they remove them from the search results. If their
engineers have been working, as you say, it's the slowest fucking project
ever.

What you don't realize is that since Google went IPO, it's not all about
building an amazing search engine. It's a lot about revenues and net profits.
There are lots of businessmen in charge who want to make lots of money
quickly, and couldn't care less about the quality of the search results.

~~~
spiffworks
>What you don't realize is that since Google went IPO, it's not all about
building an amazing search engine. It's a lot about revenues and net profits.
There are lots of businessmen in charge who want to make lots of money
quickly, and couldn't care less about the quality of the search results.

First of all, Google went public in 2004. I would like you to point me to a
study showing the steady decline in quality since that time. Who are these
'lots of businessmen' you talk about? One of the complaints about Google has
actually been that they prefer to have engineers in all sorts of roles
including management.

Now, since you've gone all conspiracy theorist, I have a theory of my own, I
hope you will excuse it. My theory is that the pro-Apple blogging community
has gone ape-shit over a perceived threat to iOS from Google, and are now
looking at them as 'The enemy', and this whole rash of blog posts maligning
Google is part of a larger 'Death by a thousand paper cuts' strategy to make
Google seem uncool, and therefore, as we all concur, dead. I also believe that
said bloggers are in fact undercover employees of Apple who get paid by the
number of times they mention Apple products in reviews of their competitors'
products, but that's just me being nuts.

~~~
rorrr
You seriously believe financial and strategical decisions are made by tech
people at Google? That's just silly. Look at glassdoor. They have finance
people, directors and executives of all kinds. They also have shareholders.
Those are the people who make important decisions.

And then you call me a conspiracy theorist, immediately followed by a pretty
insane conspiracy theory about Apple community. I'm not even an Apple fan.

~~~
spiffworks
First, let me just say sorry for the previous comment. Clearly, my attempt at
humour fell flat. I never intended to imply that you were an apple fan or
anything of the sort.

What I meant to say was that I've seen many reports which show a large
percentage of executives at Google to be engineers and programmers, although I
can't find a link right now. However, I seriously doubt that Google engineers
would have gone along without revolt down a slippery slope as the one implied
in the original article.

~~~
rorrr
It doesn't matter whether our theories are right or not.

The facts are

1) Google search results are full of spam.

2) Many of the spam sites have been reported (I reported a few myself) and
blogged/written about for years.

What can one conclude from that? I think there are 2 possible explanations

1) It's not in Google's financial interests

2) Google is failing at filtering spam

I doubt it's true, because simply banning the top 1000 spammy domains would
cut more than 99% of the search spam. It's something that an intern can do.

If Google wants a pure algorithmic solution instead, well, good luck to them.
I think until AI is smarter than humans, such solution doesn't exist. Spammers
can always find loopholes in algorithms.

~~~
TeHCrAzY
I doubt they are failing at spam, at least not in the common sense. My gmail
account received 1000's of spam emails a week, and it's a rare for one to pass
the filter, and even rarer for a genuine email to not.

It most likely a much simpler explanation: web results with content explicitly
generated for a specific target search string is simply getting closer and
closer to being actual genuine content, and thus harder and harder to
differentiate and filter.

------
navyrain
The adage "cock-up over consipiracy" comes to mind when reading this. What the
author implies is that Google is knowingly keeping spam sites in their results
so as to profit from them. Google is still the largest player in search by a
good margin, so I suspect it is more the case that spammers are targeting
their SEO spammy skills to Google's algorithm, and google just fails to
withstand the onslaught.

~~~
codeup
Google may be experimenting to find the right balance between spam and search
quality. "Right" meaning the most profitable.

~~~
brudgers
Not doing so would a breech of their fiduciary responsibility to their share
holders.

~~~
jarrett
Their duty to the stockholders is to do what's in the best interest of the
company. It's not a breach of duty for a company to forego short-term profits
if, in the management's judgement, those short-term profits can only be had by
damaging the company's long-term prospects.

Google stands to make the most money in the long run by being the preferred
search engine for the most people. Right now, they are. But that could change
if another search engine can convince people they deliver results with less
spam.

~~~
brudgers
You are assuming that balancing "ad spam" with utility is something new. One
only needs to consider how long Google's search results have been tailored to
the local from which they originate to get a sense of how long Google may have
been adjusting results.

------
codeup
Google profits more from spammy search results than from quality search
results? I don't think so. _Definitely not in the long run_.

~~~
samatman
I believe the point was more that Google profits both from spammy search
results AND from quality search results.

That's a conflict of interest, one well spelled-out in the original post: When
removing all spam (let's pretend it's possible) will negatively impact
revenue, even short-term, there's less motivation to do it.

~~~
Andrew_Quentin
why would it impact revenue if spam was removed, even temporarily?

Wouldn't the people who went to spamy sites go to the "good" sites and still
click on these sites thus still make google money?

Unless you are suggesting that the spamy sites are the only ones available and
if they disappeared searchers would not be shown results, I do not quite see
why removing spam would hit the bottom line.

~~~
gojomo
There are several mechanisms where marginal increases in spam could increase
Google's profits, as long as the effect isn't so bad people leave for another
alternative entirely.

(1) If the natural search result landing pages are slightly lower quality –
awful writing created via lowest-bidder/freelancer content mills – but the
AdWords landing pages remain higher-quality, due to more investment and
ability-topay filtering, then AdWords results become more attractive to users.
That's more clickthroughs and more revenues.

(2) If the first page of results reviewed by a user doesn't resolve their
intent, so they reformulate a more specific query, Google gets to display a
second, perhaps better-targeted set of ads on the second query – a second
chance at satisfying the user goal via a paid placement rather than an unpaid
result.

(3) If there are good natural results sites for a topic that lack ads, but
others can be created with roughly the same info that have AdSense, unless
Google specifically punishes sites for containing AdSense, the relative number
of AdSense sites with duplicate info will multiply over time. Even if they are
ever-so-lightly better (or 'as good' to low-literacy readers), and thus appear
'good' in some of Google's algorithms, they wind up wasting time in aggregate
with duplication, and dilute the traffic/community of original ad-free source
sites. More likely, though, they are just-enough-better in the cynical 'SEO'
dimensions to display other less profit-maximized sites, and thus may be worse
in the (not directly measurable) value-to-readers dimensions. Still, in the
short- and medium-term, the multiplication of such sites, and replacement of
AdSense-free sites with Adsence-drenched sites, makes Google money.

~~~
Andrew_Quentin
low-literacy readers?

11159 karma points does not give you the card to play mr arrogant gojo.

But thank you for your explanation.

I think the first point is quite clever and does make sense, but it has to be
balanced with the number of visitors that google may loose by allowing the
index to become spamy.

The second point seems to be what everyone is thinking of and perhaps the
strongest argument to suggest that google might not want to tackle spam. For
that argument to gain any credibility however it needs to be shown that other
search engines, i.e. bing, which perhaps do not have a conflict of interest
are doing better at showing quality results. Otherwise, I think it makes it
more likely that there are other reasons than conflict of interest.

As to the third point, the cases that it would apply to I would think are
miniscule in the grand scheme of search, not least because a very tiny
minority of websites does not use ads, excluding e-commerce.

Finally, I think a strong argument for google to tackle spam, and against the
proposition that google does not want to, is that google would make much more
money by converting more clicking visitors into customers for the adword
landing page. If spamy sites have a much lower conversion of clickers for the
adword customers, then cutting off these spammers would increase the
conversion rate, leading to higher payments for clicks to google by adword
users.

~~~
gojomo
<http://www.useit.com/alertbox/20050314.html>

Pretending these low-literacy users — 43% of the US adult population! — don't
exist, or can't be discussed frankly, is less respectful than seeking to
understand them.

It's quite likely that these people are late adopters of web search, and thus
their proportion of Google users is still rising. Consider, also, that there
is evidence that lower-income, lower-education users are more likely to click
on ads.

I've been perplexed when discussing content mills with Googlers how they
quickly assert (often in similar phrasing, suggesting a party line passed
along in internal or official communications) that reasonable arguments (or
very subtly, internal studies) imply many users appreciate the content mills'
writing.

Perhaps it's the low-literacy users who are happy to land on a mill site.
Either they can't tell the difference from a quality, authoritative treatment
— or maybe even the plodding, keyword-stuffed, repetitive writeups are
actually reassuring for slower readers.

There has to be a logical reason, other than Google venality, that such awful
spamglish writing decked in AdSense ranks so highly.

~~~
gamble
It ranks highly because the content farms target long-tail searches where it's
easy for marginal content to dominate the results. They are in a sense
arbitrageurs. They use software to located search terms where the revenue from
ads outweighs the cost of commissioning and ranking a low-quality page.

------
rit
This is Mutually Assured Destruction though. There's a delicate balance
between Google profiting from spammers running google ads and the profits
dropping as users trust google less and search elsewhere.

In the long run, it's a losing prospect for Google: short term profits at the
cost of long term reputation and trust doesn't make sense for a company as
large and as public as they are.

------
toddmorey
Google's service to me, the customer, is fast, useful search results. There is
no way they've confused their real customers for spam sites... that's just too
short-sighted and there are zero barriers to people switching if a competitor
with better results came along. But here's what I don't get: Lots of companies
implement customer feedback to improve their services. I can't decide why
Google hasn't done some sort of browser plugin (or other approach) that allows
you very simply thumbs down a site when it's a not useful result. Enough
negative feedback from real users and a site could be demoted in organic
search. Is that just too hard to build in a way that couldn't be easily gamed?

Edit: I know they experimented with voting arrows on the results page and I'm
curious why some variant of that hasn't been deployed more widely.

~~~
nzmsv
There's a Chrome extension for reporting spam:
[https://chrome.google.com/extensions/detail/efinmbicabejjhja...](https://chrome.google.com/extensions/detail/efinmbicabejjhjafeidhfbojhnfiepj)

No idea how seriously these reports are taken, and whether or not something
like the fake Mastercam blog would be considered spam.

------
Andrew_Quentin
Would have been interesting to learn if the "decent" websites converted better
into buying than the "spam" sites.

------
OmarIsmail
There's one thing that I really don't understand with this recent trend of
Google-bashing. The lack of distinction between the scraper spammers and
content farms like Demand Media. The author here lists some examples of
searches that have scraper results, but then goes and talks about Demand Media
without any specific DM examples. Why is that?

I'm not going to say that Demand Media content is world class, or even good,
but it's a far cry from scraper-spam and I don't think it's honest to group
the two.

~~~
gamble
I'm not sure there's any moral distinction between paying a human being to
rewrite Wikipedia or using a scraper to copy it. The key difference is that
the later is a copyright violation, which a company like Demand Media can't do
so long as they have ambitions toward legitimacy. If they could generate the
content algorithmically, they would.

~~~
OmarIsmail
I think this kind of thinking is quite dangerous. Again, you're trying to lump
in Demand Media/Content Farms with these scrapers and that can have dangerous
consequences, because where do you draw the line?

Even in an automated scraper scenario, things aren't clear cut. How much
processing needs to be done on source data before it goes from illegitimate
copying to value-add? Fundamentally Google is just a really really amazing
scraper system.

Same thing with content farms. At what point does original content go from
"paraphrasing Wikipedia" to providing useful summary information? Should
Wikipedia now hold a defacto monopoly on summary-style encyclopedic
information?

------
jeffreyrusso
I agree that it's important to distinguish between the different types of
content everyone is registering grievances against...

Scraped content and empty directory pages (that washing machine reviews page
that has nothing more than an H1 and some crummy auto-generated sentences)
vs... Mediocre human written content from sites like e-How and other Demand
media properties.

The first kind of content is pretty clearly spam, and doesn't deserve a place
in the index. Everyone can agree on that, and I'm perplexed as to why Google
hasn't taken more action against these types of sites. I have a hard time
believing they can't do it algorithmically.

The second type isn't as clear cut. Most of the readers here and on the cited
tech/search blogs know what this content is and where it comes from, and are
probably a bit more critical of it than the average user for that very reason.
I agree that these article directory pages are usually painfully mediocre, but
I wonder if they aren't "good enough" from the point of view of someone who
doesn't know better. After all, I havent seen much of an outcry about search
quality coming from the mainstream; it seems limited to HN and other tech
circles.

------
lysium
Wouldn't people that click on the ad links of the spam sites also click on the
ad links of the non-spam sites?

So what's Google's benefit to annoy everybody with non-quality search results?
I'd guess, the SEO is currently just too good.

~~~
Yzupnick
I think (as in I am making this up, but it makes sense to me) that spam sites
bring in more revenue to Google as they are specifically designed so that
people click on the adds. While non-spam sites are generally designed to
deliver content.

~~~
uxp
I am no SEO expert, or even amateur. I understand the concept of pagerank, but
I really don't care much for it, yet.

I agree with your hunch; however, bounce rate is a relatively bad thing. If,
using analytics, you see that most users navigate away from your page within
30 seconds, you probably don't have engaging content. How is that different
than the user clicking on an AdSense ad because the content on a spam site
isn't connecting emotionally or intellectually with the users? If anything,
Google should be giving a negative score to sites that have a huge percentage
of their users that actually click on ads to continue the search for what they
were originally looking for... though that is basically the opposite of their
business model, to have users click on ads.

------
shawnee_
Silly _robots_ , Google _Instant_ promotes spam, not relevant search results.
. .

~~~
lukev
Instant delivers the same results as with Instant turned off...

