About a week ago, I mentioned that I would be trying out Microsoft’s Bing as my default search engine in my main browser (Firefox 4.0) on my main computer, for at least seven days. So, the week has passed… what are my impressions of Bing, when I had used Google for years?
Just a brief note: I’ve just updated my script for graphing referrals from Google, Bing and Yahoo!. I intended it to reset each first day of a month, so one can see whether each month is “better” than the one before… however, it didn’t handle stats without any referrals at all from those search engines, which is likely to happen every time the current month changes. It’s fixed now.
Being still less than a month old, it’s normal that I keep looking at this blog‘s stats quite often, and one of the most interesting bits — especially if you care about SEO — is the top search keyphrases list. I use both AWStats and Google Analytics for this site, and, concerning the former, I had been curious for a while about why the top search leading people to my blog was apparently for “chromebar“.
Now, I’m pretty sure I had never mentioned that term on this blog, so I thought AWStats might have been doing something wrong. A quick grep on my logs, and where “chromebar” came from became obvious: it’s from the StumbleUpon add-on for Google Chrome. The whole combination was necessary; StumbleUpon users with other browsers weren’t triggering anything like this.
Now, here’s what the “referrer” part of a hit from StumbleUpon, using Chrome with the SU add-on, looks like:
"https://www.stumbleupon.com/toolbar/litebar.php?device=chromebar&version=chromebar%202.9.8.1&ts=1301274701"
While a hit’s referrer from SU with another browser looks like this:
"https://www.stumbleupon.com/refer.php?url=http%3A%2F%2Fwinterdrake.com%2Fbad-comic-panels-3-its-a-membership-card-in-a-subversive-communist-front-organization%2F"
See the difference? The Chrome version has a query string: ?device=chromebar&version=chromebar%202.9.8.1&ts=1301274701 , while the other one doesn’t. And “chromebar” appears at the beginning of that query string.
Now, StumbleUpon is listed as a “search engine” in AWStats, in a file called lib/search_engines.pm , and that file optionally specifies which part of a query string (e.g. “q=“) from the referrer is the actual search. For some search engines, however, no such part is specified — meaning that they don’t provide it in the “referrer” part of the hit. Such is the case with StumbleUpon, where the query string part is an empty string.
But there seems to be a bug (or maybe it’s an intended feature?) in AWStats here: if no part of a query string is specified, and yet there is a query string, AWStats seems to use the first part it catches. As you can see above, that was ?device= , and it was always set to chromebar.
The easiest way around this problem (I don’t know if the AWStats authors will consider this a bug or not; I’ll try to report the problem in the near future) is to edit lib/search_engines.pm and add a non-existing query string part to StumbleUpon. Open that file with a text editor and look for this line:
'stumbleupon','',
and change it to:
'stumbleupon','qwerty=',
Presto! No more “chromebar” entries in your stats in the future. (It won’t delete current entries, though, unless you clear your AWStats cache files for that month and generate new stats.)
After my last couple of posts looking at the comparative hits between Google and Bing and investigating the reasons for the huge discrepancy (Bing bringing me a puny 2-3% of Google’s hits), I’m still curious about this subject. So far, it appears that Bing brings me 2-3% of what Google brings because it has 2-3% of the users Google has… which seems too low a number to me. Yes, “google” as a verb has entered everyday language (“I’ve just googled for it”), and it’s the default browser on most non-Microsoft browsers (though I’ve heard that a couple of them changed their defaults to Bing recently), but, well, Internet Explorer is still widely used (unfortunately), and it certainly defaults to Bing.
Further investigation may be in order, but, unless both companies reveal their search volume, there’s no way to know with certainty. We can look at other statistics such as Alexa, which are certainly far from definitive, and which lists google.com as #1 worldwide (and remember that there are many other “international” Googles), and Bing as #20 worldwide (#14 in the US). Google (again, just the “.com” version) is listed as having a “reach” of 49.05000, while Bing supposedly has a “reach” of 4.77600. This puts Bing as 9% of Google, but if you add in the international Googles, the 2-3% actually sound like a plausible result.
Still, all of this, and we haven’t actually asked an important question: how good actually is Bing, for users? Are people — even Internet Explorer users — using Google out of habit / popularity, or is Bing in fact a worse search engine — say, providing worse, less relevant search results, giving you exactly what you want at the #1 position less often, or not even finding something that exists when Google does? Only one way to find out: my 7-day-long Bing trial. (Actually, 8-day-long, since I started it yesterday, but I think that ending on a Saturday is nicer than on a Friday).
Until April 2, I’m going to be using Bing as the default search engine on my main browser (Firefox 4), and I’ll post my impressions after that time. If you’re interested in SEO, or simply curious, I’d suggest you try the same — learning is always good, isn’t it? 🙂 To do it in Firefox:
about:config, accept the so-called “risk” if it’s your first time doing so, search for keyword.URL, and change it to “https://www.bing.com/results.aspx?q=“, without the quotes. Yes, it ends with an equal sign. To switch back to Google, change it tohttps://www.google.com/search?ie=UTF-8&oe=utf-8&q=“. I believe this is unchanged if you’re using Firefox 3.x instead of 4 (though you really ought to upgrade, Firefox 4 is great).In my previous post, Search engine hits: Google vs. Bing (part 1), I showed how three of my sites — a recent US-hosted blog in English, a popular Portugal-hosted forum in Portuguese, and a search engine-weak US-hosted aggregator in English — had a much greater proportion of search engine hits from Google than from Bing — about 100 Google hits for each 2-3 Bing hits. Naturally, the next step would be to investigate whether this is just a matter of quantity — in other words, Bing has 2-3% of the users Google has –, or whether there are other factors affecting this, such as the three sites having better rankings in Google than in Bing. This, then, is the subject of this post.
The method I’ll be using is the following: for each of the same three sites used in the last post, I’ll be searching for a number of phrases that I think should, reasonably, lead a user to the site. I won’t look at my stats to get the top phrases from there; as most hits come from Google, I’d just be using phrases that I already know work for that engine. And, for each phrase, I’ll note the position my site came in — it at all.
Remember that a lower number is better. Also, a caveat: I am not really testing how “good” each search engine is, only how good it is at sending people to my sites from appropriate queries. I have, however, avoided “gaming the system”; the queries are reasonable and not “custom-designed” to lead people to where I want, and the pages / sites are relevant to them.
Note: all queries were as written below; i.e. without quotes.
For this very blog, I searched for things I know I posted about — but not the exact titles. Instead, I tried strings that I myself would use if searching for those subjects:
[table id=1 /]
For that MyBB forum with mostly Brazilian and Portuguese users, I didn’t go for particular posts, just some strings that a Portuguese-speaking person could possibly use, and I counted the first forum post that appeared. I tried to diversify; hence the “dragonball”, “dragon ball” and “dbz”. I also tried both Google.com and Google.pt, which seem to prioritize pages differently (perhaps because that site is hosted in Portugal); Bing.pt, however, was exactly the same as Bing.com in terms of results, only with the user interface in Portuguese.
[table id=2 /]
And, finally, my FeedWordPress-based blog aggregator. As I explained in the previous post, PA doesn’t ever show posts individually; there are only the main page (and, page 2, page 3, etc.), author pages, and archives by date. On all of these, the post links go to the original blog posts; therefore, search engines treat all of PA as non-original content (and this is by design); because of that, searching for something in a particular post is mostly useless. My alternative was to search for more general terms:
[table id=3 /]
(1) category pages that included the post were indexed, but not the post itself
(2) not in the first 100 results, at least
For Winterdrake, Bing was a pleasant surprise, winning 5.5-0.5. Not only did it apparently index every post in the site, even some very recent ones, but it also returned a first page result for everything I threw at it — including two “number ones” that Google doesn’t even seem to currently index, for some reason. Therefore, a Bing user actually has a greater chance of getting to Winterdrake — assuming he or she is interested in something on it — than a Google user. In other words, Bing wins on “quality” — but still only sends me 3% of the users Google does. Are there really so few people using Bing? I had figured something like a 10-20% market share, not 2-3%…
As for the few non-indexed posts on Google, when the category pages have been indexed after the posts were written, I assume the reason for that is the fact that Winterdrake is still very new, and has almost no incoming links. In a couple of months, I expect that new posts will be quickly indexed by Google, as Bing already does. Still, there’s currently an advantage for Bing here.
DragonBall-PT gets mixed results. Compared to Google.com, Bing wins 3-2. Compared to Google.pt, however, Bing loses 1.5-3.5, meaning that “which” of the Googles people use is an important factor. Still, if we assume that most people use “their” national Google (I don’t, but I’m weird…), Google wins this time, and that was reflected in the results on the last post: Bing sent me only 2% of the people Google (all of them put together) did.
Finally, for Planet Atheism, Bing is a big disappointment, losing 0-3 to Google. 🙁 Either because it “hates” non-original content to a much greater degree that Google does, or because it is less “intelligent” in adapting to / understanding similar terms, it failed to show PA in the first 100 results for two thirds of the search strings I used, while Google did great with two of them, and even had a reasonably decent result for an incredibly competitive, 1-word term such as “atheism”.
Another thing I noticed is that most of the first couple of results pages in Bing for my atheism-related queries actually showed anti-atheism blogs (“atheism sucks”, “stop atheism”, and several others). That’s like searching for “democratic party” and getting mostly Republican propaganda pages. 🙂 I don’t want to sound paranoid… it’s probably just an algorithmic thing, instead of there being a less-than-honest religious programmer at Microsoft… 😉
EDIT: I’ve just noticed that the three tables above look quite bad on the RSS feed. If you want them more readable, please see the original blog post. I apologize for the inconvenience.
Having been interested in SEO for a long time, one of the things I naturally do is look at how many people are coming from the several search engines. While many people seem to care only about hits from Google, I think that 1) monopolies are bad, and 2) precisely because so many people optimize only for Google, there may be an “untapped market” of people who use other search engines. So, I’ve been doing a little investigating, and here’s what I found. Today, I will only be comparing Google with Microsoft’s Bing; I may look at Yahoo! or Ask.com some other time.
So, let’s begin with this very blog, Winterdrake. Hosted in the US, it’s a very young site, having only launched on March 3, 2011… which means that it’s just 22 days old today. Naturally, one shouldn’t expect a lot of search engine hits on such a recent site (and there aren’t), but it allows us to look at yet another interesting factor: which search engine is quicker to index and send hits to new sites? Let’s look at a (terrible-looking; I’m no graphic designer, and it shows) chart:
Not very good in terms of Bing hits, is it? Let’s look at another, more popular and established site: DragonBall-PT, a Portuguese-language forum, hosted in Portugal, launched in 2007, with mostly Brazilian and Portuguese users, which has a few thousand hits per day:
As you can see, while, as an absolute number, there are a lot more hits from Bing (a little below 2000), the proportion in relation to Google is even worse.
This can be explained in several ways. First, Bing seems to be more optimized for the US and/or English language sites, while Google appears to be more “international”. Another possibility, of course, is that almost no one in Portugal or Brazil uses Bing. But, to discount the “optimized for the US/English” factor, let’s look at a third site, one that is 1) relatively old, and 2) in English, and hosted in the US.
Planet Atheism is the world’s top aggregator of atheism-related blogs. It’s (by design) not good for search engines, as all its content is duplicated from the member blogs (and every member either asked to join or accepted an invitation; no one is aggregated there without permission), and PA doesn’t show individual posts ever. Click on a post title, and you are taken to the post on the original blog. In other words, PA is by design “below” every single one of its blogs in terms of search engine positions; most hits come from people who actually search for PA itself, or for “atheist blog(s)” or something like that.
On the other hand, it’s an established site (launched in 2006), hosted in the US, and in English. Let’s see how it looks like:
Ouch. Not very good for Bing, is it?
So, we’ve shown that Bing is sending about 2-3 hits for every 100 sent by Google, a very small percentage. But now for the million dollar question: why? Logically, there are two possibilities:
Note that, in either case, Google has a lot more users; the question is whether, for the same number of users, Bing is performing worse, better, or the same. And I think we can find out… but this post is getting a bit long, so I’ll leave that for Part 2.
From my previous post, AWStats tip: creating static pages (and why it’s a good idea):
for instance, I’m currently using MRTG to plot a graph of Google and Bing referrals, using the AWstats-generated static pages as input
So… anyone curious? 🙂
Note: again, beware of word wrapping below. I’ve added an empty line between any “true” line of text; if you see two or more lines together, it’s supposed to be just one.
cat /root/bin/winterdrake-se.sh
#!/usr/local/bin/bash
GOOGLE=`grep \>Google\</a\>\</td\>\<td\>
/var/www/htdocs/AWstats/awstats.winterdrake.com.html | awk -F"<td>" '{print $2}' | cut -d "<" -f 1`
if [ -z $GOOGLE ]; then
GOOGLE=0
fi
echo $GOOGLE
BING=`grep \>Microsoft\ Bing\</a\>\</td\>\<td\>
/var/www/htdocs/AWstats/awstats.winterdrake.com.html | awk -F"<td>" '{print $2}' | cut -d "<" -f 1`
if [ -z $BING ]; then
BING=0
fi
YAHOO=`grep \>Yahoo\!\</a\>\</td\>\<td\>
/var/www/htdocs/AWstats/awstats.winterdrake.com.html | awk -F"<td>" '{print $2}' | cut -d "<" -f 1`
if [ -z $YAHOO ]; then
YAHOO=0
fi
echo $(($BING + $YAHOO))
uptime | awk -F"up " '{print $2}'
uname -n
EDIT in April 1, 2011: fixed the script so that it deals with non-existing entries (typically on the first day of the month).
By the way, this is a FreeBSD box; that’s the reason why bash is in /usr/local/bin. In Linux, it’s probably in /bin.
The mrtg.cfg file is mostly standard: it invokes the script shown above, and labels “I” as hits from Google, and “O” as hits from Bing and Yahoo! put together.
If you want to see what the result looks like, look here.
If you’ve been paying attention, yes, both values will reset to zero the first day of every month. That’s exactly how I want it: I want to see if every month it “rises” faster than the month before, or slower, or whatever. 🙂
I have been blogging with WordPress since 2004, and during all these years I’ve maintained several blogs; some of them exist to this day, some were eventually shut down, and some “lie dreaming until the stars are right.” I hadn’t started a new one in quite a while now, and so, when I created Winterdrake a couple of weeks ago, I did what I usually do in terms of which plugins to install: instead of just copying the /wp-content/plugins/ directory from an existing blog, I start afresh. I see what plugins are the most popular and/or have the most positive votes, both for each requirement (e.g. one to edit your comments, one to show related posts, one for caching, one to improve SEO, and so on) and in general: by doing the latter, I can (and do) find something extremely useful that I didn’t even know existed.
The result is that this blog currently has an almost completely different set of plugins from my other blogs (which will eventually be changed so they use most of these ones, too), and I’d like to share a few of them, that old WP bloggers such as myself may not even know exist:
Hope you find this useful. Two weeks ago, the only one of these I knew about was W3 Total Cache (and that one I’m already using on my other blogs), and I’m in love with the other three right now. 🙂
I’ve just read a thought-provoking article, Has SEO Peaked?, which links to another article called Someday, the sun will set on SEO — and the business of news will be better for it, and I’d like to share my thoughts on that subject.
One of the questions raised in the above articles is this: if search engines were perfect, then everyone could use the “I’m feeling lucky” option on Google (which takes you immediately to the search’s first result), or its equivalent on another engine. After all, the first result would be the best one, wouldn’t it? The fact that very few people ever do that is proof that people don’t trust search engines, because they often fail. Either they put a bad quality page on top, or one that isn’t really what the user was searching for.
But what if they didn’t? (more after the break)