On or around April 22nd 2017, this blog (andreweifler.com) was removed from the Google search index.
The impact was devastating.
(Daily sessions driven by Google Search to andreweifler.com between October ’16 and October ’17)
As you can see, some days I was getting upwards of 50 sessions per day from Google (organic search). When my site was delisted, that dropped to practically zero.
This is what a search for the term “andrew eifler” should look like:
(Screenshot of Bing search for “andrew eifler” taken 10-16-17)
Notice the deep linking, images – the whole nine yards. This is almost exactly what Google looked like before I was blacklisted.
This is what a Google Search for “andrew eifler” returned after my site had been removed from the Google index:
(Screenshot of Google search for “andrew eifler” taken 8-12-17; this is what this search result looked like from mid-April to mid-August)
Neither Bing nor Yahoo (which is powered by Bing) removed me from their index at any time – yet still, just getting removed by Google was crushing.
You can see the overall impact on my monthly traffic below.
(Total sessions on andreweifler.com between October ’16 and October ’17)
I first noticed that something was wrong in mid-June ‘17, about two months after this site had been removed from Google. I’m a casual blogger, so I don’t often look at site analytics, but when I did, I saw that overall traffic had been decimated. After poking around, I realized traffic from Google had dropped to near zero.
At the time, I had no idea why Google had delisted me, although my mind immediately jumped to the very critical article I had written about Google just a few months prior.
After consulting with my good friend Jordan (who generously hosts this blog on 3rd party servers), I realized that Google actually puts the onus on individual website owners to manage their profile with the search engine. Google provides a self-service web portal for webmasters to manage their presence on Google (located here). Upon visiting, I couldn’t help but see the irony in their friendly looking landing page:
Having recently been blacklisted, Google was expressing their desire to help me in a very counter-intuitive way.
After spending 10+ hours on the site, I realized that the reason Google had blacklisted me is because my sitemap.xml file had been hacked due to one of the many security holes in WordPress (which I use as a CMS). Hackers had inserted about 1000 false entries (not malware – just non-existing pages on andreweifler.com) into my sitemap.xml file. The Google crawler, after failing to reach these false entries, kicked me out of the index. No notice, no notification, nothing.
Now, trying to be objective for a moment – I can’t blame Google for removing my site from their index. I had a security breach on my site and it was causing their crawler a few pennies of extra bandwidth. Some notice would have been nice – but I understand. (Note: special shout out to Bing who stuck with me through the problem without delisting me.)
What happened next was the part that I found most frustrating.
It has taken a very, very long time to get re-listed on Google.
Even after fixing the problem, re-uploading my sitemap, going through and manually resolving the error notifications in the Google webtools portal – I was still was not listed. I checked daily for weeks. Going into their self-service interface – manually requesting Google crawlers to visit my page, and waiting. I spoofed Google’s crawler user agent with my browser to make sure my site was loading correctly for their bots, I re-re-uploaded my sitemap, I fiddled with a half dozen other settings – still nothing.
Then slowly… very slowly, my blog started to rise back up through the Google search rankings.
After about three weeks post-fix my blog appeared on the second page of search results.
After about six weeks I had worked my way up to the fourth spot (sans deep-links, of course):
Then I rose up to the third slot.
Now (today) almost three months after fixing the problem – my blog is still stuck in the 2nd slot for the search term “andrew eifler”.
I’m up to the top slot on the keyword “andreweifler.com” (although, the listing has no deep links).
Lastly – for the term “andrew eifler blog” – the appearance of the search result is almost back to normal (with just four deep links instead of the optimal six, and no pictures):
I think this is completely crazy.
And remember – this was just for very specific search terms that contain my name. Basically, people that type “andrew eifler” into Google trying to find my blog. This isn’t Google helping me reach a larger audience or sourcing new traffic for me, this is just Google helping people who already want to visit my site successfully navigate there. Even when people specifically want to find you, when your site doesn’t appear on Google – you might as well not exist.
Now, fortunately I don’t rely on my website for income, but imagine for a second that I did. With one action, Google cut my traffic by upwards of 70% and left it that way for about four months. Traffic has just recently started to pick back up (as I’ve slowly risen up through the rankings), but my traffic is still far below the levels I was seeing before the problem started.
As a fan of net neutrality, I found this whole situation to be pretty offensive. One of the things that makes the internet diverse and wonderful is small independent website publishers. With little more than time and ambition, the internet allows anyone to publish their thoughts and reach an audience as large as the fortune 100 media conglomerates. However, in this case, Google is making it very difficult for me to realize that vision.
In the end, I don’t really blame Google for their actions. They acted rationally and in their own self-interest. What else can we really expect corporations to do? The commentary here for me is just how powerful Google is.
I often like to think that the internet is a global democracy where everyone participates evenly no matter your wealth, race, religion, gender, or nationality. There are many parts of the internet that are like that, which makes me very happy. However, when it comes to website traffic acquisition, the internet is not a democracy at all. It is a dictatorship, and Google is the dictator.