On or around April 22nd 2017, this blog (andreweifler.com) was removed from the Google search index.

The impact was devastating.

(Daily sessions driven by Google Search to andreweifler.com between October ’16 and October ’17)

As you can see, some days I was getting upwards of 50 sessions per day from Google (organic search).  When my site was delisted, that dropped to practically zero.

This is what a search for the term “andrew eifler” should look like:

(Screenshot of Bing search for “andrew eifler” taken 10-16-17)

Notice the deep linking, images – the whole nine yards.  This is almost exactly what Google looked like before I was blacklisted.

This is what a Google Search for “andrew eifler” returned after my site had been removed from the Google index:

(Screenshot of Google search for “andrew eifler” taken 8-12-17; this is what this search result looked like from mid-April to mid-August)

Neither Bing nor Yahoo (which is powered by Bing) removed me from their index at any time – yet still, just getting removed by Google was crushing.

You can see the overall impact on my monthly traffic below.

(Total sessions on andreweifler.com between October ’16 and October ’17)

I first noticed that something was wrong in mid-June ‘17, about two months after this site had been removed from Google.  I’m a casual blogger, so I don’t often look at site analytics, but when I did, I saw that overall traffic had been decimated.  After poking around, I realized traffic from Google had dropped to near zero.

At the time, I had no idea why Google had delisted me, although my mind immediately jumped to the very critical article I had written about Google just a few months prior.

After consulting with my good friend Jordan (who generously hosts this blog on 3rd party servers), I realized that Google actually puts the onus on individual website owners to manage their profile with the search engine.  Google provides a self-service web portal for webmasters to manage their presence on Google (located here).  Upon visiting, I couldn’t help but see the irony in their friendly looking landing page:

Having recently been blacklisted, Google was expressing their desire to help me in a very counter-intuitive way.

After spending 10+ hours on the site, I realized that the reason Google had blacklisted me is because my sitemap.xml file had been hacked due to one of the many security holes in WordPress (which I use as a CMS).  Hackers had inserted about 1000 false entries (not malware – just non-existing pages on andreweifler.com) into my sitemap.xml file.  The Google crawler, after failing to reach these false entries, kicked me out of the index.  No notice, no notification, nothing.

Now, trying to be objective for a moment – I can’t blame Google for removing my site from their index. I had a security breach on my site and it was causing their crawler a few pennies of extra bandwidth.  Some notice would have been nice – but I understand.  (Note: special shout out to Bing who stuck with me through the problem without delisting me.)

What happened next was the part that I found most frustrating.

It has taken a very, very long time to get re-listed on Google.

Even after fixing the problem, re-uploading my sitemap, going through and manually resolving the error notifications in the Google webtools portal – I was still was not listed.  I checked daily for weeks.  Going into their self-service interface – manually requesting Google crawlers to visit my page, and waiting.  I spoofed Google’s crawler user agent with my browser to make sure my site was loading correctly for their bots, I re-re-uploaded my sitemap, I fiddled with a half dozen other settings – still nothing.

Then slowly… very slowly, my blog started to rise back up through the Google search rankings.

After about three weeks post-fix my blog appeared on the second page of search results.

After about six weeks I had worked my way up to the fourth spot (sans deep-links, of course):

(Screenshot taken 8/17/17)

Then I rose up to the third slot.

Now (today) almost three months after fixing the problem – my blog is still stuck in the 2nd slot for the search term “andrew eifler”.

(Screenshot taken 10/15/17)

I’m up to the top slot on the keyword “andreweifler.com” (although, the listing has no deep links).

(Screenshot taken 10-15-17)

Lastly – for the term “andrew eifler blog” – the appearance of the search result is almost back to normal (with just four deep links instead of the optimal six, and no pictures):

(Screen shot taken 10-15-17)

I think this is completely crazy.

And remember – this was just for very specific search terms that contain my name.  Basically, people that type “andrew eifler” into Google trying to find my blog.  This isn’t Google helping me reach a larger audience or sourcing new traffic for me, this is just Google helping people who already want to visit my site successfully navigate there.  Even when people specifically want to find you, when your site doesn’t appear on Google – you might as well not exist.

Now, fortunately I don’t rely on my website for income, but imagine for a second that I did.  With one action, Google cut my traffic by upwards of 70% and left it that way for about four months.  Traffic has just recently started to pick back up (as I’ve slowly risen up through the rankings), but my traffic is still far below the levels I was seeing before the problem started.

As a fan of net neutrality, I found this whole situation to be pretty offensive.  One of the things that makes the internet diverse and wonderful is small independent website publishers.  With little more than time and ambition, the internet allows anyone to publish their thoughts and reach an audience as large as the fortune 100 media conglomerates.  However, in this case, Google is making it very difficult for me to realize that vision.

In the end, I don’t really blame Google for their actions.  They acted rationally and in their own self-interest.  What else can we really expect corporations to do?  The commentary here for me is just how powerful Google is.

I often like to think that the internet is a global democracy where everyone participates evenly no matter your wealth, race, religion, gender, or nationality.  There are many parts of the internet that are like that, which makes me very happy.  However, when it comes to website traffic acquisition, the internet is not a democracy at all.  It is a dictatorship, and Google is the dictator.

The Power of Google: My Story of Being Blacklisted
Tagged on:         
  • Bren Eifler

    Thanks for taking the time to share this Andrew! It certainly underscores the need for security on WordPress blogs, and to maintain a close watch on 404 errors. Some ideas:

    1) Do you have your email in Google Search Console (formerly Webmaster tools)? I was working with a site recently that had a spike in 404 errors and alerts were sent to the email address listed in Search Console. There was an option to review and fix the 404 links, and the email had a notice that said if prompt action was taken the site would not be affected. I pasted the full text here: https://pastebin.com/p2j6kn67

    2) Do you have a security plugin that can alter the admin login and block IPs that are doing brute force login attempts? – After installing iThemes Security on some client wordpress sites and altering the admin login I stopped getting blocked IP notices. (E.G. Rather than /wp-admin you use something custom like /xyz-backend-admin). Here’s the plugin: https://ithemes.com/security/

    3) It’s possible to setup alerts from Google Analytics when 404 errors spike so you’re not just relying on emails from Search Console. You can setup a filter in Gmail to forward these alerts to your phone via your phone number’s email so you don’t miss the notice. (Does anyone else got 500 emails a day?) Here’s how to find your phone’s email address: https://www.techwalla.com/articles/how-to-find-out-my-cell-phones-email
    and here’s an older article about setting up GA 404 alerts. It should at least point you in the right direction if it’s too outdated: https://www.seroundtable.com/google-analytics-404-alerts-17220.html

  • Steve Truxal

    Awesome article Eif! Great example of the power google has as an aggregator and divisiveness it can cause with pubs. Google has this not only with search but also with chrome and android.
    Definitely one of my favs of your blog.

  • Bren,
    Super helpful! I have “wordfence” installed and have been monitoring recently for brute force attacks (of which there are many!). Good idea on changing the admin URL – I will try that. I’ll also make sure google has my email so I can get alerts. Kind of a pain to have to actively manage so much of this stuff – but that how software goes sometimes I guess.

    Thanks for the notes!

  • Thanks Trux! It’s kind of amazing – I’ve had many people reach out privately and on Facebook saying they had a similar experience. Makes you wonder how much work (just in terms of raw hours) folks are putting into appeasing Googles somewhat specific requirements (and managing the downside when Google punishes them for not doing exactly what Google wants).

  • Mike

    Welcome to 2005, and the news that Google is dominant in search! Usually I love your writing Eifler and I really love most posts that bash Google, but this time the CTO and “anti-fraud” guy in me just comes out and I gotta say….

    1) You take zero accountability for being a shitty web admin. Had you registered your domain with Google Webmaster Tools before, you would have gotten an email notification that you’d been hacked. “Wow, wordpress sucks, thank god Google told me.”. By the way, your domain name is registered anonymously. So… you didn’t tell them who you were up front and have an anonymous domain and they were supposed to email you? How exactly?

    2) Your lackluster performance as a Webmaster resulted you helping the bad guys. You don’t talk about the dark side of SEO. When people hack wordpress, they modify pages and insert links, to modify search listings. You mentioned one thing Google found, did you scour your blog for other malicious behavior? Could you be hosting malware? How else did the bad guys use your blog to influence Google? Is Google really the bad guy here, or is your passive support of the dark side of the web?

    3) Your title is total click bait. More accurate would have been — “The story I learned that WordPress sucks and Google’s job of keeping a clean search index is incredibly difficult”

    Long story short… running a web server requires taking some safety precautions. It’s like tinder dating. You better use protection if you’re going to have sex. You went out, didn’t use protection, got an STD and didn’t get tested. Google figured it out and so didn’t send you any more dates until you could take some drugs. Then, Google waited a few weeks to make sure you were definitely clean before referring dates to you again… but… is still a bit cautious, because, … you’ve shown yourself to be willing to conduct unsafe behavior.


    PS: You were probably generating impression fraud as well.

  • Mike – thanks for the note and for reading (even if, as you state, I drew you here under false pretense). I don’t have much defense on the topic of my web admin skills. I think your critique is accurate and i’m trying to be better (although the job seems to get harder each year as the bad guys get more clever – and/or wordpress gets more security compromised). As far as I know I’m not currently hosting any malware (i had a brief problem with malware in 2015, so I think i know what to look for). This offense was pretty narrowly limited to having false entries in my sitemap.xml (links that led no where and all generated 404 errors).

    I think the point i’m trying to make isn’t that I should get a pass for being a crappy web admin, but rather – should Google get to decide how “compromised” a site is before it’s effectively removed from the web? I think my offense here was pretty minor and somewhat easily fixed after i realized what it was. It also didn’t do any consumer damage (no actual visitors were using my sitemap.xml to navigate my site, only bots).

    So, sure – maybe Google, after detecting my minor infraction, doesn’t send me any more users for broad or generic search terms (e.g. the term “How does google behave like a monopoly?”) – but for people who search my name? That’s pretty clear intent that they’re just using Google as a navigation tool rather than a tool to discover new content.

    Practically speaking – Google Search serves two functions today:

    1) A tool to find new content: e.g. Searching for “How to get cat pee out of a leather couch” (long story)
    2) A tool to navigate to where you already know you want to go, but don’t exactly remember the URL, or you’re too lazy to type it in properly

    Because Google has become the de-facto way to navigate to where you already want to go – does that make it more like a public utility than a private software?

    In trademark law, if your brand becomes the de-facto way to refer to something (e.g. Kleenex) you don’t get to keep the same level of trademark protection – that term has ceased being yours – it now belongs to everyone. Maybe a little bit of a stretch, but should some of the same logic be applied to Google’s role as a navigation tool?

    What i’m really trying to say here is – yes – i caught an STD, but it was a really minor one. Sort of like i showered at the gym without sandals and got some toenail fungus. It’s not very pleasant to look at, but it’s definitely not syphilis.

    Should Google get to decide my toenail fungus disqualifies me from dating? (Using your analogy where Google is like Tinder). What if someone types my name into Tinder looking specifically for me… does Tinder show me, or do they say “sorry that person doesn’t exist”? (disclaimer – i have no idea how Tinder works). I get that Tinder as a platform wouldn’t send me new strangers or recommend me to people who don’t know me, but people searching for me specifically – if all i have is toenail fungus, i think i should still get to exist at that level.

    This is not to say that i’m not at fault, but more that Google gave me a serious penalty for a minor infraction. Not to evoke a slippery slope, but if Google gets to decide whether or not i exist on the web, then what’s to stop them for removing me for other minor infractions… misspellings? Poor grammar?

    And even if you think my penalty was justified – should a private company whose mission is to maximize their own shareholders value be able to make these kinds of decisions? The Tinder example is entertaining because Tinder is a fringe way to meet a companion – and there are many other ways to accomplish this goal. But consider for a second if Tinder was the ONLY way to meet a potential relationship (or they had 95%+ market share) – would we feel comfortable giving that much control over something so important to a single private company regardless of how they handle that responsibility?

    I’m not sure if I have a good solution here, more of just a commentary on the world we live in.

    PS. Let me know if you have any recommendations (resources, courses, etc.) on how to be a better web admin and prevent this stuff in the future :)

  • I recall being at an ad agency — selling SEO services — when out client got blacklisted around ~8 months after we took over the account from the previous agency. Fortune 100 client blacklisted.

    The restoration process was a fair bit more complex (involved disavowing links, identified by scraping backlinks). What was interesting in that case is that — while some previous SEO agencies had engaged in minor SEO “hacking” — the actual offenses that Google kicked back (courtesy messages that included links that met their infraction criteria) had nothing to do with the client’s site or the client’s behavior. It was third party organizations that were trying to leverage the good name // domain power of the client.

    I agree that the amount of information Google provides — even to pretty important companies! — is not a lot. Restoration can be troublesome.

    Re: the security issues, I’ve been thinking more and more how much of a shame it is that WordPress beat Moveable Type. Static site generators are potentially a much better solution than WP, because they don’t come with the security risks.

  • Totally agree with you on wordpress point. I feel like every few months they release another version with even fancier photo-sharing features, e-commerce, etc. (stuff that i never use) and with each update comes a new batch of security problems. Words, pictures… that’s all i really need.