Home arrow Banned Website - 1
Banned Website

 

 

Banned Websites and Search Penalties


Part 1: [this page] Introduction to search engine bans / search penalties. Effects of penalties and bans. The two main classes of ban.
Part 2: Search penalties continued. Malware warnings, malware bans. Duplicate content filter. Search engine guidelines. Further resources.
Part 2a: More on search penalties - how to tell if a site is banned; search penalty myths.
Part 3: Search penalty resolution by A3webtech. Advice on avoiding penalties.
Part 4: Search penalties: an overview
 

Banned sites (at work or college) - How to view websites banned at work or college
Banned from a website - How to get around being banned from a site


Part 1
Website bans - Introduction
The two types of search engine ban

Terminology
Search Penalty definitions
Search engine aliases
Penalties from the Big 3

Why do search engines ban websites?
Effects of a search penalty
Effect of a Google ban 

SP (search penalty) levels
Major hosting issues
Algo v manual penalties

The 'post-mortem' penalty

Introduction: website banned by search engines

Welcome to what is considered to be the Net's most comprehensive resource on search penalties and all forms of website ban or browser censorship. Thank you to all who have contributed and commented by email. Thanks also to the senior search engine staff who have praised this section.

At one time or another, many sites get largely or partially banned by one or more search engines - although these events are much more accurately referred to as penalties. Because this is such a common phenomenon now, we think it probably happens to around 5% of websites at some stage, in varying measures of severity. On the other hand, what might be referred to as a 'total' ban is rare (the various definitions of bans are examined later on). It seems that this area is becoming increasingly important in icommerce and web community life.

This page is designed to present the fullest possible resource on search engine ban and penalty types and their effects. Note however that, like the search engines, we do not discuss possible causes or cures in detail, for the reasons stated at the end. In our case, these boil down to the fact that if you don't know enough to keep from being banned, you probably don't know enough to get your website cleared safely, and how to run clean and stay safe in the long term, without assistance.
 
Culpability: site owners or search engines ?
The vast majority of websites need to put in place much higher standards of housekeeping; this can hardly be achieved at their current knowledge levels if they have received a search penalty. There is another aspect, which is attitude: website owners in general need to show more willingness to read, understand and comply with search engines ToS or guidelines. Terms of Service may vary slightly between the search suppliers; please see the section on this lower down the page. There is a counter argument, though, that the ToS in general are too sketchy, especially with regard to the technical issues that can result in a search penalty and with little or no blame attached to the site owner.
 

Search engine bans are of one of two types:

1. An SP or search penalty
This is the most common type of ban, and in effect is simply a SERPs demotion - it demotes a website's position in the search results. The majority of bans are of this type, and have various levels of severity.

Since around 240,000 websites seem to have had a malware warning ban (this figure from stopbadware.org - see Resources at foot of page), there must be millions who have had search penalties. That statement is made because we see many more sites with SPs than with malware warnings.

2. A malware* ban 
Strictly speaking this type is better termed a malware warning (or badware warning as some of the major SEs prefer), rather than a penalty or ban, since the site in question is often not removed or demoted. The ban does not affect the search positions, or page rank; however, even though a site receives its normal search result positions (for example at Google #5), the result cannot be clicked-through to the site as per usual - there is no live link to the site. In addition, a warning is placed in the result that says "This site may harm your computer." Clicking the result takes you to a warning ('interstitial') page, and not the site. You have to copy & paste the text of the URL - they want you know the site is risky, and not be guilty of sending you there unknowing.

It also allows a site owner to see the search result and take action. If the site was removed form the results, it is more likely that the owner would not be aware of the problem.

* Malware, aka scumware or badware, is software that makes changes to a user's system, or leaks information to third parties, without the user's consent. A popular description might therefore be 'evil' software. It can comprise simple applications such as trojans that collect information; or diallers that 'phone home'; or small applications that enable other larger ones to be installed unknown to the user. Malware that stays resident on a server can have many purposes, none of them beneficial to visitors; one function being to install unwanted software invisibly on visitors' computers.

Website ban terminology 

Strictly speaking, in terms of the accuracy of the word, and the fairly moderate level of action in most cases by the search engines in response to non-compliant actions, we should not be using the term 'ban'. A more accurate term would be something like 'search results penalty', 'search results restriction' or 'search position demotion'. This would be a better way to describe a search engine's graduated response to non-compliance, and in fact their response is generally at the tolerant end of the scale rather than the strict. However, the power of modern parlance being what it is, ban it is, and ban it will no doubt stay. Alternatives might include blocked, blacklisted, deleted, or de-indexed.

Therefore it is more accurate in most cases to refer to these events as a Search Results Penalty or Search Penalty - or SP - and this is the term we will use whenever convenient. There are indications that some search engine staff use similar terms. A malware ban or warning is a different type of reaction.

More search penalty terms defined

AI
artificial intelligence - the SE computer + algo
algo
the algorithm that controls a search engine's index and results
deletion
a site removed from the index by human intervention
manual adjustment
a demotion allocated by human intervention
penalty
an automated demotion of search position
PR
PageRank, the value of a page calculated by the number of backlinks plus a traffic factor
SE, SEs
search engine/s
SE spam
a catch-all term meaning junk webpages seen in the search results
SERP, SERPs
search engine result page/s
SP
search penalty
Spider
a search engine's page-indexing bot. There can be several on your site from one SE at any one time. Spiders are a welcome visitor.

TBPR
Toolbar PageRank
TOS, ToS
Terms of Service - search engine guidelines
website ban
used as a generic term for any kind of penalty or warning

Search engine aliases

Search engines, like any business, change their main business name occasionally; or use different names for distinct divisions of their enterprise; or have connections to other similar businesses. Here are some connected names and variants:

Yahoo! = YPN, Overture. Linked to: AllTheWeb, AltaVista.
Bing (ex-Live Search) = MSN, Live.com, Microsoft Publisher Program.
Ask.com = Ask Jeeves, Teoma. Linked to: Lycos, Hotbot.

Google, Yahoo, and MSN search penalties

From a financial perspective it only makes sense to consider penalties applied by the Big 3 search engines. This is because other SEs may only affect under 5% of site traffic in total; and their reaction to infringements of guidelines is likely to be the same as or less severe than that of the Big 3. In addition, many other SE results are simply based on those of the Big 3 - those of their affiliate SEs, for example, and of the meta search engines.

Further, it makes sense to consider Google's reaction in particular for three reasons: they are more sensitive to infringements than the others; it is easier to measure problems and recovery; and their instructions and advice are generally better. No SEs communicate well, though at the current time (Q4 2009) Bing (LiveSearch / MSN) are doing best in this area.

A search penalty generally results in some form of SERPs demotion, and this is closely linked to page rank. Although this measure is used by other SEs in one form or another, in practical terms, when talking of page rank, we are usually referring to PageRank - Google's registered version of the term.

Why do search engines ban websites?

Simple: because of attempts to manipulate page relevancy or page rank - or actions perceived as attempts to do so. Page relevancy and PR are crucial to search results.

Page relevancy is how relevant a page's content is to the search in question. Page rank (in Google's case, PageRank) is the ranking applied to a page that places it at a level of importance compared to others. These two factors determine both a web page's position in the SERPs, and, obviously, its usefulness to the end-user. They act along with slightly less-powerful factors such as overall site strength and the trust rank of the site.

A page that has an unjustifiably high position in the search results does no one any favours. At one end of the scale, such pages are simply less useful than they should be; at the other end, they are completely irrelevant. At some point between the two they become spam. 

All the other various contributory factors to search position such as the strength of the website the page is on have an effect - but the two core factors are relevancy and PR. Therefore, if either is artificially enhanced, a page will be given a position in the results to which it is not entitled. That would mean the search results would be useless, so of course the SEs cannot tolerate that, and must react. 

To simplify these factors, page rank is primarily based on links (their number and importance), and relevancy is primarily based on naming (which includes keywords and their number, position and type, among other things). 

As brief examples of such non-compliant actions, from a non-technical angle, consider:

  • Buying links - an attempt to artificially increase PR
  • Selling links - complicity and assistance with such attempts
  • Keyword spam - stuffing keywords everywhere possible to make a page (or site) appear more relevant
  • Redirecting either searchbots or visitors - and sending them to different pages according to who they are (of which 'cloaking' is one type - delivering keyword-stuffed pages to spiders but hiding them from normal site visitors)

Such actions are violations of search engines' guidelines or TOS. Anything done to artificially 'improve' a page's ranking is non-compliant.

Adding valuable content, and when other important resources link to you naturally, are beneficial. Improving the accessibility and usability of a site are beneficial. Improving a site by making it simpler, faster, and better-organised is beneficial.

Improving the marketing on a site is neutral, but improves earnings.

Trying to fool somebody somewhere is negative, and going in the wrong direction.

Effects of a Search Penalty

You soon know about severe search penalties because sales and traffic fall through the floor. A partial ban is harder to recognise, because normally it only means that your site doesn't have the page rank it deserves. In some cases you may be as much as three points below your true rank; in others the site may only be one point down, and this may seem hard to quantify as a ban.

Of course at some position on this graph, every badly-performing website is experiencing a measure of ban, simply because they are not complying with search engine requirements. This means in effect that basic SEO will fix it for many sites that are just not achieving their potential.

Google website ban

It is important to realise that a Google ban (or that of other SEs) can have various levels. In fact it is rare for a site to disappear completely from a search engine's results, and not be spidered at all. In the past, it would probably have been necessary to attack a search engine in some way to achieve this. However, as the war between the spammers and paid promoters versus the search engines intensifies, no doubt manual deletions of sites will increase.

A website might be banned by Google alone, or by another of the big three search engines; or by all of the major search engines. Often, if you are penalised by one, you will be penalised by all - though it is hard to measure the effects precisely for some of the smaller search engines unless you have a full set of historical site data.

Bans, penalties and warnings come in two basic types: search penalties and malware warnings. The severity of search penalties rises as follows:
  • a filter
  • a penalty
  • a manual adjustment
  • a deletion

A filter, at the lowest level of search penalty, means that an on- or off-site factor has resulted in a negation of any positive value being attributed to your page, or less commonly, site. Such factors might include duplicate content.

Penalties are a more severe allocation of a negative value to pages / sites. Severe penalties perhaps justify the use of the term ban.

Major bans are a serious matter that need fixing quickly, otherwise your revenues will remain poor to almost non-existent - unless you have a large amount of search engine independent traffic, which is unusual. A high figure here would be 33% of traffic and/or revenue from independent sources such as other links, meaning that even if banned your revenue stays at 33% of the previous level rather than the much more likely 20%. There are cases where a site with a very large number of quality inbound links can remain afloat even if 'hard' banned, with a traffic level of perhaps 40% of normal.

Duplicate content filter

There is no such thing as a ban or search penalty for duplicate content. Instead, a filter is applied, meaning that such pages are mostly filtered out of the search results. If you think about it, when several pages have content X, one of those pages will need to be at #1. There is in fact no need to show other pages that also feature content X at all, except that the sites may be more relevant to a searcher's enquiry, or there may possibly be some other feature of the page that is of interest to the searcher. 

For this reason the SE must choose what it considers to be the original page with content X, or sometimes the 'best' page with content X, and put that at #1. Other pages that also have content X are included in order to cover other eventualities. And if there are a great many pages with content X, most will appear very low in the results for that content - or not at all.

A filter is not an SP, it is just a logical way to prevent many instances of the same content from taking the top positions. There is only one drawback to this: if your site is the original creator of content X, on occasion you will not be given the credit for this, and content scrapers can appear above you. It does not happen often - but it does happen. 

Search Penalty (SP) levels 

  • A total ban - site deleted. The website disappears, and for the search engine concerned, it doesn't exist.
  • A selective keyword deletion - the site's best keywords are deleted from the search results.
  • A 90% SP (or hard ban) - in effect, 90% disappearance, as described in detail in the next section; aka a minus 90 or -90 ban.
  • A partial SP (or soft ban) - a less severe ban that results in a page rank drop, affecting SERPs position; aka a minus 30 or -30 ban.
  • A snap SP - or 'temporary glitch' ban. An SP that is applied rapidly and has no obvious cause. It also resolves itself fairly quickly - at least in search engine response terms.
  • A filter - a page has any positive search value removed, probably due to a duplicate content issue. No negative penalty value is allocated, though.



The total ban
A total ban or 100% SP means:

  • The site is removed from the index and the search results. No pages are indexed, no searches present the site name. Searching for 'example.com' does not return the site.
Even clients coming to us with a history of multiple non-compliant actions have not managed this. We would have to regard this as unlikely* except for active aggressors; and by this, not just excessive auto-querying is inferred but some form of exploit / scraping their database.

This action can only be taken manually - in other words the AI cannot make this decision. For this reason (along with similar manual penalties) it has the slang name of a 'hand-job'.

It is reported that Google has taken this route with some directories, which were essentially paid link farms, and were causing a disruption of search results. If that were indeed the case, it could be argued that the standard search penalty might have been more appropriate. On the other hand, it does send a message very effectively.

* In February 2006, BMW motors of Germany were reportedly blacklisted (implying a full ban) by Google (source: BBC). This was apparently for cloaking and doorway paging, both of which BMW seems to have admitted. These are usually regarded as 'black hat SEO' techniques designed to fool the search engines, though BMW argued that they simply resulted in better search information for customers. Google didn't agree. 

Our research, however, seems to show that this 'ban' lasted from the 4th to the 9th of February. If that is the case, it hardly qualifies for the term even if the site was deleted; while surely only the most embittered cynic would suggest that a small business might find the process of recovery just a teensy bit longer...

There are numerous cases of bans where a website is said to have 'disappeared' or 'been removed'. On closer inspection you find they have receded to #90 or thereabouts in the SERPs; which probably amounts to the same thing as being removed: a #90 search position is equivalent to deletion in practical terms, as there certainly won't be any traffic from search results like that.

Selective keyword deletion
This is applied manually to a site with serious errors.

  • The site's best keywords are deleted from search results, so that the site cannot rank for them.

Only the selected keywords are deleted - the site still gets low traffic from other less valuable keywords. This is a manual adjustment that is made to sites with a major problem. Several other SP effects will also be seen.

The effect is to reduce website traffic drastically. Where an important term such as a major market single-word term has been deleted, which previously placed at #1 or #2 for example in the SERPs, the site traffic drops right off. This penalty is not the mythical -950 penalty (qv Part 2a) because a top keyword that placed at #1 or 2 cannot be seen in the top 1,000 results. It has been deleted.

The site is still spidered and indexed at 90% of norm, and possibly more. Traffic still comes in from other keywords, placed at their normal SERPs position (but these keywords have low traffic).


The 90% SP
This typically results in:

  • The site falls to #90 or so for all its keywords, in any search, perhaps from previous page 1 or 2 positions;
  • Pages rapidly drop out of the search engine's index;
  • Very few pages end up being indexed;
  • All pages register zero PageRank, even if previously at PR6 etc.; All pages except the index are greyed-out on the Google PR browser toolbar;
  • The result is commonly a 75 - 90% traffic drop, if all the top search engines are involved. Some sites with quality links do not suffer as badly.
  • But: the site is still spidered regularly by the search engine/s.

This might be referred to as a 'major' ban for example, and happens to those with website errors of a serious nature; or to those who have been somewhat over-adventurous with their website management; and especially for a combination of the two.

When specifically referring to Google, this can be be termed a minus 90 ban, since keywords appear at #92 or #93 in the SERPs that were previously at #2 or #3, for example.

Note the last, seemingly anomalous, bullet point: the site is still spidered. This scenario mainly refers to a Google ban, since it is the easiest to quantify. The various factors taken together tell you that you've got problems, but they can be fixed. 

Again, it is necessary to state that there are a lot of variations in how bans appear to affect websites - there is absolutely no fixed set of 'symptoms' even for one class of ban (or more correctly Search Penalty).

For example: a very large site we saw experienced a severe penalty. Most of their pages dropped out of the index, PR went to zero for all pages, and no search results were returned except for very minor terms that resulted in few visits per day. All other terms, e.g. 'larger' ones, were demoted to about #90 in the results. Only around 1% of their pages remained in the index.

Situation serious... After some repairs were made, pages indexed rose from a couple of thousand to 250k, which was probably most of the currently-indexed pages (again, this was a large site). But even though there were 250,000 pages in the index, PR and therefore search results were still zero. There is no fixed set of rules that apply to ban effects - perhaps especially so in the big and complex cases, since there may well be a mix of human and AI (algo) factors involved with the search penalty levels. zxc4vbn

So a site can be banned and have thousands of pages in the index. In fact this is much more common than the total removal scenario - which is far easier to diagnose of course.

It is probable that the majority of total bans / blacklist / deletion / 'removed-from' cases are in fact a 90% SP; the effects are more or less the same, certainly as regards traffic. There are many pages in the index but the site is under a severe penalty, with some or all keywords at minus 90 places from their previous position.

 
The partial SP
This has the effect of: 

  • A two or three-point drop in PageRank
  • A keyword search position drop - 30 places is common
The Partial SP can result from a simple single error, or group of errors, that when fixed results in the site jumping back up, for example, from PR 0 to PR3. The partial SP is resolved much more quickly than the 90% SP. 

We have seen many cases of this type of ban recently, where fixing a small problem on the site resulted in an almost immediate jump back up to the correct PageRank and SERPS positions - in only around three days.

When specifically referring to Google, this can be be termed a minus 30 ban, since keywords appear at #32 or #33 in the SERPs that were previously at #2 or #3, for example. 

It is probable that more sites are suffering one of these partial bans than any other type. This is because the effects can be insidious: there is no obvious complete lack of search results, they are just poor. Together with the fact that the majority of website owners have no idea at all of their page rank or even comparative search performance, a partial ban simply means their sites don't perform well, and the owners don't appreciate why. The cure is often basic SEO, since this can fix the more serious non-optimal site factors that caused the SP in the first place. gfd3saq  

The SEO consultant's job here is to tune out these handicaps and clean up the site's operation. This is a major improvement for everyone, because the huge numbers of sites currently experiencing a partial SP cannot be good for anyone - the owners, the customers, and even the search engines themselves will surely benefit. This is yet another good reason why the apparent position of some search engines on the desirability of site improvement consultants makes no sense at all.*
[*update: it used to be the case that search engines stated forcefully that site owners should not employ anyone to fix problems on their site, and that sites should exist organically with whatever faults the web designer had included and that the owner introduced. This position proved illogical and untenable, and fortunately had to change. Now, it is universally recognised that improvements (and the contractors/consultants who implement them) can either be of good quality or of poor quality.]

 

The snap SP
This SP has its own unique effects:

  • PR drops to zero
  • Very low SERPs position
  • Pages drop out of the search engine's index

These bans are applied rapidly, are inexplicable, are severe, and have no logical cause. Their effect is usually identical to a 90% SRP, a stringent ban. They can apparently occur even if no pages or anything else has, as far as is known, changed on or off-site for weeks. The affected website finds that traffic has dropped right off due to low SERPs results. As no changes are obvious to the owner, resolving the issue is difficult. 

The effect is just as if the robots.txt has been changed and all bots have been blocked, and/or as if page metatags have had a 'noindex,nofollow' attribute added. Indeed, it has been reported that search engine data even showed some URLs blocked by robots.txt, although that was not found to be the case. 

These cases generally resolve themselves over a short time period (one month) and the page rank returns, pages return to the index, and less pages are reported as being 'blocked by robots.txt'. It is tempting to say that it may have been a data glitch at the SE concerned, which resolved itself fairly quickly; or it may in a very few cases be a result of security exploits at the website. There is no recourse since the problem clears within a month or so, though it takes longer to fully recover.

Hosts implicated
However, when one of these snap bans was investigated, it was found that in fact the server had been changed over at that time. The suspicion, then, is that a network error resulted, which had the appearance of making the site seem to be involved in some sort of nefarious activity. Perhaps the site hosts had somehow 302'd the site or something equally suicidal. Therefore a proportion of these 'glitch' bans may in fact be attributable to similar causes.

The snap SP has the distinction of being the fastest applied, and the quickest (of the severe bans) to be resolved. It seems that it can be applied almost instantly. It resolves itself in a comparatively short time, by search engine damage repair standards. No action is needed by the site owner, and indeed this may only confuse the issue - it is unlikely that the matter could be resolved any quicker.

The snap search penalty can be distinguished from a 90% SP by the facts that: 

  • There is absolutely no clue to its cause, and the site itself may have high standards of housekeeping
  • Improvements start to be seen at 3 or 4 weeks, with no work input
  • A search engine may report inexplicable data such as 'URLs blocked by robots.txt', when this is clearly not the case

Major hosting issues

There are some major errors in site operation caused by hosts that can easily sink a site. Some of them can be confused with the last type of penalty. They have the effect of rendering the site unavailable at times, or of blocking a searchbot. The site will suffer badly in search results as a consequence.
 
Site unavailable
The first of these is the 'site unavailable' error reported by an SE, or the 'database server unreachable' error reported by the website's main web application. Both of these have the same cause and the same effect: the site is offline. It is caused by an overloaded database server, and happens where website hosts have too many DB-driven websites on the same server. The site will have extended page load times and/or be offline frequently. This is a hosting fault. The site's SERPs positions will suffer.
 
The only fix is to move hosts, as simply moving to another server means staying with a host not competent to host dynamic (DB-driven) websites (ie most modern business websites).
 
Robots.txt unreachable
Next comes the 'your robots.txt was unreachable' error reported by a search engine. This results in a big drop in search positions and pages dropping out of the index. It is a hosting fault caused by the server firewall blocking searchbots. It happens because the firewall default settings block concurrent connections from the same IP. However, a search engine may have multiple instances of a bot on a site - and on other sites on the server as well. The firewall blocks them out. The fix is to whitelist the IP ranges used by searchbots, on the firewall. Most hosts have firewalls, and most hosts need to implement this fix. Occasionally a new server introduced by an otherwise competent host may have this issue because they forgot all the required procedures for building a new box.
 
There are many similar possible problems on new servers, such as the su_php, Apache file ownership issues, and so on.

Domain mismanagement
Ignorance of correct domain management procedures by hosts is also a primary cause of search penalties. This is why you should never allow your hosts to hold your domains - they must be lodged with a reputable domain registrar such as namecheap.com (an unfortunate name for a quality service). A quick fix for some minus-30 penalties is to retrieve all your domains from the webhost, if held by them, and place them with a proper registrar and with all the settings checked and corrected - your penalty will disappear in 3 days. Any SEO consultant can explain the multiple reasons for this to you.

Please do not make the mistake of thinking that webhosts are universally capable of managing domains, or that domain owners are either. Domain settings and associated issues can easily sink a site.

Algo / manual penalties

In some complex cases, or those where the severest penalties have been applied, there is a likelihood that a manual adjustment has been made and that the penalty is not simply a result of the algorithm's effects. This can be seen where there is a difference between the search positions for 'big' terms and those for 'small' terms.

For instance, a penalised site may have disappeared down to #90 or thereabouts for its big terms - let's say in the insurance area, for 'home insurance'; but still be ranked at number 3 or so for 'Butte City home insurance', which brings in about two visitors a day.

It would be hard to see this as purely an algo-influenced result, since the site should be equally poorly-ranked for all terms. This is sometimes seen where there are multiple problems to be resolved.

In other cases, the website's most important terms may have been totally deleted. This of course is a manual adjustment to the search results. The 'biggest' terms that the site competes for have thus been eliminated, in order to ensure the website cannot gain unjust search positions for those terms; and instead of appearing at perhaps #1 to #3 for such terms, the site does not appear for them at all (at least, not in the top 1,000, which is as far down as we are allowed to measure).

The 'post-mortem' penalty

Some violations of search engine guidelines will raise flags that quickly attract attention. Even so, the site owner may fix these issues before search engineers get to look at the site. If this is the case, but the engineers feel that the non-compliant action was of sufficient severity to warrant it, they may still apply a penalty. For less severe cases, or where the site appears to be run honestly apart from the single transgression, they appear to give you the benefit of the doubt and allow you some leeway for the one occasion. 

This type of 'after the act' penalty might be avoided, then, if you fix a problem swiftly, and normally run clean. Good metrics are the key to both.

 

Continued in Part 2 - see main menu item: Banned Websites - Part 2

Did you find this page useful?
If so, please consider linking to it. Thank you.


author: Chris Price

originally published: 2007-08-12
last edited: 2009-09-09
minor updates: 2012-06-15

The weird and misspelt postscript with the strange terms this page gets found for:  unban site - how to open banned sites - how to know banned websites - how to open banned website - how is open baned website - baned websites. And hardly any of the obvious ones you would think of, since apparently no one is searching (yet) for search penalty or search penalties . No point in being subtle then...

 
Web Business Managers