Website Bans / Search Penalties

Part 1

Website bans - Introduction
Terminology
Search Penalty definitions
Search engine aliases
Penalties from the Big 3
Why do search engines ban websites?
Effects of a search penalty
Effect of a Google ban
The two types of search engine ban
SP (search penalty) levels
Algo/ manual penalties
The 'post-mortem' penalty
Problem areas
How to avoid website bans

Introduction: website banned by search engines

At one time or another, many sites get largely or partially banned by one or more search engines - although these events are much more accurately referred to as penalties. Because this is such a common phenomenon now, we think it probably happens to around 5% of websites at some stage, in varying measures of severity. On the other hand, what might be referred to as a 'total' ban is extremely rare (the various definitions of bans are examined later on). It seems, then, that this area is becoming increasingly important in search engine optimising, and more and more often we are asked to clear a banned website (and sometimes we have to refuse, as our business objectives do not coincide).

The vast majority of websites need to put in place much higher standards of housekeeping; this can hardly be achieved at their current knowledge levels if they have received a search penalty. There is another aspect, which is attitude: website owners in general need to show more willingness to read, understand and comply with search engines TOS or guidelines. Terms of Service may vary slightly between the search suppliers; please see the section on this lower down the page.

This sort of resource can never be described as complete, for many reasons. It will be edited and added to whenever possible.

Website ban terminology 

Because this is a fairly new subject for public debate, and also because the search engines themselves do not communicate widely or openly in this area, we must use untested terminology. In fact it is necessary to develop our own if we wish to discuss the subject fully. Therefore, please be aware that many of the terms used here are new, and not necessarily accepted by all. When a consensus on terminology is arrived at, we can adjust the terms to suit.

Strictly speaking, in terms of the accuracy of definition of the word, and the fairly moderate level of action in most cases by the search engines in response to non-compliant actions, we should not be using the term 'ban'. A more accurate term would be something like 'search results penalty', 'search results restriction' or 'search position demotion'. This would be a better way to describe a search engine's graduated response to non-compliance, and in fact their response is generally at the tolerant end of the scale rather than the strict. However, the power of modern parlance being what it is, ban it is, and ban it will no doubt stay.Alternatives might include blocked, blacklisted, deleted.

It therefore seems more accurate in most cases to refer to these events as a Search Results Penalty or Search Penalty - or SP - and this is the term we will use whenever convenient. There are indications that some search engine staff use similar terms. (However, a malware ban or penalty is a different type of reaction.) In fact we would hardly use the term 'ban' at all, since it has alarmist connotations, except for the fact that this page needs to be found by people searching for information on the subject - so the word has to be repeated frequently...

More search penalty terms defined

Search engine aliases

Search engines, like any business, change their main business name occasionally; or use different names for distinct divisions of their enterprise; or have connections to other similar businesses. Here are some connected names and variants:

Yahoo! = YPN, Overture. Linked to: AllTheWeb, AltaVista.
MSN = Live Search, Live.com, Microsoft Publisher Program.
Ask.com = Ask Jeeves. Linked to: Lycos, Hotbot.

Google, Yahoo, and MSN search penalties

From a financial perspective it only makes sense to consider penalties applied by the Big 3 search engines. This is because other SEs may only affect 1 or 2% or so of site traffic; and their reaction to infringements of guidelines is likely to be the same as or less severe than that of the Big 3.

Further, it makes sense to consider Google's reaction in particular for three reasons: they are more sensitive to infringements than the others; it is easier to measure problems and recovery; and they tend to communicate better. This last is relative, of course, since in practical terms they communicate about as well as the Masons. A minor point is that, in addition, their terminology is more complete.

A search penalty generally results in some form of SERPs demotion, and this is closely linked to page rank. Although this measure is used by other SEs in one form or another, in practical terms, when talking of page rank, we are usually referring to PageRank - Google's registered version of the term.

Why do search engines ban websites?

Simple: because of attempts to manipulate page rank or page relevancy - or actions perceived as attempts to do so. Page rank (or in the case of Google, PageRank) and relevancy are everything to search results. These factors determine both a web page's position in the SERPs, and, obviously, its usefulness to the end-user.

A page that has an unjustifiably high position in the search results does no one any favours. At one end of the scale, such pages are simply less useful than they should be; at the other end, they are completely irrelevant. At some point on the graph they become spam.

There are other contributory factors to search position, such as the strength of the website the page is on - but the two core factors are relevancy and PR. Therefore, if either is artificially enhanced, a page will be given a position in the results to which it is not entitled. That would mean the search results would be useless, so of course the SEs cannot tolerate that, and must react.

To simplify these factors, page rank is primarily based on links (their number and importance), and relevancy is primarily based on names and naming (which includes keywords and their number, position and type, among other things).

As brief examples of such non-compliant actions, from a non-technical angle, consider:

1) Buying links - an attempt to artificially increase PR.

2) Selling links - complicity and assistance with such attempts.

3) Keyword spam - stuffing keywords everywhere possible to make a page (or site) appear more relevant.

4) Redirecting either searchbots or visitors - and sending them to different pages according to who they are.

Such actions are violations of search engines' guidelines or TOS - terms of service. Anything done to artificially 'improve' a page's ranking is non-compliant. Adding valuable content, and other important resources linking to you naturally, are beneficial. Improving the accessibility and usability of a site are beneficial. Improving a site by making it simpler, faster, and better-organised is beneficial. Improving the marketing on a site is neutral, but improves earnings. Trying to fool somebody somewhere is negative, and the wrong direction.

Effects of a Search Penalty

You soon know about severe search penalties because sales and traffic fall through the floor. A partial ban is harder to recognise, because normally it only means that your site doesn't have the page rank it deserves. In some cases you may be as much as three points below your true rank; in others the site may only be one point down, and this may seem hard to quantify as a ban.

Of course at some position on this graph, every badly-performing website is experiencing a measure of ban, simply because they are not complying with search engine requirements. This means in effect that basic SEO will fix it for many sites that are just not achieving their potential.

Google website ban

It is important to realise that a Google ban (or that of other SEs) can have various levels. In fact it is rare for a site to disappear completely from a search engine's results, and not be spidered at all. In the past, it would probably have been necessary to attack a search engine in some way to achieve this. However, as the war between the spammers and paid promoters versus the search engines intensifies, no doubt manual deletions of sites will increase.

A website might be banned by Google alone, or by another of the big three search engines; or by all of the major search engines. Often, if you are penalised by one, you will be penalised by all - though it is hard to measure the effects precisely for some of the smaller search engines unless you have a full set of historical site data.

• a filter
• a penalty
• a manual adjustment
• a deletion
  

A filter, at the lowest level of search penalty, means that an on- or off-site factor has resulted in a negation of any positive value being attributed to your page, or less commonly, site. Such factors might include duplicate content. Penalties are a more severe allocation of a negative value to pages/ sites. Severe penalties perhaps justify the use of the term ban.

Major bans are a serious matter that need fixing quickly, otherwise your revenues will remain poor to almost non-existent - unless you have a large amount of search engine independent traffic, which is unusual. A high figure here would be 25% of traffic and/or revenue from independent sources such as other links, meaning that even if banned your revenue stays at 25% of the previous level rather than the much more likely 10%. There are cases where a site with a very large number of quality inbound links can remain afloat even if 'hard' banned, with a traffic level of perhaps 35% of normal.

Search engine bans are of one of two types:

1. An SP (search penalty). This is the most common type of ban, and in effect is simply a SERPs (search engine results pages) demotion - it demotes a website's position in the search results. The majority of bans are of this type, and have various levels of severity.

Since around 240,000 websites seem to have had a malware warning ban (this figure from stopbadware.org - see Resources at foot of page), there must be at least a million who have been SRP banned. That statement is made because we see clients at a ratio of about 5 to 1, and that ratio is also confirmed (indeed amplified if anything) by search results in general.

2. A malware* ban. Strictly speaking this type is better termed a malware warning (or badware warning as some of the major SEs prefer), rather than a penalty or ban, since the site in question is often not removed or demoted. The ban does not affect the search positions, or page rank; however, even though a site receives its normal search result positions (for example at Google #5), the result cannot be clicked-through to the site as per usual - there is no live link to the site. In addition, a warning is placed in the result that says "This site may harm your computer." Clicking the result takes you to a warning ('interstitial') page, and not the site. You have to copy & paste the text of the URL - they want you know the site is risky (and not be guilty of sending you there unknowing).

* Malware, aka scumware or badware, is software that makes changes to a user's system, or leaks information to third parties, without the user's consent. A popular description might therefore be 'evil' software. It can comprise simple applications such as trojans that collect information; or diallers that 'phone home'; or small applications that enable other larger ones to be installed unknown to the user. Malware that stays resident on a server can have many purposes, none of them beneficial to visitors; one function being to install unwanted software invisibly on visitors' computers.

Search Penalty (SP) levels:

• A total ban - site deleted. The website disappears, and for the search engine concerned, it doesn't exist.
• A 90% SP (or hard ban) - in effect, 90% disappearance, as described in detail in the next section.
• A partial SP (or soft ban) - a 'minor' ban that results in a page rank drop, affecting SERPs position.
• A snap SP - or 'temporary glitch' ban. An SRP that is applied rapidly and has no obvious cause. It also resolves itself fairly quickly - at least in search engine response terms.
• A filter - a page has any positive search value removed, probably due to a duplicate content issue. No negative penalty value is allocated, though.

A total ban (100% SRP) means:

• The site is removed from the index and the search results. No pages are indexed, no searches present the site name. Searching for 'example.com' does not return the site. 
  

Even clients coming to us with a history of multiple non-compliant actions have not managed this. We would have to regard this as unlikely* except for active aggressors; and by this, not just excessive auto-querying is inferred but some form of exploit / scraping their database.

This action can only be taken manually - in other words the AI cannot make this decision. For this reason (along with similar manual penalties) it has the slang name of a 'hand-job'.

It is reported that Google has taken this route with some directories, which were essentially paid link farms, and were causing a disruption of search results. If that were indeed the case, it could be argued that the standard search penalty might have been more appropriate. On the other hand, it does send a message very effectively.

Our research, however, seems to show that this 'ban' lasted from the 4th to the 9th of February. If that is the case, it hardly qualifies for the term even if the site was deleted; while surely only the most embittered cynic would suggest that a small business might find the process just a teensy bit longer...

There are numerous cases of bans where a website is said to have 'disappeared' or 'been removed'. On closer inspection one finds they have receded to #90 or thereabouts in the SERPs; which probably amounts to the same thing. A #90 search position is equivalent to deletion in practical terms, as there certainly won't be any traffic from search results like that.

A 90% SP typically results in:

• The site falls to #90 or so for all its keywords, in any search, perhaps from previous page 1 or 2 positions;
• Pages rapidly drop out of the search engine's index;
• Very few pages end up being indexed;
• All pages register zero PageRank, even if previously at PR6 etc.;
• All pages except the index are greyed-out on the Google PR browser toolbar;
• The result is commonly a 75 - 90% traffic drop, if all the top search engines are involved. Some sites with quality links do not suffer as badly.
• But: the site is still spidered regularly by the search engine/s.

Note the last, seemingly anomalous, point: the site is still spidered. This scenario mainly refers to a Google ban, since it is the easiest to quantify. The various factors taken together tell you that you've got problems, but they can be fixed.

• A two or three-point drop in PageRank

• A keyword search position drop - 30 places seems quite common

This type of ban can result from a simple single error, or group of errors, that when fixed results in the site jumping back up, for example, from PR 0 to PR 2 or 3. The partial SP is resolved much more quickly than the 90% SP.

It is probable that more sites are suffering one of these partial bans than any other type. This is because the effects can be insidious: there is no obvious complete lack of search results, they are just poor. Together with the fact that the majority of website owners have no idea at all of their page rank or even comparative search performance, a partial ban simply means their sites don't perform well, and the owners don't appreciate why. The cure is often basic SEO, since this can fix the more serious non-optimal site factors that caused the SP in the first place.

Of course, there are many types of improvements, and many types of consultants.

 

A snap SP shows:

• PR drops to zero

• Very low SERPs position

• Pages drop out of the search engine's index

These bans are applied rapidly, are inexplicable, are severe, and have no logical cause. Their effect is usually identical to a 90% SRP, a stringent ban. They can apparently occur even if no pages or anything else has changed on the site for weeks (or so it is reported). The affected website finds that traffic has dropped right off due to low SERPs results. As no changes are obvious to the owner, resolving the issue is difficult.

• There is absolutely no clue to its cause, and the site itself may have high standards of housekeeping.

• Improvements start to be seen at 3 or 4 weeks, with no work input.

• A search engine may report inexplicable data such as 'URLs blocked by robots.txt', when this is clearly not the case.
  

Algo / manual penalties

For instance, a penalised site may have disappeared down to #90 or thereabouts for its big terms - let's say in the insurance area, for 'house insurance'; but still be ranked at number 3 or so for 'Butte City home insurance', which brings in about two visitors a day.

The 'post-mortem' penalty

Some violations of search engine guidelines will raise flags that quickly attract attention. Even so, the site owner may fix these issues before search engineers get to look at the site. If this is the case, but the engineers feel that the non-compliant action was of sufficient severity to warrant it, they may still apply a penalty. For less severe cases, or where the site appears to be run honestly apart from the single transgression, they appear to give you the benefit of the doubt and allow you some leeway for the one occasion.

It is advisable, as a good operational principle, to maintain as much data on the site as possible, from as many different sources as possible, both internal and external. In many cases where an SP needs investigating, historical data is extremely valuable. It is also possible that, had those metrics been available and studied in the first place, a penalty would not have been applied to the site at all.

As far as paid links go, you need to avoid them. Don't buy them, and don't sell them. It is true to say that if your site is small and has a low profile, then you will most likely get away with things that bigger and louder offenders won't. If you stick your head above the parapet, expect to get it shot off.

How to avoid website bans

Avoiding a search penalty is done by running clean, operating in honest mode, and trying to avoid technical choices that might conflict with SE quality guidelines. For larger sites, and those with complex management routines, the latter requirement might not always be so easy; but all should take careful note of the first two points. Therefore, you should:

1. Read the Terms of Service of all those search engines whose indexes you wish to appear in. Because these guidelines are a little sketchy, you should read as many of them as you can find, and then you will appreciate the requirements better.

2. Monitor your website's operation. This means you should obtain data, and check a variety of metrics, on a regular basis. It is not someone else's job - it is yours. Get a proper website statistics package - or two - that allows you to fully monitor your website data and especially errors. If you don't know the precise page errors occurred on, what file was requested, which page was requested, and the requesting IP - then clearly you are not complying with this very important requirement. How does your server perform? What DNS checks have you carried out? What have your hosts done about redirects? How is your mailserver set up? Exactly what is the state of your domains, in your domain registrar's account? And please don't tell me you haven't got one and you use a host to hold them. What does your network monitoring service say about your server? What does your security audit say? These questions - and many, many more like them - determine if your site runs safely or not. If you don't know the precise answers here, you are wasting your time worrying about marketing or other issues - your website housekeeping is abysmal and you are inviting a ban.

3. Don't use the cheapest hosting deal you can find; and don't stick with a host you have been using for years, just out of habit - unless they are proven reliable and of good value. Each of those options is risky; one because you only get what you pay for, and the other because in our experience some of the longest-established hosts are the worst. Get your hosting audited by someone who knows what they are looking at. Read our SEO Hosting section for an idea of modern requirements. Just as SEO requirements change drastically every 18 months, so do hosting requirements, though admittedly over a longer term. A host whose operation is essentially the same as it was in the 90s - and there are plenty of those, especially among the big names - is a bad bet now.

 

Continued in Part 2 - see main menu item:

>> Banned Websites - Part 2

 

---

 

The weird and misspelt postscript:-

- with the strange terms this page gets found for: unban site - how to open banned sites - how to know banned websites - how to open banned website - how is open baned website - baned websites (nice). And hardly any of the obvious ones you would think of, since apparently no one is searching (yet) for 'search penalty' or 'search penalties'. No point in being subtle then...

It seems there is some confusion here between websites under search penalty, and websites banned from being viewed in educational, military, government or commercial establishments and so on - or even whole countries. Unfortunately we cannot help with the latter, modern life requires us to specialise.