Website search penalty issues continued:

How can I tell if my website is banned?
How to avoid website bans
Search penalty myths
Our opinion on search engine bans
Search engine guidelines - the SE's TOS
Resources - weblinks to more information

How can I tell if my website is banned?

Presumably things can't be that bad if you need to ask. Or perhaps you mean that traffic has dropped and you aren't sure if it's a ban?

In the case of a malware ban, there is no doubt, because you will still get good search results but you cannot click through to the site from them; and the warning notice placed in the search result is hard to miss.

The symptoms described previously for search penalties should allow you to make some sort of judgement for that type. A new site may need to wait as long as five months for some PageRank (assuming they have some links of course), so lack of it for a fairly new site means nothing. Even then, the site will probably have zero rank, even a year or so later, until it gets at least 30 links, which is the minimum required for PR1. If you are using the Toolbar PageRank gauge to measure this, be aware that the site will need 30 or so links in order to register on this gauge; it is basically a link number gauge in the first place. Other factors certainly influence it - but that's what it is. It has little or no importance except for measuring like with like, or previous with current; though for that purpose alone it is very useful. PageRank is an ethereal concept that may or may not have relevance in any particular situation. Only the SERPs are important, and PR only has any importance when it affects them. It doesn't always - and Toolbar PR has even less overall importance. It is mainly of use for determining the advertising value of a page.

Problems can arise in several common situations, though: if traffic drops and you don't know why; or if page rank drops and you don't know why. If your PageRank drops by three points or more, and/or if your search positions drop by more than 30 places, it's very likely a ban. Otherwise, it can be hard to tell whether you are being penalised, or if it's just an algorithm change that is hurting you. Without data, it's hard to tell; so if you don't know what your previous figures were with the big 3 search engines: for your page rank, keyword search positions, indexed page numbers and so on - then it becomes tricky.

It's a question of previous keyword search positions versus current SERPs position; previous page rank versus current page rank; previous traffic versus current traffic (in visits / visitors / uniques of course - hits or whatever are irrelevant); previous sales versus current sales. The prudent will have detailed records of this data; and if they also have network monitoring data, then their costs for resolving errors will probably be reduced. On the other hand, site operators with this standard of housekeeping are less likely to suffer an SP in the first place.

There are many unexplainable anomalies during bans that cause some head-scratching. So, you definitely can't say "If there are plenty of pages in the index, you're not banned", for instance; as we saw with the example given previously of the large site with 250,000 pages in the index that was hard banned without a doubt: zero page rank instead of PR5, and pitifully low traffic, from no search results, when it used to be on page 1 for many terms.

Take careful note of the following points:

• Clearing a website banned by search engines is a specialist division within search engine optimising. It's probably best left to experts in this exact field, unless there is one simple and obvious cause.
• Don't start the reinstatement process with a search engine unless you are certain the problems have been resolved. This is not normally a decision best left to the webmaster or host support techs, for obvious reasons. However, if you can reel off twenty or more possible causes, the probable fixes, and the ways to stop it happening again, you may qualify. Presumably in this case the ban must have occurred because the site owner could not be deterred from a course of action that was known to be dangerous.
• If you introduce an aggressive tone into your dealings with a search engine's representative, you cannot expect a happy outcome. They are only human, and desperately understaffed.

How to avoid website bans

Avoiding a search penalty is done by:

• running clean
  

• operating in honest mode
  

• maintaining efficient oversight of your site's metrics (operating data)

• maintaining good records of your site's metrics

• trying to avoid technical choices that might conflict with SE quality guidelines

For larger sites, and those with complex management routines, the latter requirement might not always be so easy. Therefore, you should:

1. Read the Terms of Service of all those search engines whose indexes you wish to appear in. Because these guidelines are a little sketchy, you should read as many of them as you can find, and then you will appreciate the requirements better.

2. Monitor your website's operation. This means you should obtain data, and check a variety of metrics, on a regular basis. It is not someone else's job - it is yours. Get a proper website statistics package - or two - that allows you to fully monitor your website data and especially errors. If you don't know the precise page that errors occurred on, what file was requested, which page was requested, and the requesting IP - then clearly you are not complying with this very important requirement. How does your server perform? What DNS checks have you carried out? What have your hosts done about redirects? Exactly what is the state of your domains, in your domain registrar's account? And please don't tell me you haven't got one and you use a host to hold them. What does your network monitoring service say about your server? What does your security audit say? These questions - and many, many more like them - determine if your site runs safely or not. If you don't know the precise answers here, you are wasting your time worrying about marketing or other issues; other problems could occur that make them redundant.

3. Don't use the cheapest hosting deal you can find; and don't stick with a host you have been using for years, just out of habit - unless they are proven reliable and of good value. Each of those options is risky; one because you only get what you pay for, and the other because in our experience some of the longest-established hosts are the worst. Get your hosting audited by someone who knows what they are looking at. Read our SEO Hosting section for an idea of modern requirements. Just as SEO requirements change drastically every 18 months, so do hosting requirements, though admittedly over a longer term. A host whose operation is essentially the same as it was in the 90s - and there are plenty of those, especially among the big names - is a bad bet now.

Search Penalty myths

The whole subject is shrouded in myths and misinformation. Here are some of the most obvious myths that are commonly seen perpetuated on forums and in blog posts:

• 'Search engine ban'. The #1 myth, of course. Search engines rarely 'ban' a website, if by this is implied that it has been removed from their index. To achieve this you normally need to be running a 100% spam site such as an auto-generated one with thousands of pages of keyword-stuffed gibberish, which is then cloaked so that spiders cannot easily recognise it as such. These sites get banned; other more normal websites receive a search penalty of varying severity.
  
  
• 'The -950 penalty'. This implies that a keyword is demoted to #950 or lower in the search results. We haven't seen this as all other penalties are far more common - if this exists at all. Perhaps there is some confusion with the -90 Penalty, where a keyword that previously appeared on page 1 at #3 for example, now appears at #93. Without any evidence that a -950 Penalty exists (where a keyword would appear at #953 if previously it was at #3), we have to assume this is another myth or a misnomer of some kind. Keywords are penalised to -90 or deleted - we have not seen any instance of keywords penalised to minus 950 places.
  
  
• 'Duplicate content ban'. There is no such thing. Pages that are substantially duplicates of others will receive no PR. They cannot receive benefit of any kind because only one instance of any given content can be ranked and appear in search results (in theory at any rate - although this is imperfectly implemented). However spam sites with a large percentage of pages that are 'scraped', ie a duplicate of other websites' content, will be deleted or have a severe penalty applied. These are known as scraper sites and are almost always MFA sites (Made For Adsense - built specifically to host large amounts of PPC adverts and with no useful original content of any kind).

Our opinion on search engine bans

Firstly, we have to apologise for using that particular phrase, purely for search results of course...

What we have tried to do here is present a balanced view that does not apportion blame or criticism unfairly, and also presents our opinions. We may or may not have succeeded.

This is a complex area, within a relationship that is itself complex. A search engine is at the business end of a three-cornered arrangement between a supplier, partners, and customers. Neither can operate without the others; each depends heavily on the others; whatever each does, deeply affects the others. The search engine is the supplier (of the search results); the site owners are the partners (they own the resources); the customers are the searchers, who are looking for the resources of the site owners and are directed there by the search engine. The system works because it is an organic entity of the free-market type - or tries to be. Of course, there are commercial pressures that affect this situation.

If one of the three, by design or accident, skews the results of the supply chain by artificial means, the others are not only affected but must act. The search engines react by penalising those who introduce elements that would affect the organic nature of the system. Such penalties are applied by degree, and in the opinion of those who support an organic system strongly, are not draconian. One might be tempted to dream of a more social-oriented system, but the fact is that unfortunately such communal-based systems do not seem to work on a large scale, only for smaller entities. The free-market mix of commercial and free resources (and pressures) seems to work best in practice, especially as the scale gets larger.

Of course there are what might be called operational problems. Because of the vast numbers involved, the system has to be run by AI (computers). There just aren't enough people or hours in the day to do otherwise. Therefore, problems can not only occur but also take a long time to be resolved. There are many cases where a website owner has no idea why they are being penalised, and even outside consultants brought in to fix it do not have an easy task figuring that out as a first stage. In these cases if in no others, a much better communication system between the site owner's agents and the search engine's staff is needed.

Because commercial necessities are an obvious factor here - staffing levels would have to increase - the obvious conclusion is that perhaps payment of a fee could resolve matters more swiftly. Not to make the problem go away - simply to provide a report on what the problem is. This would surely help to fix the problem more quickly, to the benefit of all three parties - the customers, the search engine and the partners.
 
We think:

• Search engines need to operate with a high degree of secrecy in order to protect both the system and their proprietary solutions.
• In the vast majority of search penalties the site owner or their agents were responsible, usually to a major degree (and this includes by ignorance, inaction, and accepting the cheapest possible solution in some areas).
• It is the custom now to immediately blame someone else whenever something goes wrong. It might be best to check first if your own house is in order.
• We don't think the penalties applied by search engines are too severe. If anything, they tend to react on the tolerant side of the scale. In any case, there needs to be an element of punishment in order to clearly send the message that errors and transgressions are a bad idea.
• In some cases there is no way to divine what the problem was, because it is no longer there.
• In some cases, simply as a statistical certainty, the site owner was innocent. There must be a few cases where the owner or agents were blameless, and the error occurred within the search engine's part of the process. It would be absolutely impossible to exclude errors from a giant distributed AI system connected to a global network - and anyone who says different has never been connected with any kind of engineering.
• There are many cases where technical factors involving the running of the site are responsible in some way for a penalty. All parties are - in essence - innocent, but the matter is difficult to resolve due to lack of communication.
• In complex cases, assistance from the search engine operators in resolving the matter would be very much appreciated. Resolving these cases is of benefit to all, including of course the customers.
• A fixed-fee solution, perhaps $200, might be the best way forward. A fee could be paid to provide a report on what the problem is. The fee would not be for progressing the issue, or even for reinstatement; but as technical assistance for the site owner's agents, so they could at least fix the errors accurately. Having made repairs, there might then be a slightly different reinstatement path. You would at least be able to tick off 'We fixed this, and we fixed that'.
• A core part of the contract for a fee-paid report would have to be that no blame could be attached to the search engine operator, whether or not a problem was found, and whatever or wherever that problem might be - even if it transpired that it was with the search engine. The site owner would have to release the SE from any blame or duty, in the interests of the matter being speedily resolved. The Internet / search engine / website system may just be the most complex organism on the planet; it is impossible to exclude errors from every single part of such a vast system. It involves perhaps 10 billion web pages on one end, and thousands of computers on the other.
  
• We appreciate the fact that staff levels are in no way sufficient at present to provide such services; which is why a fee payment for a simple report would be a way forward.
• Statements by owners who are not experienced in this area, that they are innocent, do not hold water.
• A statement by a search engine operator that they never make mistakes is equally unlikely to be true.
• We do not agree with the suggestion that a search engine should contact a site owner before applying a ban. Our opinion is that penalties need to be applied immediately a problem is detected. There are all sorts of reasons for this, mostly to do with the fact that visitors need to be protected.
• We don't see why search engines can't attempt to contact site owners and tell them their site has been penalised. This is especially true in manual adjustment cases. It would depend entirely on staff resources and a contact email being available on the site; when this was tried briefly before, messages sent via site hosts etc seemed to be resented rather than welcomed. It is a moot point whether the Webmaster Account is sufficient for this purpose; at any rate, it is being used for this purpose in possibly less than 1% of cases.
• Website owners who use a $10-a-year host running Apache v1 and PHP3, and with 1,000-plus sites on the same server and IP; who have no accounts with search engines; whose site code contains a lot of JavaScript cooked up by the guy next door and which doesn't appear in any textbooks or of course validate; whose site content is shared with several others; who are listed in several spam directories and link farms; and who have no records whatsoever of any of their site or network performance data - aren't really in a position to complain when they get hurt. And if you think this isn't you, so does everybody for some reason.
• Website owners in general need to display a much higher level of good maintenance practices before any credence can be given to the idea that they are often blameless.
• Search engines need to communicate much better or they cannot complain if regulations or restrictions are imposed on them.
• If a website owner or his agent buys links, and especially if they are bought from obvious sources or have labels like '$20 link exchange' attached, then the result is fairly predictable.
• If you sell links, and especially if you do so blatantly, then the result is predictable.
• If you set out to game the system in some way; or try to elevate your ranking in a clearly artificial manner, the system may bite you back. Why would you be surprised?

 
Right now there are a lot of voices calling for regulations to control search engines. Part of the problem is that they are not seen to be addressing issues that severely affect commercial and community enterprises. The establishment of some sort of system to resolve the frequent dilemmas connected with search penalties is one solution.

 

Search Engine Guidelines

In general the Terms of Service, or Quality Guidelines provided by search engines, are very similar. Some provide more material than others. They are intended to give advice and guidance on running a website without using any methods that may be construed as seeking an unfair advantage.
 
Depending on which country you are in, links vary. These links are mainly to central TOS pages, and to those specifically on topics that relate to search penalties in some way.

AltaVista Terms of Use

Ask.com Terms of Service

MSN Site Owner Help/ Webmasters

MSN Site Optimising

Yahoo info central

Yahoo! Terms Center - a list of TOS links

Yahoo Search - links, including TOS non-compliance

Yahoo list of TOS violations

Yahoo Webmaster resources

Yahoo Content/ Spam Guidelines

Google TOS

Google Webmaster Guidelines

Google on link schemes

Report paid links to Google

 

Lack of knowledge

You can often tell if people who claim to be able to help you actually know what they are talking about. All the following statements (when applied to normal website owners and not the undesirables) are untrue, and reveal a lack of knowledge:

• 'banned for life' - there is no known case of this happening, even for the worst offenders. Everyone gets another chance. They might be closely monitored, though...
• 'banned for years' - equally improbable since you would need to keep doing whatever they banned you for in the first place. You would probably go bust before that.
• 'eventually the spiders will return' - wrong. Spiders still go round your site even if you are experiencing a hard ban. They want to know when you have cleaned up your act.
• 'start a new site' - just wrong: (a) you don't need to; (b) you'll probably make the same mistakes again in any case. On the other hand, if this statement infers that you should be seeking another host - then in some cases you could be right.
• '...wait years to be re-indexed' - wrong. You will always have some pages in their index; and when you fix the problems, three or four months is more like it till you start getting the bulk of your pages back in. On the other hand, if you attack a search engine or operate a paid link farm, then precisely what do you expect?
• 'If you have lots of pages indexed you're not banned' - quite wrong, as we have seen. You can have thousands of pages indexed and still be hard banned - the only important factor, after all, is the search results positions.
• 'remove spam content' - fine, but that's one reason out of 30 or more you can get penalised. It is no more likely than any of the other 29 main reasons, therefore it doesn't make much sense to emphasise it. In fact there are much quicker and more certain ways to get banned. And you are unlikely to get banned for duplicate content, if that's what is referred to (see next item).
• '...banned for duplicate content' - wrong. The most likely result is that the duplicated page alone will receive no page rank / search results - which is only to be expected. Not only is this not a ban, it is not even a penalty: it is a filter. You just don't get any bonus (PR, good SERPs position) if you simply copied it from somewhere else. This refers to a normal site that has unwisely plagiarised a page or two; spam sites are a different proposition.  

 

Why we didn't list search penalty causes

You will of course have noted that we did not include any of the multiple possible reasons for being banned by a search engine; and especially the technical details. This is for many good reasons, among which are:

• Some cannot be fixed by non-technical personnel.
• When things are tough to sort out, as in every technical field there may well be two or more faults acting in concert; non-specialists will have a lot of problems resolving these interlinked faults.
• A partial list is of little use, since there is a good possibility that the factor/s responsible will not be on the list.
• Providing a partial list that does not resolve the error may result in us being considered to lack knowledge in this area.
• If a site owner fails to resolve the issues and then comes to us, our task is likely to be more difficult since the waters will have been muddied.
• Pure logic dictates that those who were associated with the site when it got banned are not the people with the knowledge and skills to unban it safely.
• Site owners who have been banned to a greater or lesser extent have clearly demonstrated a lack of knowledge of SEO and/or search engine compliance. They need strong direction, not a self-help guessing list.
• Even though we provide a range of instruction Guides to clients, they still need to ask many questions on site policy and related issues. It isn't possible to write down a precise guide to action in every situation because there can be so many variables.
• Reasons for getting banned can be found in many places if you search. They may or may not be relevant to your case. However, if you've been walking close to the edge of the cliff, you might be better off asking someone else's opinion, rather than going forward in roughly the same direction again.
• In some cases the blame lies with a third party. In this case, experts need to be in at the ground floor to ensure the case is handled correctly.
• In the many cases where a site owner has been badly let down by his agents, obviously the sensible answer cannot be to then give them a Get Out Of Jail Free card. It will probably happen again.
• And naturally: we love to help our clients - but why should we train our competitors in advanced SEO?
  

Resources

Search penalties (website bans)

Search Penalties Google Group
A place to ask questions and share knowledge on search penalties, malware search warnings and associated topics.

Live Search Webmaster Center Blog
The most helpful and communicative of the Big 3 search engines' senior staff, Jeremiah Andrick, posts here. The following post is particularly useful and should be read carefully:
I'm not ranking in Live Search, what can I do?

Live Search Webmaster Center Forums (MSN)
Program Manager Jeremiah Andrick engages with forum users on their search problems. This marks a Number 1 for the big search engines - a member of staff who will actually discuss your search problems with you, and investigate issues. Well done, Mr Andrick.

Malware

GeeksToGo/HijackThis
One of the original fighters against scumware, and still some of the best resources.

Badware.org
- the home of corporate anti-malware, backed by Google et al. This is your first stop if malware-warning banned by a major search engine.

...and their very hard to find forum. Some good resources, as is often the case, on this forum. If you can find it, you've probably done well.

Here's a script from that forum to alert you to a malware website hack. It needs some packaging, but would be a good place to start for a coder who wanted to put an alarm system in place.

www.searchenginehonesty.com
Presents the viewpoint that SEs should be more accountable for their actions.

SERPs penalties:
Stuntdubl's ban page
Didn't come across this page till late September '07, otherwise we'd have linked to it earlier. This is probably the best other resource on search penalties.
Found it via Google Alerts - a useful tool.