Malware bans / malware warnings
'Zero guilt' bans
Search penalty problem areas
Complying with unpublished guidelines
Buying and selling links safely
Duplicate content
Search optimising: good or bad?
Recovery and reinstatement
Recovery from a search penalty
Reinstatement of website
 

Malware bans / Malware search warnings

Search engines will either de-list websites implicated in spreading malware, or (in the case of Yahoo and Google) place a clear warning on the search result and remove the click-through attribute. This is normally a result of:

(1) Malware of some kind being available on the site for download
(2) Linking to a site that has malware
(3) Displaying adverts that link to malware sites
(4) Displaying adverts that are infected by malware
(5) A website security exploit leading to malware implications

We have seen examples of all of these; but (4) and (5) are the most common. Point (5) implies the site has been compromised in some way - there is more than one way this can happen - and links to malware sites, or malware itself placed on it; sometimes in such a way that viewed pages are infected. The result may be simply that a tracking device is downloaded and attached to the PC, so that surfing may be monitored; or a more serious type of trojan may be involved.

These invaders are invisible to the PC owner, though the use of a normal PC security policy would probably restrict the level of damage. Both a hardware and software firewall, together with a top-rated antivirus app that updates online daily and at least two separate antispyware apps, are the minimum standard now required for safe surfing on a DSL connection. Using a more secure browser than the usual choice may also help.

In contrast to the situation for search penalties, there are some good resources for technical personnel who need to investigate a malware ban. See our list at the foot of the page. Yahoo are currently seen to be the most active in the area of malware warnings and site issue resolution.

Displaying adverts infected by malware seems to be an increasing problem. What happens is this: a website shows display adverts from an advertising network other than that of the search engine concerned. One advert has been infected with malware: a mini-program that causes visitors who click on it to download a trojan of some sort. These attacks are probably only successful if the visitor is using an insecure browser and has either no security software; ineffective security software; or non-updated security software (firewall, antivirus, anti-spyware). However, this may well apply to a substantial proportion of Internet users, as is evidenced by reports that up to 20% of PCs are botnet slaves. Even half this number would be astonishing, but many researchers seem convinced of these figures.

Many respectable sites have been caught by this, and the problem can be both invisible and incredibly frustrating. By the time the penalty is observed, the guilty adverts have disappeared, and there is no visible cause of the ban. The situation is exacerbated by the fact that search engines will not usually comment on their reasons for applying the penalty*, so that the resolution / reinstatement process becomes extended, complex and frustrating.
* Note: improvements have been seen here of late.

Because of this, some search engines have come under severe criticism. In addition, large and important websites that displayed infected adverts and therefore compromised many visitors' machines were not penalised. This has resulted in claims of a double standard: mom & pop shops are hit hard, but the big names are invulnerable. No doubt this is true; but that of course is life, and it would be extremely naive to think that small businesses receive the same treatment as household names. It would be far too costly (in a multitude of ways) for a search engine to give large enterprises anything other than preferential treatment; business is business, after all.

There might be another explanation for their silence, which is that the search engine itself may not actually know the answer. People have speculated on all sorts of reasons why they never reply, such as legal reasons or to maintain security; but as yet no one has suggested that they don't know. In order to have the answers, there would need to be a logfile showing all the pertinent operations. A logfile on the indexing of 20 billion pages or more (Bing stated this figure in the past) would be fairly large; and one thing you couldn't say and maintain any credibility is "We don't know." Better to keep silent.
 

Zero Guilt bans

Being penalised due to unwitting display of infected adverts has come to be known as a zero guilt ban. In fact the procedures for lifting the ban are clearer, more likely to result in progress and faster than those concerning a search penalty - especially a manual adjustment SP. Some other cases may qualify for this particular appellation, possibly the Snap SP.

Search penalty problem areas

A website and its operation can be broken down into four areas:

• The network, the host's facility, and the server
• The site and page code
• The site content
• Offsite policy

Problems are equally likely to occur in all three of the site areas, and also in the matter of offsite management. It would be unwise to emphasise any one area as the main source of errors. Offsite, there is one major area of problems: paid links, which should be avoided.

The site owner has limited or no access to some of these areas; in others they are totally responsible. Or, in cases where the owner has no personal interaction with the site, his agents are responsible.

It is advisable, as a good operational principle, to maintain as much data on the site as possible, from as many different sources as possible, both internal and external. In many cases where an SP needs investigating, historical data is extremely valuable. It is also possible that, had those metrics been available and studied in the first place, a penalty would not have been applied to the site at all.

As far as paid links go, you need to avoid them. Don't buy them, and don't sell them. It is true to say that if your site is small and has a low profile, then you will most likely get away with things that bigger and louder offenders won't. If you stick your head above the parapet, expect to get it shot off.

Complying with Unpublished Guidelines

Making sure that one does not violate a search engine's published Terms of Service is only part of the equation - complying with their unpublished TOS is more difficult. Since those unpublished requirements might equal at least double the volume of the published ones, it is not an easy task.

These requirements are the mysterious technical details that are never mentioned, are numerous, and non-compliance with which (even though they are undefined) can result in a search penalty. The only way to comply with these is by trying to follow the spirit of the published guidelines, and then by trial and error; and only trial and error (in the absence of any form of communication) works. The basic problem is that management of a large modern website is so complex that there will inevitably be some technical apsects that are not covered by any guidelines; and these in no way whatsoever verge on the unethical: they are simply alternative technical paths.

In this field especially it is extremely frustrating when search engine operators refuse to communicate* on possible problem areas. As an example, it is sometimes necessary to redirect visitors for entirely legitimate reasons - their language, country of origin, browser type, and so on. Or, calls may have to be made to databases or other resources, that involve page redirects. This area is one in which non-compliance may seem to be detected by a search engine (for the best of reasons), where in fact none has taken place.
*Now, at last, a representative of one of the Big 3 is communicating with website owners, on the Live Search Webmaster Center Forums (MSN). Program Manager Jeremiah Andrick engages with forum users on their search problems, at:

http://forums.microsoft.com/webmaster/

Mr Andrick should be congratulated on doing something that none of his competitors have been able to achieve.
 

Buying and selling links safely

Neither of these are possible, because they clearly violate search engine guidelines. Paid links are not allowed because they disturb the link popularity value rating, one of the two core factors in the page value equation.

However, this has nothing whatsoever to do with advertising. Obviously, you can buy or sell adverts freely. Why would that not be permitted? It is one of the two core requirements for Internet visibility, together with natural search results. You could argue about which is of more use (in which case you are either an optimiser or a marketer), but both are required to do the job properly.

The key difference between adverts and links is that ads are not links. They are 'masked' links, in that the graphic or text does not link directly to the resource being advertised, it is connected via a JavaScript or other neutered 'link'. Some search engines can read and follow these links, some can't - but all of them recognise the fact that this form of link designates an advert - a type of paid link, if you like, that is above board because everyone can see it for what it is: a paid advert.

Commonly, there are problems with text links (i.e. paid links). This is because people think they can surreptitiously acquire them commercially and still increase their PR; or think they can sell them freely without penalty. Why that would be is a mystery, since it clearly infringes the SEs' TOS. Perhaps, because for a long while it was possible to do just this and escape penalty, people expected that situation to be permanent. However, staffing levels have improved dramatically at some search engines, and that has no doubt allowed them to devote more resources to preserving the integrity of the search results.

NOTE VERY CAREFULLY that placing some sort of text message in the vicinity such as 'Sponsored Links' is completely pointless and irrelevant. Only the type of link is of importance. Where valuation of these matters is performed by AI, the algo cannot see any 'disclaimer' notice nearby as relevant; and manual inspection only considers the link type, not adjacent text or warnings.

Adverts are safe because the linking method clearly designates them as sponsored. Part of the reason you pay an advertising network is to supply a range of sponsored listings that are protected from harming either advertiser or publisher, since the links are correctly masked.

The same effect can be gained by tagging a paid text link (or any other) with the rel=nofollow attribute, which is recommended as it allows non-coders to add a simple HTML tag to do the job (otherwise, JavaScript obfuscation is commonly used). Here are some ways of using it - these two methods are used in the page code to tag a link as not to be followed:

<a href="http://www.mysite.com" rel="nofollow">Ethical SEO Agency</a>
<a rel="nofollow" href="http://www.mysite.com">Ethical SEO Agency</a>
   
The second example above is how many dynamic applications such as CMS 'repackage' the link as given in the first example, and is accepted by the SEs (check your page code to see this happening).

This link format is how things turn often turn out in practice, and is followed by an image source etc.:

<a href="http://www.mysite.com" rel="nofollow">

This metadata tag is added to the page header metadata and indicates that no links anywhere on the page are to be followed:

<meta name = "robots" content = "nofollow">

This method was introduced in early 2005, so is not new, and is supported by the big three search engines; and now possibly others. There is plenty of evidence that search engines do in fact follow these links, and even allow a small amount of linkweight for them; but the links are safe because they have been clearly designated as either paid or unwanted. The latter designation is appropriate for blog and site comments, where inappropriate linking cannot be prevented; though it may be preferable to 'dofollow' them and moderate them carefully if possible, in order to allow useful contributors full value. Nofollowing links, as is the default in WordPress for example, means that both useful comments and useful contributors will be far less in evidence - at least as far as most sites are concerned.

Duplicate content

It is normally unlikely that 'dupe content' will attract a ban. Search penalties are not given for this sort of thing, simply because it is a normal and accepted - and quite respectable - business practice on the Net.

Consider the case of an ecommerce store: it may have ten thousand product pages, but perhaps only five hundred category and brochure pages. It is quite common for most or even all of the product pages to be identical on every other ecommerce store selling the same items. All 5,000 of them. Does that mean that 4,999 sites receive a search penalty? Of course not!

What is much more likely to happen is that these pages attract a filter, not a penalty. There is absolutely nothing wrong with 5,000 sites displaying multiple identical pages, as you can see - there is no attempt to deceive anyone, and the content certainly wasn't stolen. Of course, as is patently obvious, it is hardly a good idea though - the pages attract a filter, because of the duplication, and receive no page rank whatsoever. This is not a ban, it is a filter, preventing thousands of sites ranking well for the same content. They rank zero, since a major principle of search results is that there can only be one page with any given content ranked highly in the SERPs (otherwise the results would be useless). This is a feature that is extremely hard to enforce, for various reasons - there will always be a multitude of pages returned that have fairly similar or even identical content - but, in general, there is always a clear winner. The snippets shown may well be identical, which allows you to choose the highest-positioned page and ignore subsequent pages.

For that reason, it is therefore unwise to display the same bare product details as every other site. Make the page different enough, and the filter will no longer apply. You can test this out by checking all the duplicate product pages out there; they have zero PR - and in fact a greyed-out Google TBPR (1) bar, if you have that installed - indicating even less than zero (2).

1 Toolbar PageRank
2 Well - if an all-white toolbar gauge = PR zero, then what does a greyed-out bar equal? Obviously, less than zero... And I believe this has been mentioned in passing by a search engineer.


There is a different situation, though, where spam sites show scraped content or entire copied sites. These do attract the attention of Mr G, who quite rightly gives them a search penalty. So, when talking about duplicate content, it is necessary to distiguish between rightfully-duplicated (but worthless) bulk content; occasional duplicate pages (such as reprinted articles); and spam sites. Duplicate pages will get filtered; spam sites will get SP'd.

Duplicate content will not get you banned, even if you go for it in a big way as many poorly- advised ecommerce sites do; but spam sites are a different proposition.
 

Search Optimising: good or bad?

Without correct search engine optimising, your website is unlikely to receive the search position it deserves, because so many technical and content factors will be sub-optimal. To state otherwise would be foolish; and in fact most of the major search engines, and some of their staff members where in the public eye, are on record as advising site owners to optimise their sites. Here is one senior SE staffer talking about the necessity for SEO and VO (visitor optimising).

There are problems here, though, since while there are safe and conservative consultants, there are also rapid results specialists, cheap submissions operators, $50 #1-position purveyors, and various other forms of ignorant or unscrupulous contractors. It is an unfortunate fact that unethical or ignorant operators of this type have caused a lot of problems for many site owners. The methods they use to get results faster and cheaper often result in sites eventually being penalised. Google go to great lengths to advise on the type of consultant you should hire.
You can see their advice here.

In mid-2008 this page was updated. Much new information was included, but more importantly the entire attitude and tone of the page has changed. Peviously the page tended to denigrate the value of SEO to a certain extent; this has clearly changed and it's now a very balanced viewpoint. They need to point out the negatives, and do so very well. It's certainly a good starting point for anyone thinking of hiring an SEO consultant.

We should keep in mind that the search results are being attacked on a wide scale, and therefore Google et al are on the receiving end. On a separate but very closely related note, they have at last pointed out that you really don't want too much JavaScript on your website pages - in their usual rather subtle way. You have to learn to read between the lines when looking at anything from the Secret Society.

It is perfectly true that owners and webmasters can research the techniques used and make a fair job of both presenting the site well, and promoting it correctly and ethically. However, in a competitive market the 40 to 50% level of correct presentation they might achieve (if they devote a very great deal of time to research and to the process) will not be sufficient. As in any other complex and expanding technical field, specialists always do a better and more complete job. Since there are more than a dozen specialities within SEO now, it could hardly be otherwise. You might be able to represent yourself in court in the matter of a parking offence, but it would be unwise to do so in a complex corporate law case.

Poor quality control

One last point, on this new Google page advising on SEO issues. Indeed it has been upgraded with some concession to sanity, but unfortunately the quality level of the writing, and occasionally logic, is quite low. The page has not been spell-checked, proof-read or logic-checked (at 2008-09-27). This seems strange when considering the public face of a multi-billion dollar company, but obviously they are careful with cost control and don't waste resources where not considered vital. There are upwards of ten spelling, grammar and logic errors, and multiple duplicated sentences. Oh well; the SEO area never was a popular one at Google, one supposes.

Most of these can be dismissed as the result of junior staff being made responsible for public areas without being checked; but one serious error cannot be allowed to pass. Here is a quote:

There are a few warning signs that you may be dealing with a rogue SEO. It's far from a comprehensive list, so if you have any doubts, you should trust your instincts. By all means, feel free to walk away if the SEO:

# owns shadow domains
# puts links to their other clients on doorway pages
....... ..... .....
# requests your FTP account information or root access to your server

The items in this list are all eminently sensible apart from the last, which is a hilarious bloomer. An SEO consultant is a website developer by definition; just how he is expected to work on a website without access to it is not easy to understand! By telepathy perhaps?

Or perhaps Google's view of SEO is that it has no connection with improving a website's content and code, it's just about buying links and managing the PPC spend. That does explain quite a lot...
 

Recovery and Reinstatement

Although there have been many complaints about the various search engines' reluctance to communicate meaningfully with users, Google probably provides more information and assistance than most. This is primarily because the toolset and metrics in the webmaster/ site account are much better than their competitors provide.

The first step to recovery from a search penalty is to open a website account with the top three search engines. The links to these follow:

MSN (LiveSearch) account

Yahoo Site Explorer (site account) 

Google webmaster account

There is usually a three-week timelag seen in anything to do with search optimising: whatever you do, results show after three weeks, unless the site is spidered on an hourly basis, as some are. Therefore, full data may not be seen for a while, though you should see enough useful metrics after a week.

Issues: you may have multiple Google webmaster accounts for your websites, but only one Adwords account is allowed (for the PPC paid advertising side). Be very careful, then, not to duplicate accounts in the paid search department.

Then, go to the official forums for the search engine concerned. At early '08, you will perhaps have most success on the MSN / Live Search forum, since MSN staff are willing to assist with specific issues. This is the only known instance of SE staff personally assisting with customers' problems in this area on a regular basis; we need to be fully appreciative of the effort they are making here. You are recommended to support them.

Recovery from a search penalty

Recovery from algorithm-allocated penalties can often be rapid, and less than a week is often seen here for the simplest cases. This rapid recovery schedule is by no means uncommon and may even apply to the majority of simple, single-cause, one-off mistakes; on-page errors and page code errors are typical here. The algo seems to respond very quickly in a positive way, when these simple errors are removed. Of course, a fast resolution can only take place here if the site is spidered regularly, in order that changes are seen and reported.

However, regaining page rank - and especially TBPR - takes longer. In some cases, algo-induced penalties take up to a month to clear, and this is more likely to apply when a network error has occurred (i.e. something affecting the website before an actual page is reached).

The first results are seen in the search results (and certainly not in TBPR). Then, pages in the index are seen to be increasing. PageRank as measured by online tools interrogating the datacentres is seen to rise or recover. Lastly, TBPR finally recovers.

Recovering from manual adjustments (human-allocated penalties) however can be a fraught and long-drawn-out process; two months can pass before the request for reconsideration is even acknowledged.

First, an application for reconsideration is needed: this means a message to the SE concerned, applying for reinclusion in the index and/or removal of the penalty. Then, a long wait can ensue that will probably last a considerable time. After this, all being well, search results start to improve and recovery takes the same path as that for algo-induced penalties. The process is very much more extended than that for algo penalties.

There is a common problem at this point, involving a proportion of those affected and especially those with large sites or complex cases, where problems may be hard to detect for the site owners. In some cases this is absolutely not their fault, as it involves complex technical factors. When search engines will not communicate at all with the site owner on what the nature of the problem might be, it can be difficult to resolve an issue.

Here is a good resource to help you recover. It explains the issues well, and provides a route for reinclusion that has a good chance of success. This is partly because you can also communicate directly with the search engineers responsible, in their forum.

http://www.bing.com/community/blogs/webmaster/
archive/2009/03/19/getting-out-of-the-penalty-box.aspx
[rebuild this URL to one line with no spaces]

Reinstatement of website

It is our firm opinion that THE INDIVIDUAL SITE OWNER OR HIS AGENT SHOULD NOT ATTEMPT TO GET A WEBSITE REINSTATED. 'Agent' here refers to anyone working for the site owner at the time the website received the penalty.

The simple reasons for this are:

• In many of the cases we have seen, the owner was actively trying to have the site reinstated while there were still errors in place. This inevitably makes the whole process more complicated and extended.
• An owner or his agents are not likely to have the background to resolve these problems successfully. This is only logical, since they were managing the website when the problems occurred. It can be hard enough for experts.
• Attempts to have the website reinstated after any type of ban should not be entertained unless you are absolutely and 100% certain the problems have been fixed - and are unlikely to return.

In 99 out of 100 cases the site owner tries to get their site reinstated while there are still issues. We can't emphasise enough that you first need to be absolutely sure the problems have all gone. It is extremely unlikely that anyone involved with the site when it received the penalty or penalties will be able to clarify the issues. This especially applies to website managers or developers who were employed at the time, who in our experience are usually more guilty than non-technical owners.

Another factor is that your hosts can easily sink you, and sorting this out is beyond a site owner (and obviously any current website manager).

It goes without saying that arguments or abuse are unlikely to help your cause.
Continued: Banned Websites Part 2a >>