SBSM Mailbag: Four URLs, But Only One Page … Duplicate Content?

Filed in Featured, MY BEST POSTS, SEO by Matt McGee on May 18, 2011 2 Comments

In 10-12 years of doing SEO, this is one of the most common problems I’ve seen when doing site audits. A website has one page of content that’s available at several different URLs. SBSM reader Keith M. took me up on the recent post where I asked for mailbag questions, and sent this in for consideration:

mailbag-dupecontent

I’ve been reading about canonicalization and am curious about how this should be addressed. In looking at my old website, domain.com, www.domain.com, domain.com/index.html and www.domain.com/index.html all lead to my homepage, but the urls are all different and distinct. It sounds like a 301 redirect is the way to accomplish this, but:

1. Is it better form to use the www or non-www for your homepage?
2. Do all of the other 3 need permanent re-directs?
3. What of subfolders? Again, domain.com/about and www.domain.com/about go to the same page with different urls. Do they all need redirects or is this taken care of with the main urls?

My reply

First, Keith, let me say this: It’s generally not the end of the world when this happens in small doses. But when it’s happening across an entire website, I believe it can lead to less frequent crawling and unneeded distribution of trust/authority/juice across several URLs.

In either case, it’s something that really should be cleaned up. And yes, 301 redirects are the way to handle this. You want to tell the search engines that Page “A” is a permanent replacement for pages B, C and D.

Let me go through your questions in order:

1.) Is it better form to use the www or non-www for your homepage?

It depends. Here’s what I’d do:

  • Check to see which home page URLs are indexed. If only one is indexed, that’s the one you want to keep since it’s already there.
  • If more than one URL is indexed, I’d use some of the link tools that are available (like Open Site Explorer or Majestic SEO) to determine how many inbound links point to each URL. If one of them has substantially more than the other(s), I’d choose that one. If there’s not much difference in inbound links, you’re pretty much free to choose whichever URL you want.

2.) Do all of the other 3 need permanent re-directs?

Yes, after you choose the permanent URL, all others should be redirected (even if they’re not indexed).

3.) What of subfolders? Again, domain.com/about and www.domain.com/about go to the same page with different urls. Do they all need redirects or is this taken care of with the main urls?

If you’re doing a sitewide redirect — i.e., all www.domain.com URLs are going to be redirected to domain.com — you can set that up as part of the 301. I believe the term for what you’ll use is a “wildcard.” The end result is that one line of programming can tell the server and crawlers to replace all old URLs with the matching new URLs.

How you setup the 301 redirect depends on what kind of server your host uses. Apache servers generally have 301s setup in the .htaccess file, while IIS servers are a different beast. Either way, do some Google searches or maybe check the WebmasterWorld archives for the instructions.

Your turn: How would you have answered Keith’s question? Did I get it right? Do you have something to correct or add? Comments are open, so please share.

(image courtesy of Shutterstock)

Comments (2)

Trackback URL | Comments RSS Feed

  1. I would have answered largely the same way.

    .htaccess has been the go-to file for hand-editing in the 301s, but I tell ya, I seem to fat-finger it way too much and end up breaking something. I had taken to adding ONE 301 at a time, then saving and checking to make sure the site and redirected pages were fine.

    Recently, I installed the WordPress Redirection Plugin. It has been working like a champ – faster, easier and less prone to typos.

    For people that aren’t experienced systems people, a plugin is much easier than hand-editing .htaccess for apache.

  2. JM says:

    I agree with Alison, for bloggers using self hosted WordPress a plugin is the best way to go, but for large more complex sites (many of ours have .PHP url rewrite scripts that present dynamic URLs as static) a manual 301 redirect in the .htaccess file is preferable.

    And yes, backup the original file, make one redirect, test it, backup and so on…

Leave a Reply

Your email address will not be published. Required fields are marked *