A problem which many webmasters face is how to get search engines to index dynamic content. Although search engines these days are happy to index dynamic content, or at least happier then they used to be, the still have to be able to get at it some how.

Enter dynamic content

Yonder back we had a problem - countless different articles used countless amounts of pages and updating something on every page was something we wanted to avoid. Hence the arrival of databases and dynamic content. From one single page we could simply select the information from a database using an ID number and that same page could display any article from the database by changing a number in the URL.

The problem was that search engines had never dealt with this before. The same page changed and many had problems with this. Luckily search engines are equipped to deal with them now. Far from being confused they are now well educated on the dynamic content systems run by many websites.

The problem now is though that search engine also realise there would be infinite possibilities to the pages. If a database contained a million different news postings all linked to each other it could spend ages just indexing one site. So for this reasons some websites only index the page without any variables.

For example instead of indexing article.asp?ID=5 it would only index article.asp, messing up all your scripts and not showing any of your content. Therefore none of the content in its new beautifully designed page with you can easily update is lost to the search engine.

Searching for a solution

The first problem here of it not analysing variables can be counteracted; though it's not pretty. The first solution I thought of was to display every article on the page on after another. This way if a search engine does happen to knock off the variables if will still index all your content.

There are two problems with this though. The first is that if you have a lot of material this page is going to be huge, it will put a large load on the server. And visitors will also be sent to this page rather than the dynamic page with only one article on it meaning that not only is there more load on the server but also users are presented with this huge page where they then have to find what the want - if they can.

Not very good. Therefore the next solution I came up with cuts down on the indexing so all your content will be missed however it will get some keywords into the search engines index. If the variables are missed off then bullet point all your story titles (or if you have loads then just the latest 100) with links to the dynamic pages with the content.

Therefore your articles will sadly not be indexed however the story titles will and if these contain good keywords and phrases users will arrive at this page from the search engine and easily be able to find the article they want and click the link to the full story. With any luck it may also send the search engine into a loop as the dynamic content link will lead them to the same page - serves them right for knocking off our variables ;).

Hidden content

Another problem with search engines is that they can only follow links. While your users may be able to type something into a search form and hit go to find an article search engines will be stuck. Therefore you need to make sure there is a link to each of your articles that the search engine can follow.

Search engines don't need to have this link forever. So for example if you only have the latest 10 articles linked to on your homepage, as long as the search engine has that link already you don't need to place the link to the article. Once its indexed in all the search engine will do is return to the specific URL to make sure it still exists and check for any new content which won't be an issue here seen as its an archived article.

Although you must make sure that the search engines have time to index the article in before it is pushed off your homepage for good. A good counteractive to this is to make sure there is always some link to the article available somewhere on the site - visitors don't have to know its there, as long as search engines can follow it.

Creating a tunnel

Visitors will only stand for a view clicks to get to the content, because us humans are inpatient creatures. Luckily though search engines have no such flaw in them. Well they probably have something similar but that's a different story.

Therefore you can create a network of tunnels for the search engine to follow that the visitor will not be aware of. First off you want to create a page which displays the titles and links to all your articles or again if you have many just the first 100 or two. This page is just for search engines so don't link any pages to it yet.

Now go into your favourite image editing application and create a tiny graphic or spacer which is 1 pixel wide and 1 pixel high. This will be used as the link. Lets say the list of all your articles with links is called list.asp, and your homepage is called index.asp. Now open up your homepage in your HTML editor and insert the following code anywhere:

<a href="list.asp"><img src="filenamehere.gif" width="1" height="1" border="0" alt="the name of my content or keywords here" /></a>

This little button which is virtually invisible to the user unless they happen to be browsing through your source code and spot it will prove a link for search engines to pass through looking for the content keywords you placed in the ALT tag.

There are other ways to get search engines to index pages which visitors don't have access to or knowledge of. Another method would be to submit the URL to search engines as well as your homepage URL for indexing. Be aware though that no matter what method you choose visitors may end up on this list page directed there by the search engine though this is no great problem.

Conclusion

When designing a site most webmasters think about usability as for different types of browsers and operating systems. But don't forget that your site needs to be useable for search engines too. Make sure they can reach the content and won't get stuck.