What is a rewrite engine?

Contents

Examples of use
URL rewriting and search engine optimization

A rewrite engine is a component of web server software that allows you to rewrite or redirect uniform resource locators (URLs). The most popular rewrite engine is the Apache HTTP server’s mod_rewrite. There are other web servers, such as nginx or lighttpd, that provide similar functions.

Some content management systems generate cumbersome URLs. This software component is used to convert them into user-friendly URLs. The reasons for this are obvious: technical URLs such as

"http://example.com/a/index.php?title=pagetitle"

aren’t easy to remember; a rewrite engine enables the URL to be written in a much more intuitive way:

"http://example.com/article/pagetitle"

An internet user can also use this URL to access a corresponding web page. If a request like this is received by the web server, the rewrite engine automatically converts the URL into the internal one used by the server

"http://example.com/a/index.php?title=pagetitle"

The rewrite engine creates an abstraction layer between the URLs used internally by the web project and the URLs publically displayed online. This makes it possible to provide a user-friendly, consistent address scheme, regardless of internal technical requirements.

Internally, the dynamic, parameterized address can continue to be used while users from the internet access the web project via a seemingly static address. This function has the advantage that externally-presented URLs remain valid even if internal changes are made to the file hierarchy.

In addition, website operators can use the rewrite engine to implement address redirects, which can be linked to specific conditions. In effect, it is possible to set up redirects based on the user agent string or the IP address of the requesting client, in order to implement geo targeting or to have targeted, optimized websites appear on different devices. A 301 redirect is generally used, which ensures that only one version of the website is stored in the search engine’s index, despite the parallel operation of additional mobile websites or different language versions.

Website operators should make sure they stay away from cloaking practices, which is where optimized sites are specifically created and shown to search engine crawlers in order to improve rankings.

Examples of use

The rewrite engine provides various commands for manipulating URLs, which can be identified as rules in different parts of the web server software. Mod_rewrite, the rewrite engine of the Apache web server, can be used in a directory container within httpd.conf, in a virtual host section or within an .htaccess file. With nginx, URL rewriting is recorded in the configuration file /etc/nginx/nginx.conf. With lighttpd, the /etc/lighttpd.conf file is available in the vHost configuration.

In order to rewrite the dynamic URL "http://example.com/a/index.php?title=pagetitle" via the rewrite engine into the static URL "http://example.com/article/pagetitle", different commands are used with Apache, nginx, and lighttpd web servers.

Apache’s rewrite engine

To use mod_rewrite with Apache, the rewrite engine needs to be activated with the RewriteEngine on directive. This is followed by the RewriteRule, which defines instructions for URL rewriting using regular expressions (Regex):

RewriteEngine on
RewriteRule ^/article/(.*)$ /a/index.php?title=$1

If rewriting needs to be defined by a RewriteRule, then it can be broken down into two parameters: the search pattern and the target pattern.

Search pattern: this pattern describes the URLs that are to be redirected. A certain condition is defined in the form of a search pattern. If this condition is fulfilled, the URL can be rewritten according to the target pattern. In the current example, the search pattern would be the following section of the RewriteRule: ^/article/(.*)$.

Target pattern: this parameter describes the target URL. If the redirection is configured at server level, the complete URL is replaced. At directory level in the .htaccess file or within the httpd.conf, only the path from the current directory onwards is replaced. In this example, the target pattern includes this section of the RewriteRule: /a/index.php?title=$1.

An explanation of the expressions regularly used in the examples can be seen in this table:

Regular expression	Explanation
^	Beginning of string.
$	End of string.
(.*)	A place holder for any string within a URL. The parentheses store the string in a variable.
$1	A variable that allows access to cached values that are stored within parentheses.

The RewriteRule ^/article/(.*)$ /a/index.php?title=$1 defines the rule that all URLs beginning with the string /article/ should be converted to the dynamic URL scheme /a/index.php?title=$1, where $1 is the string that corresponds to anything that comes after /article/, captured by the place holder (.*).

If a user enters the static URL "http://example.com/article/pagetitle" into the web browser, the web server converts this internally to the dynamic URL "http://example.com/a/index.php?title=pagetitle", based on mod_rewrite. This is invisible to the user. In this case, the place holder (.*) and the variable $1 correspond to the 'pagetitle' string.

If URL rewriting is to be linked to certain options that control the behavior of the mod_rewrite, they are listed in square brackets after the RewriteRule, and separated by commas, if there are several. In this way, external redirecting via HTTP status code can also be implemented. The following table shows a selection of options for the RewriteRule. A complete list can be found on the official website of the Apache Software Foundation.

Option	Flag	Function
Redirect	R	The [R] flag causes an HTTP redirect to be issued to the browser with the 302 status code. To use another status code, append it with an equal sign to the flag, e.g. [R=301].
Forbidden	F	This instructs the webserver to send the web browser a 403 (forbidden) HTTP status code.
Gone	G	The webserver sends the browser the HTTP status code 410 (gone) and shows that a resource that used to be available is no longer available.
Last	L	This flag causes mod_rewrite to stop processing the rule set.
Nocase	NC	When verifying whether a URL meets the rewriting conditions, upper and lower case do not matter.
Chain	C	The next RewriteRule is only considered if the current condition is met.

Based on such an option, an external forwarding via HTTP status code could be implemented as follows:

RewriteEngine On
RewriteRule ^oldpage.html$ /newpage.html [R=301]

In addition to RewriteRules, mod_rewrite can also be used to define RewriteConds which can be used by website operators to set additional conditions that must be met for URL rewriting.

The syntax of a RewriteCond corresponds to the following structure and is noted before the RewriteRule:

RewriteCond TESTSTRING CONDITION

The test string typically contains server variables which are defined by percent signs and curly brackets such as %{HTTP_HOST}. The following table shows a selection of server variables.

Server variable	Explanation
HTTP_USER_AGENT	Refers to the client used for server access. The variable is generally used to provide an optimized website to various web browsers.
HTTP_HOST	Refers to the hostname. This can include values such as domain.com, subdomain.domain.com, or the IP address.
SERVER_PORT	Refers to the addressed port (e.g. 80 for HTTP or 443 for HTTPS). The variable allows website operators to redirect visitors to a secure connection.
REMOTE_ADDR	Refers to the IP address of the user accessing the web server. This variable is sometimes used to block spam attacks.

The following example shows a RewriteCond that binds a subsequent RewriteRule to the user’s IP address:

RewriteCond %{REMOTE_ADDR} 173.45.68.79

URL rewriting with nginx

The webserver, nginx, also supports URL rewriting natively. This can be implemented by using regular expressions. In order to convert URLs, the rewriting command is simply inserted into the web server configuration file /etc/nginx/nginx.conf according to the nginx syntax in a { [...] } block:

location /article {
 rewrite ^/article/(.*)$ /index.php?title=$1 last;
}

The location /article part tells website operators that URL rewriting is referencing the subdirectory article. The regular expressions for rewriting correspond to those that are also used with the Apache web server and are initiated with the rewrite command. The last flag indicates that the rewriting should be done internally and therefore without being redirected. The following flags are available for temporary or permanent redirects:

Flag	Explanation
last	URLs are rewritten internally. There is no redirecting.
redirect	The user is temporarily redirected to the new URL by a 302 redirect.
permanent	The user is permanently redirected to the new URL by a 301 redirect.

If no flag is set, then nginx automatically assigns the HTTP error code 500.

Rewrite with lighttpd

With lighttpd, URL rewriting is implemented on the basis of the url.rewrite-TYPE function. The place holder TYPE stands for various rewriting configuration options:

Rewriting configuration options	Explanation
url.rewrite-once	One-time URL rewriting. If the search pattern is found and the corresponding URL is rewritten to the target pattern, no more rewriting needs to take place.
url.rewrite-repeat	In contrast to url.rewrite-once, url.rewrite-repeat can be followed by further rewriting.

The syntax basically follows the same pattern as with Apache when rewriting with lighttpd since the same regular expressions are used:

url.rewrite-once = (
 "^/article/(.*)$" => "/index.php?title=$1"
)

If an external redirection is to take place instead of an internal one in lighttpd, the rewriting module isn’t used, but rather a redirect module where URLs pass through the rewrite, then the redirect module.

Rewrite with Microsoft IIS

The platform Microsoft Internet Information Services (IIS) doesn’t natively have a rewrite engine. However, this can be added later on to the webserver with the Modul IIS URL Rewrite 2.0. This also allows Microsoft users to make URLs available to their website visitors, without having to interfere with the internal file management. After downloading, the URL rewriting extension is integrated directly into the IIS Manager interface where RewritingRules are entered via a graphical user interface. IIS URL Rewrite 2.0 uses regular expressions to define URL search and target patterns.

URL rewriting and search engine optimization

Through rewriting, parameterized URLs can be converted into meaningful URLs, which makes them more user-friendly. Because of this, the functions of mod_rewrite and corresponding implementations in other web server systems are often discussed when talking about search engine optimization. It’s been a topic of discussion for a while as to whether user-friendly URLs count as ranking factors since there is no clear evidence for a direct relation. However, website operators are likely to benefit from indirect efforts. In contract to cryptic parameters, rewritten URLs allow internet users to understand where a link leads. Rewriting can therefore be used to build trust and possibly increase click-through numbers. In addition, search terms in URLs are displayed in bold on the search results page, which can provide an additional incentive for clicks.