Cache Expiration

Introduction

The caching mechanisms specified in the HTTP/1.1 Protocol offer significant performance improvements to an application by eliminating as many requests to the server, and full server responses, as possible. Of the two main mechanisms: Validation and Expiration, Expiration offers the greatest caching benefits by allowing caches to avoid sending requests to the origin server all together.

The system used by the HTTP 1.1 Protocol to determine the age and "freshness" of cached server responses, is called the expiration model. This model applies to responses (entities) that have had expiration times assigned to them by the server. These entities are used by a cache on the client side to validate the response. The model does not apply to the first-hand responses forwarded immediately to the requesting client.

The use of the expiration model is straightforward and flexible. By making use of it effectively, an application will appear faster, because it will reduce the amount of data and data connections that are sent needlessly. These savings can help keep a user beneath their data cap, keep their battery from draining as quickly, and improve the responsiveness of wireless networks that have limited capacity.

This Best Practice Deep Dive describes the components of the expiration model that are specified in the HTTP 1.1 Protocol. It gives an example of the type of issue that occurs when the expiration model is not used correctly, and it provides recommendations for how to take advantage of the cache expiration mechanism in an application.

Background

Caching is most effective when a cache is able to entirely avoid sending requests to the origin server. The HTTP/1.1 Protocol specifies an expiration model that is based on server-specified expiration times. By making calculations based on the expiration time, a cache is able to return a fresh response without having to send a request to the server first.

Servers can specify implicit expiration times using the Expires entity-header, or explicit expiration times using the max-age directive of the Cache-Control header. The explicit directives in the Cache-Control header always take precedence over implicit information like the Expires date/time. This section describes the Expires entity-header, and how it is used to determine if an entity is still "fresh". For a discussion of the Cache-Control directives and how they are used, see Cache Control.

Expires entity-header

Each time it serves a response, the server adds an explicit expiration time to the response entity by putting a value in the Expires entity-header field. The Expires entity-header field gives the date/time after which the response is considered stale. This does not mean that the original resource will change or cease to exist based on the Expires date/time. It means that a cache should not return the response-entity after that time, unless it is first validated against the origin server.

The Expires entity-header field uses the following syntax (where HTTP-date is an absolute date and time that must be in the RFC1123 date format):

Expires = "Expires" ":" HTTP-date

For example:
Expires: Thu, 01 Dec 1994 16:00:00 GMT

The value of the Expires date/time can cause the following specific cache behavior:

• When the Expires date is equal to the Date header value, the response is considered to be expired.
• When a response has an Expires header field with a date/time that is in the future, then that response is considered "cacheable".
• Servers should not set an Expires date that is more than one year in the future. When it is set to one year from the time the response is sent, then the response is considered to "never expire".
• By assigning an Expired date that is in the past, the server can force the cache to validate the response. Even though this method ensures cache validation, it is not recommended. The recommended method is to use Cache-Control directives.

Determining if a cached entry is fresh

To determine if a cached entry is still "fresh", a cache must calculate two values: The age of the entry, and the "freshness lifetime" of the entry.

If the age of the entry is less than the "freshness lifetime", then the cached entry is fresh and can be used. If the age of the entry exceeds the "freshness lifetime", then the cached entry is stale and a full response must be sent.

Calculating the age of a cache entry

When calculating the age of a cache entry, it's important that a host should synchronize its clock to a globally accurate time standard by using NTP or a similar protocol. The age is calculated using date and age values contained in the Date header from the origin server, and the Age response-header in the response.

• Date header: HTTP 1.1 requires origin servers to send a Date header, if possible, with every response, giving the time at which the response was generated. The value in this header is called the date_value and it assumes that the local clock is reasonably well synchronized to the origin server's clock.
• Age response-header: HTTP 1.1 uses the Age response-header to convey the estimated age of the response message, when obtained from a cache. The accumulated time that the response has been resident in each of the caches along the path from the origin server, plus the amount of time it has been in transit along network paths. The value in this header is called the age_value and it assumes that all caches along the response path have implemented HTTP 1.1.

The following terms define the values that are used to calculate the age of a cached entry:

Term	Definition
age_value	Age field value. Received by the cache with this response.
date_value	Date field value. When the response was created by the origin server.
request_time	The local time at which the client sent to request to the server.
response_time	The local time when the cache received the response.
Now	The current (local) time.

These age, date, and time values are used to determine apparent and corrected age values as follows:
corrected_received_age = max(Now - date_value, age_value)

Because of network delays, it's important that the age of the resource be calculated based on the time it was requested and not the time it was received.
corrected_initial_age = corrected_received_age+ (Now - request_time)

The following terms define the calculated age values that are used to arrive at the current_age of a cached entry:

Term	Definition
apparent_age	The maximum of (0, response_time - date_value).
corrected_received_age	The maximum of (apparent_age, age_value).
response_delay	The response_time - the request_time.
corrected_initial_age	The corrected_received_age + the response_delay.
resident_time	Now - the response_time.
current_age	The corrected_initial_age + the resident_time.

When a response is generated from a cache entry, the cache must include a single Age header field in the response, with a value equal to the cache entry's current_age.

Calculating Freshness Lifetime

The "freshness lifetime" of a cached entry is calculated based on the date/time value in the Expires entity-header field called the date_value, or the value of the max_age directive in the Cache-Control header field called the max_age_value. The max_age_value has precedence because explicit directives in the Cache-Control header always override implicit information like the Expires entity-header date/time.

If the max_age Cache-Control directive is present, then the freshness lifetime is calculated as follows:
freshness_lifetime = max_age_value

Otherwise, if the Expires header is present and the max_age directive is not, the freshness lifetime is calculated as follows:
freshness_lifetime = expires_value - date_value

The server specified expiration times in the Expires entity-header and the Cache-Control directives max_age or s_maxageare the main controls that the server has in the expiration mechanism of HTTP 1.1. Origin servers should provide explicit expiration times as much as possible. If they do not, caches typically assign "heuristic expiration" times to the response. These are calculated based on algorithms that use other header values like the Last-Modified time to estimate an expiration time. Heuristic expiration times are not recommended because they are not semantically transparent.

Note: For more detailed information about the caching expiration model. See the HTTP/1.1 Protocol Specification (RFC2616), Section 13.2, hosted by the World Wide Web Consortium (W3C).

The Issue

While analyzing trace results from several popular mobile web applications using the AT&T Video Optimizer diagnostic tool, it was discovered that many of the applications downloaded specific files repeatedly, even though the local versions of the files were still identical to the versions on the server. This indicated that these applications had not implemented a response cache (also known as a response-entity cache).

In most cases, mobile web applications use content that has already been optimized for the mobile web. This means that each resource already has the required caching information in its header. When this is the case, an application should implement a local response cache.

The following example of an HTTP response header from a popular news feed demonstrates this:

HTTP/1.1 200 OK
Server: Sun-ONE-Web-Server/6.1
Date: Fri, 25 Feb 2011 23:20:28 GMT
Content-type: text/html
Set-cookie: adxcs=-; path=/; domain=example.com
Expires: Thu, 01 Dec 1994 16:00:00 GMT
Cache-control: no-cache

In this example, the server was configured to use two different methods for ensuring that the content wouldn't be cached. Not only were the methods used incorrectly, but the practice of ensuring that content won't be cached by forcing revalidation, makes the application that uses this content very inefficient.

The header in this example forces revalidation in two different ways:

• By deliberately specifying an Expires value that has already passed. When you compare the Date value to the Expires value, you can see that the Expires value is set to be over 7 years in the past.
• By using the no-cache Cache-control directive.

As specified in the HTTP/1.1 Protocol Specification, Cache-Control directives override the expiration mechanism, so that the value set in the Expires field is ignored. The net effect of this server configuration is to force revalidation with the no-cache Cache-Control directive. This adds network overhead and latency—which actually degrades the user experience.

Best Practice Recommendation

The Recommended Best Practice is to ensure that caching is enabled on the server-side, and that a response entity cache is implemented in your application that makes the best use of the HTTP 1.1 expiration model.

Once caching is enabled, you should develop a caching strategy for the server. This strategy should consider the different types of resources in your application, and estimate how long clients should consider each type of content fresh. The following table shows an example of some expiration estimates for typical types of resources.

Type of Resource	Expiration Estimate
Sports team logos	Usually have a long shelf life (e.g., five years).
Blog posts	Several minutes to several hours.
News images	Twelve hours.
News stories	One or two minutes.
Weather information	Five to ten minutes.

Using the expiration estimates for the content types in your application, configure the server accordingly. For each type of resource that is served in your application, the server will calculate Expires values for responses, and populate the Expires entity-header field of each one.

Once caching is enabled, and the server is delivering resources that contain caching information, you should implement a cache in your application. A cache is a process that runs locally, as a service. It behaves transparently, and operates as a middle-man standing in between the client and server processes. A cache serves locally-stored copies of Response Entities, and can be referred to as a Response Entity Cache.

There are several ways to implement caching. There are libraries available for this, and some operating systems have functions for it, but the most direct approach is to incorporate Response Entity Cache functionality into your client software code. This is done by implementing a class that wraps a collection of Response Entities. This class should encapsulate a searchable container of Response Entity objects and include methods for:

Transparently intercepting Inbound Response Messages.

• Determining if a Response Entity is Cacheable.
• Adding Cacheable Response Entities to the collection.

Transparently intercepting Outbound Request Messages.

• Checking to see if the associated Response Entity is a cached item.
• If not (a cache miss), relaying the Request Message to the Origin Server.
• If yes (a cache hit), determine if it is appropriate to serve the cached response.
• If it is, then serve the cached response.

Note: For information on creating cache storage in Android, see Using External Storage in the Data Storage section of the Android Developer's Guide.

Once caching is implemented in your application, and caching information is attached to the content by the server, your application can take advantage of the HTTP 1.1 expiration model to determine when an entry in the cache has expired and needs to be requested from the server.

Using the expiration model, there are two ways to determine whether a cache entry has expired:

• By checking the value of the Expires entity-header field against the Date value to see if the content has reached its expiration date.
• By checking the Cache-Control field for the max-age directive. If this directive is present, it will provide a value that can be used with the Date value to determine expiration.

Note: Cache-control directives take precedence, so the max-age directive overrides the Expires value. For more specific recommendations about using the Cache-Control directives, see Cache Control.

If a cache entry has not expired, then the application can use the cached resource and should not send an HTTP request to the server.

If a cache entry has expired, then the application should perform a cache validation to determine if the resource has been modified. This is done in one of the following ways:

• By checking the value of the If-Modified-Since header field in the request against the value of the Last-Modified header field in the cache entry.
• By checking the value of the If-None-Match header field in the request, against the value of the ETag header field in the cache entry.

By validating the entry, it provides the server an opportunity to return with an HTTP 304 (Not Modified) response instead of sending a full response that contains the resource.

It is possible for the server to mark a resource as non-cacheable, by using the Cache-Control directive no-cache, or by using Pragma: no-cache. This should only be done when absolutely necessary, because of the extra overhead that will be required by client applications to use this resource.

Whenever a client requires a resource that has been downloaded previously, it should use the version that is in the cache—not the one on the origin server. If, and only if, the cache is unable to fulfill the request, should the cache pass the Request Message on to the origin server on the client's behalf.

By implementing a response-entity cache and enabling caching on the server-side that contains accurate caching information in the Expires Header or the max age Cache-Control directive, your application can make full use the HTTP 1.1 caching mechanism. The caching mechanism allows your application to make accurate decisions based on the freshness of cached content that will reduce the amount of data and data connections that are sent needlessly—making your application faster and more efficient.