Using Cookies with 4th Dimension Web Server
By Eric Saltzen, 4D, Inc. Technical Support
Technical Note 00-17
Technical Notes for Technical Notes for 00-04 April 2000
Introduction
Cookies are a general mechanism which server-side connections (such as CGI scripts) can use to both store and retrieve information on the client side of the connection. A server, when returning an HTTP object to a client, may also send a name=value pair which the client will store. Included in that state object is a description of the range of URLs for which that state is valid, consisting of the domain and a URL path. Any future HTTP requests made by the client which fall in that range will include a transmittal of the current value of the state object from the client back to the server. The state object is called a cookie for "no compelling reason" according to Netscape. Here is the basic format of a cookie sent by a web browser along with an HTTP request inside of the HTTP Header lines:
"Cookie: cookieName=cookieValue"
Again, this line will only be included by the user–agent in requests destined for a domain and a path matching that specified by the original cookie. When the server wants to set the value of a cookie, it sends:
"Set-Cookie: cookieName=cookieValue; expires=Sat, 31-Dec-2005 08:00:00 GMT; path=/"
It is worth noting that you cannot see cookie information by performing a "View Source" command from within a web browser — that only shows you the contents of the HTML file itself, which is the payload of the HTTP delivery. The protocol mandates that command and control information for the exchange between client and server be sent separately from the actual file contents that are being requested or served. This is the "HTTP Header," which inside of a 4D web process can be accessed via the $2 parameter. Do not confuse the HTTP Header with the <HEAD> section of an HTML formatted document, which is completely different. This also explains the need for the 4D command SET HTTP HEADER which allows us to add information to the HTTP response header that 4D will send alongside any requested documents.
Cookies are useful for having a browser remember some specific information which the web server can later retrieve. This enables state management and session management. As a user browses the web, any cookies which servers might send to their web browser are stored in their computer's memory. Note that another name for a web browsing program is a "user–agent." When the user agent is quit, any cookies that haven't expired are written to a cookie file(s) so they can be reloaded upon the next execution. If you are using Netscape Navigator on a Macintosh this file is named "MagicCookie", on Unix it's "cookies", and on Windows it's "cookies.txt". You can look at this file with a text editor to see exactly what cookies are stored there, or delete the file to get rid of all of the cookies. If you are using Internet Explorer on Windows the information is stored in multiple files inside of the Windows directory (or Winnt directory) in a subdirectory named "Cookies". If you are using Internet Explorer on a Macintosh you can inspect the cookies stored on your computer through the Preferences dialog under Receiving Files -> Cookies.
CookieMonster4D
"CookieMonster4D" is a sample database (available in both Macintosh and PC format) that shows how to look for a cookie in a user–agent HTTP request header, how to look up a past visitor based on their cookie value, and how to set a cookie for a visitor who has none (someone who hasn't visited the site or has lost or deleted their cookie since the last visit).
If you examine the Database Properties dialog, you will find that CookieMonster4D is set to "Start without Context," and has no "Default HTML Root" or "Default Home Page" set. Also, "Use 4DVAR Comments instead of Brackets" is checked. The database itself contains only three methods: On Web Connection, ParseCookies and COMPILER_WEB. The HTML pages for the site were designed using Macromedia Dreamweaver and Flash, with appropriate calls to 4DVAR and 4DCGI inserted as XML comment tags such as "<!--4DVAR userName
" to pull dynamic data from 4D.
Let's take a look at the On Web Connection database method. We'll follow it through twice: first assuming the user–agent does not have a cookie set from a previous visit to CookieMonster4D, and once again with a user–agent that does have a previously set cookie.
On Web Connection
` CookieMonster4D Sample Database, April 2000 4D, Inc.
` by Eric Saltzen, 4D, Inc. Technical Support
` Database Method: On Web Connection
C_TEXT ($1; $2; $3; $4; $5; $6; userName)
C_INTEGER (numCookies)
ARRAY TEXT (cookieName; 0)
ARRAY TEXT (cookieValue; 0)
REDUCE SELECTION ([WebVisitors]; 0) ` clear any previous selection
` in this web process
Case of
` request for root of site or a 4DCGI served page, otherwise let request pass thru
: (($1 = "/") | ($1 = "/4DCGI/home.shtm") | ($1 = "/4DCGI/direct.htm")
| ($1 = "/4DCGI/schedule.htm") | ($1 = "/4DCGI/specials.htm")
| ($1 = "/4DCGI/shots.htm") | ($1 = "/4DCGI/AssignName"))
numCookies := ParseCookies ($2; ->cookieName; ->cookieValue)
$primaryID := Find in array (cookieName; "primaryID")
If ($primaryID > 0)
QUERY ([WebVisitors];[WebVisitors]Cookie_ID = Num(cookieValue{$primaryID}))
End if
If (($primaryID = -1) | (Records in selection ([WebVisitors]) = 0))
` unable to find cookie in user agent response, or cookie does not match records
If (Records in selection ([WebVisitors]) = 0) ` didn't find a matching record
` create a unique ID by calculating the number of seconds since Jan 01, 2000
` converting to a string, and append two extra digits representing centiseconds
` this gives us a range of cookie values from 0 to (2^31)-1 and then via
` numeric overflow from -2^31 back up to zero, which is 497 days worth of
` unique, first-hit cookie identifiers at a maximum of one every 100th of a
` second (8.6 million per day). Thus for approximately 1.36 years we are
` guaranteed a unique ID for every visitor to our web site. Just in case we will
` double-check the uniqueness of the cookie value against previously issued
` cookies and generate another if there is a collision with an existing cookie.
Repeat
$tempCookie := Num (String (Current date - !01/01/2000! * 86400
+ Current time) + String (Milliseconds % 100))
QUERY ([WebVisitors]; [WebVisitors]Cookie_ID = $tempCookie)
Until (Records in selection ([WebVisitors]) = 0)
CREATE RECORD ([WebVisitors])
[WebVisitors]Cookie_ID := $tempCookie
End if
[WebVisitors]DateVisited := Current date
[WebVisitors]TimeVisited := Current time
[WebVisitors]RequestedURL := $1
[WebVisitors]HTTP_Header := $2
[WebVisitors]ClientIPAddress := $3
SAVE RECORD ([WebVisitors])
` our cookies are set to expire on Dec 31 2005
SET HTTP HEADER ("Set-Cookie: primaryID = "
+ String ([WebVisitors]Cookie_ID)
+ "; expires = Sat, 31-Dec-2005 08:00:00 GMT; path = /")
lastDate := String ([WebVisitors]DateVisited)
lastTime := String ([WebVisitors]TimeVisited)
IPAddress := [WebVisitors]ClientIPAddress
URLRequested := [WebVisitors]RequestedURL
requestHeader := Char (1) + Replace string ([WebVisitors]HTTP_Header;
Char (13) + Char (10); "<br>")
SEND HTML FILE ("index.htm") ` request the user to enter a user name
` for our records
Else
[WebVisitors]DateVisited := Current date
[WebVisitors]TimeVisited := Current time
[WebVisitors]RequestedURL := $1
[WebVisitors]HTTP_Header := $2
[WebVisitors]ClientIPAddress := $3
If (Length (userName) # 0)
[WebVisitors]UserName := userName ` from index.htm form, if present
End if
SAVE RECORD ([WebVisitors])
SET HTTP HEADER ("")
C_TEXT (userName; lastDate; lastTime; IPAddress; URLRequested;
requestHeader)
userName := [WebVisitors]UserName
lastDate := String ([WebVisitors]DateVisited)
lastTime := String ([WebVisitors]TimeVisited)
IPAddress := [WebVisitors]ClientIPAddress
URLRequested := [WebVisitors]RequestedURL
requestHeader := Char (1) + Replace string ([WebVisitors]HTTP_Header;
Char (13) + Char (10); "<br>")
SEND HTML FILE ("home.shtm")
End if
: ($1 = "/4DCGI/@") ` handle requests for inline images on 4DCGI pages
$fileName := Substring ($1; Position ("/4DCGI/"; $1) + 7)
PLATFORM PROPERTIES ($platform)
If ($platform < 3) ` fix file path for use on Macintosh
$fileName := ":" + Replace string ($fileName; "/"; ":")
End if
$extension := Substring ($1; Length ($1) - 3) ` CookieMonster4D uses
` 3 letter extensions by convention (e.g. "gif")
DOCUMENT TO BLOB ($fileName; myBLOB)
SEND HTML BLOB (myBLOB; $extension; True) ` send out file w/ type
` specified by filename extension, non-context mode
End case
The first thing in the method are some variable declarations. These are especially important to include if the database will ever be compiled. Then we enter a Case statement that looks to see if the request was for either the root of the site ("/") or one of the 4DCGI links that I included in the text navigation controls on the home page (home.shtm). If the request is not recognized, then it falls through to the next expression in the Case statement which detects requests for elements of pages that were filtered through a call to /4DCGI and need to have their URL processed and a file returned. Without this second Case expression, any pages that were requested through a /4DCGI call would be missing their inline images or any other files that were referenced from the basic HTML document.
Going back to the first expression in the Case statement, the first thing we do is to call another project method, "ParseCookies," passing it the complete HTTP Header and pointers to two text arrays which will contain the name=value pairs for all cookies present in the header upon return. ParseCookies is a straightforward text-processing method:
ParseCookies
` CookieMonster4D Sample Database, April 2000 4D, Inc.
` by Eric Saltzen, 4D, Inc. Technical Support
` Project Method: ParseCookies
` for more information on how Cookies are formatted and used, see
` http://www.cis.ohio-state.edu/htbin/rfc/rfc2109.html
` $1 : the HTTP header value to parse (TEXT)
` $2 : text array pointer to contain names
` $3 : text array pointer to contain values
` $0 : returns the number of cookies found
$HTTPheader := $1
$nameArrayPtr := $2
$valueArrayPtr := $3
$curPos := 0
$numCookies := 0
$location1 := Position ("Cookie:"; $HTTPheader)
If ($location1 = 0)
$0 := 0
Else
$HTTPheader := Substring ($HTTPheader; $location1 + 8) ` chop off everything
` up to and incl "Cookie: "
$location1 := 1 ` location of our first (and perhaps only) name, value pair
While ($location1 > 0)
$location2 := Position ("="; Substring ($HTTPheader; $location1)) ` find the
` end of the first cookie name
INSERT ELEMENT ($nameArrayPtr->; 0)
$nameArrayPtr->{1} := Substring ($HTTPheader; $location1; $location2 - 1)
` assign the cookie's name
$returnAt := Position (Char (13); $HTTPheader) ` look for a carriage
` return delimiter
$semicolonAt := Position (";"; $HTTPheader) ` look for a semicolon delimiter
If (($semicolonAt # 0) & ($semicolonAt < $returnAt))
$location1 := $semicolonAt ` semicolon delimiter comes first
Else
$location1 := $returnAt ` return delimiter comes first
End if
If ($location1 = 0)
$location1 := Length ($HTTPheader) + 1 ` we reached the end of the header
` w/out delimiters
End if
INSERT ELEMENT ($valueArrayPtr->; 0)
$valueArrayPtr->{1} := Substring ($HTTPheader; $location2 + 1;
$location1 - $location2 - 1) ` assign the cookie's value
$numCookies := $numCookies + 1 ` increment cookie counter
If ($location1 = $returnAT)
$HTTPheader := "" ` we're done, blank out rest of header
Else
$HTTPheader := Substring ($HTTPheader; $location1 + 2) ` not done yet,
` just trim header to begin of next cookie pair
End if
If (Length ($HTTPheader) = 0)
$location1 := 0 ` terminate cookie processing loop
Else
$location1 := 1 ` continue with cookie processing loop
End if
End while
$0 := $numCookies ` return number of cookies found
End if
After returning from the ParseCookies method, execution of the On Web Connection Database Method continues with a call to Find in array that looks for a cookie with the exact name "primaryID" which was arbitrarily chosen for use in this database. Since we are assuming on this first visit that our user-agent has no cookies set, 4D will skip over the QUERY and enter the second If statement because Find in array returns -1 if it can't find the value. At this point, the selection in the [WebVisitors] table will still be empty, so the Repeat loop will run until a unique ID number for the cookie value is generated. See the source code for details on the generation of this 32-bit long integer used as the primaryID value. At this point the On Web Connection Database Method creates a new record in [WebVisitors] for our cookie-less user-agent, and assigns all the information from the HTTP Header along with the current date and time and the primaryID value as a long integer. Finally, a call to SET HTTP HEADER inserts our newly created cookie into the response header, then sends an introductory page ("index.htm") using SEND HTML FILE.
index.htm
This is a simple HTML page containing a table with a logo picture and a text box that will, after being processed by 4D, contain all the information that was originally sent by the browser in the HTTP header. The form also contains a single text entry field, which allows the user to enter their name for future reference. Note that submitting the form initiates a request for "/4DCGI/AssignName", which means that once again the On Web Connection Database Method will be called — this time there is a cookie stored on the user-agent that will be sent along with the form data.
When the user types their name and clicks the submit button (or presses Enter, since there is only one text entry area on the page), the user-agent initiates another HTTP request for CookieMonster4D. This time the URL being requested is "/4DCGI/AssignName", so the On Web Connection Database Method begins execution in the first Case expression with a call to ParseCookies. Assuming there is only one cookie set on this user-agent by our domain and path, $primaryID will contain the value 1, and the QUERY of the [WebVisitors] table will commence searching for a record with a matching cookie value. At this point the If statement following the QUERY will be completely skipped and execution will continue at the Else clause. Here we simply update the data in the [WebVisitors] table, including the userName if it is available to us from the form submission, then call SET HTTP HEADER ("") to clear it. The final step is to assign the values of the variables that are destined to appear on the CookieMonster4D home page included as <!--4DVAR myVar> tags. Note the call to Replace string to replace the Carriage Return/Line Feed pairs (ala UNIX) with 4D standard CR (ASCII 13) delimited lines. Regardless of which page was requested originally (the site root, a form submit from index.htm, or one of the other /4DCGI links), CookieMonster4D simply sends the "home.shtm" file.
home.shtm
The JavaScript at the top of this page is used to enable the mouse-over effects in the navigational controls on the left side of the page. The links inside of the left navigational bar are direct links to other static HTML files within the site hierarchy, and when selected they will cause 4D to simply serve the static page without a detour through the On Web Connection Database Method (place a break point and try it). The text navigation links at the bottom of each page have been changed to /4DCGI calls which forces 4D to execute the On Web Connection Database Method, allowing variables to be set for inclusion via 4DVAR tags. An alternative approach would be to place direct 4DACTION calls in every page that requires dynamic data, therefore tying each dynamic page (or set of pages) to a particular 4D method designed to handle the setup.
The most interesting thing to note on this page is the display of the dynamic data, including the self-selected userName field, date and time of visit, etc. Keep your eye on the time field to make sure you are not viewing a cached copy of the page at any time (just hit the refresh button in your browser).
Summary
This technical note and sample database together provide a foundation upon which an intelligent web site that promotes a positive relationship with its viewers (the customers) can be based. Below is a diagram of the essentials of cookie processing and state/session management. Possible improvements to this database include the ability for user-agents to be reassigned to a different record in the [WebVisitors] table ("Hey, I'm not Greg — I just let him use my computer yesterday and now this site thinks I am him!") And the ability for a new visitor (a user-agent without a cookie) to self-select an existing entry in [WebVisitors] ("Darn, I reformatted my hard drive and now this web site has lost my shopping cart data and a week's worth of my online shopping time!") Another possibility would be to create an entry in [WebVisitors] for every connection to the database, allowing duplicate key fields, and then simply search for the most recent one when processing incoming cookies. Of course, if this technique were used in a heavily trafficked web site there would be a danger of rapidly growing the data file and slowing performance. However, a great deal of information could be gleaned from studying the patterns of users' clickstreams (the sequence of interactive steps between a user-agent and a web site). Which pages are the most popular? What kind of searches are people doing? What is the profile of a visit to our web site that ends in a purchase? What is the profile of a visit to our site that ends with the user going somewhere else?
See also
For more information and links to appropriate RFCs about cookies:
http://home.netscape.com/newsref/std/cookie_spec.html
http://www.cis.ohio-state.edu/htbin/rfc/rfc2109.html