The MolProbity3 User Interface Framework

This document describes the MolProbity3 graphical user interface (GUI) framework used in constructing web pages for the interactive side of MolProbity. It also provides an overview of how the HTTP protocol works and the kinds of limits that the web-service model puts on an application.

How the Web works: HTTP

HTTP is the hyper-text transport protocol, a plain-text language that web servers and web browsers use to communicate. The communication has a very strict formula: the client initiates a connection to the server, sends a command, receives a response, and disconnects. This points out two very important limitations on web applications.

First, the client (browser) initiates all connections, so the server can never "push" information out to a client. If new mail arrives in your webmail's inbox, you have to refresh the page in order to see it -- the server has no way of alerting you to its arrival. (We'll talk about how webmail designers have worked around this later.)

Second, each connection is unrelated to the ones before it. In fact, a web server has no memory of individual users; it sees no connection between the various requests it receives. The server can't tell the difference between one user clicking her way through the site and 10 users each requesting different single pages. As you might imagine, this makes it difficult to implement a web application, which must keep multiple simultaneous users separate while maintaining "state" information (e.g. which mail folder you're looking at, who's in your address book) for each user.

We'll discuss how we overcome these difficulties in a moment, but first we need to return to HTTP commands. We said that when a client initiates a new connection to the server, it sends a command. The HTTP protocol defines a dozen or more different commands, but only two -- GET and POST -- are used with any regularity.

A GET message is generated by normal hyperlinks in a web page. The web browser just sends something like this to the server:

    GET /some/web/page.html HTTP/1.1

In reply, the server (let's say it's www.example.com) sends back the HTML content of http://www.example.com/some/web/page.html, along with a few headers that describe the content. It's so simple you can easily emulate a HTTP GET "by hand" using a Telnet client. One GET is issued per file, so if the web page includes some in-line images, a Flash animation, or a Java applet, additional GETs are made by the browser to retrieve those files.

The other kind of HTTP command is a POST, which is usually generated by submitting ("posting") a form on a web page. In this case, the browser is still requesting the content of some URL, but it's also sending to the server the contents of the form you just filled in. If the URL is a normal HTML page, the result will be the same as a GET -- you'll see the requested page, and the contents of the form will effectively be discarded. However, not all URLs are created equal...

Dynamic web pages & sending data to the server

When a web server receives a request for a file, it usually just sends the file back to the user. But some "files" are actually programs, and rather than sending the program to the user, the server runs the program and sends the program's output to the user. The program might be a so-called "CGI script" written in Perl or Python, it might be a Java servlet, it might be a Microsoft ASP, or it might be a PHP script (used by MolProbity). For example, you could write a PHP script to generate a web page displaying the current date and time. Such a page would look different every time a user visited it.

These server-side programs are capable of receiving the data from a web page form (submitted with a POST command) and using it to make decisions. For instance, a very simple system would have one static HTML page with a form asking for your name, and one dynamic page that says "Hello, <your name>." A more complicated script would receive the various fields of an email message from a HTML form (To, From, Subject, the message body, attachments, etc.) and using them to actually compose and send an email from the server.

It turns out that you can send data via a GET command too, just not as much of it: the data is encoded in a special format and tacked on to the end of the URL. Variable names are separated from values by an equals sign, name/value pairs are separated by ampersands (&), and the whole thing is separated from the URL by a question mark. Any special characters also have to be "URL encoded", which involves translating them into numeric codes starting with a percent sign (%). (In PHP, there are urlencode() and urldecode() functions built in.) For example, this link could be used to send a preformed email message, assuming send_email.php exists and knows how to handle this input:

    <a href='send_email.php?to=Bob&subj=Reminder&msg=WAKE_UP_BOB'>Wake Bob up</a>

The limitation on GET commands is that the total URL can't exceed a few thousand characters, roughly the length of a short email message. If you have a lot of data to transmit, you should use POST instead. POST can even be used to upload whole files to the server, which is useful for adding attachments to email messages and for getting PDB files into MolProbity.

A simple web application: encode all data

In building a web application, the simplest model is one where all the "state" data for the application is stored and accumulated in the hyperlinks (URLs) and HTML forms. This way, the server doesn't have to keep track of individual users or distinguish between them, because all the information needed to do its job is right there in the POST or GET command.

For example, you could implement an Amazon.com shopping cart this way. Every time the user adds something to the cart, you append that to the URLs you're generating (or add type=hidden INPUT fields to the form):

    shop.php
    shop.php?book1_isbn=123
    shop.php?book1_isbn=123&book2_isbn=456
    shop.php?book1_isbn=123&book2_isbn=456&book3_isbn=789

Not exactly elegant, but it works. When you click on the checkout link, the server knows which three books you want to buy. This is fine for really simple applications, but once you add a shipping address, a billing address, gift-wrap options, and a credit card number, it becomes pretty unmanageable. It's also a real pain to ensure you're getting that information into every URL and every form: if you miss just one, all that data gets lost and the user has to start over.

For completeness, I should mention that cookies are essentially an automated way of doing this: data gets accumulated in the cookies, which is sent back and forth with every GET or POST. However, lots of people disable cookies in their web browsers, and it's still really not suitable for large amounts of data, because you have to keep bouncing it back and forth over the network. Besides, there are much better options...

A better web application: sesssions

The better way to approach this problem is to allow the server to keep track of the application data. It can store your shopping cart on its disk, so if you want to buy 50 books and ship them to 7 different people, its no problem -- you've got plenty of storage space. (If you're a real business, you probably store this in a SQL database instead, but a file on disk is simpler to maintain and works just fine for MolProbity.) In fact, MolProbity creates a whole directory structure for each user, allowing them to store multiple PDB files, kinemages, and so forth, along with the accumulated meta-data about them.

Now wait just a minute, you say, how does the server know that my shopping chart or PDB files belong to me, and not to someone else? The system we've described has no way of distinguishing users, so all of our customers would be using the same shopping cart -- annoying, to say the least. The key to overcomming this is to assign every visitor a session identifier when they first enter the site. This ID is associated with a shopping cart or with a directory of PDB files. We still have to propagate the session ID into every URL and every form, as we described above, but now it's just that one piece of data to keep track of. Every time the server creates a new page, it stores the ID into the forms as hidden INPUT fields and into the URLs as encoded name/value pairs. That way, when the user clicks a link or submits a form, the server receives the ID and can identify the user again. In this way, one web server can support multiple simultaneous users on the same application, and it can keep all their data and activities separate.

PHP has built-in library functions for session management, which are extended by the MolProbity code. Each MolProbity script starts with a little recipe that retreives the session ID and loads the data into a special array called $_SESSION. If no ID is specified, a new session is created. Any new data placed into $_SESSION is automatically saved to disk again at the end of the script. The file variables.html documents all the things MolProbity stores in $_SESSION.

It is also possible to put the session ID into a cookie, which should free us from having to remember to put it into all the forms and all the URLs. In practice, I find that cookies cause weird problems due to their expiration dates and that they end up being more trouble than they're worth. Plus, they won't work users who have cookies disabled. Thus, MolProbity doesn't use cookies and instead puts the session ID into all forms and all URLs.

Flow control: event handling

If we were building a normal desktop application, we would regard the HTML pages as the graphical user interface (GUI) we presented to the user, and we would regard his clicking on links and submitting forms and events to be handled. In a normal application, the code that creates the GUI and the code that handles the events from that GUI are tightly coupled together, and are insulated from other GUIs and event handlers in the program. That is, all the code for driving one dialog box should appear together, and it should be independent of the code driving other dialog boxes.

Web applications don't work that way. One page contains the HTML, but it submits the data to a second page for processing. That second page also has to generate some HTML (so the user doesn't get a blank screen), which is the form that will be processed by a third page, which will in turn generate some more (possibly unrelated) HTML for processing elsewhere. Each PHP page still has a GUI-generating section and an event-handling section, but the two are unrelated to each other, and are tightly coupled to the previous page (source of events) and the next page (destination for new events).

This creates all kinds of problems. For one thing, branching is hard. Say you have two different links on a page, and they take you to different parts of the application (say, Compose Mail and Read Mail). Now the code for one page worth of activities is spread over three PHP scripts (one GUI and two event handlers)! The problem is even worse for forms, because a single form can have only one destination. Wanted to choose a path based on the settings of some radio buttons? Good luck -- the possible solutions aren't pretty. Another problem is the browser's Back button. The current state of the application as stored in $_SESSION is counting on the user being on page 4, but if the click back to page 2 and try to change something, your application will most likely crash. Remember, the server doesn't see the Back button, so it gets no warning that user has backtracked and no opportunity to veto that move.

To solve these problems, I've built a framework around the index.php page. With some exceptions discussed below, the entire MolProbity application is contained in this one user-visible page. Don't worry, it's not 10,000 lines long. It delegates all of the real work -- GUI generation and event handling -- to the scripts in the pages/ folder. Each delegate script has the code to generate one HTML page worth of interface and the code to handle all the "events" (clicks on links and form submits) that might be generated by that page. The index.php page keeps track of which delegate is currently active for displaying the GUI, and it assigns numbered event IDs to all the links and forms that associate them with the appropriate event-handling code. (Events are registered using the makeEventURL() and makeEventForm() functions.) This defeats the evil Back button: if the user backtracks and tries to click on some other link, index.php (which is the only URL the user's browser has ever seen at the site) will recognize it as a "stale" event and ignore it. Futhermore, it will remember which "page" it was on, and display the appropriate (i.e., current) HTML page in response to the user's misguided click. As far as I can tell, this is more robust and error-proof than 98% of the e-commerce sites out there.

There are a few pages that exist outside of index.php's protective shield. These are generally the pop-up file viewer pages, which want to exist outside of the linear flow of the application. That's OK, because these pages don't lead anywhere else, and can't go back to anywhere either (since they popped up in a new window). Thus, there are no events to handle and no real complications to deal with.

Flow control: call & return

We've described how index.php simplifies event handling and how it protects us from the Back button, but we haven't explained how control is handed off from one delegate script to another. In every pass through index.php, we first ask our current delegate script to handle the user-submitted event (e.g. the user clicked a hyperlink) and then we ask our current delegate to generate a HTML GUI. The trick is, the event handler can change the current delegate (using pageGoto()), so that a different delegate is responsible for generating the HTML. In this way, we make it look like clicking on the link caused the user to go to a new HTML page. In fact, they just clicked on yet another link to index.php, but the event ID encoded into that URL caused index.php to invoke an event handler from the "previous" page's delegate script that in turn choose a new delegate script to create some different HTML, making index.php look like a different page.

This system is great for linear flow control, and for branching. (We can do conditional pageGoto()'s based on the contents of a submitted form, for instance.) It also enhances security by preventing the user from "teleporting" to restricted pages just by typing in their URL. (Remember, index.php is the only legal or accessible URL, aside from a few file viewers.) However, we often want "subroutines" that we can call from various locations. For instance, I want to be able to click on Compose Message from many different places in my web application. It might take me to a series of screens (address book, message window, file upload) that I use to write an email. When I'm done, I want be dropped off where I started from. That requires that the last page of the compose process to know where I started from. Rather than having to record this manually, we create a stack mechanism with pageCall() and pageReturn(). The pageCall() function works just like pageGoto(), except that the current delegate is pushed onto a stack before control is transfered to another delegate. That delegate may then transfer to several other pages using pageGoto() before the task is complete. When it is, that page can invoke pageReturn() and control will be returned to the calling delegate, which will be popped off of the stack. In general, groups of pages will be designed for access via pageGoto() or pageCall(), but not both.

Flow control: background jobs

We don't want our event handlers to do anything that might take more than a second or two of processing time, because otherwise our client is left staring at an hourglass, waiting for some HTML to appear in their browser. The solution to this is to launch time-consuming tasks as UNIX background jobs. (You do this at the command line by putting an ampersand at the end of the command.) We can then monitor the job, and notify the user when it's finished.

The problem is that there's no way to "push" a notification from the server to the client; the client must ask for the job status in order to see it (e.g. by clicking the Refresh button in the browser, or by clicking a link or button). Our solution is the job_progress.php script, which displays the background job's status to the user while periodically updating itself. It uses the <meta http-equiv='refresh'> tag in the page header to request that it be reloaded every couple of seconds. JavaScript could do the same job, but we don't want to rely on JavaScript for any critical tasks because it doesn't work in all browsers and because some users disable it for security reasons.

Here's how the choreography works. The user clicks a link or submits a form, thereby invoking an event handler. The event handler calls the launchBackground() function to start a background job, then calls pageGoto("job_progress.php"). The job progress page continues to refresh itself every 5-10 seconds, watching a flag in $_SESSION to see when the background job finishes. It's very careful not to overwrite $_SESSION however, because the background job is also a PHP script, and it's using the information in $_SESSION to run programs like Reduce, Probe, and Prekin. When the background script sets its flag and exits, job_progress.php then calls pageGoto() to get to a results page.

It all sounds pretty complicated, but the hard parts have already been written. In effect, you put a little recipe in your real event handler, and then you code a sort of extended event handler as a separate PHP script. That handler, which lives in jobs/, appears to run in a background thread while giving you an easy way to display progress messages, and when you're done, everything recovers gracefully and proceeds forward.

I believe that's about it. I'll work on writing another document that's more of a tutorial on how to add a GUI module to MolProbity, but this should have provided a solid overview of the architecture and motivations.