The MolProbity3 User Interface Framework
This document describes the MolProbity3 graphical user interface (GUI)
framework used in constructing web pages for the interactive side of
MolProbity. It also provides an overview of how the HTTP protocol works
and the kinds of limits that the web-service model puts on an
application.
How the Web works: HTTP
HTTP is the hyper-text transport
protocol, a plain-text language that web servers and web
browsers use to communicate. The communication has a very strict
formula: the client initiates a connection to the server, sends a
command, receives a response, and disconnects. This points out two very
important limitations on web applications.
First, the client (browser) initiates all
connections, so the server can never "push" information out to a
client. If new mail arrives in your webmail's inbox, you have to
refresh the page in order to see it -- the server has no way of
alerting you to its arrival. (We'll talk about how webmail designers
have worked around this later.)
Second, each connection is unrelated to the ones before it. In fact, a
web server has no memory of individual users; it sees no connection
between the various requests it receives. The server can't tell the
difference between one user clicking her way through the site and 10
users each requesting different single pages. As you might imagine,
this makes it difficult to implement a web application, which must keep
multiple simultaneous users separate while maintaining "state"
information (e.g. which mail
folder you're looking at, who's in your address book) for each user.
We'll discuss how we overcome these difficulties in a moment, but first
we need to return to HTTP commands. We said that when a client
initiates a new connection to the server, it sends a command. The HTTP
protocol defines a dozen or more different commands, but only two --
GET and POST -- are used with any regularity.
A GET message is generated by normal hyperlinks in a web page. The web
browser just sends something like this to the server:
GET
/some/web/page.html HTTP/1.1
In reply, the server (let's say it's www.example.com) sends back the
HTML content of http://www.example.com/some/web/page.html,
along with a few headers that describe the content. It's so simple you
can easily emulate a HTTP GET "by hand" using a Telnet client. One GET
is issued per file, so if the web page includes some in-line images, a
Flash animation, or a Java applet, additional GETs are made by the
browser to retrieve those files.
The other kind of HTTP command is a POST, which is usually generated by
submitting ("posting") a form on a web page. In this case, the browser
is still requesting the content of some URL, but it's also sending to
the server the contents of the form you just filled in. If the URL is a
normal HTML page, the result will be the same as a GET -- you'll see
the requested page, and the contents of the form will effectively be
discarded. However, not all URLs are created equal...
Dynamic web pages & sending data to the server
When a web server receives a request for a file, it usually just sends
the file back to the user. But some "files" are actually programs, and
rather than sending the program to the user, the server runs the
program and sends the program's output
to the user. The program might be a so-called "CGI script" written in
Perl or Python, it might be a Java servlet, it might be a Microsoft
ASP, or it might be a PHP script (used by MolProbity). For example, you
could write a PHP script to generate a web page displaying the current
date and time. Such a page would look different every time a user
visited it.
These server-side programs are capable of receiving the data from a web
page form (submitted with a POST command) and using it to make
decisions. For instance, a very simple system would have one static
HTML page with a form asking for your name, and one dynamic page that
says "Hello, <your name>."
A more complicated script would receive the various fields of an email
message from a HTML form (To, From, Subject, the message body,
attachments, etc.) and using them to actually compose and send an email
from the server.
It turns out that you can send data via a GET command too, just not as
much of it: the data is encoded in a special format and tacked on to
the end of the URL. Variable names are separated from values by an
equals sign, name/value pairs are separated by ampersands (&), and
the whole thing is separated from the URL by a question mark. Any
special characters also have to be "URL encoded", which involves
translating them into numeric codes starting with a percent sign (%).
(In PHP, there are urlencode()
and urldecode() functions
built in.) For example, this link could be used to send a preformed
email message, assuming send_email.php
exists and knows how to handle this input:
<a
href='send_email.php?to=Bob&subj=Reminder&msg=WAKE_UP_BOB'>Wake
Bob up</a>
The limitation on GET commands is that the total URL can't exceed a few
thousand characters, roughly the length of a short email message. If
you have a lot of data to transmit, you should use POST instead. POST
can even be used to upload whole files to the server, which is useful
for adding attachments to email messages and for getting PDB files into
MolProbity.
A simple web application: encode all data
In building a web application, the simplest model is one where all the
"state" data for the application is stored and accumulated in the
hyperlinks (URLs) and HTML forms. This way, the server doesn't have to
keep track of individual users or distinguish between them, because all
the information needed to do its job is right there in the POST or GET
command.
For example, you could implement an Amazon.com shopping cart this way.
Every time the user adds something to the cart, you append that to the
URLs you're generating (or add type=hidden INPUT fields to the form):
shop.php
shop.php?book1_isbn=123
shop.php?book1_isbn=123&book2_isbn=456
shop.php?book1_isbn=123&book2_isbn=456&book3_isbn=789
Not exactly elegant, but it works. When you click on the checkout link,
the server knows which three books you want to buy. This is fine for
really simple applications, but once you add a shipping address, a
billing address, gift-wrap options, and a credit card number, it
becomes pretty unmanageable. It's also a real pain to ensure you're
getting that information into every
URL and every form: if you
miss just one, all that data gets lost and the user has to start over.
For completeness, I should mention that cookies are essentially an
automated way of doing this: data gets accumulated in the cookies,
which is sent back and forth with every GET or POST. However, lots of
people disable cookies in their web browsers, and it's still really not
suitable for large amounts of data, because you have to keep bouncing
it back and forth over the network. Besides, there are much better
options...
A better web application: sesssions
The better way to approach this problem is to allow the server to keep track of the
application data. It can store your shopping cart on its disk, so if
you want to buy 50 books and ship them to 7 different people, its no
problem -- you've got plenty of storage space. (If you're a real
business, you probably store this in a SQL database instead, but a file
on disk is simpler to maintain and works just fine for MolProbity.) In
fact, MolProbity creates a whole directory structure for each user,
allowing them to store multiple PDB files, kinemages, and so forth,
along with the accumulated meta-data about them.
Now wait just a minute, you say, how does the server know that my
shopping chart or PDB files belong to me, and not to someone else? The
system we've described has no way of distinguishing users, so all of
our customers would be using the same shopping cart -- annoying, to say
the least. The key to overcomming this is to assign every visitor a session identifier when they first
enter the site. This ID is associated with a shopping cart or with a
directory of PDB files. We still have to propagate the session ID into
every URL and every form, as we described above, but now it's just that
one piece of data to keep track of. Every time the server creates a new
page, it stores the ID into the forms as hidden INPUT fields and into
the URLs as encoded name/value pairs. That way, when the user clicks a
link or submits a form, the server receives the ID and can identify the
user again. In this way, one web server can support multiple
simultaneous users on the same application, and it can keep all their
data and activities separate.
PHP has built-in library functions for session management, which are
extended by the MolProbity code. Each MolProbity script starts with a
little recipe that retreives the session ID and loads the data into a
special array called $_SESSION.
If no ID is specified, a new session is created. Any new data placed
into $_SESSION is
automatically saved to disk again at the end of the script. The file variables.html documents all the things
MolProbity stores in $_SESSION.
It is also possible to put the session ID into a cookie, which should
free us from having to remember to put it into all the forms and all
the URLs. In practice, I find that cookies cause weird problems due to
their expiration dates and that they end up being more trouble than
they're worth. Plus, they won't work users who have cookies disabled.
Thus, MolProbity doesn't use cookies and instead puts the session ID
into all forms and all URLs.
Flow control: event handling
If we were building a normal desktop application, we would regard the
HTML pages as the graphical user
interface (GUI) we presented to the user, and we would regard
his clicking on links and submitting forms and events to be handled. In a normal
application, the code that creates the GUI and the code that handles
the events from that GUI are tightly coupled together, and are
insulated from other GUIs and event handlers in the program. That is,
all the code for driving one dialog box should appear together, and it
should be independent of the code driving other dialog boxes.
Web applications don't work that way. One page contains the HTML, but
it submits the data to a second page for processing. That second page
also has to generate some HTML (so the user doesn't get a blank
screen), which is the form that will be processed by a third page,
which will in turn generate some more (possibly unrelated) HTML for
processing elsewhere. Each PHP page still has a GUI-generating section
and an event-handling section, but the two are unrelated to each other,
and are tightly coupled to the previous page (source of events) and the
next page (destination for new events).
This creates all kinds of problems. For one thing, branching is hard.
Say you have two different links on a page, and they take you to
different parts of the application (say, Compose Mail and Read Mail).
Now the code for one page worth of activities is spread over three PHP
scripts (one GUI and two event handlers)! The problem is even worse for
forms, because a single form can have only one destination. Wanted to
choose a path based on the settings of some radio buttons? Good luck --
the possible solutions aren't pretty. Another problem is the browser's
Back button. The current state of the application as stored in $_SESSION is counting on the
user being on page 4, but if the click back to page 2 and try to change
something, your application will most likely crash. Remember, the
server doesn't see the Back button, so it gets no warning that user has
backtracked and no opportunity to veto that move.
To solve these problems, I've built a framework around the index.php page. With some
exceptions discussed below, the entire MolProbity application is
contained in this one user-visible page. Don't worry, it's not 10,000
lines long. It delegates all of the real work -- GUI generation and
event handling -- to the scripts in the pages/ folder. Each delegate
script has the code to generate one HTML page worth of interface and
the code to handle all the "events" (clicks on links and form submits)
that might be generated by that page. The index.php page keeps track of
which delegate is currently active for displaying the GUI, and it
assigns numbered event IDs to all the links and forms that associate
them with the appropriate event-handling code. (Events are registered
using the makeEventURL()
and makeEventForm()
functions.) This defeats the evil Back button: if the user backtracks
and tries to click on some other link, index.php (which is the only URL the user's browser has ever
seen at the site) will recognize it as a "stale" event and ignore it.
Futhermore, it will remember which "page" it was on, and display the
appropriate (i.e., current)
HTML page in response to the user's misguided click. As far as I can
tell, this is more robust and error-proof than 98% of the e-commerce
sites out there.
There are a few pages that exist outside of index.php's protective shield.
These are generally the pop-up file viewer pages, which want to exist
outside of the linear flow of the application. That's OK, because these
pages don't lead anywhere else, and can't go back to anywhere either
(since they popped up in a new window). Thus, there are no events to
handle and no real complications to deal with.
Flow control: call & return
We've described how index.php simplifies event
handling and how it protects us from the Back button, but we haven't
explained how control is handed off from one delegate script to
another. In every pass through index.php,
we first ask our current delegate script to handle the user-submitted
event (e.g. the user clicked a
hyperlink) and then we ask our current delegate to generate a HTML GUI.
The trick is, the event handler can change the current delegate (using pageGoto()), so that a
different delegate is responsible for generating the HTML. In this way,
we make it look like clicking on the link caused the user to go to a
new HTML page. In fact, they just clicked on yet another link to index.php, but the event ID
encoded into that URL caused index.php
to invoke an event handler from the "previous" page's delegate script
that in turn choose a new delegate script to create some different
HTML, making index.php
look like a different page.
This system is great for linear flow control, and for branching. (We
can do conditional pageGoto()'s
based on the contents of a submitted form, for instance.) It also
enhances security by preventing the user from "teleporting" to
restricted pages just by typing in their URL. (Remember, index.php is the only legal or
accessible URL, aside from a few file viewers.) However, we often want
"subroutines" that we can call from various locations. For instance, I
want to be able to click on Compose Message from many different places
in my web application. It might take me to a series of screens (address
book, message window, file upload) that I use to write an email. When
I'm done, I want be dropped off where I started from. That requires
that the last page of the compose process to know where I started from.
Rather than having to record this manually, we create a stack mechanism
with pageCall() and pageReturn(). The pageCall() function works just
like pageGoto(), except
that the current delegate is pushed onto a stack before control is
transfered to another delegate. That delegate may then transfer to
several other pages using pageGoto()
before the task is complete. When it is, that page can invoke pageReturn() and control will
be returned to the calling delegate, which will be popped off of the
stack. In general, groups of pages will be designed for access via pageGoto() or pageCall(), but not both.
Flow control: background jobs
We don't want our event handlers to do anything that might take more
than a second or two of processing time, because otherwise our client
is left staring at an hourglass, waiting for some HTML to appear in
their browser. The solution to this is to launch time-consuming tasks
as UNIX background jobs. (You do this at the command line by putting an
ampersand at the end of the command.) We can then monitor the job, and
notify the user when it's finished.
The problem is that there's no way to "push" a notification from the
server to the client; the client must ask for the job status in order
to see it (e.g. by clicking
the Refresh button in the browser, or by clicking a link or button).
Our solution is the job_progress.php
script, which displays the background job's status to the user while
periodically updating itself. It uses the <meta http-equiv='refresh'>
tag in the page header to request that it be reloaded every couple of
seconds. JavaScript could do the same job, but we don't want to rely on
JavaScript for any critical tasks because it doesn't work in all
browsers and because some users disable it for security reasons.
Here's how the choreography works. The user clicks a link or submits a
form, thereby invoking an event handler. The event handler calls the launchBackground() function to
start a background job, then calls pageGoto("job_progress.php").
The job progress page continues to refresh itself every 5-10 seconds,
watching a flag in $_SESSION
to see when the background job finishes. It's very careful not to
overwrite $_SESSION
however, because the background job is also a PHP script, and it's
using the information in $_SESSION
to run programs like Reduce, Probe, and Prekin. When the background
script sets its flag and exits, job_progress.php
then calls pageGoto() to
get to a results page.
It all sounds pretty complicated, but the hard parts have already been
written. In effect, you put a little recipe in your real event handler,
and then you code a sort of extended event handler as a separate PHP
script. That handler, which lives in jobs/, appears to run in a
background thread while giving you an easy way to display progress
messages, and when you're done, everything recovers gracefully and
proceeds forward.
I believe that's about it. I'll
work on writing another document that's more of a tutorial on how to
add a GUI module to MolProbity, but this should have provided a solid
overview of the architecture and motivations.