CGI Filter/Pipe Interface

Note: this web page is from 1996/1997. Pipeline and HFPM have gone through their life cycle and are now deprecated.

I came up with an idea the other day involving modularizing CGI-scripts, which I call the CGI Filter/Pipe interface, and have implemented for Unix using Perl version 5. I think this idea will improve life for the Web community by providing extra functionality to CGI-scripts.

This idea is further described in an article in the July 1996 issue of WEBsmith magazine.

The Idea

Hello, thanks for stopping by. This page is about an idea that I have to increase the potential modularity of CGI scripts, so that instead of having one monolithic CGI script, a number of smaller, more easy to maintain, modules and a driving program can be used in sequence instead.

My idea is to have a CGI script (call it "pipeline") take input in the usual manner (GET or POST method), build an internal representation of the fields, then call a specified sequence of routines (in separate files/modules). These routines can do some processing, possibly modify fields and environmental variables, possibly output to stdout or to a file, and then return control to "pipeline" with the modifications.

An example should help illustrate this. Say a user or organization has an HTML for users to fill out. The following actions take place on the submission of the form:

1. "pipeline" reads in the fields and stores them in an internally convenient manner. It also determines what modules to run and in what order.
2. The script calls a module that looks for fields that haven't been filled out, but are required to be filled out.
3. If it finds any, it module reports this to the user, so that the user and execution stops.
4. The script calls the next module.
5. This module fills in default values for fields that have not been filled out.
6. The script calls the next module with the new fields and same environmental variables.
7. etc.

More possible script actions are mentioned elsewhere.

This idea is similar to using UNIX pipes, with the scripts corresponding to filters. The main differences are the fact that the pipes are completely buffered, and it isn't complete programs being executed.

Anther good thing about this is that it benefits portability. For example, one of the main reasons Getcomments hasn't been ported to platforms other than UNIX is that it sends mail using UNIX-specific mail programs, although most of the rest of the perl code is fairly OS-independent. If Getcomments gets divided into separate parts, including one to send mail, then there could be a UNIX, Mac, and Windows version of the mail code.

What do you think of this idea in general? Give your opinion or look at what other people have said.

Possible Modules

Some modules have been written already and are included in the HTML Form Processing Modules (HFPM). Here are some other module ideas that myself and others have thought of that would be good candidates for a filter (modular) approach to processing:

• A script to report an error to a user if field(s) aren't filled out to match an indicated regular expression.
• A script to somehow try to validate a provided e-mail address, possibly looking at the REMOTE_HOST field as a sanity check.
• A script to remove HTML tags from specified field(s).
• A script to remove certain words/text from specified field(s). This might be used to remove curses from the text.
• A script to pass the from input to a non CGI FPI compliant script via a POST or GET method. This would probably the last thing in the pipeline.
• A script to pipe the fields to an external ordinary program, i.e. htpasswd.
• See those already implemented in HFPM
• See those mentioned in the WEBsmith article
• Ideas other people have mentioned
• Do you have another idea?

Getting It Implemented

The initial implementation, Pipeline, has been done in Perl version 5 to run on Unix. Implementations for other routines and languages are welcome.

Let me know if your willing to do this.

Efficiency

I think this approach is fairly efficient since the I/O is only done once. The users can add as many things to the pipeline as they think are prudent for their situation.