SESSION 4: Complex Information Processing <br><br>
<h1>4.2: <ahref="https://dl.acm.org/doi/pdf/10.1145/800197.806036"onClick="return popup(this, '000', '1000', '800','1000','500')">A File Structure for The Complex, The Changing and the Indeterminate</a></h1><br>
<h2>T. H. Nelson Vassar College, Poughkeepsie, N.Y.</h2><br><br>
THE KINDS OF FILE structures required if we are to use the computer for personal files and as an adjunct to creativity are wholly different in character from those customary in business and scientific data processing. They need to provide the capacity for intricate and idiosyncratic arrangements, total modifiability, undecided alternatives, and thorough internal documentation. <br><br>
The original idea was to make a file for writers and scientists, much like the personal side of <aclass="VB01"href="VB01.html">Bush's Memex</a>, that would do the things such people need with the richness they would want. But there are so many possible specific functions that the mind reels. These uses and considerations become so complex that the only answer is a simple and generalized building-block structure, user-oriented and wholly general-purpose. <br><br>
The resulting file structure is explained and examples of its use are given. It bears generic similarities <aclass="r"href="#">to</a> list-processing systems but is slower and bigger. It employs zippered lists plus certain facilities for modification and spin-off of variations. This is technically accomplished by index manipulation and text patching, but to the user it acts like a multifarious, polymorphic, many-dimensional, infinite blackboard. <br><br>
The ramifications of this approach extend well beyond its original concerns, into such places as information retrieval and library science, motion pictures and the programming craft; for it is almost <aclass="r"href="#">everywhere</a> necessary to deal with deep structural changes in the arrangements of ideas and things. <br><br>
I want to explain how some ideas developed and what they are. The original problem was to specify a computer system for personal information retrieval and documentation, able to do some rather complicated things in clear and simple ways. The investigation gathered generality, however, and has eventuated in a number <aclass="r"href="#">of</a> ideas. These are an information structure, <aclass="r"href="#">a</a> file structure, and a file language, each progressively more complicated. The information structure I call zippered lists; the file structure is the ELF, or <aclass="r"href="#">Evolutionary</a> Lis't File; and the file language (proposed) is called PRIDE. <br><br>
In this paper I will explain the original problem. Then I will explain why the problem is not simple, and why the solution (a file structure) must yet be very simple. The file structure suggested here is the Evolutionary List File, to be built of zippered lists. A number of uses will be suggested for such a file, to show the breadth of its potential usefulness. Finally, I want to explain the philosophical implications of this approach for information retrieval and data structure in a changing world.<br><br>
This work was begun in 1960 without any assistance. Its purpose was to create techniques for handling personal file systems and manuscripts in progress. These two purposes are closely related and not sharply distinct. Many writers and research professionals have files or collections of notes which are tied to manuscripts in progress. Indeed, often personal files shade into <aclass="r"href="#">manuscripts</a>, and the assembly of textual notes becomes the writing of text without a sharp break. <br><br>
I knew from my own experiment what can be done for these purposes with card file, notebook, index tabs, edge-punching, file folders, scissors and paste, graphic boards, index-strip frames, Xerox machine and the roll-top desk. My intent was not merely to computerize these tasks but to think out (and eventually program) the dream file: the file system that would have every feature a novelist or absent-minded professor could want, holding everything he wanted in just the complicated way he wanted it held, and handling notes and manuscripts in as subtle and complex ways as he wanted them handled. <br><br>
Only a few obstacles impede our using computer-based systems for these purposes. These have been high cost, little sense of need, and uncertainty about system design. <br><br>
The costs are now down considerably. A small computer with mass memory and video-type display now costs $37,000; amortized over time this would cost less than a secretary, and several people could use it around the clock. A larger installation servicing an editorial office or a newspaper morgue, or a dozen scientists or scholars, could cost proportionately less and give more time to each user. <br><br>
The second obstacle, sense of need, Is a matter of fashion. Despite changing economies, it is fashionably believed that computers are possessed only by huge organizations to be used only for vast corporate tasks or intricate scientific calculations. As long as people think that, machines will be brutes and not friends, bureaucrats and not helpmates. But since (as I will indicate) computers could do the dirty work of personal file and text handling, and do it with richness and subtlety beyond anything we know, there ought to be a sense of need. Unfortunately, there are no ascertainable statistics on the amount of time we waste fussing among papers and mislaying things. Surely half the time spent in writing is spent physically rearranging words and paper and trying to find things already written; if 95% of this time could be saved, it would only take half as long to write something. <br><br>
The third obstacle, design, is the only substantive one, the one to which this paper speaks. <br><br>
Let me speak first of the automatic personal filing system. This idea is by no means new. To go back only as far as 1945, Vannevar Bush, in his famous article "As We May Think "l , described a system of this type. Bush's paper is better remembered for its predictions in the field of information retrieval, as he foresaw the spread and power of automatic document handling and the many new indexing techniques it would necessitate. But note his predictions for personal filing: <br><br>
"Consider a future device for individual use, which is a sort of mechanized private file and library. It needs a name, and, to coin one at random, "memex" will do. A memex is a device in which an individual stores all his books, records, and con~nunications, and which is echanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate~ supplement to his memory. <br><br>
"It consists of a desk, and while it can pres~mebly be operated from a distance, it is primarily the piece of furniture at which he works. On the top are slanting translucent screens, on which material can be projected for convenient reading. There is a keyboard, and sets of buttons and levers. Otherwise it looks like an ordinary desk. <br><br>
"A special button transfers him immediately to the first page of the index. Any given book_ of his library /_and presumably other textual material, such as notes/ can thus be called up and consulted with far greater facility than if it were taken from a shelf. As he has several projection positions, he can leave one item in position while he calls up another. He can add marginal notes and comments, .... " (i, 106-7)<br><br> Understanding that such a machine required new kinds of filing arrangements, Bush stressed his file's ability to store related materials in associative trails, lists or chains of documents joined together. <br><br>"When the user is building a trail, he names it, inserts the name in his code book, and taps it out on his keyboard. Before him are the two items to be joined, projected onto adjacent viewing positions. At the bottom of each there are a number of blank code spaces, and a pointer is set to indicate one of these on each item. The user taps a single key, and the items are permanently joined .... <br><br>"Thereafter, at any time, when one of these items is in view, the other can be instantly recalled merely by tapping a button below the corresponding code space. Moreover, when numerous items have been thus joined together to form a trail, they can be reviewed in turn, rapidly or slowly, by deflecting a lever like that used for turning the pages of a book. It is exactly as though the physical items had been gathered together from widely separated sources and bound together to form a new book. It is more than this, for any item can be joined into numerous trails .... <br><br>"Thus he goes, building a trail of many items. Occasionally he inserts a comment of his own, either linking it into the main trail or joining it by a side trail to a particular item .... " (i, 107) <br><br>Two decades later, this machine is still unavailable*. <br><br>The hardware is ready. Standard computers can handle huge bodies of written information, storing them on magnetic recording media and displaying their contents on CRT consoles, which far outshine desktop projectors. But no programs, no file software are standing ready to do the intricate filing job (keeping track of associative trails and other stKuctures) Ithat the active scientist or thinker wants and needs. While WallaceZ reports that the System Development Corporation has found it worthwhile to give its employees certain limited computer facilities for their own filing systems, this is a bare beginning. <br><br>Let us consider the other desideratum, manuscript handling. The remarks that follow are intended to apply to all forms of writing, including fiction, philosophy, sermons, news and technical writing. <br><br>
The problems of writing are little understood, even by writers. Systems analysis in this area is scanty; as elsewhere, the best doers may not understand what they do. Although there is considerable anecdote and lore about the different physical manuscript and file techniques of different authors, literary tradition demerits any concern with technical systems as detracting from "creativity." (Conversely, technical people do not always appreciate the difficulty of organizing text, since in technical writing much of the organization and phraseology is given, or appears to be.) But in the computer sciences we are profoundly aware of the importance of systems details, and of the variety of consequences for both quality and quantity of work that result from different systems. Yet to design and evaluate systems for writing, we need to know what the process of writing is.<br><br> There are three false or inadequate theories of how writing is properly done. The first is that writing is a matter of inspiration. While inspiration is useful, it is rarely enough in itself. "Writing is l~/o inspiration, 9Y/o perspiration," is a common saying. But this leads us to the second false theory, that "writing consists of applying the seat of the pants to the seat of the chair." Insofar as sitting facilitates work, this view seems reasonable, but it also suggests that what is done while sitting is a matter of comparative indifference; probably not.<br><br>The third false theory is that all you really need is a good outline, created on prior consideration, and that if the outline is correctly followed the required text will be produced. For most good writers this theory is quite wrong. Rarely does the original outline predict well what headings and sequence will create the effects desired: the balance of emphasis, sequence of interrelating points, texture of insight, rhythm, etc. We may better call the outlining process inductive: certain interrelations appear to the author in the material itself, some at the outset and some as he works. He can only decide which to emphasize, which to use as unifying ideas and principles, and which to slight or delete, by trying. Outlines in general are spurious, made up after the fact by examining the segmentation of a finished work. If a finished work clearly follows an outline, that o~line probably has been han~nered out of many inspirations, comparisons and tests**. <br><br>Between the inspirations, then, and during the sitting, the task of writing is one of rearrangement and reprocessing, and the real outline develops slowly. The original ~rude or fragmentary texts created at the outset generally undergo many revision processes before they are finished. Intellectually they are pondere~ juxtaposed, compared, adapted, transposed, and judged; mechanically they are copied, overwritten with revision markings, rearranged and copied again. This cycle may be repeated many times. The whole grows by trial and error in the processes of arrangement, comparison and retrenchment. By examining and mentally noting many different versions, some whole but most fragmentar~l the intertwining and organizing of the final written work gradually takes place***. <br><br>Certain things have been done in the area of computer manuscript handling. IBM recently announced its "Administrative Terminal SysteW'5,6,7, 8 which permits the storage of unfinished sections of text in computer memory, permits various modifications by the user, and types up the final draft with page numbers, right justification and headers. <br><br>While this is a good thing, its function for manuscripts is cosmetic rather than organizing. Such a system can be used only with textual sections which are already well organized, the visible part of the iceberg. The major and strenuous part of such writing must already have been done. <br><br>
If a writer is really to be helped by an automated system, it ought to do more than retype and transpose: it should stand by him during the early periods of muddled confusion, when his ideas are scraps, fragments, phrases, and contradictory overall designs. And it must help him through to the final draft with every feasible mechanical aid-making the fragments easy to find, and making easier the tentative sequencing and juxtaposing and comparing. <br><br>It was for these two purposes, taken together-personal filing and manuscript assembly-that the following specifications were drawn up. Here were the preliminary specifications of the system: It would provide an up-to-date index of its own contents (supplanting the "code book" suggested by Bush). It would accept large and growing bodies of text and commentary, listed in such complex forms as the user might stipulate. No hierarchical file relations were to be built in; the system would hold any shape imposed on it. It would file texts in any form and arrangement desired-combining, at will, the functions of the card file, loose-leaf notebook, and so on. It would file under an unlimited number of categories. It would provide for filing in Bush trails. Besides the file entries themselves, it would hold co~mnentaries and explanations connected with them. These annotations would help t]ne writer or scholar keep track of his previous ideas, reactions and plans, often confusingly forgotten. <br><br>In addition to these static facilities, the system would have various provisions for change. The user must be able to change both the contents of his file and the way they are arranged. Facilities would be available for the revising and rewording of text. Moreover, changes in t]he arrangements of the file's component parts should be possible, including changes in sequence, labelling, indexing and comments. <br><br>It was also intended that the system would allow index manipulations which we may call dynamic outlining (or dynamic ~indexing). Dynamic outlining uses the change in one text sequence to guide an automatic change in another text sequence. That is, changing an outline (or an index) changes the sequence of the main text which is linked with it. This would permit a writer to create new drafts with a relatively small amount of effort, not counting recordings. <br><br>However, because it is necessary to ,examine changes and new arrangements before deciding to use or keep them, the system must not commit the user to a new version until he is ready. Indeed, the system would have to provide spin-off facilities, allowing a draft of a work to be preserved while its successor was created. Consequently the system must be able to hold several-- in fact, many-- different versions of the same sets of materials. Moreover, these alternate versions would remain indexed to one another, so that however he might have changed their sequences, the user could compare their equivalent parts. <br><br>Three particular features, then, would be specially adapted to useful change. The system would be able to sustain changers in the bulk and block arrangements of its contents. It would permit dynamic outlining. And it would permit the spinoff of many different drafts, either successors or variants, all to remainwithin the file for comparison or use as long as needed. These features we may call evolutionary. <br><br>The last specification, of course, one that emerged from all the others, was that it should not be complicated. <br><br>These were the original desiderata. It was not expected at first that a system for this purpose would have wider scope of application; these jobs seemed to be quite enough. As work continued, however, the structure began to look more simple, powerful and general, and a variety of new possible uses appeared. It became apparent that the System might be suited to many unplanned applications involving multiple categories, text summaries or other parallel documents, complex data structures requiring human attention, and files whose relations would be in continuing change. <br><br>
A list is an ordered set of entries designated by the user. A given entry may be in any number of lists. <br><br>A link is a connector, designated by the user, between two particular entries which are in different lists; Figure I. An entry in one list may be linked to only one entry in another list. On the left we see two zippered lists. Between the entries of list A and those of B are dashed lines, representing the links between the two lists. <br><br>On the right is the table of links as it might look to a machine. The machine can read this table from right to left or left to right, finding entries in B that correspond to given entries in A, or vice versa. A change in the sequence of either list, or additions to either list, will not: change the links that stand between them. Changes in the link structure will occur only if the user specifically changes the links, or if he destroys entries which are linked to others. <br><br>To be technical, then, two lists are zippered if there are any pairwise links between their respective elements, each element is in no more than one link pair, and these links are unaffected by permutation of the lists, remaining affixed to the same pairs of elements. It is not required that the two lists be of the same length, or, even if they are, that all entries have a link to the other list. <br><br>The ELF's File Operations <br><br>Zippered lists are an information structure; the Evolutionary List File is a file structure. The ELF described in this paper holds its contents exclusively as zippered (or unzippered) lists. But the file structure must also include a set of operations by which it may be modified. These file operations exist for creating, adjusting or removing the entry, list and ]Link, and for manipulating the sequence relation. An ELF is actually any machine ~ich will, on command, carry out the basic operations on entry, list, link and sequence. <br><br>Entries. The user may create new entries at any time, putting anything in them that he thinks appropriate. Entries ~y be combined or divided (unless indivisible, like objects, commands, etc.) Entries may be put in any list, and the same entry may be put in different lists. The user may direct that entries of one list be automatically copied onto another list, without affecting the original list. <br><br>Lists. The user may create lists and assign entries to them. He may at will make new copies of lists. He may rearrange the sequence of a list, or copy the list and change the sequence of that copy. Lists may be combined; lists may be cut into sublists. <br><br>Links. Ths user may create links between entries that are in different lists. Any number of legal links may be created, although the upper limit of links between any two lists is determined by the l-for-I rule. When an entry or a list is copied into a list, links will remain between parent and daughter entries. Moreover, after a list-copying operation, the daughter list will have the same links to all other lists as does the parent list. <br><br>Sequences. The user may put a list in any sequence he wishes. (A copied list will maintain the original sequence until modified.) Sequences may be transferred between lists via the links: if the sequence of A is transferred to B, each entry of A linked to an entry in B takes tile sequential position of its linked No definite meaning is assigned to these entities or operations by the system; the user is free to let them mean anything he likes. A list may be a category, trail, index, dialogue, catalogue or poem, and lists may be assembled into larger structures. The ELF may be thought of as a place; not a machine, but a piece of stationery or office equipment with many little locations which may be rearranged with regard to one another****.<br><br>Note that zippered lists generate only one of various possible Evolutionary List Files. Indeed, the description of the file structure given here is in some ways restrictive: the ELF could take a number of other, closely similar forms and still be much the same thing. For example, it would be
The ELF appears to be closest, topologically and in other organizing features, to the Multilist system described by Prywes and Gray 12. Like that system, it permits putting entries in many different lists at once. However, in current intent 13 that system is firmly hierarchical, and thus somewhat removed from the ELF's scope of application. Another closely related system is the Integrated Data Store of Bachmanl4,15,16,17,18; this is intended as a hardware-software system for disc I/0 and storage arrangement, but in its details it seems the ELF's close relative. Each of these systems has a cormection logic that might be feasible as a basis for an ELF different from this one. Or, either might prove a convenient programming base for the implementation of this file structure. <br><br>Another obvious technical question =mst be considered. How can the ELF allow "unlimited" copies of entries and lists? By patching techniques, of course. Variant entries and lists can take virtually no space, being modification data plus pointers to the original. When a modified version of a list or entry is created, the machine patches the original with the changes necessary to make the modified version: Figure 2. <br><br>USES<br><br> In the discussion that follows, we will examine various possible applications of zippered lists and the ELF, and postpone discussing the file language they require. Finally we will return to this problem, and describe the file language PRIDE whose additional features are neededto adapt the ELF for the uses originally discussed. <br><br>By assigning entries to lists, the ELF may be used as a glorified card file, with separate lists used for categories, trails, etc. This permits extensive cross-indexing by the assignment of one entry to different lists. It permits subsets and sub-sequences for any use to be held apart and examined without disturbing the lists from which they have been drawn, by copying them onto other, new lists. The ELF permits the filing of historical trails or associative (Bush) trails through documents, business correspondence, belles-lettres, case law, treaties, scholarly fields and history, and the mixture of trail with categorical filing. These are the simple useR; the compound uses are much more interesting. But since we cannot intuitively fit every possible conceptual relationship into zippered lists, imaginative use is necessary. Remember that there is no correct way to use the system. Given its structure, the user may figure out any method useful to him. A number of different arrangements can be constructed in the ELF, using only the basic elements of entry, list and link. Zippered lists may be assembled into rectangular arrays, lattices and more intricate configurations. These assemblies of lists may be assigned meaning in combination by the user, and the system will permit them to be stored, displayed, taken apart for examination, and corrected, updated, or modified. <br><br>By using such combining arrangements on lists composed of text, the file can be self-documenting, with all labelling and documentation kept integrally within the file structure. It is thus possible to incorporate, in a body of information filed in the ELF, various levels of index, sunmmry, explanation and commentary. Many useful ways of listing and linking such documentation are possible. In Figure 3 we see some of the ways that documentary lists may be linked together. The lists shown are outline, suboutline, draft, subdraft, summary, commentary and source list. These are not all the possible types of documentary lists; for example, "footnotes" are omitted. The ELF will permit any number of these documentary lists; for example, "footnotes" are omitted. The ELF will permit any number of these documentary lists; observe that they can be built on one another, and indefinitely compounded. The system will have no trouble accepting a commentary on a commentary on a subdraft of an outline for a variant list of source materials. <br><br>Figure 3 shows also how two lists may contain some of the same entries. The dashed li
<hr>
* The Bush Rapid Selector 2 is a powerful microfilm instru- ment, but it is not suited to idiosyncratic personal uses, nor to evolutionary modification, as described. <br>
** It is believed that this account is reasonably correct for such writers as Tolstoy, Winston Churchill and Katherine Anne Porter. Those who can adhere to a prior outline faithfully, like James Fenimore Cooper, tend to be either hacks or prodigres, and do not need this system <br>
*** For a poignant, mordant portrayal of the writer's struggle, the reader is directed to Gorey's "The Unstrung Harp", or "Mr Earbass Writer.s a Novel". <br>
**** An ELF might even be consstructed out of cards, blocks, sticks and strings, using techniques of puppetry, but this would not be a convenient object. <br>
***** The sense of "hyper-" used here connotes exten- sion and generality; cf. "hyperspace." The criterion for this prefix is the inability of these objects to be comprised sensibly into linear media, like the text string, or even media of somewhat highe~ complexity. The ELF is a hyperfile. <br><br>
1 Bush, V., "As We May Think." :['he Atlantic Monthly, p. 101-108; July, 1945. <br>
2 Hirsch, P., "The Bush :Ropid Selector." Datamation, p. 56-57; June, 1965. <br>
3 Wallace, E. M., "Experience with EDP Support of Indi- viduals' File Maintenance." Parameters of Information Science: Proceedings of the American. Documentotion Insti- tute (American Documentation Institute), p. 259-261; 1/1964. <br>
4 Gorey, E., "The Unstrung Harp; or, Mr. Earbrass Writes a Novel," Duell, Sloan and Pearce, N. Y.; 1953. <br>
5 IBM Data Processing Division, "The IBM Administrative Terminal System," Brochure 520-1146. <br>
6 IBM Technical Publications Department, "14~0/1460 Ad- ministrative Terminal System Application Description," White Plains, New York. <br>
7 Timberlake, W. D., "Administrative Terminal System" (abstract.) STWP Proceedings; May, 1965. <br>
8 Farrell, A. C., "Evolution of Automated Writing." STWP Proceedings; May, 1965. <br>
9 Bobrow, D. G., and Bertram, R., "A Comparison of List- Processing Languages." Comm. ACM, p. 231-240; April, 1964. <br>
10 Comfort, W. T., "Multiword List Items." Comm. ACM, p. 357-362; June, 1964. <br>
11 Weizenbaum, J., "Knotted List Structures." Comm. ACM, p. 161-165; Mar., 1962. <br>
12 Prywes, N. S,, and Gray, H. J., "The Multilist System for Real Time Storage and Retrieval," Proceedings of IFIP Conference, p. 112-116; 1962. <br>
13 Prywes, N. S., "Interim Technical Report: The Organi- zation of Files for Command and Control," Moore School o[ Engineering; March, 1964. <br>
14 Bachman, C. W., and Williams, S. B., "The Integrated Data Store--A General Purpose Programming System for Random Access Memories," AFIPS Conference Proceedings, p. 411-422; 1964. <br>
15 General Elect'ric Computer Department, "I-D-S:" Com- pany Brochure CPB-425.<br>
16 General Electric Computer Department, "Introduction to Integrated Data Store," Company Brochure CPB-1048; 1965. <br>
17 Bachman, C. W., "Software for Random Access Process- ing," Datamation; April, ].965. '" <br>
18 General Electric Computer Department, "Integrated Data Store: New General Electric Technique for Organizing Busi- ,less Data"; January, 1965 '<br>
19 Corbin, H. S., and Stock, G. J., "On-Line Querying via a Display Console," Fourth Nat~o~al Symposium on Informa- t:oa Display: Technical S~'ssio~ Proceedings, p. 127-154; 1964.
var rooms = ['VB01.html','Ixse.html','Ixse2.html','Ixse4.html','EOC.html','EOC2.html','./home/doorway.html','./AxS/index.html','./0/ilinx_0.html','cyberSit.html'];
