title: "torn at the seams: considering digital vernacular"
author: "Michael Murtaugh"
papersize: a4
---
> Processing is a free, open source programming language and environment used by students, artists, designers, architects, researchers and hobbyists for learning, prototyping, and production. Processing is developed by artists and designers as an alternative to proprietary software tools in the same domain. The project integrates a programming language, development environment, and teaching methodology into a unified structure for learning and exploration.[^p1]
Teaching programming with free software to media design students for years, I've resisted Processing as it has always seemed to me to embody a particular kind of solipsism of digital interactivity and graphics that I want my students to avoid.
## design by numbers
> Processing is a free, open source programming language and environment used by students, artists, designers, architects, researchers and hobbyists for learning, prototyping, and production. Processing is developed by artists and designers as an alternative to proprietary software tools in the same domain. The project integrates a programming language, development environment, and teaching methodology into a unified structure for learning and exploration. […]
>
> Most of the examples presented in this book have a minimal visual style. This represents not a limitation of the Processing software, but rather a conscious decision by the authors to make the code for each example as brief and clear as possible. We hope the stark quality of the examples gives additional incentive to the reader to extend the programs to her or his own visual language.[^p1]
[^p1]: Casey Reas and Ben Fry, *Processing: A Programming Handbook for Visual Designers and Artists* (Cambridge: MIT Press, 2007), xxi-xxii.
Teaching programming with free software to media design students for years, I've resisted Processing as it has always seemed to me to embody a particular kind of solipsism of digital interactivity and graphics that I want my students to avoid.
In the fall of 1996, John Maeda joined the MIT Media Lab to replace the recently deceased Muriel Cooper. Cooper was the first art director of the MIT Press, producing influential designs such as a 1969 catalog of the Bauhaus and the iconic MIT Press logo, a Bauhaus-inspired stylized graphical rendering of the letters “mitp”. Cooper started the Visible Language Workshop, later one of the founding groups of the MIT Media Lab, to research the intersection of publishing, design, and computation.
> Solipsism: a theory holding that the self can know nothing but its own modifications and that the self is the only existent thing; also: extreme egocentrism[^solip]
> Our forefathers at the Bauhaus, Ulm, and many other key centers for design education around the world labored to create a sense of order and method to their teaching. Thanks to their trailblazing work, teaching at the university level gradually became accepted as a meaningful and constructive activity. A drawing board, small or large, became the stage for paper, pen, ink, and blade to interact in the disciplined activity that characterized the profession of visual design.[^dbn]
In the summer of 1996, as I was finishing my Masters degree, I met John Maeda who was just joining the MIT Media Lab to replace the recently deceased Muriel Cooper. Cooper was the first art director of the MIT Press, producing influential designs such as a 1969 catalog of the Bauhaus and the iconic MIT Press logo, a Bauhaus-inspired stylized graphical rendering of the letters “mitp”. Cooper started the Visible Language Workshop, later one of the founding groups of the MIT Media Lab, to research the intersection of publishing, design, and computation.
[^dbn]: John Maeda, Design By Numbers. (Cambridge: MIT Press, 1999), 19.
My friend Robin was a student in the VLW; through her I learned things like how Muriel would say "Swiss" instead of Helvetica. I took one class in typography at the VLW; among the many constraints we were given, we could only use one font, named for swiss typographer Adrian Frutiger.
[^dbn_144]: Maeda, Design By Numbers. 144.
Maeda created the Aesthetics and Computation group in part to continue Cooper’s research. Ben Fry and Casey Reas were among Maeda's first students. Maeda published a software system called Design by Numbers. It had extreme constraints such as a fixed 100 by 100 pixel size and monochrome-only graphics. The accompanying print publication also had a square format. DBN would go on to inspire Reas + Fry to create Processing. When one starts the Processing application, the default starting code comprises two commands: createCanvas and background. The default canvas output is square, and a single numeric parameter to the background command produces a gray-scale value, both echoes of earlier DBN constraints. Both DBN and Processing were written in Java, a desktop application that exists outside of the web but which can be used to publish sketches as "applets" embedded in a web page and published online.
[^dbn_252]: Maeda, Design By Numbers. 252.
Maeda created the Aesthetics and Computation group in part to continue Cooper’s research. Maeda developed (with students Tom White, Peter Cho, Ben Fry, and later Casey Reas among others) a software system called Design by Numbers[^dbnsite]. It had extreme constraints such as a fixed 100 by 100 pixel size and monochrome-only graphics. The command set is similarly constrained with for instance only two drawing commands for lines and points. Commands like "paper" and "pen", controlling the gray value of background and foreground colors, invoke the materialty of a (pre-digital) print practice. The accompanying print publication also had a square format.
In 2004, Reas co-developed an exhibition at the Whitney gallery called Software Structures. Inspired by Sol Lewitt's wall drawings, Software Structures presented a series of abstract rules for the production of an image, including some from Lewitt. The rules were then implemented using a variety of "materials": Processing, Flash MX, and C++.
[^dbnsite]: <https://dbn.media.mit.edu/>
> A benefit of working with software structures instead of programming languages is that it places the work outside the current technological framework, which is continually becoming obsolete. Because a software structure is independent from a specific technology, it is possible to continually create manifestations of any software structure with current technology to avoid retrograde associations.[^sst]
> When designing this system for learning basic computational media design, I intentionally limited the set of commands and constructs to a minimal number of possibilities. If I had given you drawing capability beyond a line or setting a dot, the examples could have been more exciting, but the point could not be made clearly because your attention would be drawn to the picture and not to the code.[^dbn_144]
[^sst]: Casey Reas, "Paragraphs on Software Art", Software Structures, retrieved October 28 2021, <https://artport.whitney.org/commissions/softwarestructures/text.html#structure>.
## two books
This valorization of "code" over picture is evident in the book spread where a digital image of a vase is presented along with its "code": a sequence of "set" commands describing the image pixel by pixel. Though it's possible that such an image might be produced in this way (discipline!) in practice such images are created by translating a digital photograph using one of a family of techniques known as "dithering" to produce a pixelated image with limited gray values. Here these tools and techniques are unmentioned, with expansive space given instead to the listing of numbers.
In 2007 I attended a book launch of: Processing: A Programming Handbook for Visual Designers and Artists (MIT Press, Reas + Fry, with a foreword by Maeda).
I remember sitting with a friend at his Commodore 64 in the 1980s typing lines and lines of "poke" commands with digital data from the pages of home computer magazines to reproduce simple games. These pages would typically use compact representations (like hexadecimal) and include things like "checksums" and bootstrap programs to help you correctly enter and verify the data you entered.
In the case of the DBN's digital vase, the purpose does not seem to be for someone to actually type the numbers in, but rather to fill the pages of a book that seems destined for the coffee table rather than a desktop. Similar to the fields of 1s and 0s still popular as backgrounds of book sleeves and powerpoint slides to suggest "all things digital", the presentation is gratuitous and misleading in terms of actual practice. Like an observer unfamiliar with deaf culture confusing the hand gestures of finger spelling with the expressivity (and ambiguity) of actual sign language, the displays of 1s and 0s, or in this case of numeric coordinates and gray values, is a shorthand that refers digital practices without actually participating in them.
The above is just one example of DBN's lack of historicity. The year is 1999, and for instance the Sony PlayStation and access to the World Wide Web are popular phenomea. Yet the text seems remarkably devoid of any reference to specific tools or practices. Even Maeda's invocations of historical figure like the Bauhaus and Paul Rand are vague, evoking a sort of nostalgic *Mad Men* universe with the creative young men (those future forefathers) at their drawing boards and a sense of "timeless design values" like a devotion to discipline and order.
At the end of the book Maeda responds to a critique apparently made to him by one of the students doing some of the programming of the DBN system itself. Recounting how a visit to a "university-level Java class for designers [...] teach[ing] the finer points of object-oriented programming and bit masking of 24-bit color values" made him feel "lost in all the gibberish", he reasserts his pedagogical approach as an alignment with the "simplification" that is the "constant goal" of programming.[^dbn_252] Rather than trying to bridge the gulf between diverse practices, Maeda dismisses that which he doesn't (care to) understand in the name of simplification.
## processing
Processing was born as kind of fork or remake of DBN. Like DBN, Reas and Fry built Processing in Java, a desktop application that exists outside of the web but which can be used to publish sketches as "applets" embedded in a web page and published online. Reas and Fry added commands to work with color and multiple output sizes, as well as commands to draw basic shapes like circle, rectangle, and triangle.
Processing sketches consists of (at least) two functions: *setup* which is invoked once and *draw* which is invoked continuously; the default frequency being the refresh rate of the computer’s display (typically 60 times per second). By using variables combined with input functions allowing access to the mouse and keyboard, graphics can be made dynamic and responsive to the user. In addition, graphics by default are rendered using a technique known as “anti-aliasing” to appear “smoother” and less pixelated.
The book comprises over 700 pages and is organized by topics like: color, control, data, drawing, image, input, math, motion, structure, typography. The book also contains interviews with artists working with digital tools not limited to Processing. As indicated in the book's introduction, many examples are illustrated by "stark" small square images and concise code.
> Most of the examples presented in this book have a minimal visual style. This represents not a limitation of the Processing software, but rather a conscious decision by the authors to make the code for each example as brief and clear as possible. We hope the stark quality of the examples gives additional incentive to the reader to extend the programs to her or his own visual language.[^p1]
Processing: A Programming Handbook for Visual Designers and Artists is published in 2006 by the MIT Press, with Reas + Fry authors, and a foreword by Maeda. The book comprises over 700 pages and is organized by topics like: color, control, data, drawing, image, input, math, motion, structure, typography. The book follows the visual style of the DBN book, with small square (mostly) monochrome images accompanying concise listings of code.
Processing commands typically address the canvas using x,y (Cartesian) coordinates. Shape commands, like circle, rectangle, and triangle allow for the drawing of graphical forms, while other functions control parameters like the color of fill or the width of the stroke (or outline) of each shape.
Despite the claim of leaving space for others to bring their own "visual language", and thus an implicit proposition of its own aesthetics as "neutral", Processing embodies a very particular set of values and assumptions. The framework valorizes smoothness and fluidity, that leads one to imagine interactivity as that which happens on the surface of a sketch, rather than say in the network, or among collaborators. The mechanism of the "draw loop" assumes that code runs in a neglible amount of time, that is less than the refresh rate (and the default 1/60th of a second). This particular, again implicit, relationship with time places the programmer in an adversarial relation with the processor of the viewer's computer and makes it all too easy (certainly for novices) to produce code that makes the viewer's computer struggle and lag.
Input functions allow access to the mouse and keyboard to produce dynamic graphics that respond to the user. Processing sketches consists of (at least) two functions: *setup* which is invoked once and *draw* which is invoked continuously; the default frequency being the refresh rate of the computer’s display (typically 60 times per second). By using variables, and changing their values, graphics can be made dynamic. In addition, graphics are rendered using a technique known as “anti-aliasing” to appear “pixelated”. This default behavior can be modified by using the noSmooth command.
Unlike DBN, the processing book contains extensive interviews with artists working with digital tools, mostly using a variety of tools outside of processing like C++, PHP and Flash. Despite the "minimalism" of the examples, the book's subtitle claims relevance to a broad audience of "visual designers and artists". The link to processing is often unclear.
Earlier in the day I had bought another technical book "ImageMagick Tricks: Web Image Effects from the Command line and PHP" by Sohail Salehi[^salehi]. While waiting for the presentation to begin, I crossed paths with Casey Reas at the back of the room. He was curious about the book I had with me and looked briefly at it. He had never heard of ImageMagick.
There's a strange disconnect where a diversity of visual and computation practices are shown, such as sequential images to make animations, cellular automata, the only real link to processing seems to be as a kind of universal "blank canvas" onto which all kinds of (digital) practices can be projected.
Processing exhibits a huge disconnect when one crosses out of what the software makes simple. For example, while very easy to make a sketch respond to mouse clicks, responding to a mouse click on a specific shape, suddenly involves using Java classes and adding code "hooks" to your draw loop to make sure the pieces work together. While this kind of code is not itself so unusual, the transition is really abrupt because the core abstractions are so simplified, there is no bridge.
> ImageMagick started with a request from my DuPont supervisor, Dr. David Pensak, to display computer-generated images on a monitor only capable of showing 256 unique colors simultaneously. In 1987, monitors that could display 24-bit true color images were rare and quite expensive. There were a plethora of chemists and biologists at DuPont, but very were few computer scientists to confer with. Instead, I turned to Usenet for help, and posted a request for an algorithm to reduce 24-bit images to 256 colors. Paul Raveling of the USC Information Sciences Institute responded, with not only a solution, but one that was already in source code and available from USC's FTP site. Over the course of the next few years, I had frequent opportunities to get help with other vexing computer science problems I encountered in the course of doing my job at DuPont. Eventually I felt compelled to give thanks for the help I received from the knowledgeable folks on Usenet. I decided to freely release the image processing tools I developed to the world so that others could benefit from my efforts.[^cristy]
Another disconnect involves processing sketches when published online. Web pages have structures for text layout and content flowing, and structual elements (like checkboxes and links) that inherently respond to mouse clicks. There is a a built-in standard mechanism (the so-called DOM or document object model), to further customize these behaviors. Processing sketches, even situated in a web page, don't participate or give access to this model, nor does the framework offer using alternative graphical elements like SVG[^svg].
[^svg]: A web standard for "scalable vector graphics", SVG is more integrated into the DOM than are java applets (in classic Processing) or the canvas element (see p5.js).
ImageMagick, first released in 1990, is a popular free software tool that’s often referred to as a “swiss army knife” due to its ability to convert between hundreds of different image formats, and for the many built-in features to filter, manipulate and generate images. Thanks to the software being "not chemically or biologically based", Cristy was able to release the software under a Free license (echoing the way the UNIX operating system became free software due to its marginality to the interests of Bell Labs). The software is full of particularities. For instance, there are a number of built in images including a wizard (the mascot of the software) seated at a drawing table contemplating an image of the Mona Lisa.
### software structures
ImageMagick is a command-line tool, designed to be used via textual commands. The typical usage of ImageMagick is to take one image as input, applying one or more transformations to it, and output a new image. In this way the tool can be used repeatedly in a so-called “pipelines”, or otherwise composed together in structures called (BASH) scripts. In these scripts ImageMagick commands can be mixed with other commands from any software installed on the user's computer that also provides a command-line interface.
In 2004, Reas co-developed an exhibition at the Whitney gallery called Software Structures. Invoking Sol Lewitt's wall drawings as an inspiration, the project presents a series of abstract rules (or software structures) for the production of an image, including some from Lewitt. The rules were then implemented using a variety of "materials": Processing, Flash MX, and C++.
[^cristy]: John Cristy, "History", ImageMagick website. Retrieved from the Internet archive Oct 28, 2021, <https://web.archive.org/web/20161029234747/http://www.imagemagick.org/script/history.php>.
> A benefit of working with software structures instead of programming languages is that it places the work outside the current technological framework, which is continually becoming obsolete. Because a software structure is independent from a specific technology, it is possible to continually create manifestations of any software structure with current technology to avoid retrograde associations.[^sst]
Salehi’s book directly reflects the structure of ImageMagick, with chapter organized around various incorporated “tools”: convert, mogrify, composite, montage, identify, display, conjure. The examples are practical, creating logos or adding captions or a border to an image. One example renders the word “Candy” with colorful stripes. Another series of examples duplicates and inverts the image and text of classical Persian poet “Hafez” to create a kind of playing card. Another example uses ImageMagick in conjunction with PHP and HTML to produce an online “e-card maker”, a sequence of commands is demonstrated to render the text “No More War” (in a dripping paint font), deform it, and project it onto the side of a chess piece.
[^sst]: Casey Reas, "Paragraphs on Software Art", Software Structures, retrieved October 28 2021, <https://artport.whitney.org/commissions/softwarestructures/text.html#structure>.
In 2016, the Whitney published a “restored” version of Software Structures.[^ssreloaded] As technologies like Java and Flash had then for reasons both technical and commercial fallen out of popular use on the web, the new version featured many of the processing sketches adapted by Reas to use p5.js, a recent port of processing to Javascript made by artist Lauren Lee McCarthy.[^p5]
In another extended example, a British flag is constructed in steps. Rather than approaching the project as drawing geometric forms on a canvas, Salehi uses the diversity of ImageMagick’s manipulations, performing a series of commands whose textual names invoke a sense of physical construction: blocks of color are skewed, sheared, cropped, flipped, flopped, and finally spliced (with "gravity" set to center). The approach creates a number of intermediate images, thus creating the digital equivalents of "cuttings" or leftover materials in the process. Below, the (intermediate) results from the same example slightly modified to take an image as input:
Despite the project's earlier stated interest in exploring diverse "materialities", it’s telling that rather than considering the older processing implementations as a different material and presenting screenshots of them as was done for the Flash and C++ examples, the "restoration" maintains the illusion of a "permanence" to the processing sketches, placing them closer to those imagined "software structures" than to "retrograde" technologies like Java or an out-dated browser. In addition the "adaptation" elides the work of the development and subsequent implementation of the then recently standardized canvas element[^canvas], as well as that of the creation of the p5.js library.
[^p5]: Despite the seeming similarity of names, Java and Javascript are two completely independent and quite different programming languages. Adapting software from one to the other is not trivial. In making the adaptation McCarthy has explicitly stated that "diversity and inclusion as core values upon which the software is built". See: <https://lauren-mccarthy.com/p5-js> and <https://p5js.org/>. However, most of the limitations I talk about here still apply to sketches made with the current version of p5.js.
convert -size 300x188 xc:none -draw ' image over 0,0 0,0 area1.png' -draw ' image over 150,0 0,0 area2.png' -draw ' image over 0,94 0,0 area4.png' -draw ' image over 150,94 0,0 area3.png' mixed.png
convert mixed.png -background white -gravity center -splice 20x20 -background '#ce201a' -splice 40x40 flag.png
```
## constructivism
## imagemagick
In the 1920s Russian avant-gardist El Lissitsky would move to Berlin and produce work that was highly influential to the then nascent Bauhaus. In 1923, Lissitsky illustrated a publication of poems by friend Vladimir Majakovskij. In it he created graphical forms by mixing typographic elements with geometric forms created by (mis-) using spacing elements (the “blind” elements typically used to create (negative/unprinted) space between lines of type, as positives producing geometric forms. This style, sometimes called constructivism, was part of an effort to make a radical break from traditional styles of typographic layout and illustration using the means then available for print. The book is notable for its "interactivity" via iconic tabbed pages.
In 2007 I attended a book launch of the Processing Handbook. Earlier in the day, I had bought another technical book "ImageMagick Tricks: Web Image Effects from the Command line and PHP" by Sohail Salehi[^salehi]. While waiting for the presentation to begin, I met Casey Reas at the back of the room. He was curious about the book I had with me and looked briefly at it. He had never heard of ImageMagick.
In the 1970s Seymour Papert co-developed a pedagogy for teaching children mathematics and programming based on the LOGO programming language. Part of the system was a (virtual) robotic turtle that could be programmed to draw figures. The system, known as Turtle graphics, had commands like: forward, turn left, turn right, pen up, pen down, that directly addressed the “turtle” to draw shapes while moving.
> The process reminds one of tinkering: learning consists of building up a set of materials and tools that one can handle and manipulate. Perhaps most central of all, it is a process of working with what you’ve got. …. This is a science of the concrete, where the relationships between natural objects in all their combinations and recombinations provide a conceptual vocabulary for building scientific theories. Here I am suggesting that in the most fundamental sense, we, as learners, are all *bricoleurs*.[^mindstorms]
> ImageMagick started with a request from my DuPont supervisor, Dr. David Pensak, to display computer-generated images on a monitor only capable of showing 256 unique colors simultaneously. In 1987, monitors that could display 24-bit true color images were rare and quite expensive. There were a plethora of chemists and biologists at DuPont, but very were few computer scientists to confer with. Instead, I turned to Usenet for help, and posted a request for an algorithm to reduce 24-bit images to 256 colors. Paul Raveling of the USC Information Sciences Institute responded, with not only a solution, but one that was already in source code and available from USC's FTP site. Over the course of the next few years, I had frequent opportunities to get help with other vexing computer science problems I encountered in the course of doing my job at DuPont. Eventually I felt compelled to give thanks for the help I received from the knowledgeable folks on Usenet. I decided to freely release the image processing tools I developed to the world so that others could benefit from my efforts.[^cristy]
ImageMagick, first released in 1990, is a popular free software tool that’s often referred to as a “swiss army knife” due to its ability to convert between hundreds of different image formats, and for the many built-in features to filter, manipulate and generate images. Thanks to the software being "not chemically or biologically based", Cristy was able to release the software as free software (echoing the way the UNIX operating system became free software due to its marginality to the interests of Bell Labs). The software is full of particularities. For instance, there are a number of built in images including a wizard (the mascot of the software) seated at a drawing table contemplating an image of the Mona Lisa.
Papert's approach is part of a pedagogic project (also) called constructivism. In a key example, Papert describes how students can be taught about circles by imagining (or better yet themselves enacting) the turtle repeatedly performing the instruction sequence "go forward a little, turn a little". He contrasts this with the formal equation of a circle (x^2^ + y^2^ = r^2^) typically taught in an elementary school geometry class.
ImageMagick is a command-line tool, designed to be used via textual commands. The typical usage of ImageMagick is to take one image as input, applying one or more transformations to it, and output a new image. In this way the tool can be used repeatedly in a so-called “pipelines”, or otherwise composed together in structures called (shell) scripts. In these scripts ImageMagick commands can be mixed with other commands from any software installed on the user's computer that also provides a command-line interface.
In Belgium, where I currently live "bricolage" is the French language equivalent to "DIY" and is often used in a derogatory sense to indicate that something is made in an amateurish way. Papert embraces the term, a central point being how informal/intuitive methods not only appeal to "common sense" but also engage more profoundly with the materiality of its subject than would a formal approach. In the case of the circle, the "turtle" method is not only a way for the student to imagine the problem physically, it also relates to methods of differential calculus, something the algebraic formulation misses completely.
[^cristy]: John Cristy, "History", ImageMagick website. Retrieved from the Internet archive Oct 28, 2021, <https://web.archive.org/web/20161029234747/http://www.imagemagick.org/script/history.php>.
[^mindstorms]: Seymour Papert, *Mindstorms* (Cambridge: MIT Press, 1980), 173.
> For most there is a total dissociation between these live activities and the dead school math. We have stressed the fact that using the Turtle as metaphorical carrier for the idea of angle connects it firmly to the body geometry. We have called this *body syntonicity*. Here we see a *cultural syntonicity*: The Turtle connects the idea of angle to navigation, activity firmly and positively rooted in the extraschool culture of many children.[^mindstorms2]
Salehi’s book directly reflects the structure of ImageMagick, with chapter organized around various incorporated “tools”: convert, mogrify, composite, montage, identify, display, conjure. The examples are practical, creating logos or adding captions or a border to an image. One example renders the word “Candy” with colorful stripes. Another series of examples duplicates and inverts the image and text of classical Persian poet “Hafez” to create a kind of playing card. Another example uses ImageMagick in conjunction with PHP and HTML to produce an online “e-card maker”, a sequence of commands is demonstrated to render the text “No More War” (in a dripping paint font), deform it, and project it onto the side of a chess piece.
[^mindstorms2]: Seymour Paper, *Mindstorms*, 68.
In another extended example, a British flag is constructed in steps. Rather than approaching the project as drawing geometric forms on a canvas, Salehi uses the diversity of ImageMagick’s manipulations, performing a series of commands whose textual names invoke a sense of physical construction: blocks of color are skewed, sheared, cropped, flipped, flopped, and finally spliced (with "gravity" set to center). The approach creates a number of intermediate images, thus creating the digital equivalents of "cuttings" or leftover materials in the process.
## tearing at the seams
By modifying the first step to use an image[^boris], I produced the following (intermediate) results:
Initially I came to the writing of this essay as part of a proposed dialog with the artist Winnie Soon. Unfortunately because of time constraints it wasn't able to happen. Together with Geoff Cox, Soon has recently published "Aesthetic Programming: A Handbook of Software Studies"[^aestheticprogramming]. The book itself, designed by Open Source Publishing, makes promiscuous use of different tools and techniques, such as the diagramming tool graphviz, HTML, Pelican (a CMS), and paged.js, and showcases an expressive use of typography (including a reinterpretion of a font originally drawn by Swiss typographer Adrian Frutiger).
[^aestheticprogramming]: Winnie Soon and Geoff Cox, *Aesthetic Programming: A Handbook of Software Studies* (Open Humanities Press, 2020)
[^app5]: Winnie Soon and Geoff Cox, 31.
Where Reas and Fry's Handbook is demure, Soon and Cox's is generous, voluptuous even, richly mixing programming exercises with diagrams, flowcharts, and illustrations. The discussions are explicit about the sociality of code and intersections with software art and software studies. The text is densely annotated with theoretical references and links to related critical artistic projects. Directly addressing the "dark sides" of technology such as the extractive and aggregative corporate practices of big data, this handbook explicitly positions coding as an act of activism. Most of the code examples are written using p5.js. As they describe in the first section:
[^boris]: Image: <https://commons.wikimedia.org/wiki/File:Boris_Johnson_official_portrait_(cropped).jpg>, Ben Shread / Cabinet Office, UK Open Government Licence v3.0 (OGL v.3)
> This book will use p5.js, a Javascript library which was created by artist Lauren McCarthy in 2014 for the purpose of what we call "aethetic programming." ... Importantly, the core idea of p5.js is not just to deploy Processing as a web-based platform, but to address diversity and inclusivity explicitly, and take these issues seriously in software development and communication"[^app5]
## constructivism and the bricoleur
In 2016, the Whitney published a “restored” version of Software Structures.[^ssreloaded] As technologies like Java and Flash had then for reasons both technical and commercial fallen out of popular use on the web, the new version featured many of the processing sketches adapted by Reas to use p5.js.
In the 1920s Russian avant-gardist El Lissitsky would move to Berlin and produce work that was highly influential to the then nascent Bauhaus. In 1923, Lissitsky illustrated a publication of poems by friend Vladimir Majakovskij. In it he created graphical forms by mixing typographic elements with geometric forms created by (mis-) using spacing or “blind” elements typically used to create negative/unprinted space between lines of type, as positives producing geometric forms. This style, sometimes called constructivism, was part of an effort to make a radical break from traditional styles of typographic layout and illustration using the means then available for print. The book is notable for its "interactivity" via iconic tabbed pages.
Despite the project's earlier stated interest in exploring diverse "materialities", it’s telling that rather than considering the older processing implementations as a different material and presenting screenshots of them as was done for the Flash and C++ examples, the "restoration" maintains the illusion of a certain "permanence" to the processing sketches, making them thus appear closer to those imagined "software structures" than to "retrograde" technologies like Java or an out-dated browser version. In addition this "adaptation" hides the quite significant work that has gone into (1) the development and subsequent implementation in different browsers of the newly standardized canvas element[^canvas], and (2) McCarthy's work creating the p5.js library to bridge from the legacy processing code to this new standard.[^javascript]
[^javascript]: Despite the seeming similarity of names, Java and Javascript are two completely independent and quite different programming languages. Adapting software from one to the other is thus not trivial.
Constructivism is also the name given to the pedagogic project associated with Seymour Papert. In the 1970s Papert co-developed a pedagogy for teaching children mathematics and programming based on the LOGO programming language. Part of the system was a (virtual) robotic turtle that could be programmed to draw figures. The system, known as Turtle graphics, had commands like: forward, turn left, turn right, pen up, pen down, that directly addressed the “turtle” to draw shapes while moving.
> The process reminds one of tinkering: learning consists of building up a set of materials and tools that one can handle and manipulate. Perhaps most central of all, it is a process of working with what you’ve got. …. This is a science of the concrete, where the relationships between natural objects in all their combinations and recombinations provide a conceptual vocabulary for building scientific theories. Here I am suggesting that in the most fundamental sense, we, as learners, are all *bricoleurs*.[^mindstorms]
Alfred North Whitehead, writing on the sciences, established an influential idea of a "fallacy of misplaced concreteness". The idea is that making abstractions, such as what happens when a particular phenomenon is named, is a simplification that works by suppressing "what appear to be irrelevant details".[^whitehead] In Media Ecologies, Matthew Fuller extends this thinking to consider technical standards as "a material instantiation"[^fuller_p127] of Whitehead's misplaced concreteness, and considers how technical devices through a process of *objectification* "expect in advance the results that they obtain".[^fuller_p104] Fuller cites the example of Laurence Lessig, who makes an argument for regulation of the Internet based on the assertion of standard objects (such as networking protocols) being layered and reconfigurable.
Papert described the pedagogic project of LOGO in Mindstorms[^ref]. In a key example, Papert describes how students can be taught about circles by imagining (or better yet themselves enacting) the turtle repeatedly performing the "go forward a little, turn a little". He contrasts this with the formal equation of a circle (x^2^ + y^2^ = r^2^) typically taught in an elementary school geometry class.
[^whitehead]: Alfred North Whitehead, *Science and the modern world* (New York: Free Press, 1967), retrieved from the Internet archive Oct 28, 2021 <https://archive.org/details/sciencemodernwor00alfr/page/52/mode/2up>
> TO CIRCLE REPEAT [FORWARD 1 RIGHT 1]
> [Lessig's argument] allows us to recognize another characteristic of the standard object: that, while it may be simultaneously embedded within multiple compositions wherein it may be involved in many, separate, disjunctive, contiguous, or contradictory processes, it does provide a threshold, either side of which is differentiated enough for significant political, technical, aesthetic, and social conjunctions or conflicts to occur.[^fuller_p129]
[^fuller_p127]: Matthew Fuller, *Media Ecologies* (Cambridge: MIT Press, 2005), 127.
In a powerful central visual sequence, Mindstorms presents a series of illustrations showing the screen output of code alongside a running dialog. The conversation starts with a proposition to draw a flower like one sketched on paper. They considers what programs they might already have to make use of, in this case a procedure to draw a quarter circle. Through a series of steps, mistakes are made, plans are adjusted and retried, happy accidents lead to discoveries (it's a bird!). In the process the "ends become means" and a new tool is put to use to create a garden, and then, incorporating the "bug", a flock of flying birds.
[^fuller_p129]: Matthew Fuller, 129.
In Belgium, where I currently live "brico" is the French language equivalent to "DIY" and is often used in a derogatory sense to indicate that something is made in an amateurish way. Papert, is borrowing the term from Claude Levi-Strauss, who first used the term in an anthropological context hypothesizing how "universal" knowledge might form from myth and fragmentary cultural knowledge[^levistrauss].
[^levistraus]: Claude Levi-Straus. *The Science of the Concrete*. The Savage Mind (University of Chicago Press: 1966), Chapter 1.
For Papert, bricolage exhibits a quality whereby informal methods not only appeal to "common sense" but also engage more profoundly with the materiality of its subject than would a formal approach. In the case of the circle, the "turtle" method is not only a way for the student to imagine the problem physically, it also relates to methods of differential calculus, something the algebraic formulation misses completely. In hacker circles, bricolage is evident in an approach of embracing "glue code" and "duct tape" methods, like the pipeline, that allow different systems to be "hacked" together to do useful (new) things.
[^mindstorms]: Seymour Papert, *Mindstorms* (Cambridge: MIT Press, 1980), 173.
Alfred North Whitehead, writing on the sciences, established an influential idea of a "fallacy of misplaced concreteness". The idea is that making abstractions, such as what happens when a particular phenomenon is named, is a simplification that works by suppressing "what appear to be irrelevant details".[^whitehead] In *Media Ecologies*, Matthew Fuller extends this thinking to consider technical standards as "a material instantiation"[^fuller_p127] of Whitehead's misplaced concreteness, and considers how technical devices through a process of *objectification* "expect in advance the results that they obtain".[^fuller_p104]
Susan Leigh Star takes Whitehead's "misplaced concretism" and proposes a feminist methodology specific to information technology.[^star] Her essay develops the idea of "standards" as one type of "Boundary object", which she describes as:
[^whitehead]: Alfred North Whitehead, *Science and the modern world* (New York: Free Press, 1967), retrieved from the Internet archive Oct 28, 2021 <https://archive.org/details/sciencemodernwor00alfr/page/52/mode/2up>
[^fuller_p127]: Matthew Fuller, *Media Ecologies* (Cambridge: MIT Press, 2005), 127.
[^fuller_p104]: Matthew Fuller, 104.
> [...] those scientific objects which both inhabit several communities of practice and satisfy the information requirements of each of them. Boundary objects are thus objects which are both plastic enough to adapt to local need and common identity across sites.[^star_p157]
Star cites Donna Haraway, who wonders in *A Manifesto for Cyborgs*:
@ -165,12 +190,18 @@ Star draws on a tradition of diverse feminist thinking through the "articulation
[^star_method]: Susan Leigh Star, 148-149.
For me tools like ImageMagick embody *collectivity* from it's origins as a way to "give back" to a community sharing code over usenet, through to its continued development by multiple authors and relation to the larger free software community as an invaluable toolbox for extremely diverse pratices. I find the *experiential* in the highly flexible commandline interface, itself also an example of honoring *contradiction* and *partialness*, with often more than one way to express the same transformation. The *processual* is implicit in its construction as a tool of transformation, encouraging an exploratory iterative approach to composing transformations to arrive at a desired outcome, often leading to misusage and errors that can be happy accidents and lead one to reconsider one's goals. Finally, in its extreme support of hundreds of different formats, ImageMagick use often leads to the discovery and exploration of diverse image formats, each with related practices and histories.
For me tools like ImageMagick embody *collectivity* from it's origins as a way to "give back" to a community sharing code over usenet, through to its continued development by multiple authors and relation to the larger free software community as an invaluable toolbox for extremely diverse pratices. I find the *experiential* in the highly flexible commandline interface, itself also an example of honoring *contradiction* and *partialness*, with often more than one way to express the same transformation. The *processual* is implicit in its construction as a tool of transformation, encouraging an exploratory iterative approach to composing transformations to arrive at a desired outcome, often leading to missteps and errors that can be happy accidents and lead one to reconsider one's goals. Finally, in its extreme support of hundreds of different formats, ImageMagick use often leads to the discovery and exploration of diverse image formats, each with related practices, and contexts.
In contrast, behind a seemingly "neutral" aesthetic, Processing I think embodies a very particular set of values and assumptions. The “visual minimalism” claimed in the Processing handbook, belies the project's expansive claims on representing a pedagogic approach to a broad intersection of programming and the visual arts. The project's "neutral" aesthetics while dimly echoing a once-radical Bauhaus aesthetic, ignores the larger pedagogic program of the historical Bauhaus' experimentation with the materials of their (contemporary) technical production. The framework valorizes smoothness and fluidity, that leads one to prioritize interactivity as that which happens on the surface of a sketch, rather than say in the network, or among collaborators. As a pedagogic project, Processing has historically seemed uninterested in its own underlying materiality, preferring its students to explore the "world at large" by adding additional layers of technology in the form of sensors, rather than considering all the ways the technologies they use are *already engaged* and impacting the world.
As a pedagogic project, Processing has seemed actively uninterested in its own underlying materiality, aspiring instead to a kind of disembodied universality. Students are encouraged to explore the "world at large" by adding additional layers of technology in the form of sensors, rather than considering all the ways the technologies they use are *already engaged with the world*. The project's "neutral" aesthetics while dimly echoing a once-radical Bauhaus aesthetic, ignores the larger pedagogic program of the historical Bauhaus' engagment and experimentation with the materials of their (contemporary) technical production.
> A concatenation of operations of misplaced concreteness thus allow the gaps, overlaps, and voids in the interrelated capacities of such systems to construct a more "accurate" account of its own operations.[^fuller_p104]
Vernacular rejects the "false neutrality" of the seamless universal design solution, embracing instead the tips and tricks of specific tools, in specific contexts. A vernacular composition tears open its seams proudly displaying the glitches and gaps as a badge of honor of a world view of truth as multi-threaded, incomplete, and sometimes uncomfortable, and is engaged with the world and not afraid to be seen as a monster.
A vernacular approach rejects the illusory construction of an isolated artist sitting at blank canvas creating works from scratch. Instead the digital vernacular thrives on working the contingencies of existing systems, and embraces working with boundary objects that connect to diverse practices and practitioners without erasing their specificity, even when it involves contradictions or incomprehension.
The digital vernacular rejects the sense of disembodiment, rather if locates itself at the intersection between many bodies.
The digital vernacular is about working withing the constraints of available resources, rather than living with a fantasy of neglible time or unlimited storage.
