vernacular/vernacular.md

---
title: "systems not surfaces: digital vernacular (or why I hate processing)"
papersize: a4
---

> Processing is a free, open source programming language and environment used by students, artists, designers, architects, researchers and hobbyists for learning, prototyping, and production. Processing is developed by artists and designers as an alternative to proprietary software tools in the same domain. The project integrates a programming language, development environment, and teaching methodology into a unified structure for learning and exploration. […]
>
> Most of the examples presented in this book have a minimal visual style. This represents not a limitation of the Processing software, but rather a conscious decision by the authors to make the code for each example as brief and clear as possible. We hope the stark quality of the examples gives additional incentive to the reader to extend the programs to her or his own visual language.[^p1]

[^p1]: (pp. xxi-xxii, Processing: A Programming Handbook for Visual Designers and Artists, Casey Reas and Ben Fry, MIT Press 2007)

Teaching programming with free software to media design students for years, I've avoided Processing as it has always seemed to me to embody a particular kind of solipsism of digital interactivity and graphics that I want my students to avoid.

> Solipsism: a theory holding that the self can know nothing but its own modifications and that the self is the only existent thing; also: extreme egocentrism[^solip]

[^solip]: <https://www.merriam-webster.com/dictionary/solipsism>

In the summer of 1996, as I was finishing my Masters degree, I met John Maeda who was just joining the MIT Media Lab to replace the recently deceased Muriel Cooper. Cooper was the first art director of the MIT Press, producing influential designs such as a 1969 catalog of the Bauhaus and the iconic MIT Press logo, a Bauhaus-inspired stylized graphical rendering of the letters “mitp”. Cooper started the Visible Language Workshop, later one of the founding groups of the MIT Media Lab, to research the intersection of publishing, design, and computation.

My friend Robin was a student in the VLW; through her I learned things like how Muriel would say "Swiss" instead of Helvetica. I took one class in typography at the VLW; among the many constraints we were given, we could only use one font, named for swiss typographer Adrian Frutiger.

Maeda created the Physical Language Workshop, later renamed the Aesthetics and Computation group, to continue Cooper’s research. Ben Fry and Casey Reas were among Maeda's first students. Maeda published a software system called Design by Numbers. It had extreme constraints such as a fixed 100 by 100 pixel size and monochrome-only graphics. The accompanying print publication also had a square format. DBN would go on to inspire Reas + Fry to create Processing. When one starts the Processing application, the default starting code comprises two commands: createCanvas and background. The default canvas (image) size is square, and a single numeric parameter to the background command (220) produces a gray-scale value, both reflections of the earlier DBN constraints.

![The default code of the current online p5 editor](images/p5editor.png)

In 2004, Reas co-developed an exhibition at the Whitney gallery called Software Structures.[^ss] Inspired by Sol Lewitt's wall drawings, Software Structures presented a series of abstract rules for the production of an image, including some from Lewitt. The rules were then implemented using a variety of "materials": Processing, Flash MX, and C++.

> A benefit of working with software structures instead of programming languages is that it places the work outside the current technological framework, which is continually becoming obsolete. Because a software structure is independent from a specific technology, it is possible to continually create manifestations of any software structure with current technology to avoid retrograde associations.[^sst]

[^ss]: <https://artport.whitney.org/commissions/softwarestructures/text.html#structure>

[^sst]: Casey Reas, In Paragraphs on Software Art, <https://artport.whitney.org/commissions/softwarestructures/text.html#structure>


In 2007 I attended a book launch of: Processing: A Programming Handbook for Visual Designers and Artists (MIT Press, Reas + Fry, with a foreword by Maeda).[^mediamatic]

[^mediamatic]: <https://www.mediamatic.net/en/page/1696/processing-with-casey-reas>

![](images/pbook.png)

The book comprises over 700 pages and is organized by topics: Color, Control, Data, Development, Drawing, Image, Input, Math, Motion, Output, Structure, Typography, Extensions. It also contains interviews with artists working with digital tools not limited to Processing. As indicated in book's introduction, most of the many examples in the book are illustrated by "stark" small squares and concise code.

![](images/pbook_inside.png)

Processing commands typically address the canvas using Cartesian x,y coordinates. Shape commands, like circle, rectangle, and triangle allow for the drawing of graphical forms, while other functions control parameters like the color of fill or the width of the stroke (or outline) of each shape. 

![](images/p5editor_blurryrect.png)

Input functions allow access to mouse and keyboard to produce dynamic graphics that respond to the user. Processing sketches consists of (at least) two functions: setup which is invoked once and *draw* which is invoked continuously; the default frequency being the refresh rate of the computer’s display (typically 60 times per second). By using variables, and changing their values, graphics can be made dynamic. In addition, graphics are rendered using a technique known as “anti-aliasing” to appear less blocky due to their representation as grids of “pixels”. This default behavior can be modified by using the noSmooth command.


Ealier in the day I had bought another technical book "ImageMagick Tricks: Web Image Effects from the Command line and PHP" by Sohail Salehi (Pakt, 2006). While waiting for the session to begin, I crossed paths with Casey Reas at the back of the room. He was curious about the book I had with me and looked briefly at it. He had never heard of ImageMagick.

![](images/imbook.png)

ImageMagick is a popular free software tool that’s often referred to as a “swiss army knife” due to its ability to convert between hundreds of different image formats, and for the many built-in features to filter, manipulate and generate images. The software is full of particularities. For instance, there are a number of built in images including  a rose and a wizard seated at a drawing table contemplating an image of the Mona Lisa.

ImageMagick is a command-line tool, designed to be used via textual commands. The typical usage of ImageMagick is to take one image as input, applying one or more transformations to it, and output a new image. In this way the tool can be used repeatedly in a so-called “pipelines”, or otherwise composed together in structures called (BASH) scripts. In these scripts ImageMagick commands can be mixed with other commands from any software installed on the user's computer (that also provides a command-line interface).

Salehi’s book directly reflects the structure of ImageMagick, with chapter organized around various incorporated “tools”: convert, mogrify, composite, montage, identify, display, conjure. The examples are practical, creating logos or adding captions or a border to an image. For instance one example renders the word “Candy” with colorful stripes. Another series of examples duplicates and inverts the image and text of classical Persian poet “Hafez” to create a kind of playing card. Another example uses ImageMagick in conjunction with PHP and HTML to produce an online “e-card maker”, a sequence of commands is demonstrated to render the text “No More War” (in a dripping paint font), deform it, and project it onto the side of a chess piece.

![](images/imbook_inside.png)

In another extended example, a British flag is constructed in steps. Rather than approaching the project as drawing geometric forms on a canvas, Salehi uses the diversity of ImageMagick’s manipulations, performing a series of commands whose textual names invoke a sense of physical construction: blocks of color are skewed, sheared, cropped, flipped, flopped, and finally spliced (two times). The approach creates a number of intermediate images, thus creating the digital equivalents of "cuttings" or leftover materials in the process. Below, the (intermediate) results from the same example slightly modified to take an image as input:

![background2: intermediate file](images/background2.png) ![another intermediate file (mixed)](images/mixed.png) ![The final "flag"](images/flag.png) 

```bash
convert -size 300x200 xc:'#002377' -fill white -draw 'skewX 128 image over 0,0 0,0 in/boris.jpg' background1.png
convert -size 300x40 xc:none -background transparent -fill '#ce201a' -draw ' rectangle 0,0, 300,20' -shear 32 band.png
convert background1.png -draw ' image over -25,0 0,0 band.png' background1.png
convert background1.png -draw ' image over -25,0 0,0 band.png' background1.png
convert background1.png -crop 150x94+0+0 area1.png
convert area1.png -flip -flop area3.png
convert background1.png -draw ' image over -25,-21 0,0 band.png' background2.png
convert background2.png -flop background2.png
convert background2.png -crop 150x94+150+0 area2.png
convert area2.png -flip -flop area4.png
convert -size 300x188 xc:none -draw ' image over 0,0 0,0 area1.png' -draw ' image over 150,0 0,0 area2.png' -draw ' image over 0,94 0,0 area4.png' -draw ' image over 150,94 0,0 area3.png' mixed.png
convert mixed.png -background white -gravity center -splice 20x20 -background '#ce201a' -splice 40x40 flag.png
```

----------------

In the 1920s Russian avant-gardist El Lissitsky would move to Berlin and produce work that was highly influential to the then nascent Bauhaus. In 1923, Lissitsky illustrated a publication of poems by friend Vladimir Majakovskij. In it he created graphical forms by mixing typographic elements with geometric forms created by (mis-) using spacing elements (the “blind” elements typically used to create (negative/unprinted) space between lines of type, as positives producing geometric forms. This style, sometimes called constructivism, was part of an effort to make a radical break from traditional styles of typographic layout and illustration using the means then available for print.

![](images/forthevoice.png)

In the 1970s Seymour Papert co-developed a pedagogy for teaching children mathematics and programming based on the LOGO programming language. Part of the system was a (virtual) robotic turtle that could be programmed to draw figures. The system, known as Turtle graphics, had  commands like: forward, turn left, turn right, pen up, pen down, that directly addressed the “turtle” to draw shapes.

> The process reminds one of tinkering: learning consists of building up a set of materials and tools that one can handle and manipulate. Perhaps most central of all, it is a process of working with what you’ve got. …. This is a science of the concrete, where the relationships between natural objects in all their combinations and recombinations provide a conceptual vocabulary for building scientific theories. Here I am suggesting that in the most fundamental sense, we, as learners, are all bricoleurs.[^mindstorms]

![](images/turtle_circle.png)[^turtle]

[^turtle]: Made with <http://logointerpreter.com/turtle-editor.php>

Papert's approach to pedagogy is (also) called constructivism. In a key example, Papert describes how students can be brought to understand the construction of a circle by imagining (or better yet themselves enacting) the turtle repeatedly performing the instruction sequence "go forward a little, turn a little". He contrasts this with the formal equation of a circle (x^2^ + y^2^ = r^2^) typically taught in an elementary school geometry class.

In Belgium, where I currently live "bricolage" is the French language equivalent to "DIY" and is often used in a derogatory sense to indicate that something is made in an amateurish way. For Papert (who himself is borrowing the term from Claude Levi-Strauss), a central point is that these informal/intuitive methods are in fact superior to more formal approaches by in fact engaging more profoundly with a subject. In the case of the circle, the "turtle" method is not only a way for the student to imagine the problem physically, the method also relates to methods of differential calculus, something the algebraic formulation misses completely.

[^mindstorms]: Mindstorms, p.173

[^mindstorms2]: Mindstorms. p. 68

> For most there is a total dissociation between these live activities and the dead school math. We have stressed the fact that using the Turtle as metaphorical carrier for the idea of angle connects it firmly to the body geometry. We have called this body syntonicity. Here we see a cultural syntonicity: The Turtle connects the idea of angle to navigation, activity firmly and positively rooted in the extraschool culture of many children. And as computers continue to spread into the world, the cultural syntonicity of Turtle geometry will become more and more powerful.[^mindstorms2]

-----

![NetMonster: Mohammedb, Graham Harwood](images/VanGogh.jpg)

![NetMonster: Mohammedb (detail), Graham Harwood](images/mohammed_dtl.jpg)

> Netmonster is an example of politically-charged cultural software
on the digital commons, built and run using only free software programs, and brought to
fruition by thousands of hours of freely given immaterial labour. Netmonster exemplifies
distributed creativity, in which networks themselves - human and technological - become
the medium. [^darimini]

[^darimini]: Grazing the Digital Commons: artist-made softwares, politicised technologies and the creation of new generative realms, 2005, Francesca da Rimini, p.59

As Da Rimini notes, '[i]t is a set of processes and structures, rather than a singular tool'


----------

In 2016, the Whitney published a “restored” version of Software Structures.[^ssreloaded] As technologies like Java and Flash had for reasons both technical and commercial fallen out of popular use on the web, the new version featured many of the processing sketches adapted by Reas to use p5.js, a new implementation of the Processing using javascript and the new HTML standard canvas tag.

Despite the projects stated interest in exploring diverse “materialities”, it’s telling that rather than considering the older implementation of processing as a different material, the “restoration” extends an illusion of permanence, hiding the large amount of diverse work implicated.

[^ssreloaded]: <https://whitney.org/exhibitions/software-structures>

In this case, beyond Reas’s work “adapting” the examples is the hiden work of the development of the canvas standard, the subsequent implementation of this standard in all the different popular browsers (Chrome, Safari, Edge, Firefox), and then the work of building the new p5js library to produce similar results to one's adapted processing code.

The “visual minimalism” claimed in the Processing handbook, belies the project's expansive claims on representing a  pedagogic approach to a broad intersection of programming and visual arts. (Repetition of "international style" as "good design") The projects "neutral" aesthetics while dimly echoing a once-radical Bauhaus aesthetic (100 years later more a conservative application of a so-called "International style"), ignores the larger pedagogic program of historical Bauhaus' experimentation with the materials of technical production contemporary with its practice. The project privileges formalisms over bricolage, that like Cartesain equations of circles presenting visual "solutions" that deny intuitive and profound contact with underlying concepts. Processing, as a pedagogic project, has historically seemed uninterested in its own underlying materiality, encouraging students to explore the "world at large" by adding additional layers of technology in the form of sensors, rather than considering all the ways the technologies they use are *already engaged* and impacting the world.

Warren Sack has recently written on what he calls the "Software Arts", tracing and re-situating the practice of programming to history of the arts and arts education, and how and why it has been re-imagined as an engineering discipline. A recurring theme in Sack's text is the notion of "digital ideology", which Sack defines as "NEED QUOTATION HERE!!!!!!!!!!". In some works of the Digital Humanities today, we can see a kind of extension of the Sack's digital ideology to the practices of the visual arts. First a digital image depicting a work (say a photograph of a Van Gogh painting) is seen as equivalent or as somehow "capturing the essence" of the original painting (or similarly, that a text file is seen as equivalent to a manuscript or book). From there, the digital image is imagined as a sort of "universal canvas" onto which software simulations can project works that then can be seen as generative equivalents to the original historical works.[^artificial_vangogh] This state of affairs has led Mongrel member Matthew Fuller to despair at the way different visual practices seem routinely reduced to "undifferentiated practices of mark making".[^markmaking]

In contrast, frameworks and tools like ImageMagick, promiscuously engaged with other standards, formats, tools and practices, offer the possibilities to support strong pedagogical situations where students are encouraged to see the "edges" and "seams" between systems, to make mistakes (and learn from them), and in general of way of using code that leads to engagement with the world. (?!) Return to the image of solipsism?

"Social software" projects like those of Mongrel / Harwood; Embracing partial, incomplete; Intersectional/Feminist approaches offer helpful tools...

> My best hope for this cyborg essay is that it will produce in you a gestalt switch where multiple memberships and multiple meanings coe to the foreground; and it will become purity and universality that routinely bear the burden of proof. (Misplaced Concretism and Concrete Situations) [^star]

> Upon further reflection, what appears to be an absence in terms of being “cultureless” works more like a superpower. … As a class, then, we begin to understand that all those things dubbed “just ordinary” are also cultural, as they embody values, beliefs, and narratives... [^benjamin]

[^artificial_vangogh]: See for instance: A Neural Algorithm of Artistic Style, Gatys, Ecker, and Bethge, <https://arxiv.org/pdf/1508.06576v1.pdf>

[^markmaking]: Remark from a personal conversation; any misremembrance is my own error/creation.

[^star]: Susan Leigh Star, Mistplaced Concretism and Concrete Situations: Feminism, Method, and Information Technology, p. 144, In Boundary Objects and Beyond. 2015. MIT Press.

[^benjamin]: Ruha Benjamin, Race After Technology. p. 4