PROJECT STEAKSAUCE: SCREWING WITH OBJECT RECOGNITION

[This is not a Briefcase; this is Christmas (0.92), Cube (0.91),  Box (0.97), Birthday (0.88), or Gift (0.83)]   

This is not a white paper released by engineers working with ground-breaking technologies.  This is a research project undertaken from another direction: graphic art.  The San Perdido Trading Co. (namely, the ridiculous art projects I do with friends insane enough to go along with them), in conjunction with the Timbostan Heavy Industries "Strange Technologies Division" (the workspace in my diningroom) have produced the first results of "Project Steaksauce": endeavors to understand how to combat "object recognition" being developed via Machine Learning and A.I. ("Steaksauce" was a codename suggested by a friend; it sounded good, so we're going with it*).  This is a project to explore what has come to be termed as "Adversarial Technologies" - how to screw with Object Recognition.   

Last week, Chinese Police in Zhengzhou proudly announced that they are using facial recognition glasses to catch criminals and monitor travelers (combined with their new "Minority Report" level Precrime endeavors, this will generate a zero-privacy/population control nightmare worthy of George Orwell).  The intent of Project Steaksauce is to foster the coevolution of Adversarial Technologies alongside that of Object Recognition.  These technologies need to be developed now, rather than farther down the road, when someday a Skynet or Big Brother already has the rest of us at a huge disadvantage.   

Last December, a team of engineers at Google published their results of the development of an "Adversarial Patch" - an image that looks like a toaster (if a toaster were painted by Salvador Dali) which, when placed next to other objects, camouflages the second object from Object Recognition: the Object Recognition sees nothing but "Toaster."  This might be because when "training" Machine Learning (showing it thousands of images of shiny metal toasters), other objects appear as reflections on the toaster surface and the Machine Learning has taught itself that "other objects" are still actually "Toaster." And simultaneously, MIT's "LabSix" (a student-run A.I. research group) published their results from 3D printing experiments designed to defeat Object Recognition.  And their results were equally spectacular: they printed a 3D "Sea Turtle" that is recognized as "Rifle" by Object Recognition, got an image of a cat to register as "Guacamole," and a 3D printed "Baseball" that reads as "Espresso."  

The starting point for method and design for this first "Project Steaksauce" endeavor begins with a dilemma from World War I: the ocean is flat, ships are not.  Watercraft cannot hide floating on the surface of the ocean, thus the camouflage strategy had to take an uncomfortable turn: "You're going to be seen, so confuse the hell out of those seeing you." Maritime artist Norman Wilkinson found a good solution and developed "Dazzle Camouflage" - creating patterns which disrupt the viewer's ability to recognize what they are seeing (and in the case of ships, even which direction they are moving).

With regard to Object Recognition, the strategy should be the same: "It is going to be seen, so make it confusing as hell."  For Steaksauce Project 1, we went with the idea that both color and shape are being used by Machine Learning to develop an idea about an object. Thus, for good Adversarial Camouflage: 1).Include a large and varied number of colors, 2).provide a pattern not common to the object concealed, 3).use a pattern which will disrupt the ability to define the profile/outline of the object against varied backgrounds, 4).provide a "false topography" - use fake highlights, shading, and line to create the appearance of a shape different than the object concealed.  For Steaksauce Project 1, we went with a colorful "noise" pattern, directed into the shape of 3 surfaces of a cube.  And added an image of a bottle of steaksauce* for both awesomeness and the hopes that Object Recognition might pick up on it as a stronger/clearer signal than "Briefcase" (but it didn't). 

The uncamouflaged back had to, of course, contain the warning "WARNING: thIs side iS NoT ProTEcTed fRom ObJect recognition."  Initial testing: online Object Recognition sites which tout immense libraries of object categories with free demos were enlisted, unknowingly. And the results were fun: none read it as a "Briefcase." And interpretations like "Cube"(0.91) and "Block/Square"(0.92) were at the top. And by color pattern, "Christmas"(0.92) and "Birthday" (0.88) were also in the top results.

(Not a real Object Recognition image, for illustrative purposes only)

Object Recognition technology can someday be used to identify tumors in X-rays, find lost cities in satellite photos, and authenticate long-lost artworks. But they will also be used to violate the privacy of individuals. And not merely for the sake of "fighting crime," but for product marketing, gossip and blackmail, and population monitoring.  And we must assume (and prepare for) that this new technology may not always be in the hands of people who intend to do good things.  

*The name for these projects, "Steaksauce," is effectively random and does not convey, in any way, an association between our screwing around with Object Recognition and any brand of delicious steak sauce, and does not convey that any brand of delicious steak sauce has any association, knowledge, or support of these art projects.