that’s not a measurement

Possibly the biggest theater-related argument I ever had with Falconworks’ artistic director Pink - who also happened to be my spouse1 - was about how to measure success.

Some version of “How does your organization measure success?” is a component of almost every grant application, whether the grant is coming from a government agency, the Ford Foundation, or a tiny donor-advised fund. It seems a sensible thing to ask. Of course, the public needs to know how its money is being spent. Of course, the Ford Foundation, with all the groups clamoring for its attention, wants to know whether its dollars will be used for good or ill. Of course - well, the family running a donor-advised fund probably asked a consultant who used to work at the Ford Foundation what they should ask, and this is the sort of thing the consultant would tell them they should ask.

(I don’t want this to turn into a rant about the foundation world. That’s a whole other book.)

Like most questions, “How does your organization measure success?” is laden with assumptions, particularly in that word “measure.” To measure implies - nay, demands - a numeric gauge.

Of everything I remember from Le Petit Prince, which for some reason we did not spend much time with in my high-school French classes - we were too busy watching Un Chien Andalou, I guess - I recall one paragraph the most:

Grown-ups like numbers.

If you’re running a hospital maybe it makes sense. How many cases of shingles did you treat last year? If you’re running a soup kitchen: How many meals did you provide, how many referrals to social service agencies did you make, how many people did you find shelter for? In the charitable sectors that you might first think of when you think of the word “charity” - a word that has fallen out of favor, like the word “poor,” and that is another post for another time - there’s some justification for the question. You count the worthy deeds done, and maybe divide that numerator by the dollars spent, and come up with some sort of effectiveness ratio that tells you - give to the Main Street Soup Kitchen, they’re dealing out 2 meals per buck, instead of the Broad Street Brothers, who are only producing 1.87 dinners per dollar.

Even there, though, what are you missing? Over on Broad Street they’re providing food in an atmosphere of support and kindness, while on Main Street they make the poor feel guilty for being poor. That difference doesn’t show up in a quantifiable way. Maybe Main Street spends less because they pay the staff less, which, yes, results in more meals per dollar but less sustainably, with lower quality, with unhappy workers who leave more frequently and don’t know their clients as well and don’t notice when one stops showing up, or stops taking apples because they need dental work. And with clients who stop showing up because of the scowls they encounter.

As if smiling itself is not worth supporting.

As a not-for-profit - a charity - you’re supposed to have a “mission statement,” and a “vision statement,” and a “statement of values.” Meet some not-for-profit careerist who for twenty years has been steeping in the industry, and the phrase “mission, vision and values” will roll off their tongue like the Pledge of Allegiance. They tend not to question it much more than your average fourth-grader questions the Pledge at the start of the day.3

“What is your statement of values?” the foundation will ask.

A necessary tangent: Whenever I think about values I remember a discussion in my Ed Psych class in graduate school. Our professor Fred Lighthall introduced the notion of a hierarchy of values, using as a case study the story of a school Lighthall had visited. The principal of this school tolerated an inebriated teacher in the classroom rather than fighting bureaucratic dragons in order to remove her.

An extreme example, to be sure, but it asked the question: given two, possibly conflicting objectives, which do you value more? Some are easy: the physical safety of children is at the top of the list. Some are trickier if less momentous: given the need to use donors’ money prudently, do you buy regular paper towels or the more expensive recycled ones?4 And some go to the heart of your work: the emotional safety of children matters, too, but all art is, in part at least, about interrogating emotions.

So with that in mind I would sit there and make something up, trying to strike the right balance between lofty and practical (more about emotional investigation, less about paper towels). This is the non-numeric part of the application, the part where you wax eloquent.

I suspect Foundation People read statements of values and say, OK, they’re not radicals (or, in our case, maybe they are). That’s good or bad, so they keep reading, or don’t. And then they get to what they care about, which is not so much what is the sound of his voice, but: How old is he? How much money does he make?5

How do we, the grown-up Foundation People, measure what you artistic children do?

I was on the verge of becoming one of those people. I came by it dishonestly, in the banking world. I had once worked under a department head - he was one of the bigger assholes in a world of assholes, but that’s beside the point - who would harangue us, “You can’t manage what you don’t measure.” This is, I suppose, a truism in the world of people who graduate from business schools. As I’ve noted elsewhere, I spent a lot of time in that world of assholes, so that kind of thinking became my way of thinking. It took me some time to unlearn it.

I don’t think it should be unlearned completely. It’s true, to a degree: There’s value in bringing rigor to what you do. Equally, however, what you measure becomes what you manage.

I didn’t really know what I was doing, running a theater company, so relied on my mostly unrelated experience and imitated those I saw around me, and often tried to do what funders seemed to want. I wrote mission statements and values statements, and I kept counts of things. The numbers of kids served. The survey results we collected after a program was done. The number of donors we had, increasing year over year. The number of butts in seats.

Then Pink and I had that fight.

Some foundation application ignited it. I don’t remember what the foundation was; almost certainly one that never went on to fund us. (Most of them didn’t; we were way too grassroots and a little too hard to understand for foundation people. You had to come and see the way we worked to get us. I complained about this once to a board member from a filmmaking program we had connected with; I said, “They want us to grow,” and he replied “… and what they don’t understand is that that would ruin it.” He got us.)

Anyway, this foundation had asked a version of that question: “How do you measure the success of your work?” I brought it to Pink. She started throwing out responses.

“We see the way the kids develop more self-confidence.”

That’s not a measurement.

“We provide a space for the community to come together and talk about the issues that young people think are important.”

That’s not a measurement.

“We ….”

We went on. I don’t know how many of iterations of this dialogue we had, but it didn’t take too long before we had figured out we were talking different languages. As managing director and the guy in charge of fundraising, I was trying to tell our story to one audience. As a theater artist, Pink had a different audience in mind. As managing director, I had my head in numbers all the time. As an artist, Pink was focused on a set of ideals that had, and have, nothing to do with calculating anything.

Our discussion got heated, probably at about the moment when each of us figured out that we weren’t going to get the other to budge. Eventually Pink did toss out one way of evaluating our work where I responded, “That’s a measurement!” It was clear she thought that measurement was bullshit.

This, then, is a dilemma in the arts. It takes money to do things. It really does: There are delusional people everywhere, maybe you’re one of them, who think artists ought to just pull up their socks and do art in their extra rooms, with a donated sofa and a few coins dropped in a coffee can. Who imagine that art can, or should, be made out of string or nothing. That way of thinking is, as I said, delusional. Even if you can manage an all-volunteer theater company, there are actual, dollar-denominated expenses. Insurance. Website hosting. Gasoline to take kids on field trips.

Worse, that delusion assumes that artists aren’t worthy of being paid. We did depend on volunteer actors and directors for Off the Hook, and there is a value to that, but from time to time an actor would say to me, “I can’t do this for free. This is my career,” and my only response would be, “I understand.”

And this too is a subject for another book.

But, assuming you do accept that money is necessary, you’re left with the question of how to come by it. In order to come by it, to what degree do you begin to measure your work, and how much does the measuring distort your undertaking?

By its nature, art defies measuring. You can’t measure the thing itself. I think about an exhibit I went to once, Everything is Index, Nothing is History. Prior to seeing the show, I had never thought about the word “index” before. I knew it, you know it, by the first definition: a list arranged usually in alphabetical order of some specified datum such as a list of topics or names in a printed work.7

I like the slightly earlier and more general definition: “A sign, token, or indication of something.” (That’s from the OED - Merriam-Webster for some reason doesn’t list this one.) The word “index” comes from the Latin “to point out,” the same as root as indicate. A thing that points us to other things - but not the thing itself.

Even better, I like the definition offered by the exhibition:

[G]estures and physical traces that establish a factual connection to the world independent of cultural codes. Nearly a century and a half ago, philosopher Charles Sanders Peirce defined index as a sign that is caused by that which it refers to. A footprint, a scar, the smoke of a fire—all are signs that simultaneously demonstrate what they signify.

We measure theater, art, by indices. The question is, do those indices tell us anything useful?

“Butts in seats,” for example. We can easily count the number of people who show up at a performance. The result is not irrelevant. Reaching twenty people is better than reaching ten, assuming you’re not staging a Klan rally at Madison Square Garden. All artists - okay, most artists - want a wider audience for their work.8 For a funder like a foundation, the size of your audience may tell them whether you have any skill at marketing or whether you just shout into the void. Maybe it tells them whether the public values your work.

Or maybe it tells them nothing of the kind. Maybe you’re a theater company in an end-of-the-world neighborhood that has been deprived of meaningful9 theater for fifty years. Building an audience, building the habit of going to see live theater, might start with twenty people, or thirty, a number that by itself would never justify the expense of a whole production, with lights and costumes and an artistic director and a program manager and actors who deserve to be paid.

Worse, your funder might assume that popularity is a measure of quality. There’s the rub. It’s easy to entertain. Early on in Off the Hook, the kids would write goofy plays, silly plays, plays where adult actors speaking the lines of a character meant to be 14 years old, or better, speaking lines of an adult character written by a 14-year-old, would provoke laughter at the incongruities. I don’t believe that was a bad form of laughter; it was the sound of the joy of recognition, of truth emerging through those incongruities.

It was easy to pack the house for those performances. But whatever the audience came away with, it wasn’t always the most meaningful experience of artistic discovery for the young playwright.

And, part of our mission was to inculcate a habit among our neighbors of going to see theater, not so much for the sake of theater, but to practice sharing an experience, engaging together with problems and conflicts, talking to each other - in the same room, face to face. To practice listening to young people whose idealism or anger or confusion is too easily dismissed with “you’ll understand when you’re older.” To grapple with difficult questions.

How do you measure the value of that? “Butts in seats” at best would miss our key goals, and at worst lead us down a primrose path.

So you do a survey. “What did you learn from this production?” (Was I supposed to learn something? the survey-taker might wonder.) “On a scale of one to ten, how would you rate your experience?” “On the basis of this play, how likely are you to return to the theater in the future?” As if we are rating customer service agents on their helpfulness.

Or you count the reviews you received, and how many stars you got. Or -

If I recall correctly - and I might not be recalling correctly, it’s been 15 years or so - Pink and I had that argument, about what was or wasn’t a measurement, at a critical juncture in the evolution of the Falconworks Theater Company. We’d been doing the kids’ Off the Hook shows, and the Police-Teen program, and other fun stuff. We were probably at the peak of our popularity. And Pink wasn’t satisfied. She had begun investigating different techniques, like Theater of the Oppressed, that encouraged the young people we worked with to dig deeper. Their plays became more serious - and the audiences smaller. The market tends to prefer fun to introspection - this is apparent if you check out what’s popular on Broadway.

The tension between Pink’s artistic and social objectives and the preferences of the marketplace - as I saw it, anyway - led not only to the argument I described at the outset, and others, but to a general air of confusion and frustration for a year or two. Even at the time I could see that was necessary to destroy the village in order to save it. That didn’t make it any more fun to live in the village while it burned.

It was a strain on the organization. We lost some friends and disappointed others. It was a strain on our marriage, for that.

In the end, you’re in or you’re out. We pulled up our socks - especially, I pulled up mine - and got back to rebuilding, newer and better. And throwing out the smoldering embers of the old, including the idea that we needed to cater to the calculating demands of the philanthropic-industrial complex. If our funders depended on a particular way of evaluating our work, we would find other funders.

Once you commit to an idea you start finding confirmation of it everywhere. It’s even worse if you’re a convert, the way I was. I located a kindred spirit in the television producer David Simon, creator and producer of the television series The Wire, who was interviewed about the show by Bill Moyers:

You show me anything that depicts institutional progress in America: school test scores, crime stats, arrest reports, anything that a politician can run on, anything that somebody can get a promotion on, and as soon as you invent that statistical category, fifty people in that institution will be at work trying to figure out a way to make it look as if progress is actually occurring when actually no progress is. … In the same way that a police commissioner or a deputy commissioner can get promoted, and a major can become a colonel, and an assistant school superintendent can become a school superintendent, if they make it look like the kids are learning and that they’re solving crime.

So if generating measurements doesn’t lead to better performance - or, in the case of theater, performances - what do you do?

Our Off the Hook playwriting program for kids originated as a few lessons in basic playwriting techniques to teenagers in a peer education program called TEACH - “Teens Educating About Community Health.” The stated goals of the program, which was sponsored by the Red Hook Community Justice Center, were to reduce substance abuse and improve reproductive health by training young people to communicate healthy practices to their counterparts. Pink had taught a dozen or fifteen participants how to write didactic little skits about HIV prevention or how to be safe around alcohol. Afterward, a couple of the young people wanted to do more writing, and she ended up sitting with them in their kitchens at dinnertime helping them write plays about wrestling and homeless ghosts in the subway.

We kept our linkage with TEACH for a while, even as Off the Hook grew into its own thing and the program director at the Justice Center changed. I became friends with the new director, Sonia, talking at various community meetings and after our brief workshops about how you engage kids when they live in a world of video games and internet chat rooms; how to handle the cases where parental support is absent or worse; how to, yes, measure results.

I once asked Sonia about her goals for the TEACH program. I was expecting an answer like “10% fewer teen pregnancies” or “20% more condom use” or “a 40% decline in STIs in 11231.”10 Thinking like a grown-up, waiting for numbers to tell me how beautiful her program was.

Instead and without hesitation, Sonia offered a different paradigm. “I want kids to come out of this program with a positive expectation for their future.”

We talked a bit more about that - about how, sure, you can keep a kid from abusing drugs or being careless about sex for a while by scaring them, or by impressing some sense of morality upon them, but in the long term the only way to get someone to care for their health is for them to believe that they have a reason to care.

And of course, not just health. Staying in school and taking it seriously. Seeking other opportunities for growth. Pursuing an artistic passion.

Honestly, that changed everything for me. I’m not sure whether Sonia realizes this. It became my talking point at fundraisers and my closing argument in grant applications. It became my personal, at times unstated, statement of values, my north star, a companion to “a positive experience of success.” For a child, a teenager - for any of us - a positive expectation about the future makes everything else go. Why exercise, why eat kale, why use birth control, why not sit in the park day drinking, why not make a buck hustling weed? Why endure tedious algebra problems and read about Ethelred the Unready and learn French, if not because you believe it will lead to something bigger? Sure, lots of people tell you that it will, but if you can’t carry a picture of your destination in your heart, it’s hard to keep your eyes focused on the road before you.

Could you develop a way to measure this? I’m sure you could - in fact, I’m sure Sonia did. She had a master’s degree in public health, and certainly knew her way around a results survey and how to prove its statistical significance. And she had government funders who were going to expect something of the kind. I never asked her about that, because it was clear from both the way she responded and the way she worked with the young people in her program that a statistical survey wasn’t how she, personally, evaluated her effectiveness. She was talking with those young people and watching them interact, and had designed her program with activities that revealed their progress. Those things are not measurements, but if carried out with intention are far more revealing and valuable than numbers on a graph.

Which leaves us with our own dilemma in our hierarchy of values. We value knowing whether we are achieving our desired outcomes, in order better to steer our decisions in the future. We can’t understand our impact - good or ill - without some means of understanding our work, and in fact of comparison. Are you better off than you were when you started this program? Did you learn something from attending this show, or at least were you entertained? Did changing the workshop plans, or the number of weeks, or where we went for the writing retreat, or even the budget for costumes, make the outcomes better or worse?

Most, or maybe all, of those questions imply measurement.

On the other side of the ledger, the arts - particularly, let’s call it, political art, by which I simply mean art that engages the polis, the community - defy measuring whether you have achieved your outcomes. Sometimes the arts defy knowing what your intended outcomes are. One of the challenges I am encountering as a writer, even in writing these posts about making theater, is accepting writing as a process of discovery. I don’t know what my intent is when I start out. How can I know whether I achieved it?

This problem is not just common in making art. It is inherent in the making of art.

Here is how I am squaring this particular circle11, today. I’m not sure I’ll believe this tomorrow, but let’s try.

You, as an artist, or a group of artists, establish your values. Maybe you spend a great deal of time developing them, maybe you just feel them, but at some point, you have a sense of what your values are.

You realize and express your values in your work, and through your work. This seems obvious, but it’s actually the challenge. As Professor Lighthall told us in Ed Psych, “the sacred of an institution … comes to light when it becomes profaned....” It’s easy, it’s almost inevitable, to find yourself doing things that run contrary to what you believe. Because you fell into a pattern without realizing it. Because some part of your company has their own, different values that they are realizing in the work. Because someone - sometimes an imaginary someone, or sometimes real - will object to the the themes you convey, or the manner of presenting them, or the language you permit kids to use on the stage. Because you’ve built up an audience and you don’t want to lose them and have to go through the pain and work of building a new audience. Because it’s cheaper. Because it’s more convenient.

You look up, and you’re tolerating a drunk teaching a class of third-graders.

Especially in an organization dependent on others for funding, the temptation is always ready at hand to adjust your values to suit a measurement - especially, for a theater, butts in seats. Maybe we were starting to do that, or I was at least. Maybe that’s part of why we had to burn the village down in order to save it.

In the end, it was a different grant application, a much better one, that helped us find a solution. The grant process included workshops whose key task was creating an “organizational profile” - a history, a description of our leadership, and, yes, a statement of our values. I was annoyed - “pissed off” may be what I said at the time - at having to jump through these hoops. Afterward I wished we’d completed that process ten years earlier.12

I’m still proud of what we developed in applying for that grant. Here are what were called our “working values”":

• Every person has equal value

• Every person deserves a voice

• Some people have had more opportunity to express their voice than others, and we will seek to be aware of that and prioritize those that have not had as much opportunity.

• We will create work that is truthful.

• Some truths have had more opportunity be told than others, and we will seek to prioritize truths that have not been told as often (even when we are telling a story that has been told many times).

• We will do the best work our budget will permit (and we will strive to do better work than our budget permits).

  

  At the same time, all we need is six feet of lumber.

• There are only so many roles in a play, and we do aspire to do the best work we can. But that said, we seek to find a way for anyone interested to participate, and we will not turn anyone away because of ability. (This is especially true for our programs for youth.)

What I like about that statement of values, what makes me still feel proud, is we phrased them as challenges to ourselves. Not, “we believe,” but “we will.” And as such, these gave us standards to measure ourselves against. Are we doing the things we said we would?

The workshops required for the grant application included as well a prompt asking us to articulate our “Measures of Success.” And here is what we said: If people are participating at all levels, it is successful. Everyone in this work is a participant - audience/actor/playwright/etc.

And we developed questions to help investigate whether people were, in fact, participating at all levels. In part, they said:

• Are participants articulating issues that they are recognizing around them?
• Are people working through difference to find common cause? Are people who would generally not be in a room together, talking about the same issue?
• Are people expressing deep emotions (laughter, tears, voicing outrage, applause)?
• Do people who did not have a public voice, develop one?
• How quickly do audience members want to leave their seats?
• Are we being asked by others to support their work addressing community issues?

Some of these look awfully close to Pink’s original suggestions: “We see the way the kids develop more self-confidence. … We provide a space for the community to come together and talk about the issues that young people think are important.” If there’s a difference - if our process of thesis and antithesis and burning down the village finally resolved into a new synthesis - it was that we now had a framework for turning observations into something like a measure - at least, we could count down the checklist and say “how many of these did we hit?” And then ask, for the ones we didn’t hit, were there good reasons? And if not, what do we do about it?

And, if there’s another difference, it’s that we - I - no longer were trying to measure the thing itself. We developed, if you will, an index - a set of indices, really, that emerged out of our values - to enable us to perceive something that in the arts is fundamentally impossible to observe, let alone express in numbers: What is this house worth?

This one is too long, and it kind of went all over the place. I appreciate your patience. It might be all over the place because we had limited opportunities to put all of this into practice - people moved, a pandemic happened, et cetera. My conclusions remain a bit theoretical. Your thoughts and feedback would be much appreciated.