CT No.53: How to measure content performance without KPIs
Considering our current outcomes, maybe we should reevaluate our measurement strategies
It’s good to be back in the Substack interface. I had a productive and restful break from newslettering and happily resisted the urge to write about the Big Tech hearings, diving instead into a couple of good research reads.
I highly recommend Race After Technology by Ruha Benjamin. It’s academic but accessible and clearly argues that big tech is developed with, at best, minimal consideration of race and, at worst, the intention to maintain white supremacist values.
Over the past few weeks, I also spoke at ContentTech Summit, the tech geek conference from Content Marketing World. It was a lovely time, and although I didn’t have a ton of capacity for extensive participation, I remembered that I love what I do. I’m still processing all the great talks I saw (and need to watch even more!), but I’ll be covering some of the concepts I saw in a couple weeks.
I was also interviewed by fellow marketing/content Substacker Mae Rice on Built In. We chatted data-driven content strategy and you can read it here.
In this week’s newsletter:
Moving beyond the KPI for content performance measurement
A review of link-saving and reading app extraordinaire, Pocket
A super buff post-vacation link roundup
New here? You can
Telling better data stories: Content deserves better than KPIs
Were this a well-researched magazine piece and not a newsletter, I would begin with the history of the KPI. But this is not that article, I’m not an economist, and we’re just going to start with where KPIs are, as I understand them, in 2020.
If you’ve never had to work toward or establish a KPI, here’s an overview: Short for key performance indicator, a KPI is the core measurement of business success. A KPI can be a calculation of multiple measures, but most often a KPI is typically a single metric, a good grade, a number to aim for when you can’t directly aim for straight-up revenue.
Why does everyone use KPIs?
KPIs are phenomenal for getting everyone on the same page. For example, together we can all agree that, say, the fewest number of deaths is a great KPI for measuring COVID response. When we agree on that KPI, then when an executive says, “But look at our case death rate,” which means case fatality ratio, we can say, “Yes, our case death rate is... ok? But it’s not our KPI, which is numbers of deaths.” Or something like that.
How many of you watched this interview and had flashbacks explaining basic data points and paper charts to high-powered executives? And let me tell you, executives absolutely love it when young women tell them, “I think you’re reading the data wrong.”
KPIs are also fantastic when programming a computer… or optimizing an advertising algorithm. A computer can only optimize toward one outcome at once. The KPI is the ur-metric, the definition of success, and the algorithm will do everything it can to meet that goal.
When I help clients develop measurement strategies for content marketing programs and campaigns, I say, “We need a true north, though. We need a KPI. Do you want to measure engagement or visibility? Soft conversions or number of sales? The computer can only optimize toward one.”
Robocop is a movie about prime directives and changing KPIs. I intentionally didn’t use the phrase “prime directive” because my partner and I also got into a long discussion about the phrase “prime directive” as it’s meant in Star Trek vs. Robocop. Welcome to quarantine, where nerding out is easy.
Optimization means choosing a KPI because the computer can only optimize toward one thing. And KPIs help focus our efforts on tactics that support that KPI. If the KPI is reader engagement, I will use a different content structure and optimization tactics than if my KPI is brand mentions.
The KPI that will live in infamy
The most famous digital KPI in all of publishing was Gawker’s scoreboard, where writers were ranked by the number of unique pageviews, compensated based on pageviews, rewarded for how much trouble they could stir up.
Following Gawker was the age of Chartbeat and “real-time analytics.” As a digital writer, I never worked in a Chartbeat newsroom, but I was fascinated with how one could measure exactly what worked to drive more pageviews or eyeballs or whatever. And I was frustrated with writers who dismissed analytics platforms like Chartbeat outright because I saw the value: with platforms like Chartbeat we could understand what our users wanted, or so we thought.
[Read my proposed alternative to pageviews here.]
For media businesses operating on a pageviews or traffic-based model, content writing and publishing rewarded exactly the same type of stories that newspapers and cable news produce: short, neatly packaged content with maybe an outrageous or ranty element, but also a beginning and end. It didn’t reward the long-term engagement and reader loyalty that drives subscriptions.
So we’ve switched our KPIs. Now media companies reward for how many subscriptions they can bring in. We’ve gone from “vanity metrics” to “business metrics.” But they’re still single KPIs.
Shifting from ur-metrics to data stories
If you’re in content marketing, you may also have some sort of performance-based rewards or flat-out compensation based on a single KPI. Did you grow engagement by 5%? Did you increase paid subscriptions with your fierce opinions?
If you’re compensating or rewarding, or if you’re reporting up to leadership on performance based on a single KPI — not an uncommon practice — please consider broadening your view to more complex content performance dashboards.
When you reward people for optimizing toward a KPI, you’re rewarding only certain optimization behaviors and not necessarily incenting long-term content impact or even recognizing the full benefits that your content marketing is providing. By rewarding based on one, or three, metrics, you’re holding people accountable to computer-based (or finance-based?) standards, when it’s increasingly clear that content businesses cannot survive that way.
Those single KPIs don’t account for all those fun economics terms like outliers and externalities, which are having very real effects on our society right now. Most KPIs reward optimization and efficiency but not, for example, inclusiveness in sourcing, depth of research and citation, story structure, innovation and originality, or any non-emotion-driven risks. Brand standards can control for style and substance, but it’s difficult to translate those into reports, or even test changes, if they’re not recognized in the KPI.
It is crucial to measure content performance. And it’s always fantastic to receive shares, engagement, subscriptions, leads. But when deciding editorial direction or identifying success, complete data stories built with long- and short-term metrics and illustrated with dashboards.
What an unsatisfying dashboard!
Why content dashboards?
Good content performance dashboards answer core business and audience questions with multiple dimensions and different types of metrics. These dashboards should always be subject to regular review, discussion and interpretation, whether it’s monthly or quarterly. (Daily is unsustainable with content reporting, especially in the age of inconsistent content lifecycles and algorithmic recommendations.)
As creators, we need to fully understand and define the metrics that tell our story and provide feedback how those metrics are changing. If you’re on the management side, start facilitating conversations with performance-obsessed leadership and those who mistrust digital measurement. Find some kind of common ground and better system with content-specific dashboards. Begin the work of telling and reading clear stories with those dashboards, interpret them with our vast and deep storytelling abilities, and provide insights not only on optimization but on new directions.
We are already paying attention to so many details and ideas in our content development processes. Our dashboards need to better reflect the process and complexity of content creation.
As an industry — both in marketing and editorial content — we need to be better at giving bad news when KPIs don’t behave the way we expect. In business, we’re always told to chase the positive, highlight the action. We celebrate spikes in performance but provide excuses for drops, when we need to be setting an expectation of consistency. We need to look at other factors that may be affecting our readers — from the number of banners or pop-ups on a page to major global changes. And we need to set up reporting systems that allow us to tell broader and longer term stories with our data.
The only positive that I can see about the Coronavirus epidemic: We’re all better at reading charts and telling complex data stories now. We know the stories need to be more nuanced and use multiple metrics. We know not only to take into account the R0, but the rate at which we are testing, the absolute number of cases of COVID, and the absolute number of deaths. To understand what’s happening in our daily world, we need to embrace all the chart-reading skills we learned in school that maybe we put behind us with our English major.
[I also wrote How are we being optimized? at the beginning of the pandemic.]
Content performance dashboard dimensions: Here’s a start
So how do you build a content performance dashboard? As with all things digital, every case is different. Developing a content performance dashboard is a collaborative process that involves editorial input and intention like any other piece of content.
Here are a few dimensions to consider when starting out with your dashboard:
Visibility - Have people seen the content?
Engagement - Do people interact with or share the content?
Sales/conversions/subscription - The money numbers
Sourcing - What is the quality of sourcing and research for the content?
Entity/pillar support - How does the content align to a full body of research?
Churn - Did you lose any support because of a content campaign? (I have never seen a performance marketing dashboard that looks at churn and that’s alarming!)
Audience loyalty - How often do users return for your content?
Volatility - How volatile are the channels where we are posting? The topics we are posting about? Are the algorithms changing frequently? Are the people changing?
For more specifics on how to build a better dashboard, watch this space. Or, if you’re in need of a dashboard and content performance measurement strategy, like, tomorrow, reply to this email.
Measurement is not going away (and that’s a good thing!), but it’s changing drastically.
Did you like this essay?
Save the content: Pocket review
Truth: I have spent the past three weeks searching for some really great professional content tools but haven’t found any that 1.) I can review fairly quickly and 2.) I like. I’m working on reviews for those most in-depth tools, but while those are in development, this week it’s another consumer product.
If you have recommendations for software to review, I beg you, send them my way! Or, if you want to find software that does a content thing and haven’t located it yet, I’d love to hear about it.
On my break I resolved to figure out a better system for saving links among the crush of content. I’ve always taken some pride in my ability to filter easily, to process web information quickly, to find what I’ve been looking for. But in developing the “beat” for this newsletter, I found myself drowned and scanning more often than not.
All this was compounded in June by the immense wealth of anti-racism resources across all the channels, in all the formats, all the time. I wanted to save everything but wound up frustrated much of the time, digging deeply for that thing I’d seen one time on Twitter or Instagram.
I don’t want to be drowned and scanning. I don’t want to be searching. I want to be reading and reacting and thinking and citing. I needed a new system.
Mozilla’s Pocket is one of the most popular save-links-for-later tools for a reason: The app integrates well with pretty much everything, and the reading experience is pleasant. Whether you want to save a tweet or a longread, Pocket’s got you covered.
Pocket at a glance
Pocket requires a few installs to work correctly: an app here, a browser extension or two, some setup on my iPhone so that it’s actually possible to bypass the “save to reading list” Safari default. I trust Mozilla enough not to sell my data, so I’m not uncomfortable building all the connections under a single log-in.
Like any other good software, once you get the hang of saving to Pocket, it works extremely well. It strips all the ad units and pop-ups out of everything so you can read content on a phone without half the screen consumed by ads for SEMRush. (I only ever see ads for SEMRush and I’m already a customer; please stop advertising to me, SEMRush.)
Some great features of Pocket:
Highlighting! You can highlight within the article, just like on a Kindle or on Medium or whatnot.
Audio transcription. Have a robotic voice read you an article that you’d rather listen to, if that’s your preferred method of content intake.
Tagging and organization, a feature I should absolutely use before it gets too late and I’m swimming in my own mess of sources.
Saving straight from Twitter on your browser. Sometimes you just want to save that one Tweet for later. (Pro tip: Do not overuse this feature for saving funny memes or rants, though, as your bookmarks on Twitter serve that function just fine. Only Pocket the important ones that you may cite later.)
Saving and viewing videos! I haven’t done this yet, as I have still never quite pivoted to video. Reader, I am also a Reader.
Some other features of Pocket:
A machine learning–assisted discovery feature, in case you are having trouble finding content on your own.
Saving and sharing with other friends on Pocket. I am fine with all of the other social networks and if I really want to share something I have other means, but sure, you may want to use this feature.
Pocket is freemium. With the paid version, there are more archiving options, no ads (although the unpaid version has very few), and no limitations on how long content can be saved. Ideally the content can be saved “forever” or at least until I glom onto another system for saving content.
Pocket: it’s no Google Reader (I hope one day Gen Z makes fun of us for idolizing Google Reader), but it’ll do.
Content tech links of the past 3 weeks
A realistic view on how Google search might incorporate a natural language generation technology like GPT-3, from the POV of an SEO expert. SEOs and marketing folks can get, shall we say, up their own butts around thinking that humans actually want to read machine-generated copy, so I’m happy for this analysis. (FYI, here’s my assessment of machine- vs. human-generated copy.)
In fact-checking (and everywhere else, but in this case, fact-checking) human eyes need to work alongside automated solutions to produce the highest quality results. From Nieman Lab.
The truth is paywalled but the lies are free. A highly shared consideration of publishing economics and information gatekeepers with sentences like “one could almost construct a serious newspaper purely from material culled from the New York Times.” Via Current Affairs.
A solid pairing: The Second Act of Social Media Activism in The New Yorker and a thorough consideration of the aesthetics of Instagram social justice slideshows in Vox. I have much to say about the new iterations of social activism, especially the findability of the good information, but these two are great places to start. Also, the NYer piece cites the book I’m currently reading, #HashtagActivism, co-authored by a brilliant former classmate of mine, Sarah J. Jackson.
A history of one of the first surveillance measurement strategies, Nielsen ratings, from Matt Locke on How to Measure Ghosts.
The other Nielsen — also known as “the good Nielsen” or Nielsen Norman Group — doubles down on why PDFs are horrible. All the content strategists and accessibility advocates cheer Jacob Nielsen for his unwavering commitment to the cause.
The shaky business model of publications’ high-margin, high-profit branded content studios remains, unsurprisingly, shaky. Via Digiday.
Please don’t beg for links for SEO, via Vice. Someday soon I’ll cover ethical link-building, but until that point, remember that the large majority of link lists indicate nothing about editorial recommendations and everything about who had the most obnoxious link-begging campaigns.
Great anti-trust investigation of Google from The Markup: How Google prioritizes its own content in search results and How Google now discourages employees from using anti-trust language, but not anti-trust thinking.
The Wired piece on the end of cookies is not perfect, but it does break down the challenges of programmatic advertising in an easy-to-understand article. (However, please let’s stop perpetuating the myth that legacy journalism failed because tech “took all the money” or whatever. Legacy journalism “broke” because of private equity, ridiculous expectations for profit margins, shitty business models that prioritized advertisers, wealth hoarding, unnecessary consolidation, unchecked sexism and racism, near-ritual abuse of employees, and ignoring the general public for 30+ years. Tech has problems, but killing journalism isn’t one of them.)
One more call to action:
Visit The Content Technologist! About. Ethics. Features Legend. Pricing Legend.