Measurement and Experiments: From Impression to Action Without Fooling Yourself

Measurement is where advocacy pages go wrong quietly. Nobody gets caught. The weekly screenshot shows reach up 40 percent, the board nods, and a year later the petition has 300 signatures and the email list has 190 names. The reach was real. It just was not the objective. This lesson is about building a measurement habit that tells you the truth early enough to change course, and about running experiments small enough for a volunteer team but honest enough that you would defend them to a statistician. For the beginner's version of what each number counts, see Reading the Numbers in the social posts course. This lesson assumes you have read it and goes on from there.

The funnel, stage by stage

A person who ends up signing your petition or joining your list passed through a sequence. The platform decided to put your post on their screen. They noticed it rather than scrolling past. They clicked, or watched, or tapped. They arrived on your page and did not bounce. They did the thing. They stayed subscribed or came back. Each step loses most of the people who entered it, and each step is measured by a different system that defines things its own way.

Stage	What happened	Metric	Who reports it	Typical leak
Shown	Content entered a screen	Impressions or Views; reach (unique people, estimated)	Platform insights	Ranking decides this; see lesson 01
Noticed	Person paused, watched 3 seconds, expanded text	3-second views, average watch time, hold rate, thumbnail click-through rate	Platform insights	Most impressions end here
Engaged	Reacted, commented, shared, saved	Engagement count and rate (per follower or per reach)	Platform insights	Rival IQ's 2025 nonprofit median is 0.046 percent per post on Facebook, 0.561 percent on Instagram, 3.04 percent on TikTok
Clicked	Left the platform	Link clicks, click-through rate	Platform insights, with gaps	Platforms deprioritize links; clicks are often a fraction of a percent of reach
Arrived	Page loaded on your site	Sessions by UTM source and medium	Your analytics	Referrer stripped, pre-fetch, bounce before load
Acted	Signed, subscribed, donated, called	Conversions; conversion rate per visit	Your form or email tool	Unbounce's 2024 median landing page conversion across industries is 6.6 percent
Stayed	Remained reachable and responsive	List churn, repeat action rate	Your email tool	M+R 2026: 4 percent of starting subscribers bounced away and another 12 percent unsubscribed in a year

Two things about this table matter more than the rest. First, the owner of the measurement changes halfway down. Above the click, the platform measures. Below it, you do. A platform's numbers and your numbers will never reconcile exactly, and a team that spends hours trying to make them match is wasting its time. Second, the leak at each stage is a multiplication, so a small improvement at a late stage is worth a large one early. Doubling your signup form's conversion rate from 3 to 6 percent doubles your list growth from the same traffic. Doubling impressions might add nothing if the extra impressions land on people who were never going to click.

One metric per objective

Advocacy pages usually run several objectives at once, which is fine as long as each piece of work is assigned to one. The failure pattern is to post for one objective and judge by another. A documentary-style explainer meant to build trust with county staff should not be judged by link clicks. A petition push should not be judged by shares.

Objective	Primary metric (the one you report)	Supporting metrics	Metrics to ignore for this objective
Agenda setting: get an issue onto the local conversation	Mentions of the issue by officials, press and other pages in the following 30 days (count them by hand)	Shares, saves, comments from new accounts	Likes, follower count
List building	Confirmed new subscribers per week, by source	Landing page conversion rate, cost per subscriber if paid	Reach, impressions
Mobilization: comments at a hearing, calls to a board	Actions completed and verifiable (speakers signed in, letters received)	Click-through on the action link, RSVPs	Video views
Trust and relationship with an audience	Return rate: share of your reach that is existing followers, email open-to-click behavior over months	Direct messages, replies from named community members	Viral spikes from outside the area
Correcting a false claim	Reach of the correction among the audience that saw the claim (hard to get; use the same page and format)	Shares of the correction, fact-check labels	Engagement rate, which outrage inflates
Fundraising	Net revenue after fees and ad cost; revenue per 1,000 emails ($54 is the M+R 2026 nonprofit median)	Donation page conversion, average gift	Everything on the platform side

The agenda setting row has no platform metric at all, and that is deliberate. The research on what advocacy communication can do, covered in lesson 09, points toward agenda setting and mobilization rather than conversion. Those outcomes happen off the platform. If your theory of change from lesson 02 says the county commission must hear from 200 residents, the number you report is residents heard from, and the page metrics are diagnostic tools for getting there.

Vanity metrics, and why the platforms keep inventing them

A vanity metric is one that goes up when you do more, regardless of whether anything you care about happened. Follower count is the classic case. Impressions and Views are the modern one. In August 2024 Instagram made Views its primary metric, replacing plays, impressions and accounts reached, and Meta said Facebook would follow; Social Media Today reported the switch on November 14, 2024. For photos, text and Stories, a View is counted every time the content appears on a screen, repeats included. One person who scrolls past your graphic three times in a day is three Views. Meta said the calculation did not change, only the name, which is true and also exactly why the number looks larger than the old reach figure sitting next to it. Watch Time became Minutes Viewed, rounded to the nearest minute.

None of this is deceptive on the platform's part. The metric does measure distribution. The trap is in your reporting: a slide that shows Views growing month over month is showing that your content is appearing on screens more often, which the ranking system controls and which may or may not bring you a single new subscriber. Put Views in the diagnostics tab of your spreadsheet, not the summary.

What each platform actually gives you

The definitions below come from each platform's own documentation checked in October 2026, which is the only safe source. Third-party guides repeat each other and drift out of date.

Meta: Facebook Pages and Instagram

Meta's Graph API Page Insights reference is more candid than the Business Suite screens built on top of it. Impressions are "the number of times any content from your Page or about your Page entered a person's screen." Reach (page_impressions_unique) is "the number of people who had any content from your Page or about your Page enter their screen," and the documentation marks every metric with the _unique suffix as approximate. Insights data is kept for two years only, and a query with a start and end date can cover at most 90 days at a time. Most metrics update once every 24 hours. Pages with fewer than 100 likes get no Insights at all. The reference also states that a batch of Page Insights metrics was deprecated across all API versions by June 15, 2026 and that calling one now returns an invalid metric error, and it marks page_impressions_unique itself as deprecated above Graph API v25, so a dashboard built on raw API metric names needs checking against the current list.

Practical consequences: export weekly, because anything older than two years is gone and anything older than 90 days takes several queries to pull; treat reach as a rounded estimate and never report a reach change under about 10 percent as meaningful; and expect Business Suite to rename or retire a column without notice.

YouTube Studio

YouTube Studio's Reach tab shows impressions, impressions click-through rate, views and unique viewers, according to the Analytics basics page. The useful detail is in the impressions help page: an impression counts only when a thumbnail is shown for more than one second with at least half of it visible, and only on YouTube itself. Impressions exclude external websites and embeds, the mobile website, YouTube Kids, YouTube Music, cards and end screens inside the player, emails and notifications, and background tabs. So a video embedded on your own site and shared by email can rack up views that never appear as impressions, and its click-through rate will look odd as a result. Impressions click-through rate is defined as how often viewers watched a video after seeing a thumbnail. It is the cleanest "noticed" metric any platform offers, because the denominator is tightly defined.

TikTok Analytics

TikTok's own help pages did not return readable text when checked, so the description here relies on Buffer's September 2026 guide, which matches the app. Analytics are available on Creator and Business accounts and come in four tabs: Overview (video views, profile views, follower change, likes, comments, shares), Content (average watch time, total play time, reached audience, full-video completion rate, views by traffic source, trending videos), Followers (counts, demographics and active times, shown only once you pass 100 followers) and LIVE. Some per-post metrics are available only for the past seven days, which means TikTok is the platform where a weekly export matters most. The traffic source breakdown, showing how much of a video's audience came from the For You feed versus your profile or search, is the single most useful diagnostic TikTok provides for an advocacy page, because it tells you whether the recommendation system picked up the video or your existing followers carried it.

Bluesky

Bluesky has no native analytics. Its FAQ describes custom feeds, the open AT Protocol and public moderation data, and says nothing about view counts or reach for accounts. The protocol itself confirms the gap: the postView definition in the AT Protocol lexicon carries likeCount, repostCount, replyCount, quoteCount and bookmarkCount, and no field for impressions, views or reach. You can count engagements, by hand or through a third-party tool that reads the public API, and that is all. Because tens of thousands of custom feeds distribute posts independently, no one can tell you how many people saw a Bluesky post. Report Bluesky by engagements and by link clicks measured on your own site.

Everything else

X's analytics offering has changed with its subscription tiers more than once; check what your account level shows the week you build your dashboard and note it in the sheet. Threads, LinkedIn and Nextdoor each offer a basic insights screen for professional accounts. Whatever they show, apply the same rule: write the platform's definition of each metric into your dashboard's notes column, with the date you checked it.

UTM tagging

The platform tells you who clicked. Your analytics tells you who arrived and what they did. The bridge between them is the UTM tag, a set of parameters appended to a URL that Google Analytics 4 and most other analytics tools read. GA4's documentation says to always include utm_source (where the traffic came from, such as facebook or newsletter), utm_medium (the type of traffic, such as social, email or cpc) and utm_campaign (the effort, such as bridge-bond-oct). It recommends utm_id and utm_source_platform and treats utm_term and utm_content as optional. Two newer parameters, utm_creative_format and utm_marketing_tactic, are accepted but not reported yet. Values are case sensitive: utm_source=Facebook and utm_source=facebook are two different sources in your reports, and a team of three volunteers will produce both within a week unless there is a written convention.

A tagged link looks like this: https://example.org/bridge?utm_source=facebook&utm_medium=social&utm_campaign=bridge-bond-oct&utm_content=video-mayor-quote. The utm_content value is how you tell two posts in the same campaign apart, and it is the field that makes creative testing possible at all.

Parameter	Convention	Examples
utm_source	Lowercase platform or page name; one spelling, written in the notes tab	facebook-main, facebook-bridge, instagram, youtube, bluesky, newsletter
utm_medium	One of a fixed list of five	social, email, paid-social, qr, partner
utm_campaign	Issue, then month, lowercase with hyphens	bridge-bond-oct, water-rates-q1
utm_content	Format and hook, short	video-mayor-quote, carousel-cost-table, text-only
utm_term	Leave empty except for paid search

Attribution and its limits

Even with perfect tagging, the story your analytics tells is incomplete in ways you cannot fix. Some of the gap is technical. Many app browsers and privacy settings strip the referrer, so a visit that began on Facebook arrives looking like direct traffic. Messaging apps, text messages and email forwards carry none of your tags unless the sender happened to copy a tagged link. Apple's Mail Privacy Protection, announced June 7, 2021, "stops senders from using invisible pixels to collect information about the user" and masks IP addresses, in Apple's words, so email open rates are partly fiction; Mailchimp's own benchmark page warns that its open rate figures "may be impacted" by the feature. Report clicks, not opens.

The deeper problem is causal. Attribution reports answer "which click came before the action." They cannot answer "would the action have happened anyway." Gordon, Zettelmeyer, Bhargava and Chapsky tested this directly with Facebook's own data: 15 randomized advertising experiments covering about 500 million user-experiment observations and 1.6 billion impressions, compared against the matching and regression methods the industry uses to estimate ad effects from observational data. The observational methods failed to recover the experimental results even with extensive demographic and behavioral controls. In half the studies the estimated lift was off by a factor of three or more. In one study the randomized lift in purchases was 73 percent while a naive exposed-versus-unexposed comparison gave 316 percent. In another the true lift was 2.4 percent and the closest observational estimate was 1,306 percent. The paper appeared in Marketing Science in 2019. The reason is selection: the people the system chooses to show your ad are the people it predicts will act, so they would have acted at a higher rate regardless.

And even randomized experiments struggle when the effect is small relative to the noise in the outcome. Lewis and Rao, in the Quarterly Journal of Economics in 2015, analyzed 25 large field experiments with major U.S. retailers and brokerages, most reaching millions of customers, and showed that the confidence intervals around the return on ad spending stayed wide even at those sizes. Their title says it: the economics of measuring advertising returns are unfavorable. An advocacy page with a $2,000 ad budget is not going to measure its return on investment to the dollar. It can measure whether a lead ad produced subscribers at a cost it can afford, which is a different and answerable question.

Building a weekly dashboard in a spreadsheet

You do not need software. A spreadsheet with one row per week per page, filled by hand from the exports above, outperforms most paid dashboards for a small network because you understand every number in it. The structure that works has four tabs.

Tab	Contents	Who reads it
Summary	One row per week: primary metric for each live objective (new subscribers, actions completed, mentions by officials), with the 4-week average beside it and the quarterly target	Board, coalition partners
Platforms	One block per page: raw weekly counts (views, reach, engagements by type, link clicks, followers) and computed rates	The team
Traffic and conversion	Sessions by utm_source and utm_medium, landing page conversion rate per source, cost per subscriber for paid sources	The team, whoever manages the budget
Notes	Metric definitions with check dates, platform changes, campaign start and end dates, outages, anything that explains a spike or a dip	Future you

Experiments on a small page

The platform research in lesson 05 tells you what tends to work on average across millions of posts. Whether it works on your page, for your audience, on your issue, is an empirical question you can answer, within limits set by arithmetic. The arithmetic is where most advocacy "tests" fall apart, so start there.

Sample size and minimum detectable effect

Kohavi, Henne and Sommerfield's Practical Guide to Controlled Experiments on the Web (KDD 2007) gives a rule of thumb for the total sample size needed: n = (4 r sigma / delta) squared, where r is the number of variants, sigma is the standard deviation of the metric and delta is the smallest difference you want to detect. It assumes 95 percent confidence and 90 percent power, and the authors note the factor of 4 may overestimate by about 25 percent for large samples. For a rate (a click rate, a signup rate), sigma squared is p(1 - p).

Evan Miller's sample size calculator does the same arithmetic with adjustable confidence and power, and shows visually how the required sample balloons as the minimum detectable effect shrinks. Use it before every test, and if the number is larger than the traffic you can get in a month, change the question or skip the test.

Pre-registration

The second failure is subtler. A team runs a test, the primary metric does not move, so they look at saves instead, which did move, and report a win. Simmons, Nelson and Simonsohn showed in Psychological Science in 2011 what this flexibility does: combining four common practices (choosing among several outcome measures, adjusting the sample size as you go, adding covariates and selectively reporting conditions) raised the false positive rate in simulation from the nominal 5 percent to 60.7 percent. A researcher using those practices is more likely than not to find a significant effect where none exists. A volunteer looking at an Insights screen has every one of those freedoms.

The fix is pre-registration, which the Center for Open Science defines as writing the plan down before the study, so that the same data is not used both to generate and to test a hypothesis. For a page, the registration is a short note in the dashboard's Notes tab, dated, that nobody edits after launch.

Holdouts for paid tests

Boosted posts and lead ads are where real money leaves the account, and where attribution is least trustworthy. The platform's own answer is a lift study. Meta's lift study documentation describes it: the audience is randomized into test and control groups, the control group is not shown the ads, and lift is the difference in conversions between them. Breakdown results require at least 100 conversions across both groups, and the tool warns when a study is underpowered. Access is limited and goes through a Meta representative, which puts it out of reach of most small advocacy accounts.

You can approximate the design yourself. Split your target area into comparable halves (two sets of zip codes with similar populations and similar past engagement), run the boost in one and not the other, and compare tagged signups per 1,000 residents between them. The comparison is crude and the sample is small, but it is a true holdout, and it will tell you whether a $500 boost produced signups you would not have gotten anyway, which no attribution report can. Swap the halves for the next campaign so neither area is permanently the control. For a statewide effort, the same logic works with counties. Record the design in the pre-registration note.

A/A tests and reading noise

Before trusting any test, run one where both arms are identical. Kohavi and colleagues recommend A/A tests continuously; with a 95 percent confidence level, the two identical arms should differ "significantly" about 5 percent of the time, and if they differ more often than that your assignment or measurement is broken. For a page, this means comparing odd-week posts to even-week posts for a month with nothing changed. The size of the difference you see is your noise floor, and no real test result smaller than that deserves a mention.

For counts, there is a quick rule. Counts of independent events scatter with a standard deviation of roughly the square root of the count. Twelve clicks has a standard deviation of about 3.5, so a post with 8 and a post with 16 clicks are indistinguishable. Four hundred clicks has a standard deviation of 20, so 360 versus 440 begins to mean something. Most weekly swings on a small page sit inside the square root band, and most "the algorithm changed" theories are born there. The American Statistical Association's 2016 statement on p-values is worth reading in full by anyone who reports test results; its principles that a p-value does not measure the size or importance of an effect, and that conclusions should not rest only on whether a threshold was crossed, apply to a Facebook test as much as to a clinical trial.

Survey methods for audience research

Platform demographics tell you age and gender bands. They do not tell you why someone follows the page, what they already believe about the water rate proposal, or which official they trust. A short survey can, if it is written and sampled with care. Pew Research Center's guide to writing survey questions is the standard, and its examples show how much wording matters: in a 2003 poll, 68 percent favored military action in Iraq without a casualty mention and 43 percent when "thousands of casualties" was added; in 2008, 58 percent chose the economy as a top issue when it was offered in a closed list and 35 percent named it unprompted. Pew warns against double-barreled questions, agree/disagree formats that invite acquiescence, and loaded words, and it randomizes the order of unordered answer options because respondents on screens favor the first option listed.

The sample is the harder problem. A survey linked from your page reaches your followers, who are not your community. Pew's 2024 analysis of online opt-in polls found that 12 percent of opt-in respondents under 30 claimed to hold a license to operate a nuclear submarine, against a true rate that rounds to zero, and that in an opt-in poll 20 percent of under-30s agreed the Holocaust is a myth against 3 percent in Pew's probability-based survey. Bogus and careless respondents cluster among the young, exactly where advocacy pages want to hear more. A follower survey is useful for learning the range of views and the language people use. It is not a poll, and reporting it as "72 percent of residents support" is a false claim that an opponent will take apart in one post.

Reporting results to a board or coalition

The audience for your measurement is often a volunteer board or a coalition of partner groups, none of whom read platform documentation. They are also the people whose trust you most need to keep. The temptation in every quarterly report is to lead with the biggest number. Resist it. The report that builds a board's confidence over years is the one that says what the goal was, what happened against it, what the team learned, and what it stopped doing.

Questions and answers

Our reach dropped 30 percent this month and nothing changed on our end. Is the algorithm punishing us?

Probably not in any targeted sense, and you cannot tell from one month. Meta's unique metrics are approximate, ranking systems shift weights constantly, and a page's reach is dominated by whether a few posts got shared. Check the 4-week average against the same weeks last year, check whether a big post in the prior month inflated the baseline, and check the Notes tab for anything that happened. If reach stays down for a quarter across all post types, then look at lesson 01 for what the ranking documentation says it rewards.

Which single number should a two-person volunteer page track?

Confirmed new email subscribers per week, by source. It sits at the "acted" stage, you own the measurement, it is immune to platform renaming, and it is the asset that survives a platform's decline. Everything else is diagnostic. See lesson 07 for why the list is the owned audience.

Is engagement rate per follower or per reach the right one?

Per reach tells you how the people who saw a post responded, which is the content question. Per follower is what benchmark reports such as Rival IQ use, because reach is not public, so use per follower when comparing yourself to a benchmark and per reach when comparing your own posts to each other. Rival IQ's 2025 nonprofit medians per post by follower were 0.046 percent on Facebook, 0.561 percent on Instagram, 3.04 percent on TikTok and 0.023 percent on X. Say which one you are using every time.

Can we trust open rates at all after Apple's changes?

Treat them as an upper bound on attention and nothing more. Apple's Mail Privacy Protection pre-loads tracking pixels for users who turn it on, so those emails register as opened whether or not anyone read them. Mailchimp's benchmark page carries the same warning. Clicks and replies are the email metrics that still mean what they say.

We cannot get 25,000 impressions. Does that mean we cannot test anything?

It means you can only detect large effects, so test large changes: video versus text, a hard call to action versus none, a question post versus a statement. The arithmetic in the worked example shows a doubling of click rate is detectable in a few posts per arm. And you can pool: a test run across ten weekly emails has ten times the sample of a single send. Pick the questions your traffic can answer.

What is a fair way to attribute a signup that came from a QR code at an event?

Tag the QR link with utm_medium=qr and a utm_campaign naming the event. Those signups are among the cleanest you will get, because the only way to reach that link was to be in the room. Record the event's attendance so you can compute signups per attendee, which becomes your benchmark for deciding which events are worth staffing.

How often should we check the dashboard?

Fill it weekly, read the summary monthly, decide quarterly. Daily checking of a small page's numbers trains you to react to noise. Kohavi and colleagues recommend running experiments in whole weeks for the same reason: day-of-week effects are real and a Tuesday is not a Saturday.

Our coalition partner reports "impressions" from Bluesky. Where did that number come from?

Ask. Bluesky's protocol exposes likes, reposts, replies, quotes and bookmarks for a post and no view or impression field, and its FAQ describes no analytics. Any impression figure for Bluesky is an estimate from a third-party tool, built on assumptions the tool should be able to explain. Report Bluesky by engagements and by tagged clicks to your site.

Hard questions

Isn't all of this overkill for a volunteer page with 3,000 followers?

The spreadsheet is an hour a week. The pre-registration note is ten minutes. What is overkill is the alternative: a year of posting judged by whichever number looked best each month, and a board deciding on a budget from screenshots. The arithmetic in this lesson mostly tells a small page what not to bother testing, which saves effort. And the one metric this lesson insists on, confirmed subscribers by source, is the one that tells a small page whether it is building anything.

If attribution is this unreliable, why tag links at all?

Because operational questions are not causal questions. Tags tell you which page's audience clicks, which link is broken, which event produced signups, which email send drove traffic, and whether your paid-social landing page converts at 12 percent like Unbounce's 2024 benchmark for that channel or at 2 percent. All of that is useful without any claim about what caused what. The caution is against reading a last-click report as proof that a boost produced an outcome, which Gordon and colleagues showed it cannot do.

Platforms have the data and the statisticians. Why not just trust their reported lift?

Meta's lift studies are real randomized designs and the documentation is candid about their requirements, including the 100-conversion minimum for breakdowns and the power warnings. The limits are access and incentive. Lift studies are gated behind a representative, so most advocacy accounts never see one, and the party reporting the lift is the party selling the ads. Lewis and Rao's 25 experiments show that even well-run experiments at scale leave wide uncertainty around returns. Use the platform's experiment tools when you can get them, read the confidence intervals, and keep your own tagged conversion data as the second opinion.

Doesn't pre-registration stop us from learning from surprises?

No. It labels them. The Center for Open Science's guidance is that changes made after seeing results should be reported as exploratory, not that they should be hidden. When the saves metric jumps in a test that was about clicks, write it down as a new hypothesis and test it next. Simmons, Nelson and Simonsohn's 61 percent false positive rate came from treating the surprise as the finding. Treating it as the next question costs nothing.

Isn't a follower survey still better than nothing?

For learning the vocabulary and objections of the people who already follow you, yes, and this lesson recommends it for that. For claiming what residents believe, it is worse than nothing, because it produces a confident-looking number that is wrong in a predictable direction. Pew's opt-in work found 12 percent of respondents under 30 claiming a nuclear submarine license. If a quarter of your "supporters" are inattentive or fake, the 72 percent support figure is noise dressed as evidence, and an opponent who knows the Pew work will say so in public.