Introduction

Whether a non-profit is a small organization working on a niche issue or a multinational giant working to end global poverty, I believe the big problem facing its administrators can be summed up this way:

How do we generate the most ____ given our budget of ____.

The first blank can be filled in with a metric of success for the organization. Things like “awareness,” “support,” “donations,” “participation,” etc.

The second blank can be filled in with whatever the relevant operating budget is for the program. Whether that number is $100 or $100 million is really irrelevant – the point is that non-profits have a finite amount of resources to accomplish a goal.

While there is a massive literature in economics, finance and management on how to maximize profit, there is not a ton of free resources on how to apply these intuitions to non-profits. The data and technical expertise required to perform these types of analyses also exceed the capacity of most small non-profits. As a result of these two factors, optimizing operation stratgies may seem like an impossible task for small organizations.

The purpose of this blog post is to convince the reader that it is indeed not an impossible task. I will do this by covering three steps:

1. Model: Present a method of modeling the impact of different actions on a non-profit’s goal.
2. Maximize: Use the findings of this model to develop a strategy that maximizes the goal’s metric given the costs of the actions and a set budget.
3. Application: Explain how your non-profit organization can begin to collect the data needed to perform this analysis, and how Intrepid Insight can perform the analysis for free.

A Fictional Example

In order to illustrate the process from start to finish, I will consider a fictional organization: the Homelessness Alleviation Foundation (HAF). One goal of the HAF is to spread awareness of homelessness.

The HAF has chosen to measure engagement using the number of Google searches performed on the phrase “homelessness” (okay, this is mainly because Google Trends provides this data so it is convenient for us to use as an example). Naturally, more searches of the phrase represent more “engagement,” since the people searching it were clearly interested in the issue enough to go over to a computer and search the phrase.

Side note: Using Google Trends data for this purpose is not a very accurate way to measure engagement for a specific organization, since it is hard to disentangle what is driving people to perform a search. It is simply used as example data for this blog post because it is publicly available and will exhibit time series patterns.

The fictional HAF has five tools it can use to increase engagement, each with an associated cost. If HEF were a real organization, these numbers would be the sum of staff costs (labor) with material costs.

For now we will pretend that HAF has been employing a variety of combinations of these methods over the course of two years while keeping records. Because HAF is not a real organization, I will generate pretend data using the awesome package simstudy, which allows me to embed correlations and create variables from a variety of distributions.

IMPORTANT DISCLAIMER: Even though the Google Trend data is real, the right-hand-side variables or predictors are all generated with artificial relationships to the Google Trends data. As a result, all the results here are for demonstration purposes only: they do not reflect real relationships.

Below is a plot of the Google Search Index series along with a plot of all of the generated HAF outreach data.

HAF has a weekly budget of $5,000 for all of its public image/outreach actions, which it needs to decide how to best allocate to maximize engagement. This is the problem I hope to solve in the course of this post!

Model

In this post I lay out the code and steps I went through to specify and estimate a model relating engagement (Google Search Index) to the various outreach channels. In brief, the process involved these steps:

1. Decide on a modeling method to account for autocorrelation of the residuals (I went with regression with MA(1) errors).
2. Identify which outreach channels should be kept in the model.
3. identify any interaction effects between outreach channels.
4. Identify any functional transformations for the outreach channels.
5. Estimate a preliminary model.
6. Select the optimal ad stock rates to apply to each channel.
7. Apply the ad stock rates and fit a final model on the transformed data.

The final two steps involving ad stock rates were new to me: I plan to talk about these in more detail later (as in, “What are ad stock rates?”, “How do I choose the optimal ad stock rates?”). For now, I will leave the reader with just a table of the optimal rates I selected for each outreach channel:

Note: Direct Mail is excluded because it was not included in the final model.

Below is the R output displaying the fit of the final model, after step 7:

## Series: together_mod[, "search_index"] 
## Regression with ARIMA(0,0,1) errors 
## 
## Coefficients:
##          ma1  intercept   email  root_tweet  instagram    blog
##       0.1546   -47.3662  0.0377     11.2370     1.0291  0.0576
## s.e.  0.1168    15.3466  0.0152      0.7534     0.5678  0.2892
##       blog_instagram  tweets_instagram
##               0.0843           -0.0110
## s.e.          0.0631            0.0094
## 
## sigma^2 estimated as 28.86:  log likelihood=-333.78
## AIC=685.55   AICc=687.37   BIC=709.77
## 
## Training set error measures:
##                        ME     RMSE      MAE        MPE     MAPE      MASE
## Training set -0.003514632 5.171217 4.030203 -0.9649885 7.475736 0.4660191
##                     ACF1
## Training set -0.01080718

Which, when written out analytically, leaves us with this equation:

\[\begin{align*} \widehat{search\ engagement_{t}} &= -47.3662 + 0.0377*EMAIL_{t}\\
&+ 11.2370*\sqrt{TWEETS_{t}} + 1.0291*INSTAGRAM_{t} + 0.0576*\\
&BLOG_{t} + 0.0843 * (BLOG_{t}*INSTAGRAM_{t})+ \\
&-0.0110* (TWEETS_{t}*INSTAGRAM_{t}) + \epsilon_t

\end{align*}\]

where the error term is modeled with an MA(1) process: \(\widehat{\epsilon_t} = 0.1546*z_{t-1} + w_t\), \(w_t\) being the white noise residuals.

Here is a graph of the final model’s fitted values (blue) against the raw data (red) with a 95% prediction interval in light green:

Maximize

Now that we have a model, let’s put it to work! Recall the main question I wanted to answer: How do we allocate our budget of $5,000 in order to maximize engagement in the next week?

Now that we have our model, we simply maximize it subject to a constraint and get the answer! We can actually do this analytically, but I will save the reader from any more boredom and do it numerically using the constrOptim function in R. The one tricky thing to remember is that we need to make sure to add the ad stock of each outreach channel in first, since these are already set in stone (they happened last week!). So the below are the optimal additional outreach actions to be performed on top of the “ad stock” we already have built up in each channel. Plugging in our model, our set of costs for each action and our budget, out pops the maximizing strategy for the next week:

Total Cost: ~$5,000
Forecasted Max Engagement Value: 71.53
95% Prediction Confidence Interval of estimated Engagement: (61.00, 82.06)

I wrote a post specifically about the optimization portion (code as well as mathematics). Here is the link.

Application

So far I have laid out how to apply this process to a fictional organization. But the same methodology can be extended to any non-profit organization that has the below information/data:

1. A metric for success: For HAF, this was the Google Search Index. For a real non-profit this could be the number of website visits, the number of views on an awareness video, the number of signatures on a petition, a donation target, or an infinite number of other possibilities. All that matters is that the metric be measurable in constant time periods, whether those periods are monthly, weekly, daily, or hourly. It is also important that the metric be relevant – there must be a strong reason to believe that the organization influences the metric in a meaningful way.
2. A set of feasible actions with associated costs: For HAF, these were the “outreach actions,” like Tweets or Instagram posts. For a real non-profit, this could be demonstrations, flyering, YouTube videos, direct messaging, phone calls, etc. Again, the important thing is that these things be measurable over time in the same way that the metric for success is measurable.
   • It is also important that the costs of each additional action be quantifiable in some way. This does not neccessarily have to be in terms of money. It could be in terms of staff time, volunteer hours, or something else.
   • It is not important the costs are known for the entire period when data was collected. All an administrator needs to perform this analysis is the current cost. Indeed this is the benefit of building the base model in terms of actions and not costs – since costs change over time.
   • All possible actions that could impact the metric for success should be considered. They may not all factor into the final model (direct mail did not in the fictional HAF example), but not considering them can lead to significant omitted variable bias (more on this in a future post).
3. Enough data to perform a meaningful statistical analysis: The HAF example involved the equivalent of two years of weekly data, equating to about 109 data points (rows), with fields for the metric for success and each of the available actions. This is actually on the light side: having around 200 data points spanning at least two calendar years would be the ideal. The larger the span of the data, the more we can consider seasonal effects, which are often present, especially in the case of fundraising. More data also allows the analyst to experiment with more advanced models, and it helps the analyst make more accurate and precise forecasts.
   • The only real exception to this rule is if the way the data was measured changed. Perhaps the non-profit changed its web analytics software, or it started tracking engagement differently. When this happens, it might be better to use only current data. At any rate, it never hurts to consider both sources, with the knowledge that there may be a structural break in the middle when the transition occured.
   • It is also helpful to have a variety of combinations of actions in the data (aka rich data). This means the organization needs to have used each of its available channels/actions in the past. Using our fictional example, HAF had to have posted on Instagram and Tweeted on Tweeter, etc for me to begin to analyze the impact of each action. Even more specifically, HAF needed to vary how many tweets it tweeted and how many pictures it posted to Instagram.
4. Inside Knowledge of How Things Work: When analyzing data and building a model, intuition about how a process works is needed to avoid blindly following statistical tests. There are an infinite number of ways to specify a model, and there is no way to rule out every possibility, even using fancy algorithms. There is also always the possibility that a model completely disagrees with a known “causal pathway,” that is a known relationship that x causes y. Inside knowledge from staff, volunteers or administrators can help refine approaches and generate sensible models.

If all four of these boxes are checked, a non-profit can proceed in performing a very similar analysis to the one I performed here. In a blog post later this week I will lay out the code and methods used to fit the model and perform the maximization. I will link it here when we publish it:

I encourage any non-profit to perform this type of analysis internally if the organization has the internal resources (someone knowledgeable enough, with time to spare). I am more than happy to provide any tips and assistance to anyone who reaches out to me. If a non-profit does not have the staff knowledge, Intrepid Insight may be able to perform the analysis completely free of charge. Simply fill out this form or send an email directly to “jake at intrepidinsight.com” without the spaces.