Am I Hitting the Inbox? KPIs to Monitor
As conversations continue about how to market in a privacy-first digital environment, questions abound around the best way to measure email success. The KPIs that were once your main indicators all was well have shifted, making running and optimizing an email program that much harder.
Getting to the inbox is hard enough as it is. Now privacy policies, like Apple's MPP, take away the effectiveness of standard email metrics. So how do you know what is and isn't working? And how do you know if your messages are making it to the inbox?
In this month’s roundup, email experts from Hurix Digital, Simon Data, Adobe, Netcore Cloud, Zeta, and myself, your very own Kickbox deliverability geek, share the KPIs you should be monitoring to ensure your program stays in tip-top shape and the first indicators your email program may need some TLC.
And off we go!
Let The Mail Path Guide Your KPIs
Understanding your email messages’ path when sending should guide the KPIs you monitor to identify success.
After you hit send, your mail must first be accepted by the mailbox provider (MBP). Then it has to be placed within a recipient’s mailbox. Then your recipients have to interact with it.
Any bumps along the way indicate something needs to be addressed.
Delivery
Mail acceptance is measured by a delivery rate, or the inverse metric, bounce rate. This is a great measurement of list health and reputation.
If you are bouncing, you aren’t making it to the inbox.
Reviewing bounce logs can help you determine why your mail is failing. Logs can indicate if the problem is your targeting, content, reputation, list source, etc.
If you are dealing with a lot of hard bounces, it could be due to your signup forms or overextending your selections. Email verification can help weed out some of these bad email addresses.
Placement
After your message is accepted, the mailbox provider (MBP) decides where it should go based on a multitude of factors — reputation, engagement, content, safety, etc.
You can infer placement based on the engagement metrics, but you can also use third-party inbox placement tools to give you an additional layer of data points to determine campaign success.
Inbox placement tools are typically based on a list of seed email addresses at a plethora of MBPs the third-party owns and monitors. If these email addresses are bulking, there is a good chance your messages are also seeing some degree of bulking.
Engagement
Engagement comes in many forms and is not just an open metric. Apple’s Mail Privacy Protection (MPP) has skewed performance metrics for marketers, but one thing is sure, if your opens dip, you may be seeing some bulking or spam issues. If your open rates drop substantially, you most definitely have inbox issues.
Low clicks can be an indicator, but a low click rate can also be a content issue, like not providing a CTA worth clicking.
Another metric is complaints. Although a low complaint rate is a sign of a well-polished and targeted program, it could also mean you are going right to the bulk folder. People can’t complain about your messages in their spam folder.
You also have proprietary metrics like website visits, revenue, etc. If you start to see a decline here, you have to determine if there is an inboxing problem, a strategy problem causing disengagement, or both.
In either case, you may be fatiguing your customer, so you’ll want to review expectations and watch their engagement flow to see when they are becoming disinterested.
Whomp! I am not in the inbox. Now what?
If you discover there’s an issue, try to identify the why. You’ll need that so that your next steps resonate with your customers and the mailbox providers.
Many other data points can help determine the WHY behind placement—blocklist, DMARC failures, postmaster reputation sites, list churn, and more.
You can also hit up your deliverability consultant (Hi, how are you? My name is Jen and I consult too!) for a deep-dive analysis of your email program or a little guidance for remediation.
Before we part
Sometimes there are discrepancies between KPIs. Minor differences are normal.
The best way to determine email performance is to use all of the KPIs at your disposal and review them over time. Pay attention to spikes and drop-offs, but really look for overall trends to rule out blips due to filter hiccups or the occasional underperforming campaign.
And for those that are not in the spam folder predicament, 👏 👏 👏 keep it up!
How To Effectively Identify & Monitor Inbox Issues
Advice for email senders: how to identify and monitor issues—before you land in deliverability hell.
Testing paves the way for early detection:
Regular testing of emails is necessary for early detection of potential deliverability issues.
Especially with Mailbox Service Providers (MSPs) like Gmail and Yahoo!, you should test your emails on multiple ids. Hard testing ids are potentially hard to inbox, and soft ones are easy to get into.
Early signs of trouble are when your test emails land in the spam box of your soft ids. That surely means the MSP has detected an issue with your email design, past interaction history or sending pattern.
A good way to check this is to have a testing framework in place where you ascertain at the end of it:
- How many percent of emails inboxed on my hard testing ids?
- How many are inboxed in soft ids?
- What were the changes in email that were tested?
Based on these metrics, either tweak and test your emails until you are sure they are inboxed or do a root-cause analysis (RCA) to get to the bottom of the problem.
If you ignore this sign and still sent a campaign to your list, no problem.
There are other red flags on the way that you will still observe:
Open rates dipping significantly:
Probably the biggest visual indicator of bad deliverability of campaigns.
If you observe that open rates have dipped marginally from your regular average on a given day, don’t get into panic mode. It could be just a bad day for subscriber attention.
But when your opens dip significantly from the average, even for your best performing segment, you know you have been confined to deliverability jail.
When MSPs decide to put your emails in spam, they do it for a majority of your subscribers, barring those who are super engaged with you. It could be a case where even your regular openers might start receiving emails in spam.
Questions to ask:
- The percentage of unengaged subscribers you have targeted in the last 30-60-90 days?
- What’s my open and click analysis of best-performing segments?
- Did you do any email design changes recently? Using URL shorteners, multiple images and links, hidden text, etc.
Keep a regular watch on your campaign opens and catch anomalies.
Note: With Apple’s MPP policy, your opens could be exaggerated, so focus on clicks as well to document issues.
Google Postmaster to the rescue:
If a majority of your audience is Gmail (which is the case for most B2C companies out there), Google Postmaster should be your guide.
It’s the Master Oogway to your Panda, Yoda to Luke Skywalker, Albus Dumbledore to Harry Potter. You get the drift. The guide won’t give you ready-made answers, but it will show you the pain areas and point out problems.
Google Postmaster is a reflection of your send patterns, not a result of it.
One glance at your Google Postmaster and you will know where your deliverability issues lie. There are a lot of metrics that can indicate bad deliverability - bad domain or IP reputation (these are the biggest indicators), rising spam complaint rates (Gmail anti-spam filters are sensitive to them), deliverability or authentication issues, and feedback loop.
Give 48 hours for Google Postmaster to update these parameters, but keep a close watch on them. Getting these parameters back to normal will get you out of the deliverability mess.
Regular Blocklist checks:
As an email sender, you need to be on top of your sending domain and IP configurations. If you are using an ESP, they need to be consulting you on blocklist checks regularly.
I have seen, from personal experience, IPs getting listed with Spamhaus, and then the entire email activity is paused till you de-list them. If you are not aware of blocklists and you keep sending emails, the previously mentioned indicators will start getting worse, and by then, it could be too late.
Even if you use an ESP, always monitor the blocklist checking tools like MXtoolbox for regular checks on your email health.
Hope the above helps!
May the Force be with you!
Good luck inboxing! :)
3 Ways To Determine If Your Messages Are Landing in the Inbox
Probably the most difficult part of sending marketing emails is knowing whether your messages are going to the inbox or the spam folder.
The mailbox providers like Gmail and Yahoo are not going to tell us how they make their decisions. And, with so many different mailbox providers in the world all behaving differently, it is exceedingly rare that a sender has a deliverability problem that is global in nature.
It’s far more likely for spam filtering problems to be present at one domain or mailbox provider and maybe not at all at another.
While this can be maddening, there are some helpful steps you can take as a sender to more confidently determine how your messages are being treated once delivered.
Monitor your campaign results by recipient domain or mailbox provider and overall results. Most often, spam filtering problems do not impact all recipient domains equally. By looking for domains in your sending stats where you see significantly poorer stats, you can more easily identify and address inboxing issues that might otherwise go undetected.
For example, if you see your messages have a 35% open rate at Gmail. A 29% open rate at Yahoo. And a 3% open rate at Hotmail. It is safe to say that Hotmail is likely treating your messages differently and filtering more of your messages to spam than other domains.
This allows you to more easily target your efforts to send only mail that you are confident will be positively engaged with in order to correct the spam filtering issue. If you do see similarly poor results across most or all mailbox providers, that is an indicator that either something is technically wrong (in which case talk to your ESP) or that your sending behavior, in general, is in dire need of adjustment.
Utilize other data points to supplement email metrics in your analysis. It is increasingly the case that the standard email metrics of opens and clicks and so on are less than accurate at telling the complete picture of what is happening with an email message.
Senders who are able to not only look at their email metrics by domain or mailbox provider but also incorporate first-party data points like conversion data into that analysis can be much more confident they understand how their messages are being treated once delivered.
The senders I’ve worked with have found it very helpful to track things like days since last order or average lifetime value of subscribers by domain or mailbox provider. In a world of inflated open rates due to things like Apple’s Mail Privacy Protection, a distinct change in conversion behavior at a particular mailbox provider might be the first sign you see that messages have started going to the spam folder in large numbers.
Monitor your email sending both by individual campaign and over time. Looking at results on a campaign-by-campaign basis can help identify acute issues such as a sudden significant drop in open rate at a particular mailbox provider and is never a bad idea.
However, it is critical to also monitor metrics over time to help identify declining results that can indicate gradually increasing spam filtering of your messages, which is far more common than a sudden change in results.
Identifying a deliverability issue early on by looking at trends over time can help you change course early and avoid problems that feel like deliverability disasters.
The More Metrics You Monitor The Clearer The Picture
In deliverability, many of us have tools (internal and external ) to help tip us off to potential problems.
Anyone worth their salt uses well-known metrics and tools like delivery rates, open rates (yes, we still use opens), click-through rates, unsubscribes, complaints, spamtrap hits, SNDS, Google Postmaster Tools, blocklist notifications, sender reputation, authentication results, mail logs and more to track deliverability.
But none of those are full proof. Platforms can be slow to report, SNDS can be having its usual bad day, one user may have sorted their inbox and reported all of your mail as spam going back 2 years, causing a massive spike in complaint rates.
These metrics are used together to paint a picture. The more metrics, the clearer that picture becomes.
There is one metric that most deliverability people do not have access to that is the best tip off to a problem, and that is the sales numbers.
After all, the bottom line is the bottom line. If sales are chugging along, steady and strong, and they are very closely related to your email promotions effectiveness, most likely, no significant problems are occurring.
If sales are slowly decreasing over time, there may be a declining reputation issue related to poor data hygiene or Feedback Loop feeds that have broken.
If sales drop off more suddenly, there could be a design issue triggering fewer "Calls to Action," such as broken links. There could also be a blocklisting at a major network causing emails to not deliver or DNS changes causing authentication to break, resulting in more spam folder placements.
Without access to those sales figures, we tend to look at delivery rates, open rates and click-through rates first. If those are consistently strong, we can relax a bit.
But periodically, we look at all of those other metrics to make sure everything is working correctly so that hopefully, if there is a problem (like a broken FBL feed, or slipping reputations) we can work to repair those items before they impact deliveries, opens and clicks and most importantly, sales numbers.
3 Areas You Should Be Monitoring to Diagonse Delivery Issues
Monitoring your email campaigns seems like a daunting task to many. Using one of several third-party tools (like the Kickbox deliverability suite) is one way to measure your campaign’s health. But there are other things you can look at in your own data systems to help diagnose delivery issues.
Domain level benchmarking:
Understanding your database and how your subscribers engage with your email is a simple starting point to review. Look at “time over time” metrics for your email broadcasts and see if your metrics are changing from day to day, week to week, or month to month.
Are your open and click rates (even with Apple MPP) consistent for each broadcast and domain? Are they following a pattern or have they recently started to decline in engagement?
Sharp declines are typically indications of bulk folder delivery at a specific Mailbox Provider you may not see in an overall set of broadcast statistics.
Reputation Monitoring:
While it’s not a universal truth that a poor reputation means your mail is in the bulk folder, it’s a pretty strong indicator you’re likely moving in that direction. Monitoring services like the Google Postmaster Tools and Microsoft’s SNDS will give you an early warning on changes to your domain performance.
It’s no surprise a domain with a high reputation one week and medium the next is on the road to poor delivery. Keeping tabs on these details can help you, in the long run, manage your reputation from one or more email providers.
Bounce Monitoring:
Review your bounces and see what they are telling you. Ensure your ESP gives you the full text of the bounce log and not an internal classification like hard or soft bounce.
You need to see bounce errors like, “Message bounced due to DMARC failure,” “IP address is sending too fast,” “User unknown,” or “Content rejected” to better understand the possible issues with your email broadcasts. Once you understand the problem better, you can adjust your sending behaviors properly.
Looking at these metrics alongside your delivery tools will give you a significant amount of insight you may not receive from your ESPs reporting or a third-party tool that is assisting you with monitoring. Tools like Netcore’s GradeMyEmail can also be used to monitor RBL listings and validate if your email authentication is properly configured.
The Metrics You Monitor May Differ Depending On Your Objective
When it comes to understanding how your email campaigns are performing, there are a few key metrics to monitor that apply when sending from an established domain or warming a new one.
The main metrics that I recommend clients monitor are domain & IP reputation, delivery rate, open rate, click rate as well as unique open and click counts.
While some of these KPIs are straightforward and easy to tell when there is an issue like delivery rate and domain and IP reputation. When it comes to warming a domain, things can get a little murkier.
For example, it is very common for open and click rates to be very high during the first days of warming since you are sending to small volumes of your most engaged contacts to build a good reputation.
But as you progress and expand the pool of contacts you’re sending to in both volume and engagement, both rates will drop, and look like you’re not getting to the inbox. This is when looking at unique counts of opens and clicks comes into play.
As we begin to increase volume and rates drop, I will look at the percent increase in delivered emails and compare them to the percent increase in open and click counts.
As long as the open and clicks are increasing at a relatively similar pace, I can feel confident emails are being delivered successfully.
When it comes to monitoring an established marketing campaign, it is important we look not only at daily performance but also at how it is performing on a weekly and monthly basis.
If we are only looking at how emails performed that day or the day prior, we won’t notice the little drops here and there that overtime adds up to larger declines in performance.
This is most common with engagement, such as opens and clicks, as it is common for rates to improve over the course of a few days. The problem with this is that new, lower number becomes the baseline every few days until your open rate is down a couple of percent along with revenue.
When we run reports looking back at the last week and month, we can see these trends start to form, investigate the cause, and devise a plan to reverse it. We can also use this data to figure out what days perform the best or worst in terms of engagement and tailor the marketing plan to these insights.
While all of these metrics are important to look at daily, in my opinion, the one that is the most important is the delivery rate. This is because it gives you the most detailed and actionable data.
With the delivery rate, you can see if there are issues at a specific provider, which can correlate to what is being seen when looking at engagement data.
For example, if I see engagement is down at Yahoo, I will check the delivery rate and see if emails are getting blocked or deferred, which is keeping email out of the inbox.
We can also use the same logic when we see overall engagement is down. If I see an overall depression and the delivery rate is low with a high volume of deferrals or bounces for reputation, I know there is a bigger issue going on. In this case, an investigation and action plan may be needed.
While the metrics themselves vary for every client, I find monitoring domain & IP reputation, delivery rate, open and click rates, and open and click counts is the best way to understand how email campaigns are performing.
Looking at the daily and trending data on a weekly and monthly basis allows you to know what is going on at that moment and gives you the ability to identify trends and pull insights into how best to manage your email program.
Email Better
Sign up for email marketing & deliverability tips, expert advice & more. Delivered monthly.
