Philips Email: Global Placement, Global Performance

Skip Fidura 6:42
Okay Good afternoon and welcome back. I hope everybody's having a great day I I didn't do my scheduling as probably as well as I should. So I just did the literally five minute mad dash down in the kitchen whip together a sandwich right back up here. So we get into this next presentation. So I hope you all have had a chance to stretch your legs to get some refreshment, get some breakfast or lunch, depending on what time of day it is for you. But now we're going into a brilliant presentation from Well, one of the presenters. Guy, Hanson is been a friend of mine for a long time, we've known each other work together in the email industry, probably longer than either one of us wants to admit. And with him today is his client, Charlie Wijen, who is from Philips. He's the digital and CRM specialist at Philips. And they're going to be presenting a case study on Philips, email and email programme. Gentlemen, over to you.

Guy Hanson 7:45
Skip, thank you so much. And really appreciate that great introduction. You're right, we've known each other for a long time, I can actually say the same of Charlie, it feels like Charlie and I've known each other a good sort of five years or perhaps even more. And, Charlie, a real pleasure to have you here with us today. And I think in terms of thinking about key takeouts from the session, I think one of them will be teaching our audience how we pronounce your surname? Would you like to sort of jump in and just educate them there quickly?

Charlie Wijen 8:18
Yeah, obviously, I hope you can all hear me. My name isn't Wijen. But I started the term happy life. And it also is introduction on how to pronounce and actually prepared. So if you would translate it into Dutch it would say re and but unfortunately, it's a very boring, last name. So it's not happy live or something cool. But it's a value and yeah,

Guy Hanson 8:41
there we go. So that's our, that's our first important learning of the day. It's a real pleasure to have you here with us today. So what are we going to be covering in our session. So I'm going to start off with the first sort of 10 or 12 minutes of the session. And we're just going to be doing some general comments, an overview about the current state of email are going to be drawing off some new research, which we prepared over the past couple of months, and basically using it to set the scene for Charlie's presentation, which is a really great session and Charlie is going to be talking about Philips email programme, global placement leading to global performance. And then we're going to finish up hopefully with enough time for a few questions and answers as well. So without further ado, I'm going to jump straight in. So what is the world of email look like at the moment? Well, one of my favourite pieces of research came out of the data and Marketing Association the DMA recently, and respondents who asked what way do you prefer to receive your marketing messages. And the key learning from this is that email is by far consumers preferred channel for receiving money. Marketing from brands. And what you'll see here is that around seven out of 10, consumers rank the email channel as their first or second preference out of nine major different channels, which is pretty impressive. And then you ask yourself, Well, why, you know, what's the email got for it got going for it, which makes it sort of, you know, so special in their minds. And I think there's two things going on here. The one is that they report that email is the most trusted, and also the most relevant channel and nearly six in 10, consumers reported on the trust factor, and around two thirds reported on the relevance factor. So I think as a result, when it comes to thinking about brands and companies consumers already use are aware of marketing emails and newsletters from those brands are their primary way of finding out about new offers products, services, and then followed in terms of other channels by TV, and online research. So as I said, we're going to dig into a few recent reports. And I'm not going to bore you to death with PowerPoint, but just to sort of tease out some of the most interesting facts and figures, especially as I say, in terms of how they frame the conversation we're gonna have with Charlie, about Philips email programme. So let's start with email challenges. And in our state of email reports, one of the study's main objectives each year, is to look at the challenges which marketers face when they using email. And this year, no surprise, competition for attention in the inbox, poor email engagement. And staffing and resource constraints have very much the top three challenges. And I think while marketers are showing progress in overcoming some of these obstacles, it's clear that they continue to face many, many challenges. So let's have a closer look.

So I think we all understand that marketing now is such a data driven discipline. And so in terms of a point of departure, it probably makes sense to first focus on how email marketers manage this most crucial aspect of their programmes. And we asked this question, how do marketers think about managing specifically the data quality of their email programmes, and it was really interesting that many of them responded that compliance with legislation and industry standards is a big, big area of focus. And this actually makes a lot of sense, because since May 2018, we've spoken a lot about what I call the GDPR halo effect, which was created by many of established best practices being written into law. And what we saw was that stronger consent, and clear resetting of expectations, and a greater provision of choice have all contributed to greater trust between consumers and brands. And that means they're more likely to provide higher quality personal data, including a primary email address. And what we're seeing from this chart is that email programmes have really benefited from this. And marketers have reported significant post 2018 uplifts against almost all of their most important KPIs, including deliverability, opens and clicks, leading to increased conversions, meaning more programme revenue. And also note that the bottom two KPIs, those are effectively your negative KPIs like OPT outs, and spam complaints have decreased significantly over the same period. So this throws data quality into really sharp relief when we examine the most important deliverability drivers. There it is. Back up at the top of the graph. Again, data quality is cited by 45% of our respondents in terms of the biggest factor that determines the deliverability of their email programmes. And when we drill down for more detail, here's some of the things that keep email marketers awake at night, bounce rates, reputation, spam complaints, low engagement, blocked listings, and all of these factors carry a cost. And marketers acknowledge this one intense say that this impact is severe. And there is actually only a small remainder who believe that it has no impact or they don't know. Now, performance visibility is often a real case of not knowing what you do. No. And as an example, less than a third of marketers believe their programmes hit the most common type of spam traps, which is recycled traps roughly a third of respondents there in the past year. And that's before we even talk about the third of businesses who either don't appear to know, that's almost 20% or claim not to have any traps at all another 15%, which certainly from my experience is extremely unlikely. So let's put this into context. It is the 90 day trap data for a well known high performance email programme. Don't worry, Charlie, it's not yours. And it hits hundreds of traps with a major impact on programme performance. So, you know, it comes back to the old Peter Drucker expression, you know, if you can't manage it, if you can't measure it, you can't manage it. And this is kind of the point that we're making on this next slide here, let's start connecting some of these dots. And from the state of email report, which I was talking about, here's three tactics which have a clear impact on delivering higher open rates. So the first of them send a reputation, which obviously helps mailbox providers determine whether emails reach users inboxes. marketers who actively monitor their sender reputation report higher open rates, and marketers who don't currently do this should absolutely consider leveraging tools like send a score, Google postmaster tools, Microsoft, smds, and so on.

The second connection of the dots is technology. So marketers who use technology to measure inbox placement are far more likely to report higher open rates to and so the second chart shows that becoming more successful at reaching the inbox, marketers increased chances of their emails being opened it stating the obvious. And then we talked briefly about block listings. And that's the third chart here. So I think most of you in the audience will be familiar with the principle of a blacklisting, which occurs if an IP address or a domain is suspected of engaging in unsafe email practices. And getting blacklisted can have a catastrophic impact on programme performance. And so those who monitor for block listings are much more likely to enjoy higher average open rates, and marketers should be pro active in doing this. So now let's look at how these tactics map onto subscriber engagement. And on the left, we examine the relationship between email marketing tactics, and inbox placement rates. And what we're seeing is that marketers with inbox placement rates of 90% or more, tend to be more sophisticated in using other email tactics as well. And the strong relationships between the high inbox placement rates and the adoption of tactics like proactive list management, multivariate testing, and integration of artificial intelligence. So on the right, we then consider the relationship with open rates. And much like with inbox placement rates, marketers who report high open rates tend to be more sophisticated in their use of other email tactics. And as with inbox placement, there's a strong relationship between above average open rates and the email tactics they use. Now, if we use email list management as an example, we can see that programmes that focus on their email list, management and quality of visibly more likely to enjoy those top tier inbox placement rates. And they are substantially more likely to have above average open rates. And I've also highlighted multivariate testing and artificial intelligence AI, especially when done together, as we're going to be hearing from Charlie on that topic in just a moment. So open rates an important metric, because the actions that marketers hope for from an email require that their recipients open them. But the act of opening an email isn't the end game. generating an action is and that action is quite often a click, although not exclusively, because it could be that you generate a phone call, you could even generate a store visit. But there's definitely a strong correlation between open rates and click rates. And the chart that we're showing here clearly shows this relationship and marketers who report higher open rates are absolutely more likely to report higher click rates. And that's why email is such an effective driver of website traffic. We've done some previous Google Analytics research, which showed that email generates around 11% of global retail web visits rising 25% when we consider paid channels only. So that's excluding direct visits and organic search. And it's highly cost effective in achieving this.

And so, while email remains a powerhouse channel for marketers, many factors influence success or otherwise for that matter, but nonetheless, twice as many marketers are reporting increased effectiveness compared with those reporting reduced effectiveness. And a big reason for that programme effectiveness maps directly onto revenue. And Charlie will tell you that there's a direct relationship, starting with deliverability and ending with revenue generated for his email programme at Philips. In the UK, email return on investment is currently a whopping 35 to one, an exceptional number that easily surpasses every other marketing channel. And this also translates into increased customer lifetime value. And what we find is that improve data quality and subscriber experience means that customers remain programme members for longer, engaging more frequently, and spending more. And coming back to the UK, email customer lifetime value in this country is currently 34 pounds and 56 pence. So a programme with a list of a million addresses is worth in the region of 35 million pounds. So that's quite a serious number. begging the question why data isn't shown on company's balance sheets in the same way that other business assets are. But that's a conversation for another day. Right now, I'm going to hand over to Charlie, hopefully, I've done a decent job of setting the scene for the great story that he's going to be telling us about how much Philips value their email programme, and the journey that they've been on to establish that value. But before you start, Charlie, I'm just going to do a quick introduction for NFS of everybody that hasn't had a chance to meet you before. So Charlie vion is a CRM specialist, he's been working with roll Phillips for the past seven years. He's part of the Digital Services Department responsible for creating and activating all digital marketing content, cross platform for Philips globally. And as you're going to hear from him back in sort of 2014 2015, Philips had some really major email deliverability problems at the time, using just one IP address and a load of what, especially these days, we were deemed to be sort of fairly bad practices in place. And over the last sort of six plus years, Philips has been working together with return path at the time now validity to become a role model in email marketing, best practices, and generating really, really sort of top tier best in class email deliverability performance for their programme. So Charlie, really great to have you with us today. Thank you for making the time and I think you've got a really great story to share with our audience. So really looking forward to hearing it. Thank

Charlie Wijen 23:20
you guy for a great introduction. also really great insights, always great to see that I see a presentation of guy at least once every two years. And that really blows my mind. They're always the data details that he shared with us. So for those who don't know fill up very quickly, Philips is by origin, a life build factory. In Holland, obviously, since they have lots of features there, they even went into other areas. And assured the point Philips literally did everything. We even made furniture but built houses built the toilet seats, etc. But most of you will probably know Philips as a consumer electronics company. Currently, it's actually becoming a healthcare company. But cool thing for my job is that even though a lot of things have been solved, like for example, televisions are no longer made, but films are actually made by TP vision, original doing all the main marketing for everything that has a fellow's branding also on the packaging or on the product except for lightning signified if they have gone their own path. But the overall branding stick still looks very familiar. I would like to share four things today with you basically once I want. First I want to take you on a journey on how actually our email deliverability problems started and how we actually resolve them across our journey doing that path. Secondly, I would like to go over some learnings that that we took away from from that. We also want to take you into some projects that we're working on and last but not least, I want to share with Due to a life market email marketing lesson stream great ideas fundamental ideas for making your email platform success three montage I think it's the best transition but let's start with the with the journey see this as a as a story basically because every good story has like a walk bottom you know a boy like and walkie for rare rapala qui dies and all these hands or Barbies mama shot 2014 was basically our our hockey moments are really a walk bottom at that time. Gordon Ramsay he was a big personality, you know, the chef and he he was working at monster chef and at the same time we were actually starting to promote the airfryer that's a fire that on June all or or or fat that just use air in order to find certain stuff fried chicken etc. And we wanted to promote in the US and they got wasn't agreement with with Master Chef so we did a combined email marketing effort. That wasn't a success because not only were we sending at that point like past and blast just everybody who has a in our database raised plus us got this email, no engagement criteria included I think there were like four or five reminders not only dead, but also from the monster TF organisation we got like a list no idea where they got it from and we sent out everything our sender score metric that we use for reputation on IP level at that point and still are using was at seven and our inbox placement was at the end do you think was at that point we only had one IP so also all other markets that are trying to send out email campaigns? Yeah, nothing which an inbox It was a long journey, which in which we really needed the help of at that point we're doing about nowadays valid, the validity to help us.

Could you go to next slide for me believe. Awesome. I'm going back to 2017. Now because we had a starting journey that look between 2014 and where we are now let's go to 2017. And that was basically the point where I think our email marketing programme was good, like average, slightly above average. And we did a lot of things to getting there, we introduced engagement criteria, we tried to having our volume at when we was being sent consistent by proper segmentation. So not just sending it to everybody, but really having a specific target audience for our emails, having a planning, making sure that all of our audience receives a regular email, all kinds of things like deca, demark, et cetera. We even at certification, with return partridges, a whitelisting programme, plus a exam that ensures that everything we're doing is a best practice. We did IP warm up for all big events, we introduced the centre score reputation monitor feedback loops, we see that all of our campaigns to really track the performance of each individual email were sent. And we went for a multiple IP strategy. Currently we're running on on average six IPS. That was 2017. And that was a point where we learned while we're doing great, we had a sender score of on average 99. And our inbox placement was up by about 85%. Because we're sending emails all over the globe. Some markers, by default perform a little bit less than others perform a little bit higher about 85%, which was a great achievement. And it was right about that time that we thought what could we do more to become actually the best because Philips didn't want to go for average, we wanted to really, really, really go for the top. So first of all, let's let's just see where we are now. Currently, our centre scores are still in the high 99. The lower the worst performing IP that we have is on average, also still 99 maybe 97. Or sometimes we have inbox placements above 90%. And we're sending over 5000 unique campaigns per year stats journeys because we're both ever going to vote using aliqua from Oracle Emery also using a Salesforce Marketing Cloud from Salesforce for some specific markets. One key thing also getting this is we built our own CMS, CMS we built for all of our stakeholders, all of our marketers, we built a drag and drop editor, making it very extremely easy for them to build an email built a segment built a campaign flow. Everything was built in a way that everything was approved. All best practices were included. I could even say that I'm 100 sure that step, CMS even works better than the stuff that you can probably find in most other CRM programmes like the one from Oracle or Salesforce or what have you. So this was basically in a couple of minutes, our journey. But I would also really like to share with you guys some challenges that we actually had during the pandemic. So roughly a year ago, we were introduced to the COVID-19. Bacteria at Corona. And although it's obviously a very negative thing, we could get some positive things looking at email marketing. One thing we thought is the type of messages that we were sending. Also as email marketing, we saw there was a growing need of get actually getting information about simple things like are you still delivering stuff are engaged, you can still order staff? How are we handling the COVID situation. These emails almost shop more like transactional emails than they felt like with email marketing messages, but we did see that they were highly engaging with our audience that these people actually wanted this kind of information. So it made us rethink the tone of force the kind of content that we actually are putting into our email messages. Could you go to the next, she's a nice animation. This was actually a large pain of a shout out at the start, but COVID related content was in for a long period of time. Another thing we saw is that our stock just went out differently because there was a high interest for certain type of products or hair clippers. Our shops literally sold out literally all the USD popular models but even the unpopular models because they for example, had too much accessories or they colour wasn't that intriguing versal load wasn't assembled single hair clipper at a certain point you could get because all the barber shops were closed

if there was a product that had something in there about for example, air purifiers that have removed bacteria, they will also getting sold out very quickly. And they're still being sold a lot. Also entertainment stuff that have Netflix on it like our high end televisions or Android televisions, etc. They're there they're sold out like like crazy. So it really also was a big effort from former former shops as well to making sure that all the products were were available. Also some some cool earnings. And the next one is about that we saw that at the start of COVID. So let's say last year's on our own tab, February March region. Mobile was by far the biggest platform that we were using or with whom our consumers and also b2b prospects were opening our emails and we saw that steadily declining and everybody going to web because obviously people had to stay at home a lot more so today we'll go beyond our desktop instead of sitting behind to their mobile phone what was funny is that certain products in our emails were starting to sell better and we also could find a very cool trend there so very often in our email we would like have three products next to each other we would have like three columns let's say in our fi or to barsha television and normally two products on the left was sold much much more often than the product on the second and third placed and we realised that on mobile because that template is actually responsive the the product on the left is shown first and then this product in the middle is actually dropped below that and the third product is well below that. So it made us test a lot more which product to actually place where because we saw the importance of having the most relevant products relevant data relevant CTAs on the top of the email all the things that we for example see is that the best time to send an email changed normally it was very early and I wish also that in the afternoon email were being opened a lot more short a lot of things changed in our in our in our behaviour. We also have some fun now obviously once we had our grooming essentials back into stock we there were some quite interesting ideas for coming from our design department. We've had some funny images from people shaving and hearts in their chest. We also had some very nice images of, of text where a certain word or a part of a word was was shaving off by the by the by the grooming tool And I will come to actually some more cooler things that we're doing. Like I said, we really want to be on top. And we really try to be the best in everything we implement. And there are quite some projects going on within within within Philips in our department to improving email campaigns because the start is obviously ensuring you have a correct process to ensure that you have all your default things that you have a welcome campaign reengagement campaign, you have your regular email marketing campaigns, you have your specific emails, for example, if you buy a toothbrush, that you get emails about replacement house, etc. But now we're actually trying some cooler stuff. And one is actually automated multivariate testing something that we are doing together with, with a partner called eight seconds. You're on screen you actually see a an example from one of the first tests we did in the North American markets, where we had a welcome campaign consisting of two emails out of which this was the first email. And we did some some testing to see how, what elements could be better. So on the left, you see the email how it's always been implemented. And only centre you see which varieties we actually included for that retest. So because of all these rights are actually 54 different emails that were being sent out in a testing period. And automatically, the winners implemented

the, the unique click through rate went from 5.4 towards 9.6. That's an increase of over 4.2%. And that basically means that 77.9% more consumers click on the contract, which is a huge increase for us. Funny to maybe say is also that we really see patterns in this, too, just to call some patterns, if you have an email where you have one button that has a colour that is contrasting the rest of the email, clicks go up 40 to 188%, on that specific button is a incredible other things that we for example learned is product colour, a black airfryer just performs better than a white product airfryer. We saw that, for example, people prefer images of products over images of people using products or, or just or just people. But on the other end of for b2b, for example, for our healthcare section, there, we saw it the other way around there, people prefer seeing imagery of doctors being doing stuff together. Other than just seeing some kind of MRI scan or image. I'm sure there's a lot of cool, interesting facts to grab from there. And also, another important thing is also to really implement it in our landscape. So in that environment, whatever whatever what I mentioned, where we were, you can basically drag and drop your email and then build it from scratch. Instead of just uploading an image. You could also just upload multiple images and everything was all the all the deaths reduced, implemented really from the start. Would you mind going to the next slide for me? Awesome. Another thing that we're actually actively working is on using AI. And we started with a first project that is actually four separate line optimization. So we have a huge amount of data. And we created a AI to where you could basically fill in a subject line and then based on past data and would give an approximants unique openweight. This horribly failed, because it was impossible to actually predict what how an email would would perform. Because a lot of things are hard to put into, into the calculator like how engaged Irish consumers are, what's the segment which which country is going to, but what we did find out is that we could 100% effectively predict the the order of which they would win, meaning we can predict if the opener is going to be 20% or 40%. But what we can predict is if we have for example three separate lines, which one is going to perform best and which one is going to perform worse. And if we then do a test on that we'd see the same result. And then in the future, we're also seeing how we can actually include AI also for the content of emails so understanding which words on which buttons are working, but even on imagery to understand which colours what kind of product image that we need, use, what kind of photos etc. So he is AI is also becoming a very big part of how we're trying to improve our email marketing campaigns. And then I would actually already like to go to some key takeaways, three mantras for Having a successful email marketing life having an happy and fulfilled email marketers live. And I'm actually going to start at the bottom because I think the top one is most important, first of all fall and learn to fail and learn. We failed miserably amount of times, we've done stuff, which is even below worst practice. But we did try it. And we feel there. And even though sometimes you expect something's going to feel something, something's actually is a huge success. At a certain point, we sent out an email that was being sent from a intern, that questionnaire that said something along the lines of Hey, can you help me with a research, very long email, I clicked on it, you got a very long quiz, it was about this Aiko or high end coffee machine, it went to a 200 contact segment. And it needed to have 150 answers on the quiz. In order for it to be successful for its force internship, that means all 170 5% of the sounds needed to be responding to the email, which I already said there's not going to happen, we continue to add it anyway. And at a certain point, even just called me, for me, I've stopped the campaign, I already have enough answers. And just to put it into perspective device that you could potentially win if you fell into questions like a coffee cup, this bag or a bag of beans. So sometimes things that you really think isn't going to work, it does work showed fail and learn, don't be afraid of failing, everybody fails. I failed miserably hundreds of times in my seven years at Philips. Next is things change, be flexible, also dare reach very often see danger or think that this is the best way to work with it. And it's very often not to call it a very small one two years ago, we did some testing with the subject line putting a motor control in it. And we saw that the unique open range went upwards 50% or something and normally was let's say 25 and ran to 37. So we thought we've we found the holy grail of serpentine optimization. But if we run the exact same test now It actually started blind run down. Apparently it was just too and so don't think you've you found it always be willing to learn. And the last thing and it's By default, the most important one is personalization, and relevancy is key. personalization, and especially relevance is key. What I mean with that. Email Marketing is the same as sending a normal email, if you send an email to a friend who's into soccer, and you send an email about soccer, that he's interested, if you send an email about hockey, then he might not be interested and a certain point he's going to lose interested. For every email you're sending, or every campaign you're sending, think about who you're sending it to. And if that is interesting for him, and if it is personal for for him. And those are the two month rush that I want to give you for free for also in improving your email marketing behaviour. Thanks,

Guy Hanson 43:01
Charlie, thank you so much. That is such an awesome presentation. So many really, really great insights that you've shared with us, I think our audience probably can't believe their luck. Skip, if you'd like to come back to the stage to join us. I just want to mention to our audience that if you're interested in any of the facts and figures that we presented, go and get a hold of the reports which I was referencing, but more importantly, I think we've got three or four minutes left and skipped. Perhaps you'd like to throw a question or two at myself or Charlie, while we're all here together.

Skip Fidura 43:38
Thanks, guy. Yeah, I would like to do that. I tell you what, though, I'll make a deal with you. I'll ask Charlie the question. And if you could throw links to those resources in the chat, that would make everybody's life a lot easier. If that's something you could easily pull together. Sure thing. So Charlie, some really interesting things. First off, it takes, I think, a lot of bravery to come on to, you know, a conference and basically say, you know, we started at rock bottom, you know, we let our email programme slip to where it couldn't slip any further. So, you know, thanks for that. And thanks for talking about how you've come back from that. One of the questions that I had though, is especially the stuff you're talking about, with the multivariate testing, and some of the AI stuff you're doing. at how big a list Do you have to have for that stuff to really start to work? So for some of the smaller brands that are on online, you know, when when will they be big enough? Do you think that they can start to take advantage of some of those some of those tactics,

Charlie Wijen 44:42
I honestly don't really think that you need to have that lesser is as important because for example, if we're looking at our, we are doing a lot of dc dc stuff, and we're doing a lot of b2b stuff, which is mostly healthcare related, right? The list of you ever need to be as mobile way smaller than the stuff that we haven't need to see, I mean, DBS and maybe millions, lower and millions and it's, and a few years if you're looking at the new stuff. In order to get good learnings, you do need to have a lot of volume. So what were for example do on smaller lists is actually do the same test in roughly the same environment on different things to actually seeing each month on the to see river more often going into a test for a campaign because they every for one campaign will already have enough list size to actually do a test. And they also see a very interesting question because on the on the screen, I'm looking at a question that came in from Clint and that was about roughly the same how many for wines would typically need for the test on each campaign, and that really comes down to the left side on what you're trying to test. So for example, if you're very small, this side, I only do an A B test or an ABC test. If we have a bigger company, I can go crazy basically, as as as a starting point, we have an average overweight of about 30% really depends a bit on the market. But taking that into consideration, we want to have every every wine seen by at least 65 people, as a bare minimum, should have means if you look at compared with the open rate, you need roughly 220 cents per Hawaiian. So basically, you can call me know, you know how much we should let Seidenberg bells into understanding wofully how much debt you could include.

Skip Fidura 46:30
Great, great, that's, that's really useful. Just to pick up on that one point about running the test multiple times on smaller lists. That's, that's a great way to get that done. And, you know, one of the push backs I always get from folks is, well, you know, the test won't be accurate. And what I would say is, look, we're selling, we're sending email marketing, I mean, we're sending emails, right? We're email marketers, we're not developing vaccines to, you know, stop a global pandemic, close is usually good enough, it doesn't have to be a perfect test, it doesn't have to be the perfect scenario. So it's great that that you kind of highlighted that.

Charlie Wijen 47:07
I mean, right. Thank you. And as a data point, that if we really did a test on multiple emails, the winner was in between 77 and 100 times with the cases the winner in all of them, right? So if you do the same test, the winner is 77% of times at least the same on average about 80%. So you could really already take some great learnings, even if there are different tests that are slightly different. Or you say, environments or different context or whatever.

Skip Fidura 47:38
Definitely different testing scenarios. Fantastic stuff. Fantastic stuff. Well, gentlemen, thank you very much, folks. If you have a chance, go pop over to the validity Expo booth. I don't know if guy's going to be there or not. But um, he would love if he's there. He'll love to chat data, and statistics and graphs with you, Charlie, thank you so much for coming along. Great, great presentation, both you. And yeah, we've only got about four minutes until our next presentation now. If you were lucky enough to be up quite early this morning. You would have seen our next presenter nasma salaam hope. She gave an amazing concert, an amazing concert, multi instrumental ukulele, banjo, uke glockenspiel, I think, anyway, Nelly has told me that it's going to be replayed. So if you get a chance, definitely, definitely check that out. But in the meantime, come back in a couple of minutes, where najma is gonna be talking about the essentials of email marketing in the third sector. In other words, the essentials of email marketing for charities,