Testing and More Testing on WooCommerce Releases with Julia Amosova from Automattic

Testing and More Testing on WooCommerce Releases with Julia Amosova from Automattic
Do the Woo - A WooCommerce Podcast

 
 
00:00 / 00:55:06
 
1X

While WooCommerce store owners and builders run tests on their sites with new releases, have you ever wondered what kind of testing happens at WooCommerce? Well, let me tell you. It’s impressive. In fact, this show is one of our longer ones, and still, we couldn’t cover everything in detail. But we managed to squeeze in quite a lot.

A Conversation with Julia Amosova from Automattic / WooCommerce

In episode 42 of our podcast, Brad Willams and I chat with Julia about:

  • What you can expect coming in WooCommerce 4.0.
  • The reason behind the release schedule and timing of beta releases.
  • How they use automated testing before releasing an update.
  • Whether visual regression testing is part of the implementation.
  • The system for testing new versions with the 400+ extensions in the WooCommerce marketplace.
  • Running tests on user sites and just how that is done.
  • Hosting the the testing plans that are part of their releases.
  • Mobile and testing the Rest API.

Thanks to our sponsors


When you take a dive into the testing that takes place before a release that goes behind the scenes at WooCommerce, expect a lot. We kicked this episode off by revisiting what is coming in WooCommerce 4.0.

From there, Julia explained just how they reached their update schedule and how it brings more stable updates, as well as beta releases for testing, to users and store owners.

She touched on both manual and automated testing, the latter being their preferred option because of the time intensive resources needed for manual testing. And we got a chance to hear some of the methods they use for testing their releases.

Brad brought up visual regression testing and asked Julia how much it is used at Woo. It was interesting to hear where it works and where it doesn’t and, based on that, how much time they have devoted to it.

Of course we couldn’t leave Julia without hearing the nitty gritty about those 400 plus extensions on the WooCommerce marketplace. From her response on their approach, it is certainly a bonus for developers who have their extensions available there.

Moving beyond the Woo confines, Julia shared how they test on user sites and the opportunity to have your site tested with some white glove services. There is even a nice reward you will get for you if you go this route.

And if you have ever wondered if they test on various hosting platforms, the answer is yes, on about six of the more popular ones. Interestingly enough, they choose random Automatticians to jump in on test sites and have a go at it, with valuable results.

Lastly, Julia touches on the testing of the Rest API and how that affects external software plugins for mobile apps, and, of course, including their own mobile app.

All I can say is that once you listen to this, not only will you understand the value of testing, but also the extremes WooCommerce goes to for that moment when you click that update button.

Announcements

  • Get Those Last Minutes Tests in for 4.0
  • Coming Soon to BobWP – Profiles in Woo
  • The WP eCommerce Show Ends

Where to find Julia on the web:

Bob: Hey everyone, BobWP here, back with another edition of Do the Woo. I'm joined by my exquisite guest, Brad, I-survived-WordCamp-Miami Williams.

Brad: Yeah, I'm back. I got a bit of sun. I was in the sun for like a minute and I got sunburned. I'm definitely not used to it this time of year. But another great WordCamp Miami. We missed you, Bob.

Bob: Yeah, I saw you doing a lot of camping on the boat and then the golf course, too, so that was good.

Brad: You know how WordCamp is. If I'm going down to sunny Miami, I gotta get a little bit of recreational time. So, yeah, I played some golf. Was on that a boat a little bit. Looked at the mansions that I'll never be able to afford and spent Saturday at WordCamp Miami. Had a great time, saw some presentations, but you know I like to do it more for the networking and seeing friends and meeting new friends. So a lot of time spent hanging out with people in the hallway and talking.

Bob: Cool.

Brad: Which is I how I like to do WordCamps. So yeah, always a great event. Had a good time.

Bob: Good deal. All right, will we have a very cool guest, but before we get into that, I want to thank our sponsors. WooCommerceis our community sponsor. Of course, you know where WooCommerce is, over on woocommerce.com and all that good Woo going on over there. Way more than you will even be able to handle between the blogs and the marketplace and everything. So check out WooCommerce. FooEvents.com is a sponsor as well. Full event-and-ticket functionality for your WooCommerce site. And we’d like to welcome WPSiteSync, our new sponsor. They let you easily push new products and other content from your staging or local install to your live WooCommerce store.

I'll be telling you more about those later in the show, so let's get right into it. Today our guest is Julia Amosova. I'm hoping I said your name correctly. How are you doing, Julia?

Meet Julia Amosova

Julia: Hi Bob. Hi Brad. Hi everyone. Bob, you did it perfectly. Thanks so much. I'm doing great. It's a nice afternoon here in New York City. Thanks so much for inviting me. I'm really excited to be here.

Bob: Now, Julie, I'm going to have you introduce yourself, but I have your official title as software quality engineer at Automattic and I'm sure there's a lot you do with that particular title. But before we get into the topic, why don't we hear a little bit about what you do over at Woo?

Julia: Sure. I'm Julia, as you already introduced. I'm a software quality engineer at Automattic indeed. I've been with the company for almost four years and I've been working on the WooCommerce division the entire time. I started as a technical support engineer and transitioned into the software quality engineer role just a bit over a year ago. I am a member of the WooCommerce core team, which is our platform team, so to speak. And we mainly work on improving performance and stability of WooCommerce.

Our team also is the team responsible for releasing and testing WooCommerce. In my day to day work, I focus on making sure our users can update WooCommerce without any issues. I do that through various testing efforts and processes that we have in place, which I am going to talk about in this episode today.

Bob: Cool. Now before we get into the testing, I know that next week we're probably going to be talking more about WooCommerce 4.0 because, by then, it will have come out. Can you give us your synopsis of what we can expect? I know we've touched on it before, but it's nice to reiterate all the good stuff coming.

What’s Coming in WooCommerce 4.0

Julia: I'm happy to do that. So next week, as you mentioned, Bob, we plan to release WooCommerce 4.0, which is a major release. It's planned to be released, currently, on Tuesday, March 10th, which is less than a week away. It will include three new features, the first and major one being a WooCommerce Admin, which is currently a feature plugin, but it's now going to be a part of core. We're very excited about that. We're hoping it's going to bring a new, improved experience for how merchants manage orders and work with products.

It brings a lot of new analytic reporting tools to merchants. The next feature, which is sort of new one, is the onboarding experience. If you've been following along in the 3.9 version, we had already included a new onboarding experience, which is basically like a set-up wizard and has been presented to new customers when they start working with WooCommerce. But it's a completely new redesign experience. In 3.9 it was only enabled to 10% of the users and with 4.0, it's going to be enabled to 50% of the users. We learned from 3.9 and now we're moving onto a broader coverage for that feature.

Finally, the third feature to be included for WooCommerce 4.0 is an action scheduler, version 3.0. It's a really big improvement. It's not really usable to the everyday WooCommerce user, but it's something that's responsible for time-based actions that are happening on the sites. For example, subscription renewals that are scheduled in advance or emails that are supposed to be sent out on different actions that a user is making on the site. This kind of functionality is powered by action scheduler and this is a brand new way, so to speak. These actions are now going to be scheduled moving forward, which increases the performance of the store tremendously. If somebody is interested, we have a blog which is at wordpress.woocommerce.com. You can search there for action scheduler and see more of the details of how it works. We published a couple of blog posts on that.

Brad: The onboarding piece you mentioned in 3.9. Some of the updates were only for 10% of the users, I think you said. Then in this release it'll be more like 50%. I didn't realize WooCommerce does that. It's fascinating that you're able to roll out a new feature, an enhanced feature in this case, to a small subset of users to gather feedback. I'm assuming that helps you craft that feature better for broader use once it rolls out. And maybe this is part of what we're getting to in the discussion around testing.

I'd love to hear a little bit more about that because I think that's actually a really smart approach, especially for something as important as the onboarding piece, to get people comfortable with setting up and using WooCommerce.

Julia: Yes, it's pretty easy for us to do on our end. Of course we built it that way, but it's something that's easy for us to turn on for a certain amount of people and increase. We then increase it in the next release and so forth. But you're absolutely right, Brad. We did it when we first rolled it out to a certain amount of people to test it out and see how it goes. Not in terms of looking for problems because we obviously did testing before we included it in 3.9, but more about our users. And our user base changes from time to time. So we'd like to know what kind of stores they're on. How many products they have on the site. How many orders they process. What kind of themes they use on the site, and so forth. Based on this information, we tailor the experience further. It's been a great iteration so far and allowed us to increase it now to 50%. I don't know exactly when are we going to turn it on for everyone, but 50% is a good amount and we're excited about what's coming next and what we can learn from it.

Bob: I just wanted to let everyone know. If you hear a little background noise while Julia's talking, the construction people next door were watching her through the window and waiting till she got online and they said, okay, let's start running the machinery.

Brad: Yeah, that happens.

Julie: Always.

Bob: So if you wonder what that noise is, a little ambiance of real life in New York city living next to construction. So we'll just continue on with that.

Brad: See, I thought it was the team testing WooCommerce 4.0 in the room next door. I just assume that's what it sounds like.

Julie: That sounds about right.

Bob: Well let's dive into this testing. You tell everybody, yeah, get out there and test, but I can imagine all the testing on your end and I'm guessing a lot of people don't even know the extent of that. So let's walk through some of that. I know you wanted to start out with the automated testing. Why don't you touch on that?

The WooCommerce Update Schedule

Julia: Sure, but before we move on to automated testing, I feel it's important to mention a bit about how often we do releases of WooCommerce and some changes that were made around that because it directly impacts how we do testing. Since about middle of last year, in 2019, we decided that we're going to switch from feature-based, WooCommerce releases to time-based. Basically we no longer wait for a certain feature to be built in order to make a release. Instead we decided that we're going to release WooCommerce every two months, no matter the state of the features.

So we're going to include whatever stable version of the feature that we want to include plus the bug fixes. And that's brought a lot of great improvements that we can already see today. If you have been following, you probably notice that 3.7, 3.8, and 3.9 releases were relatively stable. At least that's how we measured them. That's partially due to the fact that the releases are now much smaller and include less features. Of course, it's much easier to test and manage.

So we plan to continue with the same release train of every two months throughout 2020. At this point we already have a schedule of releases for the rest of the year. Of course, it makes it so much easier to test some plan in advance to know exactly what happens. We can plan around events and all the other things that go into day to day work. It's been a really great change for us that we are happy with and we're happy to continue doing the same.

And so it's two months per release, but we have this so-called pre-release process and what it means is that we release the beta version of WooCommerce four weeks before the official release. We have this four weeks to test WooCommerce. If you look at a one year period of time, if we release every two months, it means that we constantly test WooCommerce for about six months per year. So some kind of testing happens every other day. Which gives us much more confidence going into a release. I’ll go into testing soon but thought it was important to mention that fact that we're releasing every two months with the four weeks per release process.

Brad: I love the predictability of that from our standpoint, meaning an agency. We’re obviously building sites with WooCommerce, but we support a lot of existing sites. So having predictability around when new software is coming, we can plan ahead and we can work with our clients to plan ahead. So we know, hey this is coming on this date, this is when beta is and this is when release comes. Because usually we start thoroughly testing with our clients’ sites around release stage. Having that predictability versus years past with WooCommerce or even WordPress, you never quite knew when something was coming. It would just very rapidly get released when they felt it was ready. So I think that predictability, especially for something as crucial as the software that powers your online store, is a brilliant idea. And I love it.

Julia: I'm very happy to hear that. Of course we also had in mind all the developers and theme developers out there in the community who rely on core releases. And I'm happy to hear positive feedback about time-based releases. It's very helpful for us in terms of planning and we can finally move on to the actual testing, right? So with WooCommerce being such a complex ecosystem with so many various moving parts, it's really not that easy to test. Because there are so many different hosting environments available, so many different combinations of plugins that people can use, different WordPress versions, different PHP versions. And it could be quite a different experience for different people.

This episode is brought to you by WPSiteSync. Think about this. How many times do you have to update your content on your WooCommerce site? It could be 5 or 100 new products. Or other content. Often this is done live or you choose to put your site under maintenance. Well, with WPSiteSync and the newly integrated WooCommerce add on, you are now able to synchronize WooCommerce pages between you WordPress sites.

Create those new products or posts or pages. Do your testing to make sure all runs smoothly. Then simply push it to your live site with one simple click.

The product information that will easily sync over includes taxonomies, product images, image galleries, tax settings, shipping classes and pricing to mention just a few. And you can be rest assured that new customers, sales information, products reviews and other data will not be overwritten. 

WPSiteSync supports both shortcodes and Gutenberg blocks so no worries there either. I am guessing you are already thinking about the time and lost revenue that you will be saving. And I’ve test it myself and can vouch for it’s ease of use. So head on over to WPSiteSync and learn more about their WooCommerce integration. And now back to our conversation.

Automated Testing

So we have a list of different testing practices. I'll start with automated testing. There are two main categories of testing that are known in the industry: manual testing and automated testing. Manual testing is something that requires human time and effort to go into the software and perform the physical action of testing it. Whereas the automatic testing is done through automation, right? There is an automation tool available to do all kinds of automatic testing, which means less time spent on manual testing. Our overall goal, of course, is to try to automate as much as possible and only leave manual testing for things that cannot be automated.

When it comes to automated testing, we adopted a testing pyramid, which is an industry standard these days. It was created by Mike Chron, he was a contributor to the Scrum software development method. I'm not going to go into details here. You can Google what a testing pyramid looks like if you are interested, but in a nutshell, basically, in the bottom of it there is a triangle. And the bottom of the triangle is unit testing. And at the top of the triangle is the UI testing with integration and API testing in between. The higher you go up in the triangle, the less testing you should have. The majority of testing should be done through unit testing and less testing should be done on the UI side of things. This is the model we use for automated testing.

Now when it comes to unit testing, we currently have a way to measure code coverage when. For WooCommerce, right now it's 34%. It's not ideal. Ideally, we would like to have it more than that, but it's what we have right now and we've been focusing on making sure that as we develop WooCommerce further we add more unit tests, at least for new features. There is a need for a unit class for any existing features. The developer should ideally write this test and our team was very cautious about this lately, about moving forward to make sure this is done and the unit test is calculated correctly. Sort of like a core functionality or some functions that work correctly. We have this test running on every PR, so whenever somebody submits a pull request to WooCommerce core repository, these tests are being run. That gives confidence to the developer and our reviewer that nothing is broken.

We also have this unit that’s running when a PR is being merged to master, which is a master branch on WooCommerce core repository, which is basically like a project. And the same thing is being run again and nothing is broken. The goal is to make sure that we always see the green light when it comes to unit testing on merge to the master.

Another type of automated testing is something called end-to-end testing. It’s important as part of ensuring WooCommerce is working correctly and changes don't break anything. It automates the manual process of going through all the different customer admin actions on those pages. On the end-to-end test, there are different ways you can start it. One of them is where you can actually visualize it and see on the screen when the browser opens and the mouse starts moving. From the login, to creating a new product, onto the site and placing in the cart and checking out. It's a script replace manual testing. It saves a lot of time and brings a lot of confidence when we release something or need to test it.

The same type of testing runs again on every pull request that developers submit to WooCommerce core. And again, the same type of tests are being run when these pull requests are merged to master, the same logic applies. We want to see all the greens and that means we are happy. And that's the different types of testing that we have when it comes to automation.

I also want to touch base on the work we've recently done around end-to-end testing. We introduced the new framework called puppeteer. It was developed by Google in 2017 and proved to be a better tool than Selenium WebDriver. Your listeners who are familiar with with automatic testing will know what I mean. A Selenium WebDriver is a very popular framework for writing end-to-end tests. We used it before, but decided to migrate our end-to-end tests to puppeteer to make it more stable and easy to maintain. But the main reason why we transitioned into puppeteer is because the Gutenberg project also uses the same tool for writing the end-to-end test suites.

The WordPress core also adopted the same tool. So we wanted to be a part of a broader, bigger ecosystem with this tool because it allows us to use some work that they did on their end. For example, some packages that they released for writing end-to-end tests, or even setting up end-to-end testing infrastructure, both locally and on Travis CI, for example. So it's very helpful when everyone is in the same ecosystem using the same tools. We decided to take advantage of that. And so we went through this process of migration and it's not completed; we have had this end-to-end testing around core since December of last year.

We encourage everyone to give it a try. There is a documentation available; you can find it in our repository in the past folder. And something that we're also working on is the list of flows that we want to automate because right now we have very few flows that we automating, when it comes to testing with end-to-end tests. We're working on this list of flows that we want to automate and we're going to publish it soon. So everyone who is passionate about testing and wants to start, and maybe doesn't even know how to start to contribute to WooCommerce core, this is a really great way to do that. Of course the documentation is available on how to get started.

Brad: That's great. I love it. It's a super geeky topic that I love because it's something that you know, I remember back when WordPress core started integrating unit testing and really promoting the benefit of getting those unit tests in the core. Now I believe it has been for a long time and I would imagine that WooCommerce is probably similar when a new piece of functionality is introduced and that unit testing has this requirement before it will be merged in the core. I assume that WooCommerce is probably the same, right? So everything new going into the core of the software has unit tests. And if you're not familiar with unit tests, it's a very basic test around functionality of what people could expect. So if you have a function that's supposed to return an interger, the unit test just checks to make sure the function returns an integer. It's really meant to test one single thing very clearly.

So just make sure what you're shipping isn't breaking. It's great to hear how thorough the testing is. I'm curious, are there any visual regression tests as I know that's big with our clients. We do an update. There's a lot of automated tools that you can use to essentially look at a snapshot of your website before the update, a snapshot of the website after the update, then compare and look for anything that's different, which a lot of times will indicate if there's a problem. Is any visual regression testing done on your side?

Visual Regression Testing

Julie: We definitely did look at some of the tools available. There are many tools and as you explained, they all work sort of similar. The tool, when you run it, takes screenshots of the existing condition of the site or a baseline of what you want to test. Then on the second run, a day or two later, or a week later, whatever you set it at, it takes another round of screenshots . Then you can compare and take action based on what you see. We did look at it, but we didn't think that it was something that was a priority for us or something that we could actually really test in WooCommerce.

For example, in my opinion, when it comes to WooCommerce.com where we sell extensions, it’s great for this kind of testing because of how the pages are rendered. For example, a product that we have available on WooCommerce.com, when rendered, it's good to check that it renders correctly every day or every so often. When it comes to WooCommerce admin pages, I don't find it very useful at the moment or I didn't find a good use case for it.

We have other tools with WooCommerce and WordPress that are useful, for example, where you can exclude some of the areas to test with different plugins that add different functionality, different fields, different notices. So it's changing both backend and front end of WooCommerce core. And because it's changing so often it's going to be very hard. With something that is more static, like a page on WooCommerce.com, it's a better case in my opinion, which is why we haven't implemented anything yet. But we definitely did look at it at some point.

Brad: Yeah, it's interesting. I do think it definitely lends itself more to the the customization side where people are building their own themes or other customizations. Obviously if something on the backend were to get a little bit out of whack, it's less critical than something on the front end that your customer could potentially see or maybe even prevent them from ordering, which would be the worst- case scenario.

But with a lot of the testing and stuff, I think we spend a lot of time educating our clients around it because it's not something people think about. It's purely under the hood. And by and large, it's a bit of a hard sell if I'm being honest, because we're trying to pitch it as this is something that will ultimately save you time and money. But it's something you'll never know is going on. And if everything's working perfectly, you'll never even care. You know what I mean? So it's a bit of a hard sell.

The fact that we're seeing this in WooCommerce and WordPress and a lot of other open source projects, and people talking about it is doing great from an education standpoint. Because I do believe it's beneficial. I do believe it will save people time and money and headaches across the board to set up as much automated testing as possible. And that way you can identify there's a problem before anybody else does, especially your customers. So it's a really interesting topic and one I always urge our clients to take seriously and one that we can educate them on and talk about the best tests and services based on whatever their site is doing.

But yeah, really cool to hear it. I think that understanding the amount of automation, even manual testing, that's going on at WooCommerce should give people comfort and peace of mind that these releases are coming out in a pretty solid state. Right? It's not two people just kind of poking at it and crossing their fingers. It's like a team. It's a lot of different services and scripts.

With the automated testing, it's been thoroughly tested before, you know that that final release is coming out the door. I didn't even know WooCommerce is doing half the stuff you just mentioned, which is really interesting. And I'm sure there's some topics and posts about it, but I think it does give people comfort to know about all the things happening under the hood that one doesn't know about because it's not visual. Most people that aren't technical would never know.

This episode is brought to you by FooEvents. If you sell any kind of even ticketing online, and you are using WooCommerce, you may have discovered that it doesn’t have an intuitive approach to seamless ticket sales. I know, I’ve been there myself. FooEvents bring more power to your ticketing process. It’s easy to set up and you never have to worry about 3rd party fees again. Not only can you sell tickets, but managing them is really slick. And best of all, they have a growing number of extensions to fit any need you may have.

There are no worries to the amount of ticket you can send. And for the convenience of your attendees, and to help you keep track of the event better, it has a free mobile check-in app. Managing your attendees and events goes smoothly with custom event report and ticket inventory. Export your attendees for additional communication that you may need to send or to build your list. And your attendees can be rest assured that the data remains safe and private.

Whether it’s concerts, school functions, fundraisers, conferences or any number of events that you sell tickets to, FooEvents has you covered. And they have a sweet offer for you, our listener. Simply go to FooEvents.com and get the FooEvents for WooCommerce plugin for an entire year, free, no strings attached.  This offer is limited, so head on over to fooevents.com/dothewoo . And now back to our podcast.

Julia: Definitely. It's actually our goal, like I mentioned at the beginning of the podcast, to make sure that our users, our customers and developers are confident in updating WooCommerce when it comes around. I'm happy to share and talk about our testing practices today to hopefully give a little bit extra confidence to our users.

So there is a whole list of other things that we do aside from automatic testing. This was just how we do it, but I also want to speak a little bit more about some of the other parts and some of the other practices that we have, which hopefully makes everyone even more confident and comfortable upgrading.

Testing with the Marketplace Extensions

Something that I wanted to touch on is how we do extension compatibility testing. Because obviously when we think about testing WooCommerce, we are not only thinking about core because the experience of working with WooCommerce side is not just about core, right? Usually people have a bunch of plugins and a lot of extensions. We cannot test everything out there, but we can test some of the things. For example, in our WooCommerce.com marketplace, we currently have just a bit over 400 extensions that are available, both free and paid. And I would say roughly around 100 of those are developed within Automattic and about 300 of them are developed by third party developers who have extensions available on the marketplace. So in order for us to test them, we obviously can't test everything manually, right?

It's up to each plugin developer out there to make sure that they have testing practices and processes in place in their own project. But what we do have is a script that we developed at Automattic and basically we run this script against all of the extensions. We want to test WooCommerce 4.0 against all of the 400 extensions in our marketplace. It checks the compatibility on a code level. When the script runs, it's relatively fast. It gives us back the results of whether there are any deprecated functions, notices fatal errors, warnings and so forth. That applies to both in-house developed extensions and all the others extensions that we have on the marketplace. It's proved to be a really great tool because it immediately gives us results and we also can contact the developer.

Sometimes, though, this script can check code compatibility but not visual compatibility. Some of the UI may be off sometimes. In the case of WooCommerce 4.0 release, we also ask our technical support engineers team to help us test. We pick some of the popular plugins to be tested based on the new features coming. And we also go over a base functionality to make sure nothing is broken in the UI side of things. We don't do it for every release, the manual testing, but we do it for some where we think it’s important. We are constantly thinking of new ways to automate this because manual testing takes time.

Bob: That seems like a benefit for having your extension on the WooCommerce marketplace. I mean, it doesn't solve everything, but if you are a third party developer, you do have that added coverage as far as when updates are going to happen versus not getting notified, hey, we see something going on here. So that is a benefit they get for being on the marketplace.

Julia: Definitely. We have several channels through which we communicate with our marketplace developers. Some of them are public to all the marketplace. There are some private. But in general, yes, we always try to keep our marketplace developers updated on these kinds of things. We do have certain announcements for them and advance heads up on certain things. So we try to work and collaborate with them as a team. Bcause we're all in it together.

Brad: I just got to say, yeah, that sounds like a big task. I look at stores like the marketplace where you have 400 extensions, 300 being third party. That is a significant amount. But then I look at the scale of things like the iOS store, the app store. How many apps are out there, millions. And they have some type of a vetting and approval and testing process as well. So yeah, I think it's a great point, Bob. That is a nice, extra layer if you are in the official marketplace. But obviously you need to have your own processes in place to test before the release of a new version.

Julia: Indeed. Definitely.

Bob: So do you want to move on? I know that we have a few other things. How about testing on real sites?

Testing on User Sites - The Beta Testing Program of Users

Julia: Yeah, testing on user sites is actually how Bob and I connected and that led to this podcast recording today. We have this program where we offer our users a white- glove service leading up to the release to some of our users where a team member presses the update to the user’s site. We do this type of testing and look for these users through different channels, one of them being the signup link that we include in our monthly, What's New with WooCommerce mailout. I also share the link to the sign- up form in WooCommerce community Slack channel. And also Bob is now sharing this in his newsletters and I have already seen the results that your sharing actually brought to us. So thank you very much for doing that. Really appreciate it.

So yeah, this type of testing is done by our testing Guild. We do have a testing Guild that consists of our technical support engineers who are passionate about testing and we can ask this group of people to test upgrades for us whenever we need it. They perform upgrades on the staging site of the user. They work individually with every user. Once we have their interest in the investment of time, from that point on, they work individually with every customer and ask for a staging site.

Something that is good to mention. We spoke a lot about testing and testing on the staging site is always recommended. Never test on the live site. You always want to make sure you are on a staging site or something like that in order not to disrupt the work of your store. So they request the staging site and from that point on they go through the release testing instructions that is also available on our GitHub repository in Wiki section. You can find release testing instructions there that we publish for every release available to everyone out there to test. So our internal teams also follow the same testing instructions that we released to the public to follow. It's helpful and easier for everyone to know what to test as it could be quite overwhelming.

So they follow these testing instructions and report back on findings and it's a really great way for us to test WooCommerce on different environments because they are usually hosted on different platforms, run different set of plugins, different set of themes and so forth. It gives us a great chance to see what's out there and how our upgrade will perform on different sites.

As a reward for participating in a program, we provide a $200 coupon code that can be used on WooCommerce.com towards the purchase of any extension that you want to buy or any theme in our marketplace. It has no expiration date and we're always looking for users who want to participate in this program. So if you're interested for the next one, keep an eye on those channels that I mentioned earlier and express your interest through the signup form once it's available.

Brad: Is that something that you'll continue to do beyond 4.0 such as the 4.1, 4.2, major releases?

Julia: Definitely. This program has existed since 2017, but about six months ago, we improved it a bit and we are now more intentional about this testing. We target certain sites that have certain features already enabled. For example, in 3.9, we focused on WooCommerce product blocks. We were looking for sites that already utilize WooCommerce product blocks. So for this particular testing with WooCommerce 4.0, we target sites and prefer around 2000 sites that have been using WooCommerce admin plugin because now it's a part of core. We want to see how it acts and what happens once you upgrade to core and beyond. So yes, we're more intentional about this and we are doing more testing sites than before. And yes, we are planning to continue that moving forward.

Bob: Excellent. Yeah, I'm kind of just blown away by how much testing goes on. And I know we have some other things you wanted to touch on and thought I might have you synopsize those a bit because I have a feeling we could talk for a long time on this. So if you want to touch on those and maybe pull out one and talk about it a little bit more, I'll let you run with that.

Testing on Hosting Platforms

Julia: Sure. Sounds good. As Bob mentioned, there are several other testing practices that we have in place. I'd like to mention one and talk a little bit more in details about that one and I'll list the rest. It’s Something we call "hosting and third-party themes testing by randomly selected Automatticians.” It's a very long name. We have created sites that are hosted on different platforms right now: GoDaddy, Bluehost, WPEngine, Dreamhost, Liquid Web and Pressable. And these sites have different sets of plugins and themes installed on them. Some that are popular and that we know our customers use. So we have these testing sites ready to go.

They're not real sites, just testing sites. We also have a script that pulls random Automatticians for testing. We ask this group of people to test on these ready-to-go test sites. And the reason why we do that is because this group of randomly selected testers can consist of all kinds of roles. It could be developers, could be designers, somebody from the marketing team. So we want to make it easier for them, which is why we prepared this test side so they don't have to think about creating one, but instead they can just jump into testing right away, following the same release testing instructions that I mentioned before. It gives us a great way to cover testing, because we're not just looking for bugs.

We are looking for improvements and flows and usability of WooCommerce. These groups of people provide great insights for us because they're often people who don't use WooCommerce daily and they're not always from WooCommerce division. So it's just another type of testing they will also do before the release.

There are some other things around testing that I wanted to mention but won't go into detail about. Obviously, we do testing on WooCommerce.com before the release. We installed a release candidate version of WooCommerce on the staging site of WooCommerce.com because the team that works on that website, they're responsible for building and maintaining it and they're the ones who do the testing. So it also gives us a good idea of how upgrades perform on the larger sites that have thousands of transactions every day, like we have on WooCommerce.com.

Testing the Rest API

We also test the WooCommerce Rest API. An important part of our testing is done through the WooCommerce mobile app side of things because they utilize a WooCommerce Rest API and there is another quality engineer that works on the team currently building test suite to test API to make sure whatever changes we make to the API, that we include them in the releases as well. So nothing is broken for the app or for any other software plugins that utilize the Rest API as well, such as mobile apps.

And finally, something else that is important. We also test upgrades. These are upgrades from previous versions that we know users are still using and that we still support. For example, with us it’s upgrades from 3.5 to 4.0. We also test upgrades with WordPress 5.4, which is currently in beta, I believe it's beta three. So we do all these kinds of testing from all different kinds of angles to make sure that when the release comes, everybody just clicks the button, like in the mobile app, and everything is going well and we're all happy.

Bob: All right, well, wow. Brad, do you have anything to add to that?

Brad: Yeah, I think it's a great topic. It's one that, on the technical side, a lot of us understand. But I think on the end user side, even the store admin side, there's still some education to be had. So I really think it's an important topic to talk about. And one that anyone that's working with WooCommerce can help. Even if you're not directly helping with the testing or building the unit tests or automated testing or whatever.

But just to help share that information of what goes on behind the scenes and make sure that when you click that update button, you know things are going to work. That's what gives people the comfort level that WooCommerce is a solid platform and that they should continue to use it. So a really fascinating episode. Learned a lot and I really appreciate it. This has been a lot of fun and I'm sure we could keep going for another hour or two, but I don't know if Bob will make it. But I definitely appreciate everything, Julia. That was great.

Julia, Sure. I definitely can talk about this for a very long time. And yeah, it's a topic that I'm passionate about, not only because I work as a software quality engineer, but it's just something that in general, I, like most of us, just like when things are in order and working. So I guess part of my personality makes me fit for this job.

Bob: That's perfect. Well I just have a couple of announcements. But before I make mine, I want to have Julia reiterate something through her announcement, which basically ties in to what the heck we've been talking about.

Woo and Other Announcements

Julia: Sure. Guess what? As I mentioned, WooCommerce 4.0 is coming next week, March 10th. It's right around the corner. We encourage everyone to give one final round of testing, whether you're a merchant or plugin/theme developer. We have testing instructions available in WooCommerce repository in the Wiki section. You can find it there and its called release testing instructions. Also, if you want to know more details about what's included with WooCommerce 4.0, you can look it up on our developers blog, which is woocommerce.wordpress.com. All the recent posts are pretty much dedicated to upcoming releases.

Bob: And I just have a couple of things real quick. You should be watching for Profiles in Woo. This is a new piece that's coming to BobWP.com. The reason I'm doing this is I can only have one person on a week for the podcast, but I think there's a lot of people in the WooCommerce space and it'd be great to know a little bit more about them. So I'll be doing some post type interviews on my site and that will give some people a chance for you to learn about them without coming on the podcast.

Then something that's totally nothing to do with this podcast. But if you have listened to my WP eCommerce show that has been going on for four years and started out as the Do the Woo podcast, it officially ended this last Tuesday. That last show was a three-minute explanation of why I'm ending it but bottom line, it’s simple. It was time to end it. Like a TV show. Hey, it's over with. It was good. Lots of good stuff. So, I'm going to be focusing more on this podcast and yeah, that's it. What do you think, Brad?

Brad: That sounds good, Bob. I mean, this is my favorite podcast, so I think that sounds like a good plan. But yeah, let's go ahead and wrap it up. I definitely appreciate Julia being here. Awesome topic. You know, across the board a lot of great information. I want to thank our sponsors once again. First WooCommerce. Our community sponsor. And FooEvents.com. Go check them out. You can sell and manage your tickets like a pro, no ticket fees. A really, really awesome invitation there, as well as at WPSiteSync. Which I have a fun fact. I was hanging out with Mark from SiteSync and ServerPress at WordCamp Miami. So I had a good time catching up with him. But check out WPSiteSync, an easy way to sync your posts and pages and WooCommerce in just a couple of simple steps. So definitely appreciate our sponsors. Bob, anything I missed?

Bob: Nope, I think that's it. Maybe we can let people know how to connect with you, Julia. Anywhere in particular?

Where to Find Julia

Julia: Sure. You can find me on the WooCommerce community Slack. My handle there is Julia Amosova. So Slack is a great way and also LinkedIn, where I watch my messages from time to time. I just wanted to thank you, Bob and Brad. It was a great podcast. I really enjoyed talking here and sharing about WooCommerce. Thanks so much for having me, and best of luck with this show. It's amazing.

Bob: Alrighty Brad, well, call it a day, Brad.

Brad: Yeah, this is great. Thanks, everyone, for listening. Until next time, I'm Brad, he's Bob and we're Doing the Woo.