Research
Before me, Foreign Affairs had never conducted an usability test. They had marketing research and other data collected from the site but never had someone site down with different types of users to watch them go through the site. When I was first hired, I decided to do a full body check up to discover the pain points and recommend a course of treatment.
Current Users
*All users have had their name changed for privacy purposes
Foreign affairs understood their users demographic but didn't have concrete data about how they moved around on the site. Before the home page redesign, the editors were able to move around the modules based on trending features. I wanted to study the habits of current subscribers so I interviewed 5 different users and asked them to take me through how they used the site.
New Users
One of Foreign Affairs' goals is to appeal to a younger audience. Most of their cliental is pushing 60 and Foreign Affairs fears becoming irrelevant. I selected five new users between the ages of 25 - 40 to see what their thoughts were about the site.
Survey
After the first round of interviews, we decided to imbed a survey into the site to see if the interview and usability results were wide spread or if we uncovered edge cases. From there I synthasised the pain points and analyzed the unexpected habits.
Results
Searching
Both new and current subscribers, couldn't find what they were looking for. The current subscribers new the content was there but couldn't find it or got lost. New users ended up scrolling up and down the home page to try to find something and 2 out of the 5 used the footer as their main navigation tool.
The Story of Current User: Malvin
He always keeps track of the newsletters. When he has the time, he will click on a newsletter's story to enter the site to read the article.
Once he finishes that article, he wants to read another from a previous newsletter but then can never find it. Because he is lost, he has a hard time articulating the actual source of the issue.
I believe that he gets stuck either in the magazine section or on the landing page and doesn't make the distinction between the two.
The Story of New User: Gina
I gave Gina a few things to search for and told her that she couldn't user the search icon but could use the navigation bar. She assumed that the browse channel was a search field and not a drop down menu. She was throughly confused and ended up reading the entire homepage, trying to find the content.
She eventually found the footer, where she preceded to use as a navigation tool.
After another round of tasks, she discovered that the browse has a drop down navigation and she states that she wants them to be broken out in the navigation bar so that it is easier for her to find them.
Layout
There were conflicting feelings about the layout. The big pictures for the articles were very visually appealing to the new users but they all grew tired quickly of scrolling to find the information that they were looking for. After listening to two especially frustrated new users this is what I deciphered about the problems with the layout.
The first image was huge and that was the only impression that the readers got of the site. I conducted the test when trump was newly elected and one user said she wanted to bounce because she was just so sick of seeing his face. The overarching issue is that if readers don’t connect with that one article/image, then they are less likely to want to go deeper into the site.
The large images caused there to be a lot of scrolling. Two new users stated that they were overwhelmed by the length of content but there wasn’t that much there.
Across the board, new users stated that they valued the “Trending Now” Module and wanted it to be the first thing that they saw. This feedback also planted the seed to build out this module and to further study what types of modules readers felt were the most important.
Grouping
The two users that were very frustrated with the layout stated that most of their frustrations were over the organization and lack of descriptions. I think the lack of groupings and labels contributed to a current user’s search issues.
The problem really starts with the fact that users are use to big newspapers grouping their articles. Because this is expected, it is often hard for users to get use to a site that does not group their articles.
One user used the photo gallery as an example of something that needed a label. She didn’t know what it was beyond the title. Also the way her screen was sized, the gallery was too big so she couldn't read the title and look at the pictures at the same time.
Audio
Even though I didn’t ask about the audio, 1 current user and 1 usability user focused on the audio section of the articles. The current user stated that he listened to The Economist articles with his co-workers. The usability user stated that she wanted the website to take more advantage of being on the internet and have more multimedia.
Foreign Affairs just launched their audio section and the data showed that current users were going bonkers over it. It seemed advantageous to build out their media section on the homepage to focus on their audio.
Commenting
Don’t do it! Sometimes user research tells you specifically what not to do. I didn’t ask about commenting directly but when it organically came up, current users value the fact that there are no comments. They feel like comments rarely add value to an article. The editorial and development team were having a year long debate whether or not they should put in a commenting section and I shut that idea down, saving them both time and resources.
Social Media
I always asked usability test participants to think about the last news article that they read before starting the test to help frame the questions. The original five usability testers all started that they found their news on a variety of social media platforms. They would like/follow a news site or think tank, and then would read the stories that popped up in their feed.
When I interviewed current users, two of the users naturally brought up the fact that they didn’t even know that Foreign Affairs had any social media accounts. One was a 26 year old and worked for a solar company so he was definitely more tech savvy than most of our readers. It became clear that we needed to ramp up our social media efforts. They were placed on the back burner because they were not a high converting audience but it became clear with the research that we had to change our tactics to accommodate readers habits.
Our other issue was even if our social media was revised that we still had the issue of a very strict paywall. Non-subscribers could only read one article a month. Current subscribers understood this and therefore were scared to send links to the articles for fear that the recipient was going to hit a paywall. One of the users would print out the article and either mail it or physically give it to his friends to read. It is evident that Foreign Affairs' readers are fiercely loyal and can become advocates for the brand. We just needed figure out a way to give it to them without destroying our business strategy.
Working Lean
The user research didn't stop once we found our pain points and developed a plan for execution. We decided to user lean methodologies for the navigation bar, homepage and the landing pages.
The home page had the highest stakes because it was going to get the most work done, more teams were invested in it and it was the main representation of the brand. We set up four rounds of testing to gauge the direction that we would take with the site. The first two rounds were with new users because we were more focused on layout and had rougher prototypes (current users tend to get caught up in details of rough prototypes). After two tests, we needed to test the content and so I switched to current users.
Throughout the entire redesign, the lean methodology kept us on track. When we wanted to test a new idea, we would just through it into the prototype and didn't have to worry about the high stakes involved with the A/B test. Whenever someone had a question, I would see if data could answer it and if it couldn't I would test the question in the next round of tests. Testing helped me throw out bad ideas, told us what ideas were good, and unveiled further problems that I had to solve.