Amazon Personalize - One-click recommenders
This was a 2021 re:Invent project launch that aimed to create an easily consumable end-to-end Amazon Personalize workflow. This would allow our customers to spend less time mapping their problems to the traditional abstractions of the service and get to their recommendations quicker.
Role: UX Designer II
Company: Amazon Web Services
Duration: 10 months
Tools: Figma, Usertesting.com, Quip
Background
Amazon Personalize is a real time personalization and recommendation service. Personalize uses machine learning and live events from users to achieve real-time personalization in a variety of scenarios, such as giving users recommendations based on their preferences and behavior, personalized re-ranking of search results, and personalizing content for emails and notifications.
Problem description
To onboard on Personalize, there existed a multi-step process, multiple developer APIs to be integrated with, and about 100 hours spend in model understanding, model training, and campaign deployment to build a single proof-of-concept.
We wanted to cut the time it took for a customer to understand use-case to model mapping, and train models to get to recommendations by half. This project aimed at creating a simple one-touch creation workflow that would yield use-case specific recommendations for customers of our customers. Represented below is the contrast between the existing and the proposed user workflows:
The existing and the proposed user flows
Design process overview
This was my first high tier project with the Personalize team, so it was imperative for me to learn as much as I could about my users more than anything else. The process throughout this launch was about being receptive and diving deep into the subject matter, collaborating with the product manager and the engineering leads, and relying on AWS WWSO team to learn about the customers and their behaviors.
1. Discovery and persona definition
Since this was still onboarding on to the service, discovery for this project meant deepening my knowledge of the inner workings of Personalize. Learning about each and every workflows so I could empathize with my users was the first objective. Moreover one-click recommeders were aimed at relatively new userbase that Personalize was trying to target. However, one of the biggest hurdles for the UX team in this service was lack of direct contact with the customers. That is when we as the UX team relied on our SA and WWSO partners who were on the ground speaking to our customers one on one. I conducted a series of informal interviews with these SAs to formulate the proto-personas
Proto-personas
We used proto-personas by synthesizing the perception of customers our stakeholders had. The two main personas we landed on for this project in particular were these:
The main reasons for targeting them was the fact that these users were the ones spending most time in either understanding use cases and figuring out the models to train to solve for it, or were the ones who decided which use cases to solve for in the first place. By targeting the two different persona types we would be able to gather data on how the workflows should be shaped and what product requirements to prioritize for it to be successfully adopted.
2. Collaborate. Ideate. Conceptualize.
By far the most interesting sections in the double-diamond design process, collaborating with the various stakeholders during the ideation phase allowed me to conceptualize the ideas that were of the highest priority, from a requirement and implementation stand point.
The entry into the Personalize workflow, where the user would need to select an industry domain to then get a human-friendly list of use cases to choose from
Along with the necessary changes, I was pushing to make changes to the existing data ingestion flows to help better the onboarding user experience.
Once the user is done ingesting the right data for their use cases, they can quickly create 'one-click campaigns named ‘recommenders’ here.
3.a. Problem to solve for
After the initial deep dive and then testing the initial ideas with our service analysts we got a clearer idea of what domains to support for our customers, and what the best flow for project’s success would be. It also left us with one main design challenge:
How to accommodate the existing flow?
In the traditional flow for the service, the user had to create a resource called ‘Dataset group’. The issue with this concept was it not only contained datasets, but also all resources like solutions, and campaigns within it. It was not only confusing for a user to comprehend such a structure, it also had a few inconsistencies when compared to the APIs. But the users were now trained to use the structure. The other big issue it brought along was it meant creating a new structure to accommodate any new improvements could mean one of two things, either create a separate flow for one-click recommenders and not let the user choose between a mix of pre-set campaigns and custom campaigns, or persist with Dataset groups and creating a fork in the path for the user to choose between both types of campaigns. We went with the latter approach as it meant more flexibility for our customers.
4. Iteration and mid-fidelity testing
Upon validating our initial product requirements, we began to iterate on the designs with some more details and produce some higher fidelity assets to test for usability concerns
Onboarding
We explored a few ways for how would a user onboard to the workflow which would let them start one-click campaigns. We tried to design a guided approach.
During testing we realized that the users would still want to have this co-exist with the traditional workflows so that they could use both the advanced and the on-click version of getting recommendations. And also, to remain within the constraints of the design system, the above explorations were dropped to adopt this.
Project creation and recommender details
Once the user is done creating the project, they were then landed in a new designed overview page. We tried to improve the current overview pages by introducing more guiding blocks so the users could understand better what was required from them next, and how to use the new resources.
Data ingestion
As mentioned before, I kept pushing on the idea of improving the current data ingestion flows, so that user has a more streamlined workflow to from end-to-end. This was also a great opportunity to keep pushing the bar on customer experience.
This flow would allow the user to ingest all datasets in one go without over emphasizing on the optionality of some datasets, and be able to create schemas in a way easier approach.
4.a. Evaluation framework
After narrowing down to a few options from the multiple iterations, we ran into the problem of many for all the improvements that I was proposing the team. The team had been a little to averse to change, since they didn’t have data suggesting that users use console as much. On the other hand, this flow was meant to push the rookie users through to the proof-of-concept stages, which meant these users weren’t advanced API flows and would want to use the interface to get the job done. That meant, the team would want to invest in some if not all the changes, but it needed some logical framework to convince the stakeholders. I used a heuristic scoring mechanism to consider all the factors to decide between the improvements the team would work on.
Example screengrab of the evaluation scoring
5. Launched experience
After the initial rounds of iterations and testing and convincing the stakeholders of some of the improvements, we settled on an approach we were ready to get design sign-offs for. Right then, we ran into some major roadblocks, as some of the APIs were deemed out of scope for the new flows and improvements to be implemented. We had to course-correct, and start redesigning the flows to make room for the new features in the existing flow with even fewer improvements to the data ingestion and without streamlining the end to end experience of one-click recommenders (previously known as one-click campaigns).
Recommender creation
We still wanted the users to be able to create multiple recommenders by just choosing the use cases they wanted to solve for. This allowed for creating multiple recommenders at once, without having to train several models for those use cases separately.
Onboarding
The onboarding remained as is with one minor addition to let the user create a domain-specific dataset group. Which would let them create recommenders related to those particular domains
Improved overview landing page
Though the initial flows could not be improved upon, we had to make way for recommenders and custom solutions to co-exist. So we created a ‘Getting started’ guided experience for the users to know exactly what the information hierarchy is. User validation through usertesting.com helped in big way to land on this final iteration.
6. Post-launch cascade
Mainly, two things of importance happened after the launch. First, the adoption of these ‘use-case’ driven flows picked up gradually but steadily since it started to satisfy the non-traditional, non-expert Personalize users, mainly because it cut down their time to turn out experiments by almost 6 to 8 hours.
Second, it added a bunch of new items to the product roadmap. Since we realized, the whole hierarchy of resources was confusing from a users perspective, it led to a UX-led effort to improve the information architecture of the service. Teams also realized that the data ingestion flows were important to be fixed and that led to multiple spinoff efforts to not only fix the ingestion flows but also add data insights and analyses as future improvement project. We also identified how and why attribution is important for our decision making personas, and started an initiative for users to measure the business value of Personalize generated recommendations.
Also, getting UX a seat on the table as an able and reliable ally was a nice little cherry.
Challenges along the way
Of the many teams that I have worked with, Personalize continues to be one of the more challenging ones. The service like many others in AWS is a heavily engineering-first team. To have a seat on the table for UX is as big a challenge as any I have faced. Through all this one thing has helped me to be true to my users and help elevate the experience for them - and that is to earn trust with my stakeholders. The more I collaborate, the more I have been able to highlight why UX is important. Though the belief system within the team has always been to fail fast and learn and implement the next thing; UX team is trying to help them fail fast and learn and implement the next thing but in a less costly way, by making sure the users at the end of the day are more delighted than before.