.png)
"Projects" Feature for HERE Platform
Iterative Usability Research for Applying Organizational Structure to Existing Workflows
The HERE Platform is used by developers and engineers to work with, transform, and leverage location data from HERE.
A typical workflow would consist of a user bringing their own data to the platform, combining it with the location data provided by HERE, and using defined processes on the data, called pipelines, that would allow them to use the data to solve their business problem. Pipelines were reusable code that was developed off-platform and then uploaded. While the Platform included a GUI for doing this work, most users preferred to use the HERE Platform SDK and the command line.
overview
As users and their organizations accumulated more assets in the Platform, they soon ran into the fact that there was no easy way to organize their work. Within their “realm” (the workspace of their organization) the organizational structure only broke down assets by type- “Data” and a “Pipelines”- without regard to when, and how they were being used. Filtering by metadata about a particular resource was available, but not customizable.
​
Because the system didn’t support the necessary organizational structure, information about project or development stage was instead included in resource name.
The projects feature was born out of this need to better organize the resources and work done by developers on the platform.
Team
Ashley Callaway, Sr. Design Researcher
Vaughn Alderedge, Principal UX Designer​​​
Aymaan Maat, Principal UX Designer ​
Method
Remote Moderated Usability Testing
Outcome
Suggested redesign provided guidance for policymakers to improve the BCAT for redevelopment, with feedback being incorporated into the new version of the tool.
What is the Design Problem?
​During the initial beta launch of the tool (15 communities around the US), less than half successfully completed the assessment using the BCAT. Our goal was to find out why these teams were unsuccessful, and how we could increase participation and completion rates by improving experience of using the tool.
Who are the users?
The tool is designed to be used by communities of all types, although many of the communities involved in the beta were small and rural.
The users who typically make up a BCAT team include librarians, government officials, internet service providers, tribal leaders, business owners, and concerned community members. They will likely be using the tool for a duration of several months, with use ranging from around an hour total (less-involved participant) to everyday (administrator).
planning
Over the multiple rounds of testing, I collaborated frequently with two different UX designers, the product manager, the product owner under whose broader domain the feature fit in, as well as members of the engineering team.​
I knew from the outset that I didn’t have 100% buy in from the product manager, so I tried to include him in the sessions so he could hear feedback firsthand and have a sense of ownership in the research process.​​
The usability tests were planned to be remote (our participants were widely dispersed around the globe) and moderated, (moderated tests always will provide better qualitative data, but I wanted to be able to discuss and dissect participants’ ways of working, which was so key to understanding this feature).
Another benefit to moderating the tests was that we were working with an InVision prototype, so moderating would allow me to have participants expand on elements that lacked interaction but that they expected to have functionality .
Each test session would begin with a semi-structured interview to understand the participant’s experience using the platform and current workflows. Of particular interest was delving into how they currently organized their work, and what worked well/poorly about their system.
​​
Tasks to test were selected based on discussions with the product team and the UX designer, to make sure we were capturing users’ most common workflows, while testing the critical parts of the Projects feature that we had questions about. The tasks changed between iterations as we learned more and our design evolved.
​
I came onto the project after one round of design and evaluation through usability testing.
​
Once the designer had incorporated the feedback from the previous round of testing and had new designs, I would plan and run the usability testing, analyze the results, and share the insights with the designer first, then with the product team in a discussion-based sharing session, and finally a presentation open to all teams for anyone interested in hearing user feedback.
round 1: projects-first approach
The first round of design and user testing I wasn’t involved with, however I was able to go through the recordings of the tests and the report of the research.
While there were many issues with the design, the biggest problem overall was the radical change to the way people use the platform. It took resources which previously hadn’t had an organizational structure, and assigned them all to a specific project. Every resource (with the exception of certain global resources, such as HERE supplied datasets) now sat beneath the top-level object of the project. This was a problem for users who didn’t want their resources to be buried or limited to being within a single project as they were concerned about findability and lack of global resources.
Projects as main menu item
Shared resources for global resources like HERE provided Data
round 2: resource-first approach
After the first round’s recommendation that users didn’t want their resources buried inside of a project, the information architecture of projects was essentially reversed. Now, projects served more as associated metadata that would help users filter and sort resources to find relevant things to their work, rather than an organizational structure that would group and contain their work.
Projects serve more of a metadata-like function in this design
planning
Participants were recruited from within the HERE organization. We reached out to people who use the Platform as a tool for their work (not members of the Platform team). This group was the readily available, and closely matched the user profile of engineers experienced working on the kind of geolocation data projects found on the Platform (a group that is both difficult to find and expensive to recruit).
​
We had our 6 users complete 3 tasks:
-
Find a list of all projects they had access to, and then find the catalogs associated with one of those projects
-
Create a new project
-
Grant a colleague access to a catalog associated with one of your projects.
Research Questions:​
-
How do users understand projects, and how do users envision them fitting into their current workflow?
-
Are users able to find, create, work with, and manage their projects?
-
Do users prefer projects as a primary object, or a more resource-centric organization?
-
What expectations do users have around where their projects will sit, and where they can find information about these projects?
execution & analysis
The tests were conducted remotely via Webex, with the participant, designer, and myself in attendance. Also invited was the project manager and the product owner, who declined to attend. Typically, I try not to have too many people on the call as I find it can make the participants self-conscious, but I like to at least have a member of the product team invited. It gives them a sense of ownership towards the research, and that makes them more willing to take action on the findings.
​
The product ream was pretty reluctant to participate in the research process, some due to busy schedules, others due to lack of buy-in, so I wasn't surprised they didn't attend. I did share recordings and short summaries after each session to keep them in the loop.
​
After each session, the designer and I would debrief and discuss any interesting findings. Once all sessions were done, I used affinity diagraming to pull out key themes that emerged. For issues, I used a UX severity framework to assign each issue as low, medium, or high priority.
​
Before finalizing the findings in a report, I set up a meeting with the designer to go through everything, get feedback, and prepare suggestions for presenting to the larger product team.​​
​
"I would have different environments of that project as well...sitting within projects."
"I work on a lot of systems, a lot of projects- the way we organize them is we have a tree or something on the left, kind of like folders."
"I would be expecting to have a tab of my projects somewhere...kind of like we have data and pipelines"
results
This design didn’t well work either.
Users really struggled to find where and what projects were. The design had gone too far in the opposite direction- while they were able to have full visibility to their resources, projects was so far in the background it wasn’t useful, it was just confusing.
Users overwhelmingly expected “Projects” to be an object in the top-level navigation, wanting to be able to operate within a project for their work. They saw value in the idea of projects, and wanted to use them. This led to another redesign of the feature, where users would instead operate within the context of a project, but still be able to access “data” and “pipelines” as top level entities as well.
​
​
​
​
​
Other key findings included:
-
The desire to use projects for managing usage/costs, and for managing environments (dev, test, prod)
-
Fear of disruption to their work, especially related to finding resources
-
Permissions/access for projects and resources was unclear and confusing
Full Research Report
round 3: dual location approach
After reviewing and discussing the recommendations from the research report, the designer created a third concept for projects. This time, users had access to their resources AND projects as top-level navigation items.
Users had the option of working with their resources they have global access to, or selecting a Project to work in- meaning everything they do on the platform is within the context of this project.​​This was a huge improvement. For the first time, all the users were able to find projects (a key task to be able to accomplish), and the reception was overwhelmingly positive.
Research Questions • How do users make sense of working within the context of a project? • What do users do when starting a new project? • Are users able to understand the relationship between projects and the resources both in that project and on the platform? • How do users understand the concept of a resource being in a project? What expectations exist around what they can do with that? • What do users understand to happen when a project is deleted? • How do users understand the difference between a linked catalog and a home/created catalog?
adjustments to research plan
The recruitment criteria and pool remained the same as with the previous rounds, as did the methodology of remote moderated usability testing.
​
The tasks were altered slightly to capture the questions the team had about the new design.
Projects Tasks
Projects Tasks
Projects Round 3 Task 1
Projects Round 3 Task 2
Projects Round 3 Task 3
results
​​This was a huge improvement. For the first time, all the users were able to find projects (a key task to be able to accomplish), and the reception was overwhelmingly positive.​​​​
However there were still some key issues that kept recurring:
-
First, our users were developers-they were used to using environments to manage their workflow, and were already doing this in an informal way through naming conventions. Each round of testing, I heard multiple users mention that this was a way they would want to use Projects.
-
The same was found for usage/cost. Users wanted to know how many resources their work was taking up, and what the associated cost was.
-
Based on technical limitations, there was a requirement that every resource had to have a dedicated “home project” it couldn’t be removed from. There was a strong assumption that a user would be able to transfer a resource from one project to another, and the breaking of that assumption caused a lot of frustration.​
-
Permissions were the last piece that users really struggled with. Users were able to give read/write/manage access to a project, but wanted to be able do this more granularly at a resource level. The current system was seen as insufficient to the point where they didn’t know if they would use it.
Full Research Report
round 4: deep-dive on permissions
After we heard how critical getting permission controls right was for our users, and seeing how many issues still existed with the improved design, we decided to do a final iteration of testing just for permissions.
We worked these to allow for more granularity over specific resources. The new design for permissions was built around the concept of policies- a policy was a set of permissions (read, write) that would be applied to a certain type of resource (pipeline, data catalog, template). Due to a technical limitation, users were limited to 5 policies.
Overall, the reception was positive, as users liked having fine-tuned control over their resources within a project. There was confusion caused by the limit of policies, as users both didn’t realize it existed, why it existed, or how they could possibly work around this limitation. In addition to custom policies, there were also pre-made policies for common use cases (Read access to all catalogs, for example). There a few issues understanding what these were, and parsing through possible policies to find what they were looking for. Overall though, it was a vast improvement to simple project level access and permissions, and made projects something that was actually usable.
Study Focus
Overall, the goal of this research was to get user feedback on anew design for managing permissions within a project, whether they are understandable and sufficient for users to manage access to projects and resources
Research Questions
-
Do users understand the additive properties of policies?
-
How do people respond to the 5 policy limit? The 5 group limit? Are they able to find work-arounds using groups?
-
Where do people go to add policies- the policies tab or the members tab?
-
Do people understand what happens when I resource is deleted or removed from the project?
-
Do users understand why certain access isn’t available for linked vs. home projects?​
-
Is there enough information for users to accomplish their tasks? Is anything irrelevant?
-
Do users anticipate using the CLI or UI for this?
planning
Again, to evaluate the updated designs, ​​​we ran moderated remote usability tests using Webex and an Invision prototyple.
​
Scenario: You are a developer working for a big automotive company. You are working on an app that is used in the car, specifically on a search function for this app. A colleague has prepared a data catalog that should be the data source for your search.
-
Task 1 Create a project, add a user
-
Task 2: Create custom policy, add that to a user.
-
Task 3: Give a user who are currently at the max of 5 policies, access to another policy via groups.
-
Task 4: Remove a policy from a user
​
I was able to finally able get support from the account managers to connect with customers for participant recruitment. We had originally had 4 external participants for this study, one of which I ended up dropping for the study because they weren't a fit for our user criteria.
results
Overall, the reception was positive, as users liked having fine-tuned control over their resources within a project. There was confusion caused by the limit of policies, as users both didn’t realize it existed, why it existed, or how they could possibly work around this limitation.
In addition to custom policies, there were also pre-made policies for common use cases (Read access to all catalogs, for example). There were a few issues understanding what these were, and parsing through possible policies to find what they were looking for. Overall though, it was a vast improvement to simple project level access and permissions, and made projects something that was actually usable.
Full Research Report
outcome & reflection
Many of the issues on Projects kept recurring throughout our work because of pushback from a particular product manager. During my time on the Projects feature, the UX team butted heads with them quite regularly, with pushback on pretty much any suggestion or change. Part of the reason for the turnover on the Projects UX team was due to this difficulty in working together.
In defense of this PM, he was facing both technical limitations and constrains in terms of budget and developer resources. I was glad he begrudgingly took the time to do so many rounds of user testing, even when he pushed back on implementing changes based on the results.
We did win some battles. Eventually, we were able to get the ability to copy projects into the roadmap. While this wasn’t an official way to handle environments, it would allow users to create a dev project, and then copy it for separate test/prod projects as it was developed.
Further down the road, there was also additional work done to add a cost/usage dashboard that incorporated project grouping, and while I would have preferred to have this within the context of the project rather than in a separate part of the platform, it was still a win to get users this ability to estimate costs.
User Research often acts in a consulting role- jumping from feature to feature to provide support in talking to users and evaluating designs. It was nice to be able to have continuity in working on this feature over the course of so many changes, and to see the direct impact of my research in how it was ultimately implemented.