How We Prioritize Technology Feedback and Projects

In May 2014, we launched a new version of our website, mountaineers.org 2.0. Over the last six years, we have worked to make improvements and add features, considering the needs and experiences of our volunteers and members first. Here we pull back the curtain and give you insights into how our technology projects and resources are managed and prioritized.

Table of Contents

• Overview
• Managing Feedback Ideas in UserVoice
• Project Ranking System
• Project Ranking Process
• Project Ranking Team
• Project Stakeholders
• What's Agile Development?
• What’s a User Story?
• What's Planning Poker?

Overview

To help manage our many technology needs, we have a small team that meets quarterly to prioritize small projects, which are generally completed through our support contract with our developers. It provides 15 hours/month to fix bugs and make small improvements. We plan larger improvements annually and present these to a larger group for review. Anything classified as a "project" typically requires significant time to design and implement, and those are completed in the allotted "development iterations" throughout the year. Typically we have four iterations annually, but technology improvements were reduced significantly, along with all other Mountaineers programming, during the pandemic.

All technology projects deemed high priority are added to our internal Technology Projects: Planning & Prioritization sheet for group review. These are pulled from our UserVoice feedback system, where all ideas are submitted and tracked by our technology team. Members can be vote and comment on all ideas in our feedback system.

All improvement requests deemed big enough to be projects are assigned to one or more stakeholders, and a ranking team reviews all ideas and projects annually to assign priorities to new ideas, verify existing priorities and ranking scores, and assign ranking scores to newly added projects.

You can learn more about each of these individual pieces and how they fit together in the sections below.

Managing Feedback Ideas in UserVoice

We adopted the UserVoice platform to help us manage help requests and feedback ideas. All emails to info@mountaineers.org are added as "tickets", and items submitted via feeback.mountaineers.org are known as “ideas."  The orange “?” icon at the bottom right corner of every page on our website allows site visitors to add tickets and ideas. Look for it now!

Our Member Services Team reviews and responds to tickets Monday through Friday, with a goal of responding to each ticket within 48 business hours. Tickets are generally of a more personal nature (e.g. a member needs help with course registration) and, therefore, are directed right to the member services team.

New ideas are reviewed as they are added, and are given a status in our feedback system, which is communicated to the person who added the idea and anyone else who has voted for it:

• None/No Status: Newly added ideas that have not yet been reviewed. Any ideas for broken functionality or errors are converted to a ticket and immediately investigated—this is usually rare. Any ideas that are identical to an existing ideas are merged with the existing idea. 
• New: Ideas that need to be reviewed by our team and prioritized.
• In Review: Ideas that are likely to be “high priority” once enough information is gathered to fully define the idea and its benefit, or ideas that are high priority based on their benefit but need better defined specifications to estimate their cost.
• High Priority: Ideas with a significant enough benefit to our organization to justify an in-depth review and further prioritization within our development iterations or monthly support. These ideas are added to our internal Technology Projects: Planning & Prioritization sheet. We typically keep this list to typically 25 ideas or fewer.
• Medium Priority: Ideas with benefit to our organization, but not as high a priority as other ideas and/or not enough benefit to justify the cost.
• Low Priority: Ideas with only a small benefit to our organization and not worth the cost.
• Scheduled: Ideas that have been deemed the highest priority and have an expected implementation time frame.
• In Progress: Ideas that are underway and should be completed soon.
• Long-term Project: Ideas that are ongoing projects or issues or that have a very long completion horizon. These are often neither technological in nature nor easily solved with improvements to our technical systems (e.g. creating better lighting in the Magnuson Park parking lot).
• On Hold: Ideas that were started but are not able to be completed until after whatever is blocking their completion is resolved or for items we think are high priority and need reviewed but “on the back burner” for now.
• Canceled: Ideas that we no longer need or were subdivided into smaller ideas.
• Declined: Ideas that do not fit or are in conflict with our mission or our strategic initiatives, or ideas that are untenable.

All ideas deemed high priority are entered into GitHub, the system where we manage monthly support and development iterations (projects) with our developers. Projects with significant effort may require more than one item in GitHub. GitHub assigns a number to each entry, and those “GH numbers” are appended to the idea’s title in UserVoice. Note that many of the ideas in UserVoice have an “SD number.” This refers to user stories in ScrumDo, the platform in which our developers formerly tracked projects. We will move projects from ScrumDo to GitHub as needed.

Project Ranking System

High priority ideas are added to our internal Technology Projects: Planning & Prioritization sheet as projects and we determine an overall rank based on the weighted rank from five factors. Ranking scores range from ten to one, with a higher score indicating higher importance. The items with the highest score are scheduled first.

Strategic Alignment (35%): This indicates how well this project will help us achieve one or more of our strategic initiatives. The more critical a project is to achieving one or more strategic initiatives, the higher score it gets. We also consider infrastructure projects like platform upgrades in the “strategic alignment” rank. These are projects like upgrading the Salesforce Nonprofit Success Pack application from NPSP v2 to NPSP v3, and our website platform from Plone 4 to Plone 5. The more critical it is that we make an upgrade—security, alignment with current best practices, etc.—the higher the score.

Revenue & Cost Savings (15%): We determine if this project will create a new source of revenue, increase an existing source of revenue, or decrease costs and/or staff time. Added revenue generation can come from a revenue increase in things like memberships, donations, book sales, course registrations, and lodge reservations. Cost savings mostly come from things that save staff time or eliminate third party systems (or fees from them) that would no longer be needed.

Intangible Benefits (25%): Intangible benefits are likely the most subjective ranking, and consist of things that make life easier for volunteers or members and/or get lots of interest and support. A high rank indicates a project is likely worthwhile despite having little to no projected financial gain or savings.

Risk (10%): Risk is a bit counter-intuitive—high rank means low risk. The more complex the implementation of a project is, the more risky it is. Ill-defined projects are more risky. The more effort (and cost) to complete a project, the more risky it is. Adoption is also a potential risk—will the enough of the group for whom this was done actually use it? Will the desired benefit be achieved? Is the technical implementation enough to achieve the benefit or does more need to be done?

Cost to Implement (15%): This rank is determined directly from the estimate of the project’s effort and cost. The less effort and cost of a project, the higher its rank. The table below lists the rank for the cost to implement a project based on estimates that our developers determine from the agile development process we use. Each project is composed of one or more user stories, and we estimate the difficulty of implementing each user story by assigning a number of points with a process called, planning poker. The number of points is translated into time and cost to implement.

Ranking Process

We use a fun method for determining the ranking scores! It’s a lot like how our developers run planning poker to estimate the difficulty of our user stories. When we meet, everyone gets 10 numbered cards. For each project and for each ranking factor, each person chooses a card with their estimate of the rank. If there are four of any one score and the distribution is tight, we go with that score. If the distribution is wide, we discuss.

Before the ranking team meets, each team member reviews ideas in our feedback system:

• New Ideas are reviewed looking for ones that should be High Priority. We need to give all of these another status, but the primary goal is to find the ones that should be High Priority.
• Medium Priority and Low Priority ideas are quickly reviewed looking for any that should be High Priority.
• On Hold Ideas are reviewed looking to see if we can unblock them, if it’s time to change their status to “In Review,” or if it’s time to de-prioritize them.
• High Priority and In Review ideas are reviewed looking for ones that should be medium or low priority. Ultimately we need to cull the High Priority list to roughly 25 ideas.

Ranking Team

Our ranking team consists of staff  from our Programs Division and our Books Division, our CEO, and our IT Manager. This enables our ranking team to be:

• Small enough to quickly and efficiently determine a score each of the four more subjective project rankings.
• Diverse enough to understand the needs and scope of the proposed projects in light of our organization’s overall needs and strategy.
• Balanced so that our entire organization is adequately represented.

Stakeholders

Stakeholders are critical to successfully completing and implementing a project. We always assign at least one staff person to be a project’s stakeholder and often one or more volunteers. Stakeholders must be able to:

• Provide information needed to define and prioritize the project.
• Provide information as requested and within 24 hours during implementation.
• Test the new or improved features during implementation.
• Manage the development of content needed once the features are live.
• Develop or update “how to” pages needed to support use of the new features.
• Communicate the new features to constituents.
• Facilitate adoption of the new features.

What's Agile Development

By definition, agile software development is a group of software development methods based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams. It promotes adaptive planning, evolutionary development and delivery, a time-boxed iterative approach, and encourages rapid and flexible response to change.

There are many specific agile development methods. Most promote development, teamwork, collaboration, and process adaptability throughout the life-cycle of the project. Agile methods break tasks into small increments with minimal planning and do not directly involve long-term planning. Iterations are short time frames (time-boxes) that typically last from one to four weeks. Each iteration involves a cross-functional team working in all areas: planning, requirements analysis, design, coding, and testing. At the end of the iteration a working product is demonstrated. This minimizes overall risk and allows the project to adapt to changes quickly.

Our use of the agile process involves two-week cycles, called iterations and often referred to as sprints. An iteration is composed of 50 hours of work by our developers, typically 40 hours are spent in the first week and 10 hours in the second. In the first week, we choose and implement several user stories, system requirements written from the user’s perspective. Choices are based on the user story’s importance, dependence on other user stories, and effort to complete the story. During this week, we meet daily to review progress and resolve issues. In the second week, our developers test and debug the functionality, and demonstrate it to us. Then we test it, report issues in a feedback meeting, and create tickets in a support system for “bugs” that need to be fixed.

Learn more about agile software development.

What's a User Story?

User stories are brief descriptions of a proposed feature of a software system written in informal, everyday language. User stories describe the user, how they will interact with the system, and the benefit they will derive from it.

An example: As a member or guest with a website account, I want to reset my password so that I can get back into the website if I forget my password.

Learn more about user stories.

What's Planning Poker?

Planning poker is a fun, gamified, consensus-based technique used to estimate the effort to implement a user story. Developers discuss the user story and ask questions of the product owner to understand the needs and benefits described in the user story. When the discussion is complete, each team member privately selects a point estimate (0, 0.5, 1, 2, 3, 5, 8, 13, 20, etc.) that represents the relative effort to implement the user story. Then the "dealer," known as the scrum master, calls for all estimates to be revealed. This method minimizes anchoring cognitive bias, whereby an initial piece of information influences subsequent decisions. 

If all the estimates are the same, that estimate is recorded on the user story. If not, the team discusses their estimates. Those who estimated high and low share their reasons. After some discussion the team agrees on an estimate for the user story.

Learn more about planning poker.