Scaling Academia.edu’s AI grants tool into a multi-product suite with 12% conversion lift
Feedback from beta users
Our beta proved users wanted this tool. This phase was about building something worthy of that trust.
The proof of concept for Grants AI had a quarter-long beta phase to demonstrate signal with our Academic user base.
We exited the beta with enthusiastic user feedback. Our NPS survey received an 84% response to I would be disappointed if this tool ceased to exist. Additionally, 93% of beta users rated the generated proposal content 4-5 (out of 5) stars.
The project’s early success was rewarded with expanded resources to develop the grant tool further. The team immediately focused on growing our database of grant opportunities and improving match relevancy.
60-second summary
The problem
Now that we had demonstrated that users wanted this tool, our attention shifted to improving the overall value. Value perception was important as we looked to define and stress test our monetization strategy.
Our possible monetization paths were: be added to the existing suite of Premium tools, launch as standalone product, and/or position ourselves as a B2B solution.
Key decisions
AI remained the foundation of the experience, but the goal was no longer to turn one proposal seamlessly into many. Instead, we made sure users felt in control of all decisions and actions.
We continued to see higher engagement with the discovery portion of the flow. Due to monetization testing constraints and our limited resources, we decided to decouple the discovery and writing tools from each other.
My role
Engineering resources grew following our beta, but I remained the only designer supporting the Grants tool.
In addition to adding net-new features and functionality, I addressed polish and quality. This included UI and UX updates that aligned the tool with the rest of the Academia products and general clean-up where speed had taken precedence during our period of rapid experimentation.
Impact
Our focused attention on the Grant Opportunity Finder led to a 11x growth in users. Continued iteration on the new opportunity-specific content saw a 3x increase in page views when we added personalized eligibility content. The weekly funding match email remained a successful touchpoint, with a 25% bump to CTR.
Finally, early monetization testing focusing on B2C strategies achieved a 12% conversion rate.
Learnings that defined our next stage of development
Change our approach to AI
A key takeaway from our beta user interviews was a general apprehension about generative AI.
This learning made us rethink our holistic approach to AI within the grant tool. Initially, we wanted the tool to feel like magic:
Upload one of your past grant proposals and…POOF! We found new grants and prepped the proposals for you!
Our grant writers did not share this desire. Their ideal tool helped to streamline the process, but they remained firmly in the driver's seat.
Prioritize grant discovery
Engagement was highest for the Grant Opportunity Finder (GOF) and we were seeing huge CTR on our weekly emails presenting new grant opportunity matches.
However, we were seeing slower adoption of the proposal writing features. Our original hypothesis, which we later confirmed, was that users were more comfortable with AI managing their grant search than with adopting it for their writing.
This insight led us to disconnect the grant discovery piece from proposal generation and start iterating on them independently.
Solve for eligibility
Funding opportunities can have many requirements that researchers must meet. Parsing these requirements and understanding eligibility surfaced as a major pain point for our researchers.
A dedicated details page for each opportunity was always our goal, but out of scope for the MVP. When architecting the next phase of content, I developed short- and long-term plans for communicating eligibility to our users.
What changed structurally?
Almost everything.
In the post-beta era, every step of the experience was questioned. Rapid iteration and testing was our process. The team was comprised of a PM, 4 engineers, a half-time data analyst, and 1 designer.
Changing our approach to AI
Our users quickly embraced the benefits AI brought to the searching phase of the funding process. Yet, almost all our users shared some version of the following sentiment:
“The writing piece feels more personal. I've spent years developing how I communicate my work.” -Karl G.
This contributed to a shift in our thinking about how AI was presented in the tool. AI remained the foundation of the experience, but the UX left decisions up to the user.
Furthermore, we adopted a mentality of showing our work. We wanted to build trust with our users by allowing them to check the information we presented and understand exactly where information was pulled.
We shifted away from thinking “AI-first” to a model that presented AI as a partner.
Human-AI
Do it with me
AI-first
Do it for me
Examples of showing our work
Control over content generation
One test we ran to address AI writing concerns was to introduce recommendations as an opt-in experience. The first step of generating a proposal section was to give users control over what recommendations are incorporated. Users selected from the list of recs paired with explanation and context.
In-line citations for all informational content
All grant content and AI Grant Assistant answers included citations for where the information was sourced.
The inclusion of citations also helped us reduce hallucinations. Unsupported outputs were flagged before they ever reached the user.
Original funding opportunity source link
A routine behavior we observed among new users was visiting the original source to verify our content against the primary source. When satisfied with what the tool surfaced, they would no longer click on the original source link.
Grant AI as a multi-product suite
Our acquisition and monetization tests were much more successful when we decoupled GOF & proposal writer and presented them as separate tools.
The roadmap and monetization strategy of the two products started to diverge. As a team, we decided to deprioritize work on the proposal writer product while prioritizing the B2C strategy. The proposal writer was determined to be more impactful for a B2B audience.
CTR for weekly match email
GOF unique users
11x
39%
A 25-point improvement from our beta CTR
Growth post-beta
Adding grant details & eligibility
We wanted our tool to provide all the details about a grant call. Most importantly, we wanted to outline eligibility requirements and proactively assess whether a researcher meets them. This would give them the clearest answer to the question, “How likely am I to get funded?”
On the side, the AI Grant Assistant is also available to answer additional questions and connect their research to the funding opportunity.
Detail page views
Chat use
3x
30%
Increase to page views when we added personalized eligibility content
Page visits that include a chat session
Improving polish & quality
The MVP was built to test an idea. This new phase was about building an experience we could ask our users to pay for.
UX improvements I prioritized:
Revised language and naming conventions
Refined interaction states
New sidebar navigation
Cleaned up loading states
Optimized mobile experience
Conversion
+12%
When monetization tests kicked off, we were able to grow conversion by 12 percent during 1 quarter of iteration & refinement.
One improvement I made was to rethink the navigation pattern for the tool.
I developed a pattern that could handle visualizing secondary and tertiary feedback patterns and allowed me to flatten the total number of pages in the experience.
Version of GOF at the end of the beta (quarter 1)
