Currently the digital products we interact with use a one-size-fits-all interface. This mismatch between how technology is packaged up and offered to users versus how it’s used in practice often causes frustration and negatively affects the overall user experience. It adds inefficiency to the user flow, can alienate users e.g. a new user trying to make sense of the product, and can lead to only a small portion of a product’s functionality being used, while the rest goes unused. Ultimately, for the business it causes preventable churn, and they miss out on potential revenue. We think the future of product development is adaptive - a world where products are more malleable and tailored to the individual end user.
What are the problems with product development today?
Product development goes through a process of ideation, prioritization and testing before being rolled out across the whole user base - the typical cycle takes somewhere between 6-8 weeks, and on average 85% of experiments fail. That’s a lot of capital to be deploying into a hugely inefficient process. So why is it inefficient, and how might AI upend the way we do things today?
Ideation
Nowadays most modern teams are data-driven and look for signals from product data and customer feedback helping validate assumptions. But data is often fragmented with sources spanning marketing, sales, customer service, product etc. To battle this, teams have found ways to consolidate different data sources into warehouses, with Customer Data Platforms (CDPs) layered on top, providing a unified view of the customer journey. But even this approach requires lots of data sifting, so teams are also using segmentation tools to group users into profiles sharing similar characteristics. Most product teams will argue these segments are often too broad, and lack product data as they’re managed by sales/marketing (e.g. they look at broader demographic data, not the underlying user behaviors within the product) - even if you hone in and increase the number of ‘profiles’ that exist, managing more than a handful of segments becomes too time-consuming.
Prioritization
When making decisions, and prioritizing which experiments to run, product teams often rely on subjective frameworks (e.g. RICE/ICE), which go some way in quantifying gut decisions against potential ROI (e.g. by assigning measures to the Reach, Impact, Confidence and Effort). In a best case scenario, these frameworks combined with a clear data funnel gives product teams the insight that they need to work out which experiments to run. In a worse case scenario, teams end up with different scores depending on who is carrying out the assessment and you can end up in lengthy discussions with everyone pushing for their own idea, peddling massaged evidence alongside it.
It’s clear, as humans we struggle with decisions. Organizations are tackling this with the introduction of the role of the decision scientist who combines data science (stats, ML, AI) with behavioral science (psychology, neuroscience, economics) as a way to better understand organizational data and make better decisions. Even so, identifying latent behaviors in complex data patterns is something we as humans struggle with but machine learning doesn’t.
Testing
Once a team has identified a problem, a hypothesis or an idea, the next step is to test live on segments of users. The current default for experimentation is A/B or multivariate testing. This approach has been great up until now - it’s allowed teams to better understand how their users interact with products, and has helped teams make iterative product changes based on real data they’re seeing from their users. But could it be better? We’ve already covered some of the problems that come with A/B testing, including the time it takes to get meaningful results, and the risks associated with testing on live users, but testing today also biases towards short-term success over long-term impact. What we mean by that, is product teams often fall into the trap of optimizing an experiment or product to improve the target metrics today, without thinking about the impact it will have on tomorrow. And finally, experimentation today optimizes for the ‘winning variant’ - product teams roll out the same change to every single user. Why is this an issue?
To give you an example, you might test different versions of a product feature change and one version could be very popular with the majority of users but 10% of your users liked the version which was the worst performing overall. By implementing the winning variant into your product across your user base, you’re leaving a lot of value on the table.
So, what does the future of product development look like?
A single product often has an endless set of user journeys, and behavior combinations. We think that the static, rules-based interfaces we know and use today will be replaced by real-time, adaptive user interfaces, accounting for different preferences, needs, intentions and context. But how do we get there?
AI-first CDP + Simulator + Code-based LLMs = AUI
Firstly, to get to this level of hyper personalization, companies need to be able to analyze and act on data at scale in a way that can’t be achieved with the current tools on offer today. This is where advancements in AI (and in particular reasoning), automation and data analysis come in. Data is cheaper than ever, and users are creating a lot more of it. We’re in this weird world right now where legacy software is trying to keep up with advancements in technology, consumer expectations are higher than ever and the cracks are beginning to show. We’re rethinking the product development stack completely, and putting AI and automation in the driving seat.
The AI-first CDP
Firstly, we see a world where machine learning is used to detect latent behaviors in product usage and complex data patterns. Combine this with behavioral science, and teams are able to completely reinvent the way they think about customer segmentation. Instead of relying on high-level, or simplified demographics, teams can build deep, psychographic profiles about their users, and the way that they currently use products. In short, building human intelligence (HUMINT) models augmented by AI to create a clear picture of how their customers interact with their products and why. We’re seeing a space open up for an AI-first customer data platform which is able to look at every combination of behavior to connect the dots.
Predictive modeling and the role of simulators
With richer behavioral profiles, or in future an AI-first CDP, teams can create synthetic users, building a sandbox environment for testing potential product experiments. This synthetic environment can help simulate, or predict, how users are likely to react to changes in a product, based on quantitative data that’s been collected directly from the product in its current form.
Why might this be relevant? As we’ve already mentioned, the average experimentation cycle is currently 6-8 weeks, and 85% of experiments run today fail. Not only does this shorten the process by weeks each and every time, but it gives teams the ability to 10x the number of experiments that they’re ‘running’. In the short term, we expect it to be a way to hack the prioritization process. In the long term, we think it might replace A/B testing altogether. New tools are already coming to market offering a range of ‘sandbox environments’ for modeling out and predicting the impact of changes on virtual populations or ‘digital twins’. We think the simulator approach will yield big changes for product development - we’ve covered how simulators can help experimentation already.
With AI tools powering interfaces, products will be able to react and mold themselves in response to in-product behavior, and predict a user’s intention. We don’t quite have a crystal ball yet…but we’re getting closer. Predictive AI is able to identify precursor behaviors within data to make a prediction around intent. We’ve seen this in some of the products we already use with predictive text or an in-app customer support bot, which when asked a question can gather context from your product activity - both historic as well as from the current active app session - to better support and answer your question.
Automating product development with code-based LLMs
Teams are working with more data than ever and they’ve found hacky ways of simplifying and automating processes with user segments, A/B tests, triggers and if-then logic but it’s starting to buckle. The tools of the future will sit much closer to the raw data stored in warehouses/lakes, and will have a level of agency to edit code in real-time. We’ve already seen tools like Devin launch - and as these tools become more sophisticated, teams are likely to trust them to begin to make adaptations to products based on how users are interacting with them. The benefit, of an AI-first approach is models can build up an understanding over time. They can learn from what product changes did and did not work for particular users to get better at matching product changes to the user’s preferences.
The ‘experimentation’ layer that powers AUI
Product development and experimentation is about to go through a big change - and we’re likely to see legacy tools across CDPs, A/B testing, and dev-tooling evolving. We’re already seeing Product take a more central role in organizations with the rise of product-led sales and product-led growth opening up room for more key product-led infrastructure to sit across the different functions of an organization. The long term is to deliver truly personalized experiences to end users and a more natural way of interacting with technology - real-time adaptive UI. In short, we think the next SAP or Salesforce is going to be the company that can successfully build and implement an ‘experimentation’ layer for digital products. Learn more about how we’re building towards that layer today.