Why UX is Key to Differentiation in ML
Software 2.0
If you’ve spent time in Tech Twitter or browsed Medium you’ve probably run into the term Software 2.0. To summarize Andrej Karpathy’s essay, Software 1.0 is writing logic via code to achieve the desired output. In Software 2.0, the machine learning model (e.g. a neural network) learns the logic from examples of the desired output.
My personal interpretation of Software 2.0 is that writing complicated and deterministic code to solve a class of problems is too much of a pain/impossibility so we let a neural network learn the logic for us. The output is code that works well but randomly fails now and then.
We’re sacrificing determinism for probabilistic software that is more convenient to build but more difficult to debug.
And there lies the challenge of building products on top of ML models - handling “silent bugs” where the application shows incorrect model outputs with no explicit error messages.
My prediction for the recent wave of ML products is whichever ones have the UX that most gracefully handles these types of model failures will win.
Data moats aren’t everything
Wait hold on Anish, what about data moats?! Doesn’t having the most amount of data to tune your model behavior mean you’ll have the best product?
Obviously, your model to needs to achieve an acceptable level of performance to even build a usable product. Collecting more data and re-training your model on collected user feedback certainly improves model accuracy. But even if your trained ML model achieves an 80%, 90%, or even 99% level of performance, there are inputs that can fool your model before you have the chance to adapt it.
Your product should allow users to navigate and work through model failures so they can accomplish the task at hand. Otherwise, I think you risk losing their trust and their business. A hunch I have is that there’s probably some psychological phenomenon where bad model outputs matter less if you allow the user to discard bad model outputs and continue working with good ones.
A successful user-facing ML product treats handling model failures as a top priority in the earliest stages of product development. Simply put, you cannot just slap a UI over an accurate fine-tuned GPT-3 model and call it a day. I see a lot of people (I’m guilty of this too) over-index on model accuracy and I think it might be because ML Engineers/Scientists tend to start ML companies instead of UX folks.
Examples of Successful ML Product UX
Let’s take the scalding hot niche of AI-assisted writing of marketing copy.
Copy.ai and Jasper are two successful companies in the space largely because of the thoughtfulness of the UX around the creation of marketing copy. Some examples of this are:
Having heuristics that filter out suggestions that have too many repeated phrases
Allow users to choose suggestions they like and generate similar suggestions while discarding ones they didn’t like
Striking the balance between showing different flavors or tones of suggestions without overwhelming the user
Breaking up copy generation by specific use case (e.g. blogs vs social media) so the product can guide the user as they specify what they want to generate
None of these differentiating and non-obvious features are solely reliant on their trained copy generator models. The code and design to support the stellar UX is the moat that makes it difficult for someone to copy Copy.ai (no pun intended) and Jasper. Users love these products because it makes them feel productive and get into a flow state. Building a product that makes users feel like this only comes from a deep understanding of the domain and the user’s workflow.
Source: https://copywritingcourse.com/ai-writing/
The success of ML products such as Jasper and RunwayML has taught me that UX is crucial in building successful ML products. ML product developers should embrace and accept the probabilistic behavior of models and work backward from their to build a stellar experience.
I’m currently putting these opinions to the test in building an LLM-enabled tool to help lawyers draft legal documents much faster. If you found this insightful or totally disagree, I’d love to hear your thoughts!