The AI debate is really about free will
AI replacement theory assumes a perfect universe, but we don't live in one!
When I first started in the field of data science, I was so fascinated by ML algorithms and predictive analytics. Like every good data scientist worth their salt, of course I attempted to build an algorithm to predict the stock market. Naively, I had the brilliant idea that I am secretly going to discover the grand equation behind stock movements and crack the code to the future - Retirement here I come.
Gentle reader, was I wrong.
Not only machine learning was not the magic bullet I thought it would be - it turned out predicting the stock market is not all that straightforward - in fact more people get it wrong than right.
You know why? Because markets aren’t deterministic! And neither is the real world (fight me!)
The fantasy of a perfect algorithm
Take for instance this supply chain assignment in my Operations theory grad course. Our final year grade exam was an assignment of running a real world simulated supply chain for a week. The task was to maximize profit while striking the right balance between meeting demand and reducing overstock. At its core, this was a forecasting problem - we had to accurately guess customer demand in order to plan order volume, efficient storage, transport routes etc. I cycled through a bunch of predictive algorithms from a time series to regression and xgboost. And yet in the end, the algorithm that forecasted the best was a rolling 3 week average. Not because a rolling average is statistically more powerful than an xgboost, but because real world data often is more noise than perfect trend.
A decade later not much has changed, AI and all.
AI can now fetch trends faster than you can say analyst and build sophisticated models within minutes. But the fact remains - real data is noisy. The universe is random and riddled with anomalies and exceptions.
And the best algorithms cannot eliminate uncertainty, in an inherently probabilistic world.
More compute doesn’t eliminate randomness
The AI will replace us argument fundamentally boils down to the age old debate of determinism vs free will.
At its core, the determinism argument subscribes to the viewpoint that the universe can be completely explained by a core set of rules and laws. Given the right set of variables, one would be able to explain every occurrence in a universe. This also means if you know the state of the universe at time t you are able to predict the state of the universe at t+1. In short, all events are an effect of previous state / cause - and everything is predetermined. By conclusion, the determinism argument implies, the decisions that we make are not really free choices, but preset actions triggered based on variables that led up to that moment.
For example - We know it takes 365 days for the earth to rotate around the sun. So if we know the temperature of the sun on day 1, through laws of physics we should also be able to derive the exact location of earth and the temperature on day 365. This temperature then helps determine the chance of rain on day 365 - which then determines the chance of you stepping out of the house. So determinism argues the likelihood of you stepping out on day 365 was already determined by the temperature of sun on day 1, which in turn was determined by position of the sun at the beginning of the universe and so on.
Everything that happens is determined by events preceding it
On the reverse, the free will argument argues the universe is non-deterministic and the choices we make our own. Up until the 20th century the Newtonian physics model had little evidence to negate the determinism theory. From planetary motions to atomic matter - all seemed to follow the same basic set of physical equations suggesting the universe was contained in an explainable set of mathematical equations. That is until the discovery of quantum mechanics. Quantum physicists noticed things are far more interesting when objects get very small or large or fast and at those scales outcomes aren’t pre-fixed but more probabilistic in nature. This is most fundamental argument to support the universe being non-deterministic and by conclusion the idea of free-will
At its core the AI will automate everything argument hinges on the assumption that we operate in a perfectly well-defined, explainable universe waiting to be discovered with more sophisticated algorithms and compute. But a non-deterministic universe negates that.
AI is not going to eliminate noise in a inherently probabilistic universe - just surface it faster
We are a statistical anomaly
Take for instance, the odds of life on earth. Studies shows there is less than 0.001% chance of planets like earth and intelligent life existing. And yet we exist in the bottom 1 in 10 million probability curve - on a floating rock paying taxes!
The mathematics can tell you the odds, but in a non-deterministic universe it still can’t dictate what will actually happen.
We fundamentally live in a messy, undefined random universe where things exist beyond mathematical odds and defined frameworks. Entropy can’t be automated.
The future of career ladders
This doesn’t mean I’m an AI skeptic - far from it. AI is fundamentally disrupting industries and the way we work and there is a massive shift happening in career journeys and employability, faster than we are prepared for. But AI cannot automate inherently random real-world systems - even if the market seems to think so. That also doesn’t mean we will not see massive disruptions due to layoffs and cost-cutting. The shareholders demand profits and profits the market gets. But replacing humans with AI doesn’t just magically make systems more better and efficient - it just surfaces the same noisy problems faster (and at more costly rate?). As evidenced by multiple studies showing no noticeable gains in productivity, even though AI advancement is everywhere!
That doesn’t mean every job is safe. Some job are more automatable than others. For instance - If your job is a building a set of rules and enforcing systems based on those rules - especially digital systems - these are the easiest for AI systems to replicate. Examples are data-entry jobs, scheduling, accounting, any job with more than 80% repeatable patterns
However, if your job operates with an inherently probabilistic space, AI is going to help more than harm. The time to write code or algorithms yourself is probably past (although I disagree with this too - working code != good code. But topic for another post) but the best algorithms and systems are only tools that will surface the same uncertainties - faster. The most valuable skill to learn for these roles is navigating uncertainity.
Again, these roles will not involve engineers building the actual predictive algorithms but being involved in probablistic decision making process and making the right decisions in light of these predictions.
Now you might wonder - what happens the day we have AI systems making really really good predictions. Congrats you have discovered the plot of Devs! Let’s just say we have bigger problems as humanity to worry about than our jobs then!
This article has been entirely written by human. Not because I am an AI skeptic, but because I love writing that much
Interesting post, I never considered that AI depends on determinism. You’re right to bring up quantum mechanics as an apparent gap in determinism, but I still don’t see a mechanism for free will; a conscious mind would have to recursively affect the resolution of quantum functions within the brain, or something? The model I loosely hold is that the universe is deterministic at the scale I’m familiar with, though there are technically random inputs at the quantum scale. But you did convince me that those random inputs make perfect prediction impossible (at least with our current understanding of physics). It is funny how the stock market is so hard to predict, I think it’s because participants are always trying new and different strategies for reasons that may exist outside of the actual market data itself. What if it were possible to run a machine learning model that included biometric and psychological data from every trader? I think the results would improve. Thanks for writing this (though I maintain you had no ability to do otherwise)
It is a big leap to say that if the universe is non-determinitic it implies free will. Just because there is uncertainty at the quantum level, doesn't mean our brains are doing anything other than having neural chains of events firing off to produce some effect, based on the current state plus inputs sprinkled with some uncertainty which we have no way to either control or harness as free will. People say that AI doesn't really "think" or "understand" because it is just, oh, just doing something very similar to how or brains chuck stuff out!