Enjoying this conversation on the evolution of software. One thought on: "People want reliable software" and "we want deterministic behavior in our software." IMO they are different things, but depends on your definition of "reliable." I'd say people want "software that does what they want it to do," which is going to become a looser and looser concept over time. In some cases (fintech), the rules are stricter, and LLMs are not only probabilistic but objectively fail at following strict rules. Humans have lived without determinism in practice for a long time, so I think we're okay with handling that :)
Yeah, reliable may have been a loaded word in the SRE context but what I really meant was deterministic...but people want software that's highly available with respect to up-time as well!
That said, I actually don't think that people will be looser about wanting deterministic software. I agree they want it to do what they want it to but they'll always want that thing to be deterministic unless some data changes about the process or them.
Maybe we’re just arguing semantics but I don’t think people care about determinism (same input equals same output) as much as predictability (it behaves according to expectation of the users and programmers). For predictability I think you just need strict rules that are actually enforced, which a probabilistic system is in theory capable of doing (e.g. it took a while but now chatgpt can output json every time, which wasn’t the case for a while)
Yeah, that's kind of fair. A probabilistic system that outputs 99.99999% of the time consistently is approximately deterministic. The question is how often that can be guaranteed for a given input and if you're in a highly regulated environment that 0.00001% can be consequential (which is really what I was saying). Moreover, when you have an agent that's executing multiple operations then the probability of an inaccurate sequence starts to blow up.
Yeah super agree. For example, as a user transferring money on Venmo, I believe in there's probably a 99% certainty that the money I send to a friend will get to them (maybe I'm more skeptical than most). In the 1% chance that it doesn't, I have a reasonable certainty (80-90%) that I can contact Venmo support and get it resolved. There are always exceptions (notable case where someone put "ISIS funds" as the memo and Venmo locked up the funds), but my point is that even for something as basic as a money transfer, as a user I still have uncertainty in the outcome. But I have enough certainty that it doesn't shake my faith in the system at large.
That said, on the Venmo platform side, they probably look for *much* higher than 99% certainty for transfer success, but even then there is still room for error and human intervention is required.
Maybe it's about being able to explicitly say what error bounds you want. E.g. for transfer fraud , maybe you care that the false positive rate is < 0.1%, whereas for transfers debiting one account and crediting another, error rate should be 0.00001%, and sending a success message error rate is < 1%.
So when we say "approximately deterministic" we're saying it behaves according to expectations, subject to how important those expectations are.
Enjoying this conversation on the evolution of software. One thought on: "People want reliable software" and "we want deterministic behavior in our software." IMO they are different things, but depends on your definition of "reliable." I'd say people want "software that does what they want it to do," which is going to become a looser and looser concept over time. In some cases (fintech), the rules are stricter, and LLMs are not only probabilistic but objectively fail at following strict rules. Humans have lived without determinism in practice for a long time, so I think we're okay with handling that :)
Yeah, reliable may have been a loaded word in the SRE context but what I really meant was deterministic...but people want software that's highly available with respect to up-time as well!
That said, I actually don't think that people will be looser about wanting deterministic software. I agree they want it to do what they want it to but they'll always want that thing to be deterministic unless some data changes about the process or them.
Maybe we’re just arguing semantics but I don’t think people care about determinism (same input equals same output) as much as predictability (it behaves according to expectation of the users and programmers). For predictability I think you just need strict rules that are actually enforced, which a probabilistic system is in theory capable of doing (e.g. it took a while but now chatgpt can output json every time, which wasn’t the case for a while)
Yeah, that's kind of fair. A probabilistic system that outputs 99.99999% of the time consistently is approximately deterministic. The question is how often that can be guaranteed for a given input and if you're in a highly regulated environment that 0.00001% can be consequential (which is really what I was saying). Moreover, when you have an agent that's executing multiple operations then the probability of an inaccurate sequence starts to blow up.
Yeah super agree. For example, as a user transferring money on Venmo, I believe in there's probably a 99% certainty that the money I send to a friend will get to them (maybe I'm more skeptical than most). In the 1% chance that it doesn't, I have a reasonable certainty (80-90%) that I can contact Venmo support and get it resolved. There are always exceptions (notable case where someone put "ISIS funds" as the memo and Venmo locked up the funds), but my point is that even for something as basic as a money transfer, as a user I still have uncertainty in the outcome. But I have enough certainty that it doesn't shake my faith in the system at large.
That said, on the Venmo platform side, they probably look for *much* higher than 99% certainty for transfer success, but even then there is still room for error and human intervention is required.
Maybe it's about being able to explicitly say what error bounds you want. E.g. for transfer fraud , maybe you care that the false positive rate is < 0.1%, whereas for transfers debiting one account and crediting another, error rate should be 0.00001%, and sending a success message error rate is < 1%.
So when we say "approximately deterministic" we're saying it behaves according to expectations, subject to how important those expectations are.