I don't think I'm really running this right. I think some market format for this might be interesting and this was worth trying but I don't plan to add new options atm

I think there are excellent first-principles reasons to want trading fees to go to market creators. If it would be interesting I might do a write-up on this.

It would be interesting to me!

This seems pretty weird to me. Control is hard and needs to be meticulously maintained, while chaos is the natural state! It is a bit like betting against the second law of thermodynamics.

Exhibit A: North Korea-style dictatorships still fail to control a lot of stuff, like people watching South Korean movies https://www.voanews.com/a/in-video-north-korea-teens-get-12-years-hard-labor-for-watching-k-pop-/7448111.html.

Exhibit B: Even though we (humans) are in many sense much more powerful than slugs, cockroaches or ants, we cannot completely control them in any meaningful sense even on relatively small scales (ask anyone who's tried to get their house/garden rid of those things).

@MartinModrak I am not the market creator, but there's a tweet from Eliezer on your insect metaphor which I think is highly relevant: https://twitter.com/ESYudkowsky/status/1790050422251761969 (skip to Problem Two)

Control is hard, but the optimization power of individual humans is limited, we can't modify our own neural architecture, and trying to scale organizations of multiple humans runs into massive inefficiencies. An artificial superintelligence with no need to waste time in various ways, much faster thinking, and the ability to develop new algorithms on the fly and modify itself as needed would have much more optimization power.

@MartinModrak I am skeptical of arguments like ‘the 2nd law of thermodynamics is a robust statistical law’ to ‘multiagent equilibria get more and more chaotic’ but even if I did, I think a more chaotic multiagent setup is more likely to end with a singleton, bc it’s a stable endstate that is hard to get out of, unlike the situation where all the agents are competing on tech that might at any point give one of them a decisive advantage. What am I missing?

@Bayesian I think you are making a tacit assumption that AI capabilities are likely to improve beyond all limits in all regards, making the AI basically a god. I think that's wrong - both the AI's pure intelligence and its ability to exert control over physical space will quite likely run into limits and I would bet those are substantial. Under a wide range of such limits, a set of (semi-)autonomous agents can easily prove more efficient and resilient than a single entity.

This also allows some pathways out of the singleton state - if the singleton still needs to provide some autonomy to subunits, it is plausible that some of those units will manage to increase their autonomy (in a weak analogy to cancer).

Another, distinct line of problems is your definition of "everything" - there are many things which are intrinsically hard to control (weather, space weather, bacteria, ...) are you excluding those?

@MartinModrak Not beyond all limits, not a god. There are physical reality limits, there are complexity theoretic limits, stuff like that, and those things don’t seem very limiting in absolute terms, so I don’t see these limits preventing an AI from passing, for example, the manufacturing and technological capcity of a country, or all countries, or all countries ten times over, while being a distributed system that cannot realistically be destroyed. I’m curious why you think the limits would be anywhere close to our current power level.

I think if subunits of the singleton are just programs / other ais, they can be error corrected and have lots of fancy math that makes cancer statisticslly impossible in a similar way to how decreasing entropy is impossible.

And yeah, in this case control everything is one where controling nature doesn’t matter as long as other agents cannot realistically act against you

@Bayesian The limits I have in mind are more of the engineering type: making more efficient computers/turbines/photosynthesis/steel production... I find it completely plausible that in all of those examples the things we see currently in the world are within an order of magnitude or less to the maximum for any practical system (with the exception of computing where 2 order of magnitudes improvement seems not so far fetched).

I do have to admi that upon closer inspection the connection between those engineering limits and the limits on singleton power is weaker than I initially thought.

One extra line of attack is that the speed of improvement in AI (or really anything) isn't limited just by the actor getting better at reaching a next level, but also by the increase in difficulty in reaching the next level. So by tweaking the difficulty curve you can get the (currently still quite hypothetical) AI takeoff to have any shape you like from exponential (the increase in difficulty between successive "levels" is negligible compared to the increase in skill) to linear (increase in difficulty comparable to increase in skill) to basically flat (the increase in difficulty is exponential with larger base than the increase in skill). There's very little information that can be used to guess the relative shape of the improvemnt/difficulty curves (though exponential increases in difficulty are definitely not implausible).

So I don't think you can flatly rule out the possibility of AI never really being that much smarter than a human, which would preclude singleton control over the whole planet - anything at very large scale would have to be performed by groups of AIs putting limits on central control vaguely similar to the limits of central control individual humans can wield.

@MartinModrak

by tweaking the difficulty curve you can get the (currently still quite hypothetical) AI takeoff to have any shape you like

Completely agree there! I think our world is one that appears more likely under some hypotheses than others, and certainly it will ultimately be some kind of sigmoid because power concentration cannot go to infinity. My guess is that it’s pretty exponential for a long time, computers are quite a few ooms away from theoretical power and efficiency limits iirc, but also they can be stacked into a planetary supercomputer if u want to keep scaling, and it’s harder to see a limit on that end. I do think on the last point of that paragraph, that we are drowned with information and the difficulty is in using it correctly.

And yeah to your last point, agreed we cant preclude jt witj certainly like everything else, for the purpose of this market if I am pretty confident it’s possible then id need to be confident it isn’t. Im concerned that too few of these will resolve positively and people will be annoyef so I might put things I’m less confident in. I think it’s plausible that human brains are within 2.5 OOM of theoretical maximum efficiency, but that a future powerful ai will just be the size of 1000 human brains or more while having a similar efficiency, at which point efficiency is not saving u, it’s lnly rly helpful in a species whose intelligence is strongly restricted by the size a head can fit as a child or wtv

@Bayesian Not an engineer, dunno about efficiency of wind turbines and photosynthesis though those are interesting questions. I don’t see them matering signiticsntly to the current question. Computer does matter, but ig power of scale seems to matter more? If current brain is the efficiency cap i still think power differential between humans and ai can be very very very large

@Bayesian Look, I didn't think about this too deeply, so I am just trying a couple options to see what sticks.

One another possible avenue is game theory. I'll assume the invevitability of ending up with powerful AIs as the only strong actors (which I think is far from inevitable, maybe even unlikely, but since you seem to be starting there, let's grant it). Presumably most entities strongly prefer existing to not-existing and have assymetric preferences in that penalty for not existing is larger than the payoff of obtaining several times more resources/power. This is also true for humans, so why hasn't war been eliminated? One answer is that this is because of the security dilemma - we assume everybody else could be preparing for war, so not preparing for war is risky. In fact it is risky not to attack any weaker actor that you plausibly can tackle, because otherwise another actor will do that and become stronger than you. This then makes war more likely. But if we grant the AI's superhuman skills why wouldn't they be able to overcome the security dilemma in some way (zero-knowledge - crypto - satellite whatever) and manage a long-term peaceful coexistence of a multitude of actors? Maybe even agreeing to restore others from backups should something bad happen.

And even if the security dilemma is not completely solved, the AIs may still play the balancing act well enough to produce long-term stable multipolar settings. Note that the AIs know that modern war is so destructive that even winning a war is almost always worse than keeping peace. They won' start a war just based on a whim of one Vladimir... One likely important contributor to starting a war is imperfect information - the attacker believes their gains will be larger than they truly are and price will be lower than it will truly be. AIs could be much better at navigating this.

Decentralization and independence also gets more likely with interplanetary conquest due to communication limitations.

It might also matter how pednatic are you about the meaning of "the most natural and probable" - e.g. if this has a probablity of 2% , while there are 98 competing outcomes with 1% probability each (with whatever messed up carving of the possible worlds into outcomes), is that still "most natural and probable"? Or is it enough to move you from say a 90% probability to just below 50%? Or something else?

@MartinModrak

Look, I didn't think about this too deeply, so I am just trying a couple options to see what sticks.

I am skeptical of this strategy but fair enough, I appreciate your points.

For the second paragraph, yeah, the security dilemma idea makes some sense to me. If there are two advanced AIs with similar capabilities, they probably can overcome the security dilemma, and manage a long-term peaceful coexistence. It may even be easier with more actors, since the balance is more easily maintained. I think my position relies on the fact that reaching that stable equilibrium, with lots of very powerful ais (ais that have reached some physics-based equilibrium where they can't rly be militarily more powerful with the same amount of energy, they're at the limit of the physical laws) is very unlikely, because if one AI gets to a point where it can scale in capability on its own, it reaches the limit, many orders of magnitude above current humanity power, quickly enough that it can stop other ais from reaching that point as well. the outcome you mentioned is still a plausible one, and one worth considering.

For the 3rd paragraph, I agree with pretty much all of what you said, and capable ais that have non-opposite utility functions will probably be able to avoid wars, or only allow it at some small probability

Decentralization and independence also gets more likely with interplanetary conquest due to communication limitations.

That seems true. If powerful AIs are developped in 300 years by the time we've colonized mars, and it takes 3 weeks to go back and forth, that makes stable multi-agent equilibriums slightly more likely. But this seems unlikely to happen, overall.

It might also matter how pednatic are you about the meaning of "the most natural and probable"

Roughly, I'm meaning something like "probability is higher than 30%, and is the thing with the highest individual probability". To account for two separate scenarios:

• lots of possible outcomes, all 1%, and this one outcome is 30% likely; it seems fair to call it the most natural and probable outcome

• 3 possible outcomes, 35%/35%/30%, it seems wrong to call the 30% one the most probable outcome

@Bayesian

A quick extra observation:

because if one AI gets to a point where it can scale in capability on its own, it reaches the limit, many orders of magnitude above current humanity power, quickly enough that it can stop other ais from reaching that point as well.

This is where this line reasoning ties well with the difficulty curve and engineering hurdles. I don't think AIs are very likely to achieve any sort of extremely fast takeoff, precisely because of the engineering difficulties of doing so. A superhuman AI will still not be able to make more intelligent AI (or factory for more powerful chips, ...) just by armchair speculating. At some point, the ideas will need to be tested against reality and/or executed and this will take real resources and time. More intelligence can definitely lead to more efficient experimentation, but I don't think it can obviate this need altogether.

So I find the scenario where one actor just runs waaaaay ahead of the others implausible. In fact, there will likely be feedback forces against this - historically, one actor making a technological breakthrough has lead to often significant, but always temporary advantage as others observe the breakthrough, gaining useful insights about the technology allowing them to catch up with less resources than what was needed to find the breakthrough in the first place. AI development is quite likely to be similar.

@MartinModrak tbc, before some kind of fast takeoff if ever there is one, as long as humans are in the loop and do most of the research, I agree with the feedback forces keeping any one company not too far in front of the others. Historical evidence is pretty useful in judging this to be pretty likely, and infosec only goes so far, and probably won't go far enough in the near future.

As for the first paragraph, I've heard a bunch of engineering ppl say stuff like that, and I'm wondering if, if it was all true, I would be able to notice it so. It seems like clear wisdom on a human scale; that if you're a human, and you see another really smart human go off to do his own thing, even if he's really smart he won't pull ahead of everyone else and end up with more advanced technological innovations or wtv, because everyone gets stuck on hurdles and they really are very hard to overcome, and require lots of people specializing and correcting one another. I am not fully confident that we can reject that analogy when it comes to much more advanced systems, but I think we largely can, and ppl seem to have a strong intuition that the analogy holds, and I suppose my guess is that it doesn't? Bits of evidence aren't treated differently if they're taken as part of some objective sounding engineering test, or if they're just you reading off of wikipedia, or predicting existing data before looking at it, or any other kind of evidence that you can make sense of by generating models that successfully predict it. You cannot obviate the need for experimentation, but you probably can experiment very well if you're clever, even if you're limited in what kind of experiments you can run; which is not to say the ais will be limited in what they can run; if a fast takeoff happens, the ai is likely to be in an environment when it can, in fact, do quick experimentation. Curious where you think this line of reasoning goes wrong.

@Bayesian The main problem is that even clever experiments take time. Let's start just with software: the AI has an idea for a better AI architecture, let's say it is a godlike coder, so the implementation is correct on the first try. Still it needs to train the architecture (takes time and resources) and see how the resulting model fares first in some simple environment than gradually closer and closer to real world. Deviations from expected results requires inspection - if this is a real failure, it must be understood why did it fail (this may require further specialized meddling = time and resources). Or maybe the new architecture is actually right and the expected answer is wrong? Then you iterate. Or maybe you just want to do a gridsearch to find optimal hyperparameters - this is already very costly in terms of absolute real-world time for current models, so not clearly obvious the cost can drop sharply for even more powerful models.

Since completely understanding even very small systems without a lot of experimentation is basically impossible for humans (like even pretty small systems of ordinary differential equations), I find it implausible that the AI will find it easy to understand systems that are by construction more complex than the AI itself. Hence it will (like we do) experiment a lot and thus be bound by available computing resources and pure wall-clock time.

Or you may realize that good quality training data for your task does not exist and so you need to collect/create it.

Things get much worse when you want to introduce new physical technologies. Just building a factory cannot be sped-up idefinitely: concrete takes time to cure, etc. Setting up any complex machine requires tons of iterative testing, tweaking and fiddling at the installation site, because real world is hard and entropy does not like you. Etc... Scaling up production from a lab prototype is usually completely non-obvious and takes time (and may even fail completely) Does that make sense?

@MartinModrak

I guess my model is roughly that the AI looks at its own architecture, and sees lots of redundancies or steps that can be optimized through better interpretability, so like, there's a huge inefficient part of itself that can do few-digit arithmetic, and it removes that part of itself, and replaces it by a calculator that takes a few bits to encode and is much more efficient, and does that on a wider scale for lots and lots of things, and also creates substructures it can prompt, until it has the same capabilities as the original model but on 100x less resources, and then scales and finds further ways to transform its reasoning into code, more or less? the ASI is probably not running on messy neural networks. idk though. I think what you described does seem like a bunch of time spent relearning a bunch of things, and the ai will find a strategy that is not that. Interpretability could be too hard for the ai to be able to simplify and optimize itself sufficiently quickly that it creates a recursive self-improvement loop, but it's hard to say.

I agree creating factories takes time even for a very smart asi, similarly to how the asi couldn't blot out the sun in a year whatever its abilities, if it starts from a lab on earth with nothing to its name or wtv. but it probably can in 10-25 years, do something like capture a large fraction of the sun's energy. not really relevant to this discussion tho ig. but yeah, the difficulty of scaling prod and dealing with entropy and messiness makes sense. further progress on ai might be harder than I expect, even for an entity that can abstract and selfimprove much more easily than any human can.

@Bayesian I think it is illustrative to compare the easiest problems we as humans cannot solve completely from the armchair with the hardest problems we can solve reasonably well through directed experimentation.

Some of the easy problems we cannot solve fully are things like 3-body problem or the Collatz conjecture or fluid dynamics (Navier-Stokes equations). But in all of those cases, we can run experiments/simulations to figure out good practical answers. But those experiments/simulations are quite costly compared to how simple the problem looks. In particular, there is little room for shortcuts, you just have to do the number crunching. Alternatively you could try to rely on some abstractions, but all abstractions leak, so you'll still need to test whether the abstract solution works in practice (fluid dynamics are a good example here - we can do a decent job of simulating and designing for fluids, but quite often real-world implementation hits substantial hiccups).

Now compare the complexity of the previous problems to the complexity of an artificial general intelligence (which we postulate is within human reach). Why should we expect the AI to have substantially narrower gap between "completely solvable by just thinking" and "solvable, but with extensive experimentation"?

For similar reasons, I don't really buy the possibility of AI just self-inspecting and improving without substantial (and thus costly) experimentation. The AI is almost by definition a complex system and predicting all possible interactions within that system is going to be extremely difficult.

On another note - you also seem to assume that the first AGI to be developed will be effectively a single entity, but I don't see why people (or the AI itself) wouldn't run multiple copies. And it is seems likely to me that multiple copies of the same AI won't act as a single actor and instead would develop their histories, relationships and agendas (multiple copies of a single software with different options/setup/tasks is just an extremely normal thing to have).

I think nuclear weapons probably increase the expected time beyond a few millenia. This timeline has clear and drastic anthropic shadow with regards to nuclear weapon use (the way world war ii played out, structure of the cold war, near misses) and I would expect most civilizations to stall technological progress by blowing themselves up repeatedly.

@SaviorofPlant ie technological progress is too hard, and big explosions are too easy, and you can't have sufficient technological progress quickly enough so as to overpower the other agents that have access to big explosions?

If so I think the part I disagree with is the technological progress being too hard. I currently think intelligence enhancements and AI make the world super duper fragile

@Bayesian Current AI is predicated on decades of developments in math, computing and hardware engineering as well as plentiful data from a very open society, and it still took 80 years after the development of nuclear weapons. I think states are irrational enough for nuclear weapon use to be highly probable in that timeframe, and I also expect nuclear weapon use to create worldstates where that type of technological progress is very unlikely.

@SaviorofPlant oh, if AI development is slow enough, and the trajectory is obvious enough, there may come a point where nuclear powers want a nuclear catastrophe over the status quo of technological defeat. It could also be the case that enough states act counter to their own best interests that they just swing nuclear weapons around and that makes everyone explode. I think, though, that overall an international nuclear war is less than 50% likely in the next 50 years? Do you disagree?

@Bayesian There is a stigma on nuclear weapons use as time passes without it occurring, which lowers the probability. Most of the probability mass for a multiparty nuclear war is shortly after development, which for us was ~1945-1965. I think the odds of this occurring are somewhere north of 90% - it seems like a highly atypical case for the technology to be developed, when a war is almost over, by a power which has no intention of using it aggressively to further its own interests besides ending a war it would have won anyway.

That said, I do also disagree with that statement (disregarding AI). I think social media makes it easier for irrational actors to come into power than before, and I expect nuclear weapons use to be more likely in the 2050s than it was in the 1990s.

@SaviorofPlant Ok I reread your original comment and your reply in the comment thread above and is your position that it’ll happen, but like in 20 millenia instead? Ig ‘a few millenia’ depends when u consider civ started, my position isn’t that we’re unusually late or anything, though that might be true, but moreso that from a point like ours singletonness is a very hard-to-avoid natural outcome

@Bayesian Yeah I don't disagree with the point you actually want to make, I just felt like pedantically arguing against the argument as you phrased it.

@SaviorofPlant LOL okay fair enough. In the future I’d encourage you stating that more clearly so I know. Precision is important but as stated in the description i consider ‘changing my mind’ to be more than ‘realize i phrased a thing technically wrong’

@Bayesian Listen, I asked you to create an answer I disagree with, this was the closest thing 😅

If it helps, here are some hot takes you might disagree with:
- I believe Goodhart's Law and outer misalignment are concepts describing the same core phenomena
- I would argue there is a substantial probability of small birds / mammals and large reptiles being moral patients (and a small probability for trillion-scale language models)
- I believe free will is an incoherent, confused way of trying to point at agency, which is plainly false as described
- My p(doom) by the end of 2025 is around 25% (and I expect a significant fraction of surviving worlds to be in an irrecoverable position)

@SaviorofPlant Good takes! I would need to think more about 1), would probably reach a conclusion like ‘i see what u mean, they are connected in their structure but still are pretty different’. Agree with 2), though being a moral patient is probably not an objective thing. 3) is imo obviously true, 4) my pdoom is slightly higher but I dont get what you mean by the stuff in parenthesis