Watching AI Accelerate as a Regular CS Undergrad

My twitter feed has become overrun with a lot of talk of AI. It floods me with papers, new and slightly “old” that anyone who wants to try to keep up with AI advances should probably read. I do this for fun nowadays, not as a serious student looking to get into AI work in the future (assuming doomers are wrong and we’ll get to live till ripe old ages). I am mostly ignorant of a lot of the amazing work that other people have done; I feel so grateful that a lot of people will publish their work openly and that I even get a chance to read, but it is also a brutal reminder that just because it’s there doesn’t mean I am ready to read it. It can be very easy to get overwhelmed with this information and not know what to do.

Is it that I don’t find working with code and building applications intrinsically exciting to me, or do I just like solving problems defined in neat little boxes? Since the only real reason I decided to major in Computer Science was for the money and to live a comfortable life, to not do any of the sort of hard labor I grew up watching my parents do in a small business restaurant, I think to myself that I should try to be as competent as I’m willing to put effort into it while maintaining time and energy to pursue other things that are actually intrinsically interesting to me. I think it’s important for all people to think and talk about it with other people about literally all the things that you do right now; (within reason) are they things that you do because you have to do them and in hindsight you’ve just gotten good at them so you only like it because it’s good and maybe you tell yourself since you’ve already invested so much of your time into it you might as well learn to like/love it or it’s something that intrinsically excites and inspires you to apply yourself hard at it? And of course, ask yourself, do you even care about asking this question? Do you need to question these things about yourself if everything is going smoothly anyways? That’s for you to decide; although just because things are fine now doesn’t mean you should look to see what is coming.

A good question to ask is, how much of your current work is or can be done by next token prediction? Are you doing secretive work that has not managed to be aggregated in the training datasets? Maybe that won’t even matter now or in the future; GPT-4 and beyond will be general enough and have a context window long enough (wow 32k tokens seems like a lot) to match your current cognitive cache and load in whatever you want it to process and spit out, relying on its massive “knowledge base” of parameters. It is raw computation that dominates. As can be found in The Bitter Lesson by Richard Sutton, GPT-4 has leveraged the transformer architecture and RLHF (aka exploitation of workers to make it as positively human interact-able as possible) and massive amounts of compute to learn from trillions of tokens (probably). They don’t release the exact technical specifications because of “the competitive landscape and the safety implications” as stated in their technical report.

Should you believe in your ability to be creative and generate something so profound that you are able to out-compete a token generation model?

It is certainly a fact that I have such a limited amount of knowledge about the world and what smarter humans have discovered; I’m not well read (i wish i could comprehend and understand a stream of words faster) nor innately endowed with great intelligence, nor am I likely to scale my personal cognitive capabilities anywhere close at all to in comparison to the FLOPs we see in silicon. It seems to me that dumb trial and error, empowered with increasingly greater orders of magnitude of compute and search space to learn from (digitized data so helpfully produced by humans), I imagine in the future that at some point they will be equipped with sensors superior to ours and learn from the universe directly, instead of from curated worlds created by us, which is but a small window to the grandiose of the universe.

Are accelerationists/optimists just smarter and are unafraid because they can see further into the gametree than others can? Or is it belief in blind hope, that raw competency is enough to correct wrongs as they come along? If the wrongs are too great, or committed by even more competent players, doesn’t that spell great doom for the rest of us, to be left behind and cast out like nothing?

Anyways, what do I know? Take everything with a grain of salt and have a good day.