Insane compute wasn’t everything. Hinton helped develop the technique which allowed more data to be processed in more layers of a network without totally losing coherence. It was more of a toy before then because it capped out at how much data could be used, how many layers of a network could be trained, and I believe even that GPUs could be used efficiently for ANNs, but I could be wrong on that one.

Either way, after Hinton’s research in ~2010-2012, problems that seemed extremely difficult to solve (e.g., classifying images and identifying objects in images) became borderline trivial and in under a decade ANNs went from being almost fringe technology that many researches saw as being a toy and useful for a few problems to basically dominating all AI research and CS funding. In almost no time, every university suddenly needed machine learning specialists on payroll, and now at about 10 years later, every year we are pumping out papers and tech that seemed many decades away… Every year… In a very broad range of problems.

The 580 and CUDA made a big impact, but Hinton’s work was absolutely pivotal in being able to utilize that and to even make ANNs seem feasible at all, and it was an overnight thing. Research very rarely explodes this fast.

Edit: I guess also worth clarifying, Hinton was also one of the few researching these techniques in the 80s and has continued being a force in the field, so these big leaps are the culmination of a lot of old, but also very recent work.

My guess was that they knew gaming was niche and were willing to invest less in this headset and more in spreading the widespread idea that “Spatial Computing” is the next paradigm for work.

I VR a decent amount, and I really do like it a lot for watching TV and YouTube, and am toying with using it a bit for work-from-home where the shift in environment is surprisingly helpful.

It’s just limited. Streaming apps aren’t very good, there’s no great source for 3D movies (which are great, when Bigscreen had them anyways), they’re still a bit too hot and heavy for long-term use, the game library isn’t very broad and there haven’t been many killer app games/products that distinct it from other modalities, and it’s going to need a critical amount of adoption to get used in remote meetings.

I really do think it’s huge for given a sense of remote presence, and I’d love to research how VR presence affects remote collaboration, but there are so many factors keeping it tough to buy into.

They did try, though, and I think they’re on the right track. Facial capture for remote presence and hybrid meetings, extending the monitors to give more privacy and flexibility to laptops, strong AR to reduce the need to take the headset off - but they’re first selling the idea, and then maybe there will be a break. I’ll admit the industry is moving much slower than I’d anticipated back in 2012 when I was starting VR research.

I really want to see if worker owned cooperatives plus AI could do help democratize running companies (where appropriate). Not just LLMs, but a mix of techniques for different purposes (e.g., hierarchial task networks to help with operations and pipelining, LLM for assembling/disseminating information to workers).

C could just be a blank and you have to bit blit the arrow on yourself.

I don’t know why, but I still can’t open a core file without going I’m in. I don’t do QA, though, and so tinkering with final breath of my program frozen in time maintains some novelty.

My two cents, after years of Markdown (and md to PDF solutions) and LaTeX and a full two years of trying to commit to bashing my head against Word for work purposes, I’m really enjoying Typst. It didn’t take long to convert my themes, having docs I can import which are basically just variables to share across documents in a folder has been really helpful. Haven’t gone too deep into it but I’m excited to give it a deeper test run over the next little bit.

Lots of immediate hate for AI, but I’m all for local AI if they keep that direction. Small models are getting really impressive, and if they have smaller, fine-tuned, specific-purpose AI over the “general purpose” LLMs, they’d be much more efficient at their jobs. I’ve been rocking local LLMs for a while and they’ve been great as a small compliment to language processing tasks in my coding.

Good text-to-speech, page summarization, contextual content blocking, translation, bias/sentiment detection, click bait detection, article re-titling, I’m sure there’s many great use cases. And purely speculation,but many traditional non-llm techniques might be able to included here that were overlooked because nobody cared about AI features, that could be super lightweight and still helpful.

If it goes fully remote AI, it loses a lot of privacy cred, and positions itself really similarly to where everyone else is. From a financial perspective, bandwagoning on AI in the browser but “we won’t send your data anywhere” seems like a trendy, but potentially helpful and effective way to bring in a demographic interested in it without sacrificing principles.

But there’s a lot of speculation in this comment. Mozilla’s done a lot for FOSS, and I get they need monetization outside of Google, but hopefully it doesn’t lead things astray too hard.

That too, and I can really efficiently manage the items going into bags given I backpack my groceries and want pretty specific configurations…

Totally agree. I forgot about those, as I’ve only encountered the weighing ones once in the past very long time and it was a mess, I can totally get hate if weighing ones are the only experience with them.

I almost exclusivity self-checkout for groceries, and it had drastically sped up my checkout time as most people in my area opt to use traditional checkout and the stores are still keeping lots of lanes open (just closing the express lanes). The last 3 times I’ve used a non-self checkout, each time I was double charged for items or didn’t have reduced prices applied and didn’t notice because I was bagging.

I appreciate the comment, and it’s a point I’ll be making this year in my courses. More than ever, students have been struggling to motivate themselves to do the work. The world’s on fire and it’s hard to intrinsically motivate to do hard things for the sake of learning, I get it. Get a degree to get a job to survive, learning is secondary. But this survival mindset means that the easiest way is the best way, and it’s going to crumble long-term.

It’s like jumping into an MMORPG and using a bot to play the whole game. Sure you have a cap level character, but you have no idea how to play, how to build a character, and you don’t get any of the references anyone else is making.

This is a very output-driven perspective. Another comment put it well, but essentially when we set up our curriculum we aren’t just trying to get you to produce the one or two assignments that the AI could generate - we want you to go through the motions and internalize secondary skills. We’ve set up a four year curriculum for you, and the kinds of skills you need to practice evolve over that curriculum.

This is exactly the perspective I’m trying to get at work my comment - if you go to school to get a certification to get a job and don’t care at all about the learning, of course it’s nonsense to “waste your time” on an assignment that ChatGPT can generate for you. But if you’re there to learn and develop a mastery, the additional skills you would have picked up by doing the hard thing - and maybe having a Chat AI support you in a productive way - is really where the learning is.

If 5 year olds can generate a university level essay on the implications of thermodynamics on quantum processing using AI, that’s fun, but does the 5 year old even know if that’s a coherent thesis? Does it imply anything about their understanding of these fields? Are they able to connect this information to other places?

Learning is an intrinsic task that’s been turned into a commodity. Get a degree to show you can generate that thing your future boss wants you to generate. Knowing and understanding is secondary. This is the fear of generative AI - further losing sight that we learn though friction and the final output isn’t everything. Note that this is coming from a professor that wants to mostly do away with grades, but recognizes larger systemic changes need to happen.

100%, and this is really my main point. Because it should be hard and tedious, a student who doesn’t really want to learn - or doesn’t have trust in their education - will bypass those tedious bits with the AI rather than going through those tedious, auxiliary skills that you’re expected to pick up, and use the AI was a personal tutor - not a replacement for those skills.

So often students are concerned about getting a final grade, a final result, and think that was the point, thus, “If ChatGPT can just give me the answer what was the point”, but no, there were a bunch of skills along the way that are part of the scaffolding and you’ve bypassed them through improper use of available tools. For example, in some of our programming classes we intentionally make you use worse tools early to provide a fundamental understanding of the evolution of the language ergonomics or to understand the underlying processes that power the more advanced, but easier to use, concepts. It helps you generalize later, so that you don’t just learn how to solve this problem in this programming language, but you learn how to solve the problem in a messy way that translates to many languages before you learn the powerful tools of this language. As a student, you may get upset you’re using something tedious or out of date, but as a mentor I know it’s a beneficial step in your learning career.

Maybe it would help to teach students about learning early, and how learning works.

Education has a fundamental incentive problem. I want to embrace AI in my classroom. I’ve been studying ways of using AI for personalized education since I was in grade school. I wanted personalized education, the ability to learn off of any tangent I wanted, to have tools to help me discover what I don’t know so I could go learn it.

The problem is, I’m the minority. Many of my students don’t want to be there. They want a job in the field, but don’t want to do the work. Your required course isn’t important to them, because they aren’t instructional designers who recognize that this mandatory tangent is scaffolding the next four years of their degree. They have a scholarship, and can’t afford to fail your assignment to get feedback. They have too many courses, and have to budget which courses to ignore. The university holds a duty to validate that those passing the courses met a level of standards and can reproduce their knowledge outside of a classroom environment. They have a strict timeline - every year they don’t certify their knowledge to satisfaction is a year of tuition and random other fees to pay.

If students were going to university to learn, or going to highschool to learn, instead of being forced there by societal pressures - if they were allowed to learn at their own pace without fear of financial ruin - if they were allowed to explore the topics they love instead of the topics that are financially sound - then there would be no issue with any of these tools. But the truth is much bleaker.

Great students are using these tools in astounding ways to learn, to grow, to explore. Other students - not bad necessarily, but ones with pressures that make education motivated purely by extrinsic factors than intrinsic - have a perfect crutch available to accidentally bypass the necessary steps of learning. Because learning can be hard, and tedious, and expensive, and if you don’t love it, you’ll take the path of least resistance.

In game design, we talk about not giving the player the tools to optimize their fun away. I love the new wave of AI, I’ve been waiting for this level of natural language processing and generation capability for a very long time, but these are the tools for students to optimize the learning away. We need to reframe learning and education. We need to bring learning front and center instead of certification. Employers need to recognize this, universities need to recognize this, highschools and students and parents need to recognize this.

Hmm… Nothing off the top of my head right now. I checked out the Wikipedia page for Deep Learning and it’s not bad, but quite a bit of technical info and jumping around the timeline, though it does go all the way back to the 1920’s with it’s history as jumping off points. Most of what I know came from grad school and having researched creative AI around 2015-2019, and being a bit obsessed with it growing up before and during my undergrad.

If I were to pitch some key notes, the page details lots of the cool networks that dominated in the 60’s-2000’s, but it’s worth noting that there were lots of competing models besides neural nets at the time. Then 2011, two things happened at right about the same time: The ReLU (a simple way to help preserve data through many layers, increasing complexity) which, while established in the 60’s, only swept everything for deep learning in 2011, and majorly, Nvidia’s cheap graphics cards with parallel processing and CUDA that were found to majorly boost efficiency of running networks.

I found a few links with some cool perspectives: Nvidia post with some technical details

Solid and simplified timeline with lots of great details

It does exclude a few of the big popular culture events, like Watson on Jeopardy in 2011. To me it’s fascinating because Watson’s architecture was an absolute mess by today’s standards, over 100 different algorithms working in conjunction, mixing tons of techniques together to get a pretty specifically tuned question and answer machine. It took 2880 CPU cores to run, and it could win about 70% of the time at Jeopardy. Compare that to today’s GPT, which while ChatGPT requires way more massive amounts of processing power to run, have an otherwise elegant structure and I can run awfully competent ones on a $400 graphics card. I was actually in a gap year waiting to go to my undergrad to study AI and robotics during the Watson craze, so seeing it and then seeing the 2012 big bang was wild.

Oh for sure. And it’s a great realm to research, but pretty dirty to rip apart another field to bolster your own. Then again, string theorist…

For me, it’s the next major milestone in what’s been a roughly decade-ish trend of research, and the groundbreaking part is how rapidly it accelerated. We saw a similar boom in 2012-2018, and now it’s just accelerating.

Before 2011/2012, if your network was too deep, too many layers, it would just breakdown and give pretty random results - it couldn’t learn - so they had to perform relatively simple tasks. Then a few techniques were developed that enabled deep learning, the ability to really stretch the amount of patterns a network could learn if given enough data. Suddenly, things that were jokes in computer science became reality. The move from deep networks to 95% image recognition ability, for example, took about 1 years to halve the error rate, about 5 years to go from about 35-40% incorrect classification to 5%. That’s the same stuff that powered all the hype around AI beating Go champions and professional Starcraft players.

The Transformer (the T in GPT) came out in 2017, around the peak of the deep learning boom. In 2 years, GPT-2 was released, and while it’s funny to look back on now, it practically revolutionized temporal data coherence and showed that throwing lots of data at this architecture didn’t break it, like previous ones had. Then they kept throwing more and more and more data, and it kept going and improving. With GPT-3 about a year later, like in 2012, we saw an immediate spike in previously impossible challenges being destroyed, and seemingly they haven’t degraded with more data yet. While it’s unsustainable, it’s the same kind of puzzle piece that pushed deep learning into the forefront in 2012, and the same concepts are being applied to different domains like image generation, which has also seen massive boosts thanks in-part to the 2017 research.

Anyways, small rant, but yeah - it’s hype lies in its historical context, for me. The chat bot is an incredible demonstration of the incredible underlying advancements to data processing that were made in the past decade, and if working out patterns from massive quantities of data is a pointless endeavour I have sad news for all folks with brains.

I understand that he’s placing these relative to quantum computing, and that he is specifically a scientist who is deeply invested in that realm, it just seems too reductionist from a software perspective, because ultimately yeah - we are indeed limited by the architecture of our physical computing paradigm, but that doesn’t discount the incredible advancements we’ve made in the space.

Maybe I’m being too hyperbolic over this small article, but does this basically mean any advancements in CS research are basically just glorified (insert elementary mechanical thing here) because they use bits and von Neumann architecture?

I used to adore Kaku when I was young, but as I got into academics, saw how attached he was to string theory long after it’s expiry date, and seeing how popular he got on pretty wild and speculative fiction, I struggle to take him too seriously in this realm.

My experience, which comes with years in labs working on creative computation, AI, and NLP, these large language models are impressive and revolutionary, but quite frankly, for dumb reasons. The transformer was a great advancement, but seemingly only if we piled obscene amounts of data on it, previously unspeculated of amounts. Now we can train smaller bots off of the data from these bigger ones, which is neat, but it’s still that mass of data.

To the general public: Yes, LLMs are overblown. To someone who spent years researching creativity assistance AI and NLPs: These are freaking awesome, and I’m amazed at the capabilities we have now in creating code that can do qualitative analysis and natural language interfacing, but the model is unsustainable unless techniques like Orca come along and shrink down the data requirements. That said, I’m running pretty competent language and image models on 12GB of relatively cheap consumer video card, so we’re progressing fast.

Edit to Add: And I do agree that we’re going to see wild stuff with quantum computing one day, but that can’t discount the excellent research being done by folks working with existing hardware, and it’s upsetting to hear a scientist bawk at a field like that. And I recognize I led this by speaking down on string theory, but string theory pop science (including Dr. Kaku) caused havoc in people taking physics seriously.

Soap. 100%.

It depends what “From Scratch” means to you, as I don’t know your level of programming or interests, because you could be talking about making a game from beginning to end, and you could be talking about…

• Using a general purpose game engine (Unity, Godot, Unreal) and pre-made assets (e.g., Unity Asset Store, Epic Marketplace)?
• Using a general purpose game engine almost purely as a rendering+input engine with a nice user interface and building your own engine overtop of that
• Using frameworks for user input and rendering images, but not necessarily ones built for games, so they’re more general purpose and you’ll need to write a lot of game code to put it all together into your own engine before you even starting “Making the game”, but offer extreme control over every piece so that you can make something very strange and experimental, but lots of technical overhead before you get started
• Writing your own frameworks for handling user input and rendering images… that same as previous, but you’ll spend 99% of your time trying to rewrite the wheel and get it to go as fast as any off the shelf replacement

If you’re new to programming and just want to make a game, consider Godot with GDScript - here’s a guide created in Godot to learn GDScript interactively with no programming experience. GDScript is like Python, a very widely used language outside of games, but it is exclusive to Godot so you’ll need to transfer it. You can also use C# in Godot, but it’s a bigger learning curve, though it is very general and used in a lot of games.

I’m a big Godot fan, but Unity and Unreal Engine are solid. Unreal might have a steeper learning curve, Godot is a free and open-source project with a nice community but it doesn’t have the extensive userbase and forum repository of Unity and Unreal, Unity is so widely used there’s lots of info out there.

If you did want to go really from scratch, you can try using something like Pygame in Python or Processing in Java, which are entirely code-created (no user interface) but offer lots of helpful functionality for making games purely from code. Very flexible. That said, they’ll often run slow, they’ll take more time to get started on a project, and you’ll very quickly hit a ceiling for how much you can realistically do in them before anything practical.

If you want to go a bit lower, C++ with SDL2, learning OpenGL, and learning about how games are rendered and all that is great - it will be fast, and you’ll learn the skills to modify Godot, Unreal, etc. to do anything you’d like, but similar caveats to previous; there’s likely a low ceiling for the quality you’ll be able to put out and high overhead to get started on a project.