• 0 Posts
  • 25 Comments
Joined 4 months ago
cake
Cake day: March 12th, 2024

help-circle








  • Setting aside the obvious answer of “because capitalism”, there are a lot of obstacles towards democratizing this technology. Training of these models is done on clusters of A100 GPU’s, which are priced at $10,000USD each. Then there’s also the fact that a lot of the progress being made is being done by highly specialized academics, often with the resources of large corporations like Microsoft.

    Additionally the curation of datasets is another massive obstacle. We’ve mostly reached the point of diminishing returns of just throwing all the data at the training of models, it’s quickly becoming apparent that the quality of data is far more important than the quantity of the data (see TinyStories as an example). This means a lot of work and research needs to go into qualitative analysis when preparing a dataset. You need a large corpus of input, each of which are above a quality threshold, but then also as a whole they need to represent a wide enough variety of circumstances for you to reach emergence in the domain(s) you’re trying to train for.

    There is a large and growing body of open source model development, but even that only exists because of Meta “leaking” the original Llama models, and now more recently releasing Llama 2 with a commercial license. Practically overnight an entire ecosystem was born creating higher quality fine-tunes and specialized datasets, but all of that was only possible because Meta invested the resources and made it available to the public.

    Actually in hindsight it looks like the answer is still “because capitalism” despite everything I’ve just said.










  • This is where ChatGPT and Codium.ai has been a godsend for me. Something that would have taken me a few hours to 1+ days to iterate on is now reduced down to anywhere from minutes to an hour. I don’t even always see it all the way through to completion, but just knowing that I can iterate on some version of it so quickly is often motivation enough to get started.

    If you’re paying for the Plus subscription, GPT-4 with Code Interpreter is absolutely OP. Did you know you can hand it a zip file as a way of giving it multiple files at once?



  • There’s a lot of little things like that, that are very different here and we just take for granted without thinking twice. I’m in Argentina and have noticed a ton of stuff like this.

    A random example is there’s a handle next to the stove where you close the gas line when it’s not in use. They don’t just have an always on supply of naturals gas ready to leak from the burner if you accidentally twist the knob.

    Another one is the freezer will seal itself shut for about 15 seconds after being opened. I thought it was broken the first time I experienced this and tried forcing it open.