Comments

You must log in or register to comment.

phobos_0 t1_j3hwuqe wrote

Dude this is dope

169

uoftsuxalot t1_j3hz4nm wrote

Not to take anything away from this project, but it’s just an api call to gpt3 with prompt “fix this error {error}”. I thought there was some training and fine tuning, but I guess LLMs can do it all now a days

144

jsonathan OP t1_j3i0txg wrote

Yeah, right now it’s just a thin wrapper around GPT-3, but there’s a lot that could be done to improve it, like using static code analysis to build a better prompt or even training a more specialized model (like this).

107

ginger_beer_m t1_j3jlhaj wrote

How did you deal with incorrectness from ChatGPT?

3

jsonathan OP t1_j3joesx wrote

I didn't. Adrenaline won’t always correctly fix your error, but it can at least give you a starting point.

39

kelkulus t1_j3k8w2w wrote

Well for one, he's not using ChatGPT. GPT-3 is not the same.

18

danielswrath t1_j3l1fkb wrote

GPT-3 has the same problem though. ChatGPT is a successor of GPT-3, so it's not the same but it's not extremely different either.

7

Glum-Bookkeeper1836 t1_j3lhgfx wrote

I'm not sure if we know this for certain, but it appears to be davinci instruct 3 with a custom prompt prefix.

3

[deleted] t1_j3l3f45 wrote

[deleted]

1

jsonathan OP t1_j3l3xm5 wrote

I haven’t used this yet, but my understanding is it can explain code and generate unit tests. It can’t explain and fix errors.

5

cloudedleopard42 t1_j3p7pr3 wrote

is it possible to fine tune GPT for static code analysis ? if yes...what would be the training set looks like?

1

satireplusplus t1_j3i1grq wrote

LLMs are our new overlords, it's crazy

18

2Punx2Furious t1_j3l29nx wrote

And it's not even AGI yet. The singularity is closer than a lot of people think.

0

TrueBirch t1_j3mcg7g wrote

I don't think AGI will ever happen, but with enough task-specific applications, the difference may become academic.

2

iamnotlefthanded666 t1_j3muxea wrote

Why don't you think AGI will ever happen?

2

TrueBirch t1_j3mwk67 wrote

Check out this comment. Some things that we take for granted from low-wage humans are incredibly hard for computers and robots. Think about valet parking. Our society doesn't think "Oh my goodness, valet parkers are geniuses!!!" But it's really really hard to build a robot that can do what they do.

1

TradeApe t1_j3rra6b wrote

If they can automate huge chunks of super busy cargo harbors, they can automate valet parking...and they won't even need AGI for that. Hell, valet parking will likely become obsolete once full self driving is here.

People also didn't think AI will make artists obsolete...but here we are.

1

TrueBirch t1_j3xlp2e wrote

Artists are hardly obsolete. Photoshop didn't make them obsolete and generative AI won't either. And I say that as someone who has extensively used Stable Diffusion for work and personal projects.

Regarding valets, I'm referring to the ability to toss your keys to a robot and have it drive your car. Even when true self driving cars are first produced (which always seems to be ten years away), we'll be a long way away from a robot being able to park a non-automated car. That's just one example of a task that seems really easy for humans but is shockingly hard for robots. Folding laundry is another one, which is especially relevant since I'm ignoring the fact that my dryer just finished a load.

1

2Punx2Furious t1_j3mopda wrote

Yeah, I see a lot of goalpost-moving, but in the end, it depends on how you define "AGI", some people have varying definitions. I think even a language model can become AGI eventually.

1

TrueBirch t1_j3mw92i wrote

There are some things that are incredibly hard. Imagine you work on a farm. You toss the keys to the ATV to a 17yo farmhand who's never worked for you before. You say, "Head over to field 3 and tell me if it's dry enough to plow. You can see where it is on this paper map. Radio back using this handheld." The farmhand duly drives the ATV to field 3, sees that it's muddy, picks up the radio, and says, "Sorry boss, field 3's a no-go."

We're a long way from a robotic farmhand being able to perform those skills, certainly not for a price comparable to a farm laborer.

You could definitely train an application-specific AI to monitor fields and report on their moisture levels. You could even have an algorithm that schedules all of your farm equipment based on current conditions and other factors. So it's not that AI can't revolutionize how we work, it's just that it'll be different from true AGI.

2

eldenrim t1_j4kmltf wrote

I'm curious how you feel about the following:

There are humans that can't do the task you outlined. Why use it as a metric for AGI? Put in other words, what about a "less intelligent" AGI, that crawls before it walks? An AGI equivalent to a human with lower IQ, or some similar measurement that correlates with not being capable of the same things as those in your example?

Second, if an A.I can do 80% of what a human can, and a human can do 10% of what an A.I can, would you still claim the system isn't an AGI? As in, if humans can do X, A.I can do X * 100 things, but there's a venn diagram with some things unique to humans and many things unique to A.I, does it not count because you can point to human examples of tasks it cannot complete?

Finally, considering a human system has to account for things irrelevant to an AGI (body homeostasis with heart rate and such, immune system, etc) and an AGI can build on code before it, what do you see as the barrier to AGI? Is it not a matter of time?

1

TrueBirch t1_j4kv71p wrote

I think "AGI" is a silly concept overall and never really happening. Computers are good at doing things in different ways from humans. Rather than chasing AGI, you can make a lot more of an impact by leveraging a computer's strengths and avoiding its weaknesses.

For my example, I picked an occupation with an average salary south of $30,000/year (source). I'm not saying everybody can do it, but the market puts a price on this kind of labor that suggests many people can do it. A true AGI system could replicate how a low-salary human does a job. In reality, a computerized system would use a few wireless sensors that call home instead of physically driving around looking at fields.

Similarly, consider meter readers, another low-wage job. Imagine what it would take to create a robot that could drive from house to house, get out of the car, find the power meter, gently move anything blocking it, and take a reading. Instead, utilities use smart meters that call home. It's cheaper, more reliable, and simpler.

It's beyond hard to create a true AGI system, and there are plenty of ways to make tons of money with application-specific systems.

1

eldenrim t1_j4l7ilw wrote

I'm currently interested in ML to alleviate the suffering of my disabled partner and myself, I just enjoy theoretical discussion with AGI.

Maybe making money will come later. :)

1

TrueBirch t1_j4ldkum wrote

I'm talking about where the funding is going. Anything remotely approaching AGI would require billions and billions of dollars of funding.

1

eldenrim t1_j4lfc8z wrote

So you don't think that repeatedly making narrow AI, and then at some point bundling them together, is a valid way to get to AGI?

1

TrueBirch t1_j4qdbf2 wrote

It'll be something entirely new, but not capable of doing everything that my toddler can do. Systems will be designed to avoid those weaknesses. Again, think about replacing meter readers with cheap sensors instead of expensive robots.

1

2Punx2Furious t1_j3nzumw wrote

> We're a long way from a robotic farmhand being able to perform those skills, certainly not for a price comparable to a farm laborer.

If we get AGI, we automatically get that as well, by definition. Those you listed are all currently hard problems, yes, but an AGI would be able to do them, no problem.

The issue is, will AGI ever be achieved, and if yes, when?

I think the answer to the first one is simple, the second one not as much.

The answer (in very short) is: Most likely yes, unless we go extinct first. Because we know that general intelligence is possible, so I see no reason why it shouldn't be possible to replicate artificially, and even improve it, and several, very wealthy companies are actively working on it, and the incentive to achieve it is huge.

As for the when, it's impossible to know until it happens, and even then, some people will argue about it for a while. I have my predictions, but there are lots of disagreeing opinions.

I don't know how someone even remotely interested in the field could think it will never happen for sure.

As for my prediction/opinion, I actually give it a decent chance of it happening in the next 10-20 years, with probability increasing every year until the 2040s. I would be very surprised if it doesn't happen by then, but of course, there is no way to tell.

0

TrueBirch t1_j4jd8af wrote

A true AGI has way too many edge cases to be possible in the timeframe you describe. It's also not necessary to create AGI in order to make a lot of money from AI. You can find the specific jobs that you want to replace and create a task-specific AI to do it.

1

2Punx2Furious t1_j4kyyhq wrote

True that you don't need AGI to disrupt everything. But I don't think the edge cases matter, it's not like it will be coded manually.

1

TrueBirch t1_j4lb7fg wrote

>I don't think the edge cases matter

Being able to handle those weird edge cases is what distinguishes AGI from the kinds of AI that companies are currently developing...

1

2Punx2Furious t1_j4lbgjj wrote

Yes, I'm saying the fact that there are edge cases doesn't matter, because it's not us who have to address them. As we get closer and closer to AGI, it will get better at handling them, we won't have to find them, and code solutions for them. I think it will be an emergent quality of AGI.

1

jsonathan OP t1_j3ido30 wrote

Thanks! Feel free to try it out here. Let me know if any of y’all get some impressive bug fixes.

7

jsonathan OP t1_j3hwo11 wrote

Try it out here: https://useadrenaline.com

50

jsonathan OP t1_j3hwx5l wrote

Right now, this is just a simple demo of what’s possible with AI-driven debugging. But I’d like to build it out so that instead of just explaining errors, Adrenaline provided a ChatGPT-style assistant that can answer questions about your error, and teach you during the debugging process.

This is open-source, so if anyone’s interested in contributing, here’s the GitHub repository: https://github.com/shobrook/adrenaline

46

ddproxy t1_j3i21my wrote

Does it come with an animated assistant in the shape of a paperclip?

53

_swnt_ t1_j3itpq1 wrote

That's would be an actually useful paperclip 😂

11

StuntHacks t1_j3jmwtj wrote

Maybe we should make more of em

3

jsonathan OP t1_j3jokpy wrote

What could go wrong?

11

lucidrage t1_j3l0695 wrote

They become self replicating.

2

ImPetarded t1_j3lo80s wrote

...it didn't end like we thought it would in the movies. There were no killer machines....there were paper clips, trillions of them....

2

[deleted] t1_j3jjc7v wrote

[deleted]

14

ksblur t1_j3kkft7 wrote

Just wait till you see what the rate for management will be after LLMs come for their jobs.

Managers are mostly people-interaction-managers, and LLMs are already 10x better at that than they are at creating novel code.

11

keepthepace t1_j3o7vy8 wrote

I was going to argue that employees will be able to bullshit their automated manager easily but well, it is not like humans are much better at handling it.

2

GoofAckYoorsElf t1_j3ilywu wrote

This is all great. The only problem is that I can't use it due to non-disclosure and IP protection of my employer. As long as I have to send code over the web, it's a no-no.

13

IshKebab t1_j3j1gkz wrote

Yeah I imagine that will be an issue for lots of people. What's the SotA in open source LLMs?

I looked it up. Apparently it's BLOOM. Slightly bigger than GPT-3. No idea if it is better.

You need a DGX A100 to run it (only $150k!).

12

LetterRip t1_j3n91mt wrote

I'd do GLM-130B

> With INT4 quantization, the hardware requirements can further be reduced to a single server with 4 * RTX 3090 (24G) with almost no performance degradation.

https://github.com/THUDM/GLM-130B

I'd also look into pruning/distillation and you could probably shrink the model by about half again.

2

--algo t1_j3kxv2l wrote

How do you deal with source code hosting?

2

GoofAckYoorsElf t1_j3l2sls wrote

A cloud hosted GitLab with customer managed keys. We have a very detailed IP and security agreement with our cloud provider.

2

keepthepace t1_j3o8avv wrote

I am willing to be that 99% of the code is overprotected and no one in OpenAI would spend valuable time looking at it.

These protections mostly exist to justify some bullshit jobs within the company.

1

GoofAckYoorsElf t1_j3po3ti wrote

Probably. I'm still getting fired if I do something like that without permission.

2

Accomplished-Low3305 t1_j3j6b0v wrote

It would be nice to have some metric to evaluate how good is GPT-3 solving bugs. In my experience it only works fine for simple bugs, such as using an incorrect variable.

9

yerop82726 t1_j3j6hp9 wrote

Nice one. Keen to see the vscode extension!

3

RKHS t1_j3k2134 wrote

This is a fairly useless example. It's simply a rewording of the error. Do you have any examples that are non trivial?

3

naiq6236 t1_j3j0ofz wrote

This would be a game changer dude

2

Jbonez87 t1_j3jc3a7 wrote

This is pretty cool dude!

2

davidswelt t1_j3kg6vo wrote

OK, how did you evaluate it? How do you tell it's working well or not?

2

fnetma t1_j3kmesf wrote

An AI debugger would be very helpful. I actually see that as a use case...

2

Mikatron3000 t1_j3l71ns wrote

Not sure if this is already there but it might be worth adding some license information here since sending closed source code over an open sourced API / model might become a no no in the future legally. I guess that would be the problem of making this an Intellij / vscode plugin

2

devinhedge t1_j3lhp0h wrote

This is cool. How do we give feedback to the training engine so that it improves over time?

2

EarthAdmin t1_j3mgzbo wrote

Would love this to be a VSCode plug-in! Happy to drop our OpenAI api key in there.

2

outthemirror t1_j3jg8l7 wrote

If this was a pycharm/vscode plugin….

1

Eastern_Care_6369 t1_j3kipko wrote

Can you turn this into an IOS app

1

Think_Olive_1000 t1_j3tmuvt wrote

Why the fuck are you coding on an iPhone - if you're going to use a phone at least be android

1

Own-Cherry6760 t1_j3m1ifz wrote

Which model are you using from openAI ?

1

LucasLeroy19 t1_j4cj9v0 wrote

My question to you is, why the name Adrenaline? How did you come up with that?

1

aminostfx t1_j51kpiw wrote

This is awesome. But i have an idea to make it even better. What if we train a RL agent to write code without errors and actually make sure there is no bugs in the code. The environment used to train the RL would be the compiler. We can start first with support python only and supporting other languages later on. DM if you’re interested to colab on this project.

1