For recreational programming sound right now I'd go with either Supercollider (if you prefer a Smalltalk style syntax) or ChucK (if you are more comfortable with a C/Java style syntax). Neither is aimed at plugins, I don't see that as a issue because I personally stopped caring about plugins after I got into programming for sound but it may be a issue if you want your work to work in a DAW or would like to distribute/sell it, in that case the C++ VST developers kit will be the way to go.
There is CSound but I feel that right now it's mainly of historical relevance; I quite like my modern language constructs.
Then there are a whole series of graphical systems; MAX/MSP or the free (as in freedom) PureData alternative, Reaktor, Tassman, Synthedit, etc. These are often seen as "easier" than text-based programming. Depending on one's temperament that may well be true but in the general case I think that once you are comfortable with the system and beyond a certain level of complexity that no longer holds.
There is no shortage of options and it really depends on what you would like to do, how much time you are willing to invest and what you feel comfortable with. If you are mainly interested in experimenting with weird sounds it would be a waste to start with reading the bigest tome on filter design in pure ASM for embedded chips that you can find, for example.
I'd suggest looking at ChucK and SuperCollider first, especially because you are already comfortable with code. At the very least look around at some of these options before investing months and/or hundreds of bucks.