From what I've read about JTAGing and modding the Xbox 360 in general, this sort of problem is not that uncommon. If its overheating, there may be an issue with original soldering done for the JTAG.
Here is what I would recommend trying:
1. Checking the soldering joints on the JTAG - are they neat and clean? In soldering, if the joints were done messily or the iron was to hot, it may have damaged some nearby components.
2. Try removing and replacing the thermal paste connecting the Heat sink to the CPU and GPU. In my experience, the factory thermal paste used in the original manufacturing of a lot of consoles are not great, to put it mildly. There is a good chance the original thermal paste has dried up reaching the end of its lifespan, so replacing it should fix the problem. You'll don't need anything to fancy - Artic Alumina thermal paste should do the trick. (Its made by Artic Silver, and you want the single syringe one, NOT the one that comes with two syringes. I believe you can get it here through iFixit, and its also the thermal paste typically sold with Xbox 360 RROD fix kits.) You'll also need the 2 Step Articlean to clean off the old thermal paste from the processor and heatsink, and prep the surface for the new paste. This can also be bought here on ifixit, along with most major computer parts and repair stores.
[guide|744|How to Apply Thermal Paste]
Hope this helps!
- Ben