This sounds to me like the battery is not supplying the right voltage (to start the unit up), but it is supplying the right current (to keep it running). What happens when you try a different battery? Update: Many electronics use a different starting voltage and a different sustaining voltage. If your battery is not delivering the proper voltage, the device won't start up properly, but if it's plugged in, the battery is taken out of the picture. My advice to you, try to borrow a known good battery from somebody, and if your Vita works properly when you use it, you know that your battery is the problem. If you can't borrow a battery from somebody, you have to buy one, try not to destroy the packaging and save the receipt, in case you have to return the battery to the store (if that turns out to not be the problem)
Many towns have a hazardous material disposal, either all the time, or once a week, or once a month, you bring the swollen battery to them. I heard Best Buy will take old Laptop Batteries for safe disposal also. Your mileage may vary on that though. It might take a little research. But you absolutely can't just put the batteries with the regular trash.