In order to make a contribution to an IRA, you must have eligible ( taxable) income. See page 8 of IRS publication 590 at
http://www.irs.gov/pub/irs-pdf/p590.pdf for a list of eligible compensation. If you contributed to your IRA in years that you did not have eligible compensation, then you should remove the amount form the IRA, as it accrues a 6-percent penalty for every year it remains in the IRA.
The interest ( on the ineligible/excess) contribution is removed only if the contribution is being removed timely, i.e. by your tax filing deadline , plus extensions.
If you file your return on time, you receive an automatic 6-months extension by the way. The earnings is taxable in the year to which the contribution applies. For instance, if you made the contribution in 2003, the earnings will be taxable in 2003, if the contribution is removed in 2003 or in 2004 ( by the deadline)