Can I get a refund for the payments I made to Affirm before returning the phone?

Unfortunately, Affirm will not refund paid interest. When you make your payments, the funds are first applied to the accrued, unpaid interest, and then the principal. Since interest is the only cost of borrowing with Affirm, it is not refundable.