How to cancel this job?

I've had these customers for maybe 5 years. The first time I met them they rubbed me the wrong way but were very nice the next time, or should I say the wife was. They're both probably in their 60's and the husband always seems to be ornery, spouting rude things from time to time. Went to powerwash their deck before staining recently and their water was off...very frustrating. After telling them that I simply didn't want to work for them anymore, they complained and then practically begged me to still do it so I agreed to do it, scheduled for next week. The other day as I was at a neighbors house, the husband comes over and tells me I need to replace some deck boards that I replaced for him last year FOR FREE! Turns out the wood filler I used in some small gaps was of poor quality and cracked, which I didn't know would of course, but the husband insisted I replace two entire boards, which would have to be cedar now (2x as expensive) since the deck is to be stained soon. I told them again that I didn't want to pw/stain their deck and they once again argued, backed down, begged. I feel very uncomfortable dealing with these people now. I don't have an official contract but there are emails confirming that I would work for them. Any advice?