I have had this happen a couple time. I got a call one Saturday morning asking about 3 recent charges, including a Men's Warehouse in Waikiki.
My response: "You are calling me at my California home number. I WISH I were in Waikiki right now. How about you reverse everything today and send me a new card in the mail?"
That reminded me of one false positive that may be of interest to the HN community:
I was buying a custom-made suit at Indochino's "traveling tailor" event. It was at a pop-up storefront in Seattle (where I live) but my bank thought their mobile point-of-sale card processing system was in Vancouver, BC and declined the transaction until I explained to them what happened.
My response: "You are calling me at my California home number. I WISH I were in Waikiki right now. How about you reverse everything today and send me a new card in the mail?"