Can't refresh FAVOURS with a controller after King of the Mist.

Hello, first time here so forgive any mistakes please.

As stated in the title I can't refresh the favours after King of the Mist. It seems to be because a console controller won't let me reach the actual button as if it needs a mouse pointer to be reached. No matter how much I scroll back and forth with sticks and D-pad it only swaps between the different items. Maybe if the favours window was full of loot I could scroll to loot close to the refresh button and then to the button itself but since there is no loot close to the button I cannot reach it.
This however is just speculation, maybe it carnot be reached with a controller at all or maybe it's not an issue for everyone with controllers.