Unable to progress story in act 2

After defeating the Caravan boss, the Caravan leader brings us to the south to find some temple. I immediately left that area before walking on the teleporter, by talking to an npc that teleported me back to the Caravan, and now I have no way to get back to that temple area.