Sponsored search positions are typically allocated through real-time auctions, where the outcomes depend on advertisers’ quality-adjusted bids—the product of their bids and quality scores. Although quality scoring helps promote ads with higher conversion outcomes, setting these scores for new advertisers in any given market is challenging, leading to the cold-start problem. To address this, platforms incorporate multi-armed bandit algorithms in auctions to balance exploration and exploitation. However, little is known about the optimal exploration strategies in such auction environments. We uti- lize data from a leading Asian mobile app store that places sponsored ads for keywords. The platform employs a Thompson Sampling algorithm within a second-price auction to learn quality scores and allocate a single sponsored position for each keyword. We empirically demonstrate two key ways in which exploration increases platform revenues. First, we quantify how exploration enhances efficiency by discovering high-quality new advertisers, a mechanism well-established in standard bandit problems. Second, un- like standard non-strategic bandit problems, we find that exploration can serve as a tool to increase market thickness, thereby boosting platform revenues. Based on these insights, we propose a customized exploration strategy in which the platform adjusts the level of exploration for each keyword according to specific keyword-level charac- teristics. We derive the Pareto frontier for revenue and efficiency and offer actionable policies that highlight substantial gains for the platform on both fronts when using a customized exploration strategy.