Ranking turns a huge catalog into a short ordered list for one user and one moment. It runs as a funnel: retrieval gathers a few hundred to a few thousand candidates with cheap methods (embedding similarity, follows, co-purchases), ranking orders them with a richer model, and re-ranking applies business rules such as diversity, freshness or removing items the user already owns.
Measure rankings the way they are used. Offline metrics such as NDCG compare against past behavior, but past clicks only cover what was shown, and items shown first get clicked more just because of position. Online experiments are the final judge. Keep the funnel within budget: features fetched per candidate multiply quickly.
Design a retrieval and ranking funnel, budget its feature fetches and measure it fairly.