Data Analyst: Ranking Team

Overview

Job Description

Responsibilities

• - Analyze Ranking Performance: Evaluate A/B tests across models, configurations, and customer segments to quantify business impact and guide ML development.
• - Investigate training data quality, model performance trends, and feature effectiveness at scale.
• - Understand Shopper Behavior: Investigate how ranking changes affect user behavior and conversion metrics.
• - Use SQL, Python, and Spark to uncover usage patterns, anomalies, and opportunities for optimization.
• - Design & Validate Metrics: Define new metrics to measure search relevance, personalization, and model performance.
• - Ensure metrics align with user experience and business goals through rigorous validation.
• - Build Analytics Infrastructure: Create scalable dashboards and reporting tools for product, engineering, and leadership teams.
• - Develop debugging tools to explain ranking decisions and identify performance issues.
• - Drive Data-Informed Decisions: Partner cross-functionally to design experiments, validate hypotheses, and communicate insights that directly influence product roadmap and ML strategy.

Required Skills

• - Advanced Analytics & A/B Testing: 3+ years analyzing complex experiments and extracting actionable insights from large, noisy datasets.
• - Experience with statistical testing and practical experiment design.
• - SQL & Big Data Expertise: Write optimized SQL queries for terabyte-scale data extraction and transformation.
• - Proficiency with distributed systems like Spark for large-scale data processing.
• - Python Programming: Strong skills in exploratory analysis, custom metrics development, and building internal tools.
• - Experience with data science libraries and automation.
• - Machine Learning & Search Systems: Understanding of ML pipelines, training data quality, and ranking/recommendation metrics.
• - Familiarity with search relevance and personalization concepts.
• - Metrics Design & Validation: Design metrics that accurately reflect model and product performance.
• - Ensure alignment between technical metrics and business outcomes.
• - Data Visualization: Create compelling dashboards using Tableau, Looker, or custom dashboards in Python.
• - Present complex findings clearly to both technical and executive audiences.
• - Cross-Functional Leadership: Influence product and engineering decisions through data storytelling.
• - Collaborate effectively across teams to drive ML and product improvements.
• - Product-Minded & Business-Focused: Deep curiosity about user behavior and business impact.
• - Connect algorithm changes to real-world customer outcomes.

Benefits

• - Unlimited vacation time - we strongly encourage all of our employees take at least 3 weeks per year
• - Fully remote team - choose where you live
• - Work from home stipend! We want you to have the resources you need to set up your home office
• - Apple laptops provided for new employees
• - Training and development budget for every employee, refreshed each year
• - Maternity & Paternity leave for qualified employees
• - Work with smart people who will help you grow and make a meaningful impact
• - Regular team offsites to connect and collaborate