Mitigating Digital Discrimination in Dating Apps – The Dutch Breeze case

Tim de Jonge; Frederik Zuiderveen Borgesius

doi:10.71265/d2f0vn41

Authors

Tim de Jonge Radboud University Nijmegen
Frederik Zuiderveen Borgesius Radboud University Nijmegen

DOI:

https://doi.org/10.71265/d2f0vn41

Keywords:

Artificial Intelligence, non-discrimination law, discrimination, dating, fairness, bias

Abstract

In September 2023, the Netherlands Institute for Human Rights, the Dutch non-discrimination authority, decided that Breeze, a Dutch dating app, was justified in suspecting that their algorithm discriminated against non-white. Consequently, the Institute decided that Breeze must prevent this discrimination based on ethnicity. This paper explores two questions. (i) Is the discrimination based on ethnicity in Breeze's matching algorithm illegal? (ii) How can dating apps mitigate or stop discrimination in their matching algorithms? We illustrate the legal and technical difficulties dating apps face in tackling discrimination and illustrate promising solutions. We analyse the Breeze decision in-depth, combining insights from computer science and law. We discuss the implications of this judgment for scholarship and practice in the field of fair and non-discriminatory machine learning.

Downloads

Download data is not yet available.

Author Biographies

Tim de Jonge, Radboud University Nijmegen

Tim de Jonge is a PhD Candidate on Fairness in Machine Learning at Radboud University, Nijmegen. He is part of iHub, an interdisciplinary research group on the societal impact of digitalization. His research focuses on digital discrimination, but extends more broadly to other interpretations of fairness. Despite a technical background, Tim has taken ‘interdisciplinary’ to heart, and combines technical, legal, and philosophical insights to tackle the questions relating to modern day fairness.
Frederik Zuiderveen Borgesius, Radboud University Nijmegen

Frederik Zuiderveen Borgesius is professor in ICT and law. He works at the iHub (part of Radboud University), the interdisciplinary research hub on digitalization and society. His research mostly concerns fundamental rights, such as privacy and non-discrimination rights, in the context of new technologies. He regularly advises policymakers.

References

Abdollahpouri H, Burke R and Mobasher B, ‘Controlling Popularity Bias in Learning-to-Rank Recommendation’, Proceedings of the eleventh ACM conference on recommender systems (2017)

Abdollahpouri H, Burke R and Mobasher B,, ‘Managing Popularity Bias in Recommender Systems with Personalized Re-Ranking’ [2019] arXiv preprint arXiv:1901.07555

Abdollahpouri H and Mansoury M, ‘Multi-Sided Exposure Bias in Recommendation’ <http://arxiv.org/abs/2006.15772> accessed 1 May 2025

Angwin J and others, ‘Machine Bias’ (ProPublica) <https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing> accessed 22 April 2025

Beigang F, ‘Yet Another Impossibility Theorem in Algorithmic Fairness’ (2023) 33 Minds and Machines 715

Bell A and others, ‘The Possibility of Fairness: Revisiting the Impossibility Theorem in Practice’, Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery 2023) <https://doi.org/10.1145/3593013.3594007> accessed 1 May 2025

Bertsimas D, Farias VF and Trichakis N, ‘On the Efficiency-Fairness Trade-Off’ (2012) 58 Management Science 2234

Biega AJ, Gummadi KP and Weikum G, ‘Equity of Attention: Amortizing Individual Fairness in Rankings’, The 41st international acm sigir conference on research & development in information retrieval (2018)

Binns R, ‘Fairness in Machine Learning: Lessons from Political Philosophy’, Conference on fairness, accountability and transparency (PMLR 2018)

Boratto L, Fenu G and Marras M, ‘Connecting User and Item Perspectives in Popularity Debiasing for Collaborative Recommendation’ (2021) 58 Information Processing & Management 102387

Chouldechova A, ‘Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments’ (2017) 5 Big Data 153

College voor de Rechten van de Mens, ‘Dating-App Breeze Mag (En Moet) Algoritme Aanpassen Om Discriminatie Te Voorkomen’ <https://www.mensenrechten.nl/actueel/nieuws/2023/09/06/dating-app-breeze-mag-en-moet-algoritme-aanpassen-om-discriminatie-te-voorkomen>

College voor de Rechten van de Mens, ‘Breeze Social B.V. Discrimineert Niet, Als Zij Maatregelen Neemt Die Voorkomen Dat Haar Algoritme Gebruikers Met Een Niet-Nederlandse Afkomst of Donkere Huidskleur Benadeelt.’ <https://oordelen.mensenrechten.nl/oordeel/2023-82>

Corbett-Davies S and others, ‘The Measure and Mismeasure of Fairness’ (2024) 24 The Journal of Machine Learning Research 312:14730

Court of Justice of the European Union, ‘Case C-83/14 (CHEZ Razpredelenie Bulgaria AD/Komisia Za Zashtita Ot Diskriminatsia)’

De Jonge T and Hiemstra D, ‘UNFair: Search Engine Manipulation, Undetectable by Amortized Inequity’, Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery 2023) <https://dl.acm.org/doi/10.1145/3593013.3594046> accessed 27 March 2024

Dehdashtian S, Sadeghi B and Boddeti VN, ‘Utility-Fairness Trade-Offs and How to Find Them’ (2024) <https://openaccess.thecvf.com/content/CVPR2024/html/Dehdashtian_Utility-Fairness_Trade-Offs_and_How_to_Find_Them_CVPR_2024_paper.html> accessed 9 May 2025

Deldjoo Y and others, ‘Fairness in Recommender Systems: Research Landscape and Future Directions’ (2024) 34 User Modeling and User-Adapted Interaction 59

Diaz F and others, ‘Evaluating Stochastic Rankings with Expected Exposure’, Proceedings of the 29th ACM International Conference on Information & Knowledge Management (Association for Computing Machinery 2020) <https://dl.acm.org/doi/10.1145/3340531.3411962> accessed 24 May 2024

Eindhoven University of Technology, ‘TU/e resumes preferential policy for hiring female scientists’ (19 April 2021) <https://www.tue.nl/en/news/news-overview/tue-resumes-preferential-policy-for-hiring-female-scientists/>

Ekstrand MD and others, ‘Fairness in Recommender Systems’ in Francesco Ricci, Lior Rokach and Bracha Shapira (eds), Recommender Systems Handbook (Springer US 2022) <https://doi.org/10.1007/978-1-0716-2197-4_18> accessed 22 April 2025

‘FAQ’ <https://breeze.social/nl/faq> accessed 8 May 2025

Feldman M and others, ‘Certifying and Removing Disparate Impact’, Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Association for Computing Machinery 2015) <https://dl.acm.org/doi/10.1145/2783258.2783311> accessed 9 May 2025

Foulds JR and Pan S, ‘Are Parity-Based Notions of AI Fairness Desirable?’ [2020] IEEE Data Eng. Bull. <https://www.semanticscholar.org/paper/Are-Parity-Based-Notions-of-AI-Fairness-Desirable-Foulds-Pan/b811870a7aa2b806bb51cbac2f149bd27566a474> accessed 27 March 2024

Gajane P and Pechenizkiy M, ‘On Formalizing Fairness in Prediction with Machine Learning’ [2017] arXiv preprint arXiv:1710.03184

Germano F, Gómez V and Le Mens G, ‘The Few-Get-Richer: A Surprising Consequence of Popularity-Based Rankings?’, The World Wide Web Conference (2019)

Hardt M, Price E and Srebro N, ‘Equality of Opportunity in Supervised Learning’, Proceedings of the 30th International Conference on Neural Information Processing Systems (Curran Associates Inc 2016)

Hitsch GJ, Hortaçsu A and Ariely D, ‘What Makes You Click?—Mate Preferences in Online Dating’ (2010) 8 Quantitative marketing and Economics 393

Hutson JA and others, ‘Debiasing Desire: Addressing Bias & Discrimination on Intimate Platforms’ (2018) 2 Proc. ACM Hum.-Comput. Interact. 73:1

Hwang W-C, ‘Who Are People Willing to Date? Ethnic and Gender Patterns in Online Dating’ (2013) 5 Race and Social Problems 28

Kamiran F, Žliobaitė I and Calders T, ‘Quantifying Explainable Discrimination and Removing Illegal Discrimination in Automated Decision Making’ (2013) 35 Knowledge and Information Systems 613

Kearns M and others, ‘Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness’ <http://arxiv.org/abs/1711.05144> accessed 24 May 2024

Kleinberg J, Mullainathan S and Raghavan M, ‘Inherent Trade-Offs in the Fair Determination of Risk Scores’ <http://arxiv.org/abs/1609.05807> accessed 22 April 2025

Kusner MJ and others, ‘Counterfactual Fairness’ (2017) 30 Advances in neural information processing systems

Mitchell S and others, ‘Algorithmic Fairness: Choices, Assumptions, and Definitions’ (2021) 8 Annual Review of Statistics and Its Application 141

Morik M and others, ‘Controlling Fairness and Bias in Dynamic Learning-to-Rank’, Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval (2020)

Naudts L, Vedder A and Smuha N, ‘Fairness and Artificial Intelligence’ <https://lirias.kuleuven.be/handle/20.500.12942/760314> accessed 8 May 2025

Nowak T, ‘Dutch Positive Action Measures in Higher Education in the Light of EU Law’ (2022) 29 Maastricht Journal of European and Comparative Law 468

Pizzato LA and Silvestrini C, ‘Stochastic Matching and Collaborative Filtering to Recommend People to People’, Proceedings of the fifth ACM conference on Recommender systems (2011)

Potârcă G and Mills M, ‘Racial Preferences in Online Dating across European Countries’ (2015) 31 European Sociological Review 326

Raj A and Ekstrand MD, ‘Comparing Fair Ranking Metrics’ <http://arxiv.org/abs/2009.01311> accessed 9 May 2025

Sapiezynski P and others, ‘Quantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists’, Companion Proceedings of The 2019 World Wide Web Conference (Association for Computing Machinery 2019) <https://dl.acm.org/doi/10.1145/3308560.3317595> accessed 9 May 2025

Selbst AD and others, ‘Fairness and Abstraction in Sociotechnical Systems’, Proceedings of the Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery 2019) <https://dl.acm.org/doi/10.1145/3287560.3287598> accessed 7 May 2025

Singh A and Joachims T, ‘Fairness of Exposure in Rankings’, Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Association for Computing Machinery 2018) <https://dl.acm.org/doi/10.1145/3219819.3220088> accessed 24 May 2024

Strathern M, ‘“Improving Ratings”: Audit in the British University System’ (1997) 5 European Review 305

Van Bekkum M and Borgesius FZ, ‘Using Sensitive Data to Prevent Discrimination by Artificial Intelligence: Does the GDPR Need a New Exception?’ (2023) 48 Computer Law & Security Review 105770

W Flores A, Bechtel K and Lowenkamp C, ‘False Positives, False Negatives, and False Analyses: A Rejoinder to “Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And It’s Biased Against Blacks.”’ (2016) 80 Federal probation

Wachter S, Mittelstadt B and Russell C, ‘Why Fairness Cannot Be Automated: Bridging the Gap between EU Non-Discrimination Law and AI’ (2021) 41 Computer Law & Security Review 105567

Wang H, Wang Z and Zhang W, ‘Quantitative Analysis of Matthew Effect and Sparsity Problem of Recommender Systems’, 2018 IEEE 3rd International Conference on Cloud Computing and Big Data Analysis (ICCCBDA) (IEEE 2018)

Wang Y and others, ‘A Survey on the Fairness of Recommender Systems’ (2023) 41 ACM Trans. Inf. Syst. 52:1

Watkins EA and Chen J, ‘The Four-Fifths Rule Is Not Disparate Impact: A Woeful Tale of Epistemic Trespassing in Algorithmic Fairness’, Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery 2024) <https://dl.acm.org/doi/10.1145/3630106.3658938> accessed 9 May 2025

Weerts H and others, ‘Algorithmic Unfairness through the Lens of EU Non-Discrimination Law: Or Why the Law Is Not a Decision Tree’, Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery 2023) <https://dl.acm.org/doi/10.1145/3593013.3594044> accessed 7 May 2025

Weerts H and others, ‘The Neutrality Fallacy: When Algorithmic Fairness Interventions Are (Not) Positive Action’, Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery 2024) <https://dl.acm.org/doi/10.1145/3630106.3659025> accessed 3 April 2025

Wong P-H, ‘Democratizing Algorithmic Fairness’ (2020) 33 Philosophy & Technology 225

Xia B and others, ‘WE-Rec: A Fairness-Aware Reciprocal Recommendation Based on Walrasian Equilibrium’ (2019) 182 Knowledge-Based Systems 104857

Zafar MB and others, ‘From Parity to Preference-Based Notions of Fairness in Classification’, Advances in Neural Information Processing Systems (Curran Associates, Inc 2017) <https://proceedings.neurips.cc/paper_files/paper/2017/hash/82161242827b703e6acf9c726942a1e4-Abstract.html> accessed 9 May 2025

Zehlike M and Castillo C, ‘Reducing Disparate Exposure in Ranking: A Learning To Rank Approach’, Proceedings of The Web Conference 2020 (Association for Computing Machinery 2020) <https://dl.acm.org/doi/10.1145/3366424.3380048> accessed 24 May 2024