Artificial intelligence (AI) is rapidly introducing groundbreaking tools that are reshaping various aspects of healthcare. While fields like genetics have engaged in complex debates about AI's integration, the application of AI tools — such as machine learning (ML), natural language processing (NLP), and large language models (LLMs) — in end-of-life care has received little attention. Given the human-centric skills traditionally associated with end-of-life care — like effective patient- centered communication, supporting memory-making processes, and enhancing quality of life through spiritual care — some might argue that this field is inherently resistant to AI's transformative impact. Recently, however, discussions have emerged about the potential for predictive algorithms to replace the role of the surrogate decision-maker, a human fiduciary appointed to make decisions for individuals no longer capable of expressing their treatment preferences. Of particular importance is the proposal by well-known bioethicists Bryan Earp and Julian Savulescu: the Personalized Patient Preference Predictor (P4). This fine-tuned LLM leverages machine learning to infer incapacitated patients' preferences using personal data, acting as a "non-emotive digital psychological twin." While this approach aims to respect the principle of autonomy, it raises concerns about quantifying a deeply existential context like end-of-life decisions. Can we, and should we, envision a role for algorithmic tools in such profoundly "human" choices? How do core bioethical principles like autonomy, beneficence, and non-maleficence intersect or conflict in this context? Are we heading toward a future where decision-making authority is delegated to predictive algorithms, even in matters of life and death?