<aside> đź’ˇ
“Oh LLM, I’m Asking Thee, Please Give Me a Decision Tree”: Zero-Shot Decision Tree Induction and Embedding with Large Language Models
https://arxiv.org/pdf/2409.18594
</aside>
Zero-shot decision trees encode rich structural dependencies
between the input features and prediction target.
Stochastic Having a random probability distribution or pattern that may be analyzed statistically but may not be predicted precisely.
Something of interest here is Knowledge Distillation i.e we let the LLM use prior knowledge while constructing decision trees.
Here’s a summary of the summary by Gpt-4o
This work explores using large language models (LLMs) to create decision trees without training data, demonstrating that these **zero-shot** trees can outperform traditional data-driven trees on small tabular datasets while being interpretable and privacy-preserving. The study also shows that combining these trees with downstream models can match the performance of data-driven approaches. However, the findings are based on small datasets and may not apply to other contexts. The authors suggest that performance could improve with **more powerful LLMs** and **better prompting techniques**. They propose that LLMs as zero-shot model generators offer a new toolset for customizing machine learning models.