Recently the capability of Large language model (LLM) on multi-modal query planning has been investigated. However, most studies focus on query plan generation and ignore query optimization, leaving a big gap towards practical LLM-based multi-modal query planner. In this paper, we investigate the query optimization ability of LLM over multi-modal data and propose LaPuda, a novel LLM and Policy based query optimizer for multi-modal data. Compared to existing LLM-based planning methods that let LLM reason how-to-optimize purely from examples (example-based), and traditional query optimizer that strictly follows pre-defined rules (rule-based), LaPuda deploys abstract policies (which are higher level summaries of the rules) to guide LLM in the optimization with the examples. Furthermore, to prevent LLM from making mistakes or negative optimization, we borrow the idea of gradient descent and propose a guided cost descent (GCD) algorithm to perform the optimization, such that the optimization can be kept in the correct direction. In our evaluation, our methods consistently outperform the baselines in most cases. For example, the plans optimized by our methods result in up to 3x execution speed compared to those by the baselines.