Other discussion

[qcjySONG]

Thank you for this interesting work, which enables the 7B - sized model to achieve such good results on BIRD as well.
The evaluation metrics of the BIRD leaderboard are relatively strict. Specifically: floating - point numbers must show full precision, the display of columns must be the same as the quantity and order arrangement in the question, and so on. These conditions and the uncontrollability of LLM have caused a relatively obvious decline in the performance of my work results. And OmniSQL - 7B can achieve an impressive score of 69.04 on DEV. May I ask if your model has solved the problems in this regard? Or assuming that an external auxiliary alignment correction is added, your model can still gain improvement again (and it will be a surprising improvement). Forgive me for not being able to fully run your model to verify this problem for the time being.
I would be very grateful if you could reply. Best regards.

[lihaoyang-ruc]

Hi, thank you for your attention to our work and for your insightful questions.

Question 1: The problem of the evaluation metric (i.e., the execution accuracy metric).

Answer 1:
For text-to-SQL, I acknowledge that execution accuracy (EX) is not an ideal evaluation metric. As you pointed out, there are often cases that would be considered correct by humans but are marked as incorrect by the EX metric. Therefore, the model's true capability is likely underestimated by the EX score. Unfortunately, we have not yet found a better evaluation metric. Some recent works have tried to leverage LLMs to simulate human experts as evaluators for text-to-SQL, which can partially mitigate issues with EX, but this approach introduces new challenges such as hallucination, unreliability, and non-determinism inherent to LLMs.
Your observation also explains why powerful closed-source models like GPT-4o may perform worse than Qwen2.5-Coder on certain datasets (such as Spider and BIRD) in terms of EX score, even though GPT-4o's actual text-to-SQL abilities are often better in real user scenarios. Qwen2.5-Coder is likely trained on the training sets of Spider and BIRD, making it more aligned with the specific patterns of these datasets and resulting in a higher EX score.

Question 2: How does OmniSQL-7B achieve 69.04% on BIRD's dev set?

Answer 2:
Actually, I did not use any external auxiliary alignment or correction techniques. The only method I used is self-consistency: I set the temperature to 1.2 to encourage greater diversity, increased the number of samples to 32, and then selected the final SQL query based on execution result majority voting.
Additionally, while submitting to BIRD, I found that the test set contains one or more very wide databases with many tables and columns, which can cause the context length to be exceeded. To address this, I added a RoBERTa-based schema filter (as used in CodeS and RESDSQL). This component did not bring significant performance gains—sometimes it even slightly hurt performance due to imperfect recall—but it ensured that the code I submitted would not raise errors.
One more thing, in my previous experiments, I actually found that OmniSQL-7B's performance on the BIRD dev set could reach a 71.3% EX score when supplemented with additional information, such as the detailed column descriptions provided in this file. However, to ensure consistency in the experimental settings, I ultimately decided not to report this score.

[qcjySONG]

Thank you for the author's patient response and for introducing so many technical details to me. Also, thank you to Renmin University of China for its contributions to the field of Text-to-SQL!