Problem
Business-intelligence questions involve time periods, comparisons, joins, and organization-specific schemas. General NL2SQL benchmarks and exact-match scores can miss these demands and conceal partially correct answers.
Approach
BIS organizes 239 questions into nine categories over two SQLite databases. It evaluates both SQL structure through abstract-syntax-tree comparisons and query results through column matching, giving partial correctness a measurable role.
Evidence
The contribution is a BI-oriented dataset and evaluation procedure. Its examples explain how query similarity and result similarity reveal different errors; they do not establish that one NL2SQL model is universally best.
Scope
The benchmark covers a bounded set of databases and business questions. Partial similarity is useful for diagnosis, but should not be mistaken for permission to execute an incorrect business query.
Sources
Bibliographic details
- Authors
- Bora Caglayan, Mingxue Wang, John D. Kelleher, Shen Fei, Gui Tong, Jiandong Ding, Puchao Zhang
- Venue
- ICSOC 2024
- Status
- Published
- Preprint posted
- 2024-10-30
- First published
- 2024-12-07
- Proceedings year
- 2025