Understanding the semantic structure of tabular data is essential for data integration and discovery. Specifically, the goal is to annotate columns in a tabular source with types and relationships between them using classes and predicates of a target ontology. Previous work either requires trained labeled data or exploits the overlapping data between the table data and a knowledge graph to predict types and relationships. However, these approaches cannot be used in a new domain with limited labeled data. To address this issue, we propose a novel domain-independent approach to estimate a score reflecting the semantic relatedness between a table column and an ontology class or property using the table metadata and data. Our empirical evaluation demonstrates that our approach significantly outperforms strong baselines based on large language models.