Interacting with vector databases by means of domain-specific language

2.1 DSLs for NoSQL databases access

The rapid evolution of NoSQL systems has prompted the development of model-driven (MD) solutions that abstract vendor-specific intricacies and provide uniform access to heterogeneous data stores [13]. Initially, querying tasks were focused on by these solutions, and subsequently, operations such as data migration, transformation, and schema mapping were incorporated.

It is acknowledged that vector databases are classified as one type of NoSQL systems, and access to NoSQL databases in general, including vector databases, is thereby provided. The investigation is hereby aimed at determining whether DSLs for vector database access are available. In this section, we provide an overview of some prominent DSLs developed for NoSQL database access, where the complexity of interacting with diverse NoSQL backends is encapsulated by these languages through tailored abstractions and syntax.

To begin with, the save our systems interface [13] was introduced in 2014 to standardize data access across heterogeneous NoSQL systems. A mechanism was provided to translate queries between various database models, such as document-oriented stores, extensible record stores, and key value stores, which allowed uniform querying and data manipulation capabilities to be applied across platforms such as HBase [14], MongoDB [15], and Redis [16].

Expanding on this work, ATHENA [17] was developed, as an ontology-driven system for natural language querying over relational databases. In ATHENA, natural language queries were mapped to an intermediate ontology query language and then compiled into SQL. Through this process, the decoupling of query formulation from the underlying schema was achieved. Although ATHENA targeted relational databases, its MD methodology provided valuable insights into abstracting complex query formulations.

Following this, attention was directed toward addressing scalability challenges in model persistence with the development of NeoEMF [18]. It was designed to support multiple database backends, including key-value databases, graph databases, and column databases. Integration into the eclipse modeling framework (EMF) allowed for seamless use with the existing modeling tools, while an advanced API was provided for fine-grained database tuning. As a result, a flexible and scalable persistence solution for handling extensive model-based data was established.

Advancements continued with the model-based methodology presented in the study by Kolovos et al. [19], which addressed the design, deployment, and querying of hybrid NoSQL and relational data persistence architectures. In this work, three DSLs were developed; however, two are particularly relevant to this discussion: (i) TyphonDL was proposed to bridge the gap between high-level designs and concrete database configurations through automated deployment, thereby addressing aspects of data definition language (DDL) and system configuration; and (ii) TyphonQL was created to compile high-level queries into database-specific commands, thereby facilitating querying across heterogeneous systems that include document-oriented databases, key-value databases, and graph databases.

In a related advancement, an extension of the Cypher query language was detailed by Castellort and Martin [20]. The integration of the Fuzzy4S framework allowed fuzzy logic to be incorporated into the querying process, thereby supporting the formulation of approximate queries and enhancing data manipulation capabilities in graph environments such as Neo4J.

Concurrently, research was conducted on schema design in column-family and document-oriented databases, leading to the development of Mortadelo [21]. In this solution, agnostic generic data model (GDM) was employed to support DDL operations and data modeling for these types of databases. A three-stage transformation process was utilized to automatically generate physical implementations (e.g., Cassandra query language (CQL)), and a textual DSL was provided to define and validate GDM instances.

Further contributions have been made in the area of structured access to heterogeneous data sources through the development of a JavaScript object notation (JSON)-based DSL [22]. In this approach, the JSON grammar is leveraged to define and process domain-specific queries and transformations, thereby facilitating interoperability between different NoSQL databases. The DSL is primarily designed for document-oriented stores such as MongoDB [15], CouchDB [23], and Firebase Firestore [24], where JSON-based data are natively managed. Moreover, DSL is employed to standardize data exchange between client-side and server-side applications, thereby ensuring consistency in data representation. This JSON-based DSL provides an abstraction layer that facilitates schema management, querying, and integration, thereby enhancing the efficiency of web applications that rely on such storage solutions.

SkiQL [25], a unified schema query language, was proposed to facilitate uniform access to NoSQL database schemas. Designed as a platform-independent query language, it is utilized for logical schema querying across columnar, document-oriented, key-value, graph, and relational databases. SkiQL was designed to enable schema extraction, visualization, and querying. Although data manipulation is not supported, schema management tasks such as modeling and transformation are assisted. By offering a standardized method for querying NoSQL schemas, SkiQL contributed to multimodel database management, ensuring consistency across diverse NoSQL environments.

We can conclude that there is a number of DSLs aimed at supporting: data modeling, DDL and data manipulation language (DML) operations, and querying across a variety of NoSQL systems. However, based on the presented literature, no comprehensive DSL has been developed to provide uniform access to vector databases managed by different VDBMSs. The increasing relevance of vector databases in applications that involve similarity search and high-dimensional data analysis is noted, indicating the need for further research on MD solutions tailored to the management of vector databases. In the following section, we explore an overview of vector database access strategies.

2.2 Overview of vector databases accessing methods

Vector databases are used as storage systems for the outputs of embedding models, with vectors – referred to as vector embeddings [26] – being preserved within their domain. The speed of similarity searches is enhanced by these embeddings being stored, with user queries being vectorized and matched to relevant vector embeddings. ML algorithms are utilized to index these embeddings [27], with specialized data structures being mapped to optimize quick similarity or distance calculations, particularly in approximate nearest neighbor searches [7].

The searching process of nearest neighbors is executed by the calculation of distances between stored and query vectors using similarity metrics such as cosine similarity [27], performed by a VDBMS [9]. The most similar vectors are subsequently provided in response to specific queries, and these computed distances are applied in various ML applications, including semantic search, recommendation systems, image recognition, and natural language processing [8].

This section focuses on comprehension of accessing methods to vector databases, identifying which languages are supported by databases of various vendors and determining whether they offer their own DSL, query language or both. During the research, ten VDBMSs and eight multi-model database management systems (MMDBMSs) with the vector data model supported were analyzed. MMDBMSs support diverse data models, such as relational, document-oriented, vector, graph, and key-value, enabling the seamless integration and management of heterogeneous data, while reducing structural complexity. The analysis is carried out on widely used database management systems (DBMSs) in 2024 and 2025. Analyzed DBMSs are as follows:

• VDBMSs: Milvus [28], Pinecone [29], Qdrant [30], Weaviate [31], Chroma [32], Acitan Vector [33], Vald [34], DeepLake [35], SvectorDB [36], and Transwarp Hippo [37].

• MMDBMSs: OpenSearch [38], Microsoft Azure AI Search [39], Apache Cassandra [40], MyScale [41], Elasticsearch [42], Marqo [43], JaguarDB [44], and Vespa [45].

• VDBMS/MMDBMS – evaluated VDBMSs and MMDBMSs;

• Native client libraries – languages supported by the DBMSs via native client libraries;

• Additional access – alternative connectivity methods supported by the DBMSs, with “ANY” is used to denote that it can be accessed using any programming language via standard APIs such as representational state transfer (REST), JDBC or ODBC; and

• DSL/query interface type – the available query interfaces or DSLs supported by the DBMSs.

Based on the analysis that has been conducted, it has been observed that access to vector databases is primarily achieved through programming code, which is facilitated by language-specific libraries and REST APIs that enable language-independent access. Support for dynamically typed, high-level programming languages that facilitate rapid development and ease of integration has been emphasized, as evidenced by Python being supported by 94.4% of the surveyed systems, while Java, Go, Node.js, and JavaScript have been supported by 55.6%, 44.4%, 27.8%, and 22.2% of the systems, respectively. Query interfaces have been implemented in either SQL-like or non-SQL forms, with only a few systems having been provided with a DSL that is typically limited to querying. Approximately 33.3% of the systems have been equipped with SQL-like or extended SQL interfaces – examples being those provided by Milvus (via Vanna), Actian Vector, Transwarp Hippo, Apache Cassandra (via CQL), MyScale, and OpenSearch – while the remaining 66.7% have been equipped with non-SQL query languages, including JSON-based, RESTful API, GraphQL-inspired, and proprietary query languages.

Challenges in interoperability and query migration have been found to be contributed by the predominance of non-SQL interfaces, owing to the steeper learning curve and syntactic heterogeneity that have been introduced. Confusion has been observed to arise when the SQL syntax, which closely resembles that of relational DBMSs, is employed to describe contextually different concepts.

Despite the availability of diverse programming languages and query methods, the mastery of vector database concepts is hindered by the inherent complexity and heterogeneity that is observed across current VDBMSs. Various query methods – including SQL-like, JSON-based, GraphQL-inspired languages, and DSLs custom-made for specific VDBMSs or MMDBSMs – have been noted, and difficulties have been introduced to integration, learning, and maintenance [10]. Furthermore, potential misunderstandings are introduced by the differing syntaxes used for accessing various vector databases and by the inconsistent application of terminology, where synonyms may represent the same concepts and homonyms may denote different concepts. This variability in terminology and syntax further complicates the learning process.

We determine that a novel language is required to provide straightforward access to vector databases and to unify heterogeneous concepts, thereby addressing the identified challenges of integration, learning, and maintenance. Here, we propose employing the MDSD paradigm to abstract vendor-specific intricacies and provide access to different VDBMSs through a single language. During the development of this language, careful consideration is to be given to the naming of concepts so that end-users may be allowed to focus on working with vector databases rather than on deciphering complex syntax. Finally, the syntax for database access is to be designed to be clear, concise, and easy to learn.

2.3 Foundation of MDSD paradigm and DSLs

Within the MD paradigm [46,47], the entire system development is regarded as being guided by models that are considered to be foundational artifacts for all phases of development. When the decision is made that complex systems are to be designed with guidance provided by models rather than with manually written program code, it is required that the multifaceted role played by these models be understood.

MDSD extends the MD paradigm by utilizing models to guide software system development. This paradigm is supported by documented observations, as it is recognized that software artifacts are rendered increasingly complex. Different levels of abstraction are employed to manage such complexity in a manner tailored to the profiles of the users involved, the stages of the development process, and the specific goals of the work. It is further observed that software is made to permeate various spheres of everyday life and that an increased demand is placed on both the development of new systems and the evolution of existing ones. Moreover, a shortage of skills in software development is periodically experienced in relation to prevailing job demands. The collaborative nature of development is acknowledged, and collaboration is frequently observed to be required with individuals who are not directly involved in software development, such as clients or managers. Their input is regarded as crucial for the description of the technical aspects of the system.

In this context, the organization of models across multiple layers of abstraction is carried out enabling automated or semi-automated model-to-model or model-to-text transformations. The generation of implementation artifacts from abstract models is thereby facilitated. The advantages of modeling over traditional methods – including direct manual coding and ad-hoc software development practices – have been documented, which include [46]: (i) enhancing the speed of software system development through automation and centralized knowledge representation; (ii) improving software quality through formalization; (í) increasing the reusability of models; and (iv) reducing system complexity through the use of abstraction levels.

In MDSD, DSLs are employed to create models and code generators are utilized to transform models into other models, text or executable code. DSLs are designed for specific domains or contexts, allowing users to define solutions using familiar concepts. In contrast, general purpose languages (GPLs) can be applied to any sector or domain but require additional time and expertise for domain-specific tasks [48].

A lack of uniform access to vector databases has been observed, and reliance on proprietary APIs and vendor-specific query languages is known to result in strong coupling between application code and individual systems, potentially hindering future migration between different VDBMSs. In addition, difficult management of VDBMSs in database systems containing different VDBMSs is encountered. It has been determined that the development of a novel language may be required to provide more straightforward access to vector databases and to unify heterogeneous concepts.

The employment of the MDSD paradigm is proposed to abstract vendor-specific intricacies and to facilitate access to different vector databases through a single language. Particular attention was devoted to the designation of the language’s concepts so as to ensure emphasis remains on interaction with vector databases rather than on interpretation of intricate syntax. It was demonstrated by Rädler et al. [11] that the adoption of MDSD principles may simplify development processes and foster standardization and reusability.

Within this context, we propose a novel DSL, named vecDSL, and offer an MDSD solution to establish a uniform method for interacting with vector databases. With vecDSL being developed as a prototype declarative language, it is anticipated that vendor-specific dependencies may be reduced, data management may be streamlined, and performance assessments may be facilitated for more efficient execution.

4.1 Abstract syntax of vecDSL

The vecDSL meta-model is presented in Figures 2 and 3, and each class of the meta-model is referenced in the text with its name written in italics. Diagrams from these figures represent a single meta-model visually divided to enhance clarity and facilitate a better understanding of its content. Also, there is a notable duality in the meta-model, with two classes playing crucial roles while clearly distinguishing different aspects of the meta-model. One class (Script) models script at a high level of abstraction, while the other (Concept) defines the concepts of a vector database on which these scripts will be executed, thereby delineating distinct concerns within the meta-model. Therefore, the concept view of the vecDSL meta-model is presented in Figure 2 and discussed first in this section, while the script view of the vecDSL meta-model is presented in Figure 3 and discussed afterward. To facilitate a comprehensive understanding of vector database concepts, an overview of key concepts essential for users is provided in this section as well.

Figure 2

The concept view of the vecDSL abstract syntax.

Figure 3

The script view of the vecDSL abstract syntax.

4.2 Example of using concrete syntax of vecDSL

In Listings 1 and 2, an example of DDL, DML and Query statements written by vecDSL are presented in the domain of faculty courses. Through the examples, a model was created to support operations within a VDBMS, demonstrating the process of creating data structures, inserting data into the database, and querying the stored data. The data structure of a university, a faculty, and its courses is created in a vector database, and after vectors representing courses are stored and changed, search through courses is performed.

Listing 1: An example of a vecDSL script model for defining the structure of a Milvus vector database.

Listing 2: An example of a vecDSL script model for data manipulation operations in Milvus vector database.

Scripts can be used for any VDBMS for which a code generator is provided. In this example, script models are designed for the Milvus VDBMS by having the vendor parameter set to “MILVUS” (Listings 1 and 2, lines 1 and 2). By defining idNumber inside of script (Listing 1, line 3), it is granted that all written statements inside that script can be referenced in another script (Listing 2, line 3), by citing that same idNumber. Also, this indicates that a user which is citing specific idNumber has all equivalent access rules and permissions within specific vector database instance as the user who created the script. The scripts are defined with a series of methods (Listing 1 and Listing 2, line 5) representing various DDL, DML and Query statements.

In Listing 1, statements for creating a vector database structure are presented. A vector database named “UniversityDb” (Listing 1, lines 6–9) is created via a connection whose host is set to “localhost” and port to “8081.” A cluster named “FacultyOfScience” is created within “UniversityDb” (Listing 1, lines 11 and 12) with its capacity defined as two collections. Thereafter, a collection named “ScienceCourses” is created within “FacultyOfScience” (Listing 1, lines 14–16), is marked as dynamic, and is provided with a description. A limit called “CourseLimit” is created for “ScienceCourses” (Listing 1, lines 18–20), with its type set to “NOFFIELDS” and its value set to “4,” so that the number of fields is restricted.

Subsequently, a nonvector field named “courseId” is created in “ScienceCourses” (Listing 1, lines 22–25) is designated as primary, typed as “INT,” and provided with a description, while a nonvector index named “CourseIdIndex” is created for “courseId” (Listing 1, lines 27 and 28) with its type set to “INTEGER,” so that efficient lookup is enabled. A partition named “UndergradPartition” is created in “ScienceCourses” for the field “courseId” (Listing 1, lines 30 and 31), with its partition strategy set to “HASH,” thereby ensuring that data segmentation is defined. Another non-vector field, “courseName,” is created in “ScienceCourses” (Listing 1, lines 33 and 34) with its type set to “VARCHAR.” In addition, a vector field named “courseDescription” is created in “ScienceCourses” (Listing 1, lines 36–39) with its type set to “FLOAT_VECTOR,” its dimension defined as “768,” and embedding model specified as “SentenceTransformer(BAAI/bge-m3),” thereby ensuring that vector embeddings are generated for similarity search. A limit called “DescLimit” is created for “courseDescription” (Listing 1, lines 41–43) with its type set to “VECTORDIM” and its value set to “768,” so that the vector dimensionality is enforced. Also, a vector index named “CourseVecIndex” is created for the referenced vector field (Listing 1, lines 45–47) with its type set to “HNSW” and its metric set to “COSINE” so that similarity searches are accelerated. Furthermore, a non vector field named “creditHours” is created in “ScienceCourses” (Listing 1, lines 49 and 50) with its type set to “INT.”

In Listing 2, a script model is presented that specifies a series of data manipulation operations for managing university-related data. Access to structures created in the previously provided script is granted by citing idNumber of the provided script (Listing 2, line 3).

The script begins with an insertion operation for “ScienceCourses” (Listing 2, lines 6–8), where data are loaded from the file located at “/data/science/courses.csv” and included first “1000” rows. Thereafter, an update operation is executed in “ScienceCourses” (Listing 2, lines 10–12) to load data from “/data/science/courses_update.csv” under conditions where “courseId” is matched with the pattern “CS%,” and a delete operation is executed in “ScienceCourses” (Listing 2, lines 14 and 15) for dataset instances under conditions where “creditHours” is determined to be greater than “5.”

A vector search is executed on “ScienceCourses” (Listing 2, lines 17–23) with parameters specifying a top k value of “10,” a metric of “COSINE,” a range of “0.8,” and a radius of “0.9,” while output fields “courseName” and “courseDescription” are selected and filters are applied so that only records with “creditHours” greater than or equal to “3” are returned. A non-vector search is executed on “ScienceCourses” (Listing 2, lines 25–28) with a top k value of “5,” output fields set to “courseId” and “courseName,” and filters applied so that records with “courseId” matching the pattern “CS%” are returned.

An alteration is executed on the cluster “FacultyOfScience” within “UniversityDb” (Listing 2, lines 30–32) to delete field “creditHours.” Finally, a drop operation is executed on the instances of the collection “ScienceCourses” (Listing 2, line 34).

4.3 Example of generated executable python script

The scripts presented in Listing 1 and Listing 2 are being converted into Python code, by a code generator made for such a purpose. As noted in Section 2, Python is most widely supported across vector databases, so transformations will be predominantly carried out into Python code, although adjustments can be made for other languages if required.

Listing 3: An example of a script model for DML and DDL operations in Milvus vector database.

When code generation templates are utilized, and after specific syntax concepts have been instantiated through a use case, code is generated based on a specific part of the template for that concept. Within Listing 3, an insight into the executable generated Python code for the Milvus VDBMS is given, where the example of the model from Section 4.2 provided in Listing 1 is taken, as well as part of the Listing 2, related to the INSERT method, to provide embedding generation source data path. Executable scripts for Pinecone and Chroma would be generated analogously to this example, if a parameter for those VDBMS is specified in the vecDSL script model, within the attribute vendor.

It has been demonstrated that the vecDSL script model encapsulates a comprehensive abstraction of vector database configuration – encompassing connection setup, database and cluster definition, collection creation with detailed field specifications, partitioning, and index configuration – in a concise and vendor-neutral format. In contrast, the generated Python code for Milvus, Pinecone, and Chroma exhibits substantial variations in syntax and structure due to the heterogeneous nature of their underlying APIs, which will be explained further in the next section. A uniform specification is provided that can be automatically transformed into target-specific code. The benefits of this approach include reduced development effort, minimized potential for human error, and enhanced maintainability when supporting multiple VDBMSs.

In the next section, the evaluation of vecDSL is outlined through a detailed analysis of its complexity, where various metrics are examined, comparisons with other database query languages are conducted, and insights into its structural efficiency and expressiveness are provided.

5.1 Diversity of languages’ syntaxes

In addition to the metric-based analysis, a brief example of a query has been provided in Table 2 to illustrate the diversity of syntax used across different VDBMSs. The following task has been designed to demonstrate a vector search query within a collection of “ScienceCourses”: “A vector search query must be constructed to retrieve university course data from a collection named ‘ScienceCourses.’ The query should return information on course names, descriptions, and credit hours. A vector similarity search should be performed using the cosine metric, retrieving the top 10 closest results within a similarity range of 0.8 and a radius of 0.9. In addition, filtering conditions must be applied to select only courses that begin with ‘Intro’ in their name, have a course ID greater than 101, and do not exceed 4 credit hours.”

Through the examination of access methods, distinct categories of DSLs have been identified, each reflecting a specific paradigm of interaction with vector databases. Representative VDBMSs or MMDBMSs have been selected for each category, allowing a comparison of their syntax and query structures. Consequently, vendor-specific examples have been developed for Pinecone, Chroma, and DeepLake, as well as for the MMDBMSs OpenSearch and Elasticsearch, ensuring comprehensive coverage of different DSL families.

We observe that JSON-based DSLs, such as those used by Pinecone, Chroma, and Elasticsearch, are constructed with a high degree of verbosity, where nested objects and arrays are explicitly defined (Table 2, Pinecone; Chroma; Elasticsearch). This explicit nesting has been seen to provide fine-grained control over filtering and selection operations, yet it has also been noted that the resulting syntaxes are more complex and less concise compared to those of SQL-like or GraphQL-inspired languages.

In addition, we determine that keywords make cross-system query translation difficult, such as:

• For field selection: “select” (Table 2, Pinecone, line 3; DeepLake, line 3; OpenSearch, line 3), “fields” (Table 2, Chroma, line 8), “_source” (Table 2, Elasticsearch, line 14),

• For filtering: “filter” (Table 2, Pinecone, line 9), “conditions” (Table 2, Chroma, line 9), “where” (Table 2, DeepLake, line 5; OpenSearch, line 5), or

• Variations in operator notation, such as “$gt” (Table 2, Pinecone, line 12), “gt” (Table 2, Chroma, line 11; Elasticsearch, line 10), “>” (Table 2, OpenSearch, line 5) or lambda expression “lambda x: x > 101” (Table 2, DeepLake, line 7), make cross-system query translation difficult.

Further, the handling of nested conditions has been implemented in different ways: JSON-based DSLs have been seen to employ operators like “$elemMatch” to address array filtering, while DSLs as those used in Chroma have been observed to utilize simpler operators such as “contains” to achieve similar objectives. The DSL of DeepLake has been designed to incorporate Python lambda functions, which have been noted to provide dynamic flexibility in constructing queries, although the potential for runtime errors has been highlighted as a drawback. In contrast, SQL-like DSLs, employed by OpenSearch and Vespa, have been designed to leverage the familiarity of SQL syntax, thereby reducing learning curves and simplifying query structures at the cost of expressiveness and granular control.

To build upon this foundational analysis of DSL diversity, a detailed evaluation of vecDSL’s complexity and structural efficiency is presented in the following section. By systematically measuring various syntactic and operational metrics, a comparative assessment is conducted to determine how vecDSL aligns with existing query languages in terms of verbosity, readability, and functionality. Through this quantitative examination, deeper insights into its usability and potential areas for improvement are gained, further complementing the qualitative observations made in the previous discussion.

5.2 Evaluation of syntactic and operational metrics of vecDSL

The evaluation in this study is conducted to assess the complexity and practicality of vecDSL in relation to various VDBMSs and MMDBMSs. The primary objective of the evaluation is to determine whether vecDSL provides a more streamlined method to interact with vector databases while identifying areas where improvements could be introduced. To achieve this, an analysis is carried out by measuring multiple syntactic complexity indicators across different query languages.

For evaluation purposes, scripts for all VDBMSs and MMDBMSs presented in Section 2 are created analogously scripts from Listing 1 and Listing 2. The evaluation of vecDSL script was performed using the original scripts from Listing 1 and Listing 2 without any modifications. By examining its structure, conciseness, and efficiency, valuable insights into vecDSL’s effectiveness are gained. A set of metrics was adopted from the established study [53] and complemented by additional measures developed during this research. The additional metrics were chosen to capture aspects of syntactic complexity specific to vecDSL, such as verbosity, structural depth, and cognitive load. They were selected to provide a more nuanced comparison with other query languages, particularly given the declarative nature of vecDSL. The resulting syntactic complexity indicators are described as follows.