Comparing Vector Databases on GCP (Google)

To See All Articles About Technology: Index of Lessons in Technology

Selecting a GCP Database for Storing and Querying 31.5 Million Vector Embeddings

1. Executive Summary

This report evaluates several Google Cloud Platform (GCP) database services for storing and querying a dataset of 31.5 million vector embeddings. The increasing prevalence of vector embeddings in applications such as semantic search, recommendation engines, and Retrieval-Augmented Generation (RAG) necessitates efficient and scalable storage and retrieval solutions. This analysis focuses on Vertex AI Vector Search, AlloyDB for PostgreSQL, Cloud SQL for PostgreSQL, Firestore, and BigQuery, assessing their capabilities in terms of architecture, storage, querying, scalability, performance, and cost. The findings indicate that each option presents distinct advantages and disadvantages depending on the specific usage patterns and priorities of the user. Vertex AI Vector Search emerges as a purpose-built solution with high scalability and low latency, while AlloyDB and Cloud SQL for PostgreSQL offer integration with the familiar PostgreSQL ecosystem and robust relational database features. Firestore provides a serverless NoSQL approach with ease of use, albeit with certain limitations, and BigQuery excels in analytical workloads and large-scale data processing. The optimal choice will ultimately depend on a careful consideration of performance requirements, scalability needs, existing infrastructure, team expertise, and budget constraints.

2. Introduction

Vector embeddings, which are numerical representations capturing the semantic meaning of unstructured data like text, images, and audio, have become a cornerstone in the development of intelligent applications . By mapping content into a high-dimensional vector space where semantic similarity is represented by proximity, these embeddings enable efficient similarity searches, powering features like finding related products, retrieving relevant documents, and building sophisticated recommendation systems . The ability of vector databases to efficiently index and query these embeddings is crucial for unlocking the potential of semantic search . The user in this scenario has a specific requirement to store and query a substantial dataset of 31.5 million vector embeddings within the GCP ecosystem. This report will delve into the capabilities of several GCP database services that support vector embeddings, providing a comparative analysis to aid in the selection process. The subsequent sections will introduce each database option, followed by a detailed comparison of their features, performance characteristics, scalability limits, and estimated costs, taking into account different usage scenarios.  

3. GCP Database Options for Vector Embeddings

3.1. Vertex AI Vector Search

Vertex AI Vector Search is a fully managed service on GCP designed specifically for high-scale, low-latency vector similarity matching . It leverages Google's Scalable Nearest Neighbor (ScaNN) search technology, making it well-suited for applications demanding rapid retrieval of similar vectors from large datasets . This service underpins various AI-powered applications, including e-commerce product recommendations, RAG systems for enhancing large language model responses with contextual data, and personalized search functionalities . Vertex AI Vector Search supports multiple types of searches, including those based on dense embeddings that capture semantic meaning, sparse embeddings focused on keyword matching, and hybrid searches that combine both approaches . Key concepts associated with this service include the representation of data as vectors (lists of float values), the semantic representation of data through embeddings, the organization of vectors into an index for efficient searching, the deployment of an index to an endpoint for querying, the measure of search accuracy through recall, and the ability to filter search results using restrictions .  

To utilize Vertex AI Vector Search, the initial step involves preparing the vector embeddings, which are typically generated using a separate embedding model . These embeddings are then used to create an index within Vertex AI Vector Search . This process often involves storing the embedding data in Cloud Storage and then configuring the index with parameters that define the similarity metric (such as cosine similarity, dot product, or Euclidean distance) and the desired number of nearest neighbors to retrieve . Once the index is built, it is deployed to an endpoint, making it accessible for querying . Users can then send search queries to this endpoint to find vectors similar to a given query vector, with the option to apply filters to narrow down the results based on specific criteria . It is important to note that Vertex AI Vector Search focuses on the storage and querying of embeddings and does not itself provide the functionality to generate these embeddings .  

Vertex AI Vector Search is engineered to handle datasets of immense scale, capable of managing billions of embeddings and supporting high query throughput with remarkably low latency . Performance benchmarks on publicly available datasets have demonstrated its ability to maintain sub-10 millisecond latency even when searching across millions or billions of vectors . Furthermore, the service supports auto-scaling, automatically adjusting resources to maintain consistent performance as the query load fluctuates . While specific scalability limits for a dataset of 31.5 million embeddings are not explicitly detailed in the provided materials, the general architecture and performance metrics suggest that it can comfortably accommodate this volume of data .  

The pricing structure for Vertex AI Vector Search comprises the costs associated with the virtual machines (VMs) that host the deployed indexes and the expenses incurred during the building and updating of these indexes . Even for moderate-sized applications, a minimal setup might incur costs under $100 per month . The primary cost components include index serving, which is determined by the machine type and the region where the endpoint is deployed, and index building, which is charged per GiB of data processed . An example involving a smaller dataset of 10,000 records indicated that serving costs can constitute a significant portion of the total monthly expenditure . The selection of an appropriate shard size, which influences the required machine type, plays a crucial role in cost optimization . Notably, the serving cost model lacks a purely pay-as-you-go option, often being calculated on a continuous 24/7/30 basis, which might appear expensive for projects with infrequent querying needs .  

3.2. AlloyDB for PostgreSQL with Vector Extensions

AlloyDB for PostgreSQL is a fully managed, PostgreSQL-compatible database service on GCP that incorporates built-in AI capabilities . It offers seamless integration with the pgvector extension, enabling users to efficiently store and query vector embeddings within a familiar relational database environment . A key advantage of AlloyDB is its direct integration with Vertex AI, allowing for the generation of vector embeddings from text data using the embedding() function directly within database queries . Furthermore, AlloyDB has incorporated the ScaNN index, a vector search technology developed by Google, which provides enhanced performance and scalability for large datasets compared to the standard pgvector Hierarchical Navigable Small World (HNSW) index . This combination of features makes AlloyDB a powerful option for building AI-powered applications that require both relational data management and semantic search capabilities.  

In AlloyDB, vector embeddings can be stored either as arrays of real values or using the dedicated vector data type provided by the pgvector extension . Similarity searches are performed using standard SQL queries in conjunction with operators provided by pgvector, such as <-> for L2 distance, <#> for negative inner product, and <=> for cosine distance . AlloyDB supports both exact and approximate nearest neighbor (ANN) searches through indexing, offering options like HNSW and the more scalable ScaNN index . A significant benefit of using a relational database like AlloyDB is the ability to seamlessly combine vector search with traditional SQL filtering and join operations, allowing for richer and more context-aware queries . Additionally, AlloyDB features inline filtering, a query optimization technique that improves the performance of filtered vector searches by evaluating metadata filtering conditions and vector search in tandem .  

AlloyDB, especially when utilizing the ScaNN index, is designed to handle massive datasets, including those with billions of vectors, while maintaining sub-second query latencies . Benchmarks have demonstrated that ScaNN can outperform pgvector's HNSW index, particularly when dealing with large, memory-intensive datasets . ScaNN also boasts a significantly smaller memory footprint, approximately 4 times less than HNSW, which can lead to cost efficiencies . Furthermore, AlloyDB with pgvector supports vectors with dimensions up to 8000 through Scalar Quantization (SQ8), offering flexibility for various embedding models .  

The pricing for AlloyDB is based on several factors, including the usage of vCPUs and memory, the amount of storage consumed, backup storage costs, and networking charges . While AlloyDB generally has a price markup compared to Cloud SQL for the enhanced performance it offers, its pricing model is considered predictable . Examples of hourly and monthly costs for vCPU and memory in different GCP regions are available . Users also incur costs for regional cluster storage and backup storage . If embedding generation is performed within AlloyDB using Vertex AI models, the costs associated with the Vertex AI Embeddings API would also need to be considered . The predictable nature of AlloyDB's pricing, coupled with its performance advantages, makes it a strong contender for demanding applications .  

3.3. Cloud SQL for PostgreSQL with Vector Extension

Cloud SQL for PostgreSQL is another fully managed relational database service offered by GCP, providing high performance, seamless integration, and compatibility with the PostgreSQL ecosystem . Similar to AlloyDB, Cloud SQL supports the pgvector extension, enabling the storage and querying of vector embeddings . It also integrates with Vertex AI, allowing users to generate embeddings using the embedding() function within SQL queries . Cloud SQL includes specific optimizations for working with the pgvector extension, enhancing its capabilities for vector search workloads . This makes Cloud SQL a viable option for users who prefer a standard PostgreSQL environment with the added functionality of vector search.  

In Cloud SQL, vector embeddings can be stored as arrays or the vector data type, consistent with the pgvector extension . Querying these embeddings involves using SQL in conjunction with pgvector operators . Cloud SQL supports both exact nearest neighbor search and approximate nearest neighbor search using indexes such as HNSW and Inverted File Flat (IVFFlat) . Like AlloyDB, Cloud SQL allows users to combine vector search with traditional SQL features like filtering and joins .  

While Cloud SQL is a scalable service, its performance for extremely large datasets might not reach the levels offered by dedicated vector databases like Vertex AI Vector Search or AlloyDB with ScaNN . However, the availability of HNSW indexes helps to reduce query latency for approximate nearest neighbor searches . Recent updates to pgvector, which are supported by Cloud SQL, include performance improvements like parallel HNSW index builds . To further optimize performance, Cloud SQL provides features such as data caching, data cache metrics, System Insights, and Query Insights, which can help monitor and troubleshoot performance issues related to generative AI applications, including vector search .  

The pricing for Cloud SQL is based on several components: CPU and memory usage, storage (SSD or HDD), networking costs, and instance pricing (applicable to shared-core instances) . Examples of hourly and monthly costs for vCPU and memory are provided for both Enterprise and Enterprise Plus editions . Users are also charged for the storage capacity they provision (with different rates for SSD and HDD) and for backup storage . Committed use discounts are available, offering potential cost savings for users with predictable workloads . If Vertex AI embedding models are used for generating embeddings, the associated costs would also apply . Cloud SQL can be a cost-effective solution, particularly when leveraging committed use discounts, and its pricing structure aligns with typical PostgreSQL deployment costs .  

3.4. Firestore

Firestore is a serverless NoSQL document database on GCP that has recently introduced vector search capabilities . It allows users to store vector embeddings directly within Firestore documents and perform nearest neighbor queries using vector indexes . This integration offers a simpler, serverless approach to vector search, particularly for applications already utilizing Firestore for their primary data storage .  

The process of using vector search in Firestore involves writing vector embeddings to specified fields within Firestore documents . Vector indexes can be created and managed using the gcloud CLI or the Firebase CLI, specifying the collection group, the field containing the vector embedding, and the vector configuration, including the dimension and index type (which is currently limited to flat) . Firestore supports distance metrics such as Euclidean, Cosine, and Dot Product for measuring the similarity between vectors . However, there are certain limitations to note: the maximum supported embedding dimension is 2048, and a nearest-neighbor query can return a maximum of 1000 documents . Additionally, vector search in Firestore does not currently support real-time snapshot listeners .  

Firestore is known for its ability to automatically scale to handle millions of users and massive datasets . However, the limitation on the maximum number of returned nearest neighbors (1000) might pose a challenge for a dataset as large as 31.5 million embeddings, as it might prevent the retrieval of a comprehensive set of similar vectors for a given query .  

The pricing for Firestore is based on document reads, writes, deletes, the amount of stored data, and network egress . A free quota is available for the default database, which might be sufficient for initial experimentation . For vector search, users are charged one read operation for each batch of up to 100 kNN vector index entries read by the query . If embeddings are computed using Cloud Functions, those costs would also need to be factored in . The cost of storing the vector embeddings themselves is included in the overall stored data costs . While Firestore's cost model is generally straightforward, the pricing of vector search based on read operations, especially with a large dataset, requires careful evaluation .  

3.5. BigQuery

BigQuery is a fully managed, serverless data warehouse on GCP that offers vector search capabilities, primarily geared towards analytical workloads . It utilizes indexes to optimize the lookups and distance computations required for identifying closely matching embeddings . BigQuery's strength lies in its ability to handle petabyte-scale data, making it a potential option for storing a large volume of vector embeddings, although its performance characteristics might be more suited for batch processing and analytical queries rather than real-time, low-latency retrieval.  

In BigQuery, vector embeddings are stored as columns within BigQuery tables . Users can create vector indexes using the CREATE VECTOR INDEX statement and perform similarity searches using the VECTOR_SEARCH function in their SQL queries . BigQuery supports various distance types, including COSINE . It is important to note that queries containing the VECTOR_SEARCH function are not accelerated by BigQuery BI Engine . For performance optimization, BigQuery allows for pre-filtering of data and the use of stored columns within the index . For datasets exceeding 10 billion entries, sharding strategies can be employed to scale beyond the limits of individual indexes .  

BigQuery is designed to handle massive volumes of data, making it capable of storing 31.5 million embeddings . However, its architecture is optimized for analytical processing, and query performance for vector search might not match the low latency offered by dedicated vector databases . For use cases requiring real-time responses, other options might be more suitable.  

The pricing for BigQuery's vector search features, including CREATE VECTOR INDEX and VECTOR_SEARCH, is based on BigQuery compute pricing . During index creation, only the indexed column contributes to the bytes processed . There are storage costs associated with active vector indexes . BigQuery offers on-demand compute pricing, where charges are based on the amount of data processed by queries, measured in terabytes . If embeddings are generated using BigQuery ML, the costs for those operations would also apply . For users with consistent workloads, capacity-based pricing might offer a more predictable cost model . Overall, BigQuery's cost-effectiveness for vector search depends on the query volume and the amount of data processed .  

4. Cost Analysis and Comparison

To provide a preliminary cost estimate, let's consider the storage requirements for 31.5 million embeddings. Assuming an average embedding dimension of 768 and using float32 representation (4 bytes per dimension), the raw data size would be approximately 96.77 GB (31,500,000 * 768 * 4 bytes).

The estimated monthly storage costs for each database option, based on typical pricing in the us-central1 region (pricing can vary by region and should be confirmed with the GCP pricing calculator), are presented in Table 1. Please note that these are rough estimates and do not include potential costs for backups or other features.

Table 1: Estimated Monthly Storage Costs (Approximate, us-central1)

Database Option	Approximate Storage Cost per GB/month	Estimated Monthly Storage Cost (96.77 GB)
Vertex AI Vector Search	Included in VM costs	To be determined based on VM type
AlloyDB for PostgreSQL	$0.0005375 / GB hour ≈ $0.39/month	~$37.74
Cloud SQL for PostgreSQL	$0.222 (SSD) / GB/month	~$21.48
Firestore	$0.150 / GiB/month	~$14.52
BigQuery	$0.02 / GB/month	~$1.94

Projected compute costs for indexing and querying are highly dependent on the specific usage scenarios.

• Storage-Heavy Scenario (Infrequent Querying): In this case, the primary cost driver would be storage. BigQuery appears to be the most cost-effective option for storage alone, followed by Firestore. AlloyDB and Cloud SQL have higher storage costs. Vertex AI Vector Search costs are tied to the provisioned VMs, which would need to be sized appropriately for the data. Indexing costs would be incurred upfront or with infrequent updates for Vertex AI and BigQuery. Query costs would be minimal.

• Query-Heavy Scenario (Frequent, Low-Latency Querying): For frequent querying with low latency requirements, Vertex AI Vector Search and AlloyDB with ScaNN are likely to be the most performant options. Vertex AI Vector Search's serving costs, based on continuous VM uptime, could be substantial. AlloyDB's compute costs (vCPU and memory) would be the primary driver, along with potential indexing costs. Cloud SQL might also be viable, but performance at high query volumes might need careful optimization. Firestore's read operation-based pricing for vector search would need to be estimated based on the expected query volume and the number of index entries read per query. BigQuery's per-TB query costs might become significant with frequent querying.

• Balanced Scenario (Moderate Querying and Storage): This scenario would involve a mix of storage and compute costs. The optimal choice would depend on the specific balance between storage needs and query frequency/latency requirements. AlloyDB and Cloud SQL, with their relational database features and vector search capabilities, could be strong contenders, offering a balance between performance and cost.

Due to the complexity and dependency on specific usage patterns, providing precise compute cost estimates without more information about the expected query volume, frequency, and desired latency is challenging. It is recommended to use the GCP Pricing Calculator with estimated usage patterns to obtain more accurate cost projections for each service.

5. Feature, Performance, and Scalability Comparison

Table 2: Feature, Performance, and Scalability Comparison

Feature/Characteristic	Vertex AI Vector Search	AlloyDB for PostgreSQL	Cloud SQL for PostgreSQL	Firestore	BigQuery
Vector Search Focus	Primary	Secondary	Secondary	Primary	Analytical
Scalability (Embeddings)	Billions	Billions (with ScaNN)	Millions	Millions	Billions+
Query Latency	Very Low	Low	Low to Moderate	Low to Moderate	Moderate to High
Embedding Dimension Limit	Not explicitly stated	8000 (with SQ8)	2000 (pgvector default)	2048	High
Max Results per Query	Configurable	Configurable	Configurable	1000	Configurable
Indexing Options	ANN	HNSW, ScaNN	HNSW, IVFFlat	Flat	IVF, TreeAH
Filtering	Yes	Yes (Inline Filtering)	Yes	Yes (Pre-filtering)	Yes (Pre and Post-filtering)
Hybrid Search	Yes	No (via SQL)	No (via SQL)	No	No
Managed Service Level	Fully Managed	Fully Managed	Fully Managed	Fully Managed	Fully Managed
Relational Data Integration	Limited	Excellent	Excellent	Good (within document)	Excellent
Ease of Use	Moderate	Moderate	Moderate	Easy	Moderate
Cost	Potentially High	Moderate	Moderate	Low to Moderate	Depends on Usage

6. Recommendations and Best Practices

Based on the analysis, the following recommendations can be made:

• For users prioritizing highest performance and scalability with a dedicated vector database service, Vertex AI Vector Search is a strong contender. Its underlying ScaNN technology is designed for large-scale, low-latency retrieval. Alternatively, AlloyDB for PostgreSQL with the ScaNN index also offers excellent performance and scalability within a relational database context.

• If integration with an existing PostgreSQL infrastructure is a key requirement, both AlloyDB for PostgreSQL and Cloud SQL for PostgreSQL are suitable options. AlloyDB offers enhanced performance and built-in AI features, while Cloud SQL provides a more standard PostgreSQL environment.

• For users seeking a simpler, serverless approach with ease of use, Firestore with its vector search capabilities is worth considering. However, the limitation on the maximum number of results per query (1000) should be carefully evaluated given the dataset size.

• For analytical workloads and large-scale batch processing of vector embeddings, BigQuery can be a cost-effective solution due to its storage pricing and powerful data processing capabilities. However, its query latency might not be ideal for real-time applications.

• For cost-sensitive applications, Cloud SQL for PostgreSQL (with careful resource management and potential committed use discounts) or Firestore (if the 1000 result limit is acceptable) might be more suitable.

General best practices for storing and querying large vector datasets on GCP include:

• Carefully select the embedding model and dimensionality that best suit the specific use case, considering the trade-offs between accuracy and storage/computational costs .  
• Optimize indexing strategies based on the dataset size and expected query patterns. For very large datasets, consider using approximate nearest neighbor (ANN) indexing techniques like HNSW or ScaNN to balance recall and speed .  
• Implement filtering mechanisms to narrow down the search space and improve query efficiency and relevance .  
• Continuously monitor the performance of the chosen database service and adjust configurations (e.g., instance size, index parameters) as needed to meet the application's requirements .  
• For extremely large datasets, explore data partitioning and sharding techniques to distribute the data and workload across multiple nodes or instances .  
• Leverage committed use discounts offered by GCP for database resources to achieve potential cost savings for predictable workloads .  
• Regularly evaluate and optimize query costs by analyzing query execution plans and considering different pricing models (e.g., on-demand vs. capacity-based) .  

7. Conclusion

This report has provided an overview of several GCP database services suitable for storing and querying a dataset of 31.5 million vector embeddings. Each option—Vertex AI Vector Search, AlloyDB for PostgreSQL, Cloud SQL for PostgreSQL, Firestore, and BigQuery—offers a unique set of features, performance characteristics, scalability limits, and cost implications. Vertex AI Vector Search stands out for its purpose-built architecture and high performance, while AlloyDB and Cloud SQL offer integration with the PostgreSQL ecosystem. Firestore provides a simpler, serverless approach, and BigQuery is well-suited for analytical workloads. The optimal choice for the user will depend on their specific needs and priorities. It is strongly recommended that the user conduct a proof-of-concept with a representative subset of their data to thoroughly evaluate the performance, scalability, and cost of the most promising options within their specific application context before making a final decision.

Tags: Cloud,Database,Technology,