Hyperspace Docs
Hyperspace Homepage
  • Getting started
    • Overview
      • Hyperspace Advantages
      • Hyperspace Search
    • Quick Start
  • flows
    • Setting Up
      • Installing the Hyperspace API Client
      • Connecting to the Hyperspace Server
      • Creating a Database Schema Configuration File
        • Vector Similarity Metrics
        • Index Type Methods
      • Creating a Collection
      • Uploading Data to a Collection
      • Building and Running Queries
        • Building a Lexical Search Query
        • Building a Vector Search Query
        • Building a Hybrid Search Query
      • Retrieving Results
    • Data Collections
      • Uploading Data
      • Accessing Data
      • Supported Data Types
    • Queries
      • DSL Query interface
        • Aggregations
        • Bool Query
        • Candidate Generation and Metadata Filtering
        • Scoring and Ranking
  • Reference
    • Hyperspace Query Flow
    • Features and Benefits
    • Search Processing Unit (SPU)
    • Hyperspace Document Prototype
  • API Documentation
    • Hyperspace Client
      • add_batch
      • add_document
      • async_req
      • clear_collection
      • collections_info
      • commit
      • create_collection
      • delete_collection
      • delete_by_query
      • dsl_search
      • get_schema
      • get_document
      • reset_password
      • search
      • update_by_query
      • update_document
    • DSL Query Framework
      • Aggregations
        • Cardinality Aggregation
        • Date Histogram
        • Metric Aggregations
        • Terms Aggregation
      • Bool Queries
        • Free Text Search
        • 'match' Clause
        • 'filter' Clause
        • 'must' Clause
        • 'must_not' Clause
        • 'should' Clause
        • 'should_not' Clause
      • Candidate Generation and Metadata Filtering
        • Geo Coordinates Match
        • Range Match
        • Term Match
      • Scoring and Ranking
        • Boost
        • 'dis_max'
        • Function Score
        • Rarity Score (TF-IDF)
  • Releases
    • 2024 Releases
Powered by GitBook
On this page
  1. API Documentation
  2. Hyperspace Client

add_batch

The functionadd_batch(batch, collection_name) uploads a batch of documents to a collection.

Input

  • batch – Represents the batch of documents to upload. The structure of each document must conform to the database schema configuration file. It must be of type list [dictionary].

  • collection_name – Specifies the name of the Collection into which to load the document.

Example 1 -

The following code snippet builds a list of documents in a temporary variable named batch and then uploads each batch using –

hyperspace_client.add_batch(batch, collection_name)
hyperspaceClient.addBatch(batch, collection_name);
await hyperspaceClient.addBatch(batch, collection_name);

Response

{'code': 200, 'message': 'Batch successfully added', 'status': 'OK'}

Example 2 -

The following example uploads batches of 250 documents for. Documents are added to the batch, and once a batch reaches 250 documents, it's uploaded to the Hyperspace Collection.

Copy the following code snippet

BATCH_SIZE = 250
batch = []
collection_name = "new_collection"
for i, document in enumerate(documents):
   batch.append(document )
   if (i+1) % BATCH_SIZE == 0:
      response = hyperspace_client.add_batch(batch, collection_name)
      batch.clear()
      
if batch:
  response = hyperspace_client.add_batch(batch, collection_name)
hyperspace_client.commit(collection_name)
import java.util.ArrayList;
final int batchSize = 250;

for (int i= 0; index < documents.size(); i++) {
    batch.add(documents.get(i));
    if ((i+ 1) % batchSize == 0) {
          List<DataPoint> batchCopy = new ArrayList<>(batch);
          futures.add(hyperspaceClient.addBatch(batchCopy, collectionName));
          batch.clear();
      }    
}

if (!batch.isEmpty()) {
    futures.add(hyperspaceClient.addBatch(new ArrayList<>(batch), collectionName));
}
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
hyperspaceClient.commit(collectionName).join();
const BATCH_SIZE = 250;
const batch: any[] = [];
const collectionName = "new_collection";

for (const [i, document] of documents.entries()) {
    batch.push(document);
    if ((i + 1) % BATCH_SIZE == 0) {
        await client.addBatch(collectionName, batch);
        batch = [];
    }
}

if (batch.length != 0) {
    await client.addBatch(collectionName, batch);
}
await client.commit(collectionName)

Where –

  • document – Represents the document to upload. The structure of each document must be according to the database schema configuration file. Must be of type dictionary.

  • BATCH_SIZE – Specifies the number of documents in a batch.

  • commit – Is required for vector search only. commit should only be performed after the data upload is complete.

In this method, each document will be assigned with an automatic identifier.

PreviousHyperspace ClientNextadd_document

Last updated 10 months ago