add_batch

The functionadd_batch(batch, collection_name) uploads a batch of documents to a collection.

Input

• batch – Represents the batch of documents to upload. The structure of each document must conform to the database schema configuration file. It must be of type list [dictionary].

• collection_name – Specifies the name of the Collection into which to load the document.

Example 1 -

The following code snippet builds a list of documents in a temporary variable named batch and then uploads each batch using –

Python

hyperspace_client.add_batch(batch, collection_name)

Java

hyperspaceClient.addBatch(batch, collection_name);

JavaScript

await hyperspaceClient.addBatch(batch, collection_name);

Response

{'code': 200, 'message': 'Batch successfully added', 'status': 'OK'}

Example 2 -

The following example uploads batches of 250 documents for. Documents are added to the batch, and once a batch reaches 250 documents, it's uploaded to the Hyperspace Collection.

Copy the following code snippet

Python

BATCH_SIZE = 250
batch = []
collection_name = "new_collection"
for i, document in enumerate(documents):
   batch.append(document )
   if (i+1) % BATCH_SIZE == 0:
      response = hyperspace_client.add_batch(batch, collection_name)
      batch.clear()
      
if batch:
  response = hyperspace_client.add_batch(batch, collection_name)
hyperspace_client.commit(collection_name)

Java

import java.util.ArrayList;
final int batchSize = 250;

for (int i= 0; index < documents.size(); i++) {
    batch.add(documents.get(i));
    if ((i+ 1) % batchSize == 0) {
          List<DataPoint> batchCopy = new ArrayList<>(batch);
          futures.add(hyperspaceClient.addBatch(batchCopy, collectionName));
          batch.clear();
      }    
}

if (!batch.isEmpty()) {
    futures.add(hyperspaceClient.addBatch(new ArrayList<>(batch), collectionName));
}
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
hyperspaceClient.commit(collectionName).join();

JavaScript

const BATCH_SIZE = 250;
let batch: any[] = [];
const collectionName = "new_collection";

for (const [i, document] of documents.entries()) {
    batch.push(document);
    if ((i + 1) % BATCH_SIZE == 0) {
        await client.addBatch(collectionName, batch);
        batch = [];
    }
}

if (batch.length != 0) {
    await client.addBatch(collectionName, batch);
}
await client.commit(collectionName)

Where –

• document – Represents the document to upload. The structure of each document must be according to the database schema configuration file. Must be of type dictionary.

• BATCH_SIZE – Specifies the number of documents in a batch.

• commit – Is required for vector search only. commit should only be performed after the data upload is complete.

In this method, each document will be assigned with an automatic identifier.