Method to upload a large batch at max speed

[marclaporte]

Could / should the Manticore PHP client be adapted?

[sanikolaev]

Hi @marclaporte

1. disable binlog

This has been mitigated here manticoresoftware/manticoresearch#879

2. write to multiple tables at once

This will be resolved once auto-sharding is completed:

• Autosharding (Buddy's part) manticoresearch-buddy#150 - done
• auto-sharding: distributed writes manticoresearch#1087 - yet to do

Also, I believe the client shouldn't manage the schema directly anyway.

3. write with a higher concurrency
4. write with a larger batch

It might make sense to integrate this into the client. The idea would be that when the client receives a large batch of documents, it starts sending them to Manticore at the maximum possible speed, automatically adjusting concurrency and batch size. This could be based on factors like the number of CPU cores and workers available in the Manticore instance, increasing until further gains are no longer feasible.

[marclaporte]

Many things have been addressed.

One remaining blocker: manticoresoftware/manticoresearch#3073

[marclaporte]

AFAICT, the last blocker is manticoresoftware/manticoresearch#1087

[marclaporte]

Now that manticoresoftware/manticoresearch#1087 and manticoresoftware/manticoresearch#3073 have been handled, what is the next step, if any?

[marclaporte]

it starts sending them to Manticore at the maximum possible speed, automatically adjusting concurrency and batch size

This is most interesting! This could help us compare various configurations and optimize resource allocation.

[marclaporte]

For ease of use, I copy-pasted part of the chat from the screenshot above:

"The PHP client doesn't do the things recommended above out of the box. If you want to measure max throughput on your server you can use a script like this (modify it, so the table schema/data looks like yours):"

#!/usr/bin/php
<?php
if (count($argv) < 4) die("Usage: ".__FILE__." <batch size> <concurrency> <docs>\n");

// This function waits for an idle mysql connection for the $query, runs it and exits
function process($query) {
    global $all_links;
    global $requests;
    global $latencies;
    foreach ($all_links as $k=>$link) {
        if (@$requests[$k]) continue;
        mysqli_query($link, $query, MYSQLI_ASYNC);
        @$requests[$k] = microtime(true);
        return true;
    }
    do {
        $links = $errors = $reject = array();
        foreach ($all_links as $link) {
            $links[] = $errors[] = $reject[] = $link;
        }
        $count = @mysqli_poll($links, $errors, $reject, 0, 1000);
        if ($count > 0) {
            foreach ($links as $j=>$link) {
                $res = @mysqli_reap_async_query($links[$j]);
                foreach ($all_links as $i=>$link_orig) if ($all_links[$i] === $links[$j]) break;
                if ($link->error) {
                    echo "ERROR in '".substr($query, 0, 100)."...': {$link->error}\n";
                    if (!mysqli_ping($link)) {
                        echo "ERROR: mysql connection is down, removing it from the pool\n";
                        unset($all_links[$i]); // remove the original link from the pool
                        unset($requests[$i]); // and from the $requests too
                    }
                    return false;
                }
                if ($res === false and !$link->error) continue;
                if (is_object($res)) {
                    mysqli_free_result($res);
                }
                $latencies[] = microtime(true) - $requests[$i];
                $requests[$i] = microtime(true);
                mysqli_query($link, $query, MYSQLI_ASYNC); // making next query
                return true;
            }
        };
    } while (true);
    return true;
}

$t = microtime(true);
$all_links = [];
$requests = [];
$latencies = [];
$c = 0;
for ($i=0;$i<$argv[2];$i++) {
  $m = @mysqli_connect('127.0.0.1', '', '', '', 9315);
      if (mysqli_connect_error()) die("Cannot connect to Manticore\n");
      $all_links[] = $m;
//      mysqli_query($all_links[0], "drop table if exists user$i");
//      mysqli_query($all_links[0], "create table user$i(area text, age int, active bit(1)) hitless_words='all' rt_mem_limit='16M'");
  }

// init
mysqli_query($all_links[0], "drop table if exists user");
mysqli_query($all_links[0], "create table user(area text, age int, active bit(1)) hitless_words='all' rt_mem_limit='16M'");

$batch = [];
//$query_start = "insert into user".rand(0,$argv[2])."(id, area, age, active) values ";
$query_start = "insert into user(id, area, age, active) values ";

$error = false;
while (count($all_links) and $c < $argv[3]) {
  $batch[] = "(0,'".substr(md5(rand()), 0, 6)."',".rand(5,15).",".rand(0,1).")";
  $c++;
  if (count($batch) == $argv[1]) {
    if (!process($query_start.implode(',', $batch))) {
      $error = true;
      break;
    }
    $batch = [];
  }
}
// wait until all the workers finish
do {
  $links = $errors = $reject = array();
  foreach ($all_links as $link)  $links[] = $errors[] = $reject[] = $link;
  $count = @mysqli_poll($links, $errors, $reject, 0, 100);
} while (count($all_links) != count($links) + count($errors) + count($reject));

echo "finished inserting\n";
echo round($argv[3] / (microtime(true) - $t))." docs per sec\n";

[sanikolaev]

Now that manticoresoftware/manticoresearch#1087 and manticoresoftware/manticoresearch#3073 have been handled, what is the next step, if any?

The next step is to finally release the sharding functionality, which should happen soon. Once that's done we'll prioritize this task.

[marclaporte]

Congratulations for auto-sharding and the multiple other enhancements in the latest release!
https://github.com/manticoresoftware/manticoresearch/releases/tag/27.1.5

[marclaporte]

Sharding in Manticore Search: automatic distribution and replication
Author: Dmitrii Kuzmenkov
Published: Jul 02, 2026

https://manticoresearch.com/blog/sharding-in-manticore-search/#benchmarks-does-sharding-actually-speed-up-inserts

[Nick-S-2018]

Since we’ll soon introduce indexer support for RT tables, it makes sense to use that mode for bulk uploads in the PHP client as well.
https://github.com/manticoresoftware/manticoresearch-php/tree/rt-indexer-bulk

In our tests, it was roughly 2–4× faster than regular batched uploads, depending on the worker count.

MS_PORT=9408 ./scripts/compare-add-documents.php --docs=10000000 --batch-size=1000 --workers=1

Target : 127.0.0.1:9408  table=taxi_indexer_bench

=== addDocuments ===
docs=10000000  batches=10000  elapsed=170.036s

=== addDocumentsStreaming ===
docs=10000000  mode=indexer_rt_bulk  selected_workers=1  elapsed=96.263s

=== summary ===
addDocuments           : 58811 docs/sec
addDocumentsStreaming  : 103882 docs/sec (x1.77)



MS_PORT=9408 ./scripts/compare-add-documents.php --docs=10000000 --batch-size=1000 --workers=8

Target : 127.0.0.1:9408  table=taxi_indexer_bench

=== addDocuments ===
docs=10000000  batches=10000  elapsed=171.031s

=== addDocumentsStreaming ===
docs=10000000  mode=indexer_rt_bulk  selected_workers=8  elapsed=43.229s

=== summary ===
addDocuments           : 58469 docs/sec
addDocumentsStreaming  : 231326 docs/sec (x3.96)

Blocked by manticoresoftware/manticoresearch#4824