feat: add worker lifecycle SQL hooks - #24
Merged
Merged
Conversation
Run connection-scoped setup and ordered concurrent finalization SQL around benchmark workloads. Preserve child exit statuses, support multiline cached INSERT templates, and regenerate zero-byte caches in place.
clt-insert👎 CLT tests in test/clt-tests/basic/worker_lifecycle_order.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
./manticore-load --drop --quiet --json --threads=1 --total=1 --init="CREATE TABLE worker_lifecycle_order(value int)" --worker-init="BEGIN; INSERT INTO worker_lifecycle_order(id,value) VALUES(1,10)" --worker-finalize="INSERT INTO worker_lifecycle_order(id,value) VALUES(3,30); COMMIT" --load="INSERT INTO worker_lifecycle_order(id,value) VALUES(2,20)"
––– output –––
{
"threads": 1,
"batch_size": 1,
"time": "00:00",
"total_operations": 1,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -e "SELECT id,value FROM worker_lifecycle_order WHERE id=1; SELECT id,value FROM worker_lifecycle_order WHERE id=2; SELECT id,value FROM worker_lifecycle_order WHERE id=3"
––– output –––
- id value
+ +------+-------+
- 1 10
+ | id | value |
- id value
+ +------+-------+
- 2 20
+ | 1 | 10 |
- id value
+ +------+-------+
- 3 30
+ +------+-------+
+ | id | value |
+ +------+-------+
+ | 2 | 20 |
+ +------+-------+
+ +------+-------+
+ | id | value |
+ +------+-------+
+ | 3 | 30 |
+ +------+-------+
––– input –––
./manticore-load --drop --quiet --json --threads=4 --total=4 --init="CREATE TABLE worker_lifecycle_parallel(value int)" --worker-init="BEGIN" --worker-finalize="COMMIT" --load="INSERT INTO worker_lifecycle_parallel(id,value) VALUES(<increment>,10)"
––– output –––
{
"threads": 4,
"batch_size": 1,
"time": "00:00",
"total_operations": 4,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -N -e "SELECT COUNT(*),MIN(value),MAX(value) FROM worker_lifecycle_parallel"
––– output –––
- 4\t10\t10
+ +------+------+------+
+ | 4 | 10 | 10 |
+ +------+------+------+
––– input –––
./manticore-load --drop --threads=4 --batch-size=2 --total=8 --init="CREATE TABLE worker_lifecycle_failure(value int)" --worker-init="BEGIN" --worker-finalize="INVALID FINALIZER" --load="INSERT INTO worker_lifecycle_failure(id,value) VALUES(<increment>,10)" > /dev/null; echo $?; mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM worker_lifecycle_failure"
––– output –––
1
- 0
+ +------+
+ | 0 |
+ +------+
––– input –––
)"
––– output –––
- {
+ bash: syntax error near unexpected token `)'
- "threads": 1,
- "batch_size": 2,
- "time": "00:00",
- "total_operations": 4,
- "operations_per_second": #!/\d+/!#,
- "qps": {
- "avg": 0,
- "p99": 0,
- "p95": 0,
- "p5": 0,
- "p1": 0
- },
- "latency": {
- "avg": #!/\d+(\.\d+)?/!#,
- "p50": #!/\d+(\.\d+)?/!#,
- "p95": #!/\d+(\.\d+)?/!#,
- "p99": #!/\d+(\.\d+)?/!#
- }
- }
––– input –––
mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM multiline_insert"
––– output –––
- 4
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'multiline_insert' in search request
––– input –––
cache=$(./manticore-load --verbose --threads=1 --batch-size=2 --total=5 --init="CREATE TABLE empty_cache_regression(value int)" --load="INSERT INTO empty_cache_regression(id,value) VALUES(<increment>,<int/1/100>)" 2>&1 | sed -n 's/.*Generating new data cache \(\/tmp\/manticore_load_[a-f0-9]*\).*/\1/p' | head -1); test -n "$cache"; : > "$cache"; inode_before=$(stat -c %i "$cache"); ./manticore-load --drop --quiet --json --threads=1 --batch-size=2 --total=5 --cache-from-disk --init="CREATE TABLE empty_cache_regression(value int)" --load="INSERT INTO empty_cache_regression(id,value) VALUES(<increment>,<int/1/100>)"; inode_after=$(stat -c %i "$cache"); test "$inode_before" = "$inode_after"; test -s "$cache"; mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM empty_cache_regression"
––– output –––
{
"threads": 1,
"batch_size": 2,
"time": "00:00",
"total_operations": 6,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
- 5
+ +------+
-
+ | 5 |
+ +------+
+
test/clt-tests/basic/worker_lifecycle.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
./manticore-load --drop --quiet --json --threads=1 --batch-size=2 --total=4 --init="CREATE TABLE worker_lifecycle(value int)" --worker-init="BEGIN" --worker-finalize="COMMIT" --load="INSERT INTO worker_lifecycle(id,value) VALUES(<increment>,<int/1/100>)"
––– output –––
{
"threads": 1,
"batch_size": 2,
"time": "00:00",
"total_operations": 4,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) FROM worker_lifecycle; SHOW TABLE worker_lifecycle STATUS LIKE 'ram_chunk_segments_count'"
––– output –––
- COUNT(*)
+ +----------+
- 4
+ | count(*) |
- Variable_name Value
+ +----------+
- ram_chunk_segments_count 2
+ | 4 |
-
+ +----------+
+ +--------------------------+-------+
+ | Variable_name | Value |
+ +--------------------------+-------+
+ | ram_chunk_segments_count | 1 |
+ +--------------------------+-------+
+
test/clt-tests/basic/insert_two.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
PROCESS_1_DELAY=0.5 PROGRESS_DELAY=0.5 ./manticore-load --drop --init="create table t" --total=10 --load="insert into t values(<increment>)" --threads=1 --delay=1 --together --drop --init="create table t2" --total=10 --load="insert into t2 values(<increment>)" --threads=1 --delay=0.5
––– output –––
======================================================================================
Process 2: Running with threads=1, batch-size=1
Process 2: Dropping table: DROP TABLE IF EXISTS t2
Process 2: Executing init command: create table t2
Process 2: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
======================================================================================
Process 1: Running with threads=1, batch-size=1
Process 1: Dropping table: DROP TABLE IF EXISTS t
Process 1: Executing init command: create table t
Process 1: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
-------------------------------------------------------------------------------------------------------------------------------------------
Time Elapsed 1:Progress 2:Progress 1:QPS 2:QPS 1:DPS 2:DPS CPU Workers Chunks Merging Disk Inserted
-------------------------------------------------------------------------------------------------------------------------------------------
%{TIME} #!/\d+/!#ms - - - - - - N/A - 0 0B 0
- %{TIME} 1.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 6
+ 14:13:26 1.5s 20% 30% 2 2 2 1 N/A 1 0 978B 6
- %{TIME} 2.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 9
+ 14:13:27 2.5s 30% 50% 1 2 1 2 0% 1 0 978B 9
- %{TIME} 3.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 12
+ 14:13:28 3.5s 30% 70% 1 2 1 2 0% 1 0 978B 12
--------------------------------------------------------------------------------------
Process 2 final statistics:
Init command: create table t2
Load commands:
- insert into t2 values(<increment>)
Total time: #!/(4|5)\.\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 2
QPS avg: 2
QPS 1p: 2
QPS 5p: 2
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 50#!/\d/!#.#!/\d/!# ms
Latency 50p: 505.#!/\d/!# ms
Latency 95p: 505.#!/\d/!# ms
Latency 99p: 505.#!/\d/!# ms
--------------------------------------------------------------------------------------
- %{TIME} 4.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 5
+ 14:13:29 4.5s 50% - 1 - 1 - 1% 1 0 489B 5
- %{TIME} 5.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 6
+ 14:13:30 5.5s 60% - 1 - 1 - 0% 1 0 489B 6
- %{TIME} 6.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 7
+ 14:13:31 6.5s 70% - 1 - 1 - 1% 1 0 489B 7
- %{TIME} 7.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 8
+ 14:13:32 7.5s 70% - 1 - 1 - 0% 1 0 489B 8
- %{TIME} 8.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 9
+ 14:13:33 8.5s 90% - 1 - 1 - 0% 1 0 489B 9
--------------------------------------------------------------------------------------
Process 1 final statistics:
Init command: create table t
Load commands:
- insert into t values(<increment>)
Total time: 9.#!/\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 1
QPS avg: 1
QPS 1p: 1
QPS 5p: 1
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 100#!/\d/!#.#!/\d/!# ms
Latency 50p: 1050.#!/\d/!# ms
Latency 95p: 1050.#!/\d/!# ms
Latency 99p: 1050.#!/\d/!# ms
--------------------------------------------------------------------------------------
test/clt-tests/basic/insert_basic.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
PROCESS_1_DELAY=0.5 PROGRESS_DELAY=0.5 ./manticore-load --drop --init="create table t" --total=10 --load="insert into t values(<increment>)" --threads=1 --delay=1
––– output –––
======================================================================================
Process 1: Running with threads=1, batch-size=1
Process 1: Dropping table: DROP TABLE IF EXISTS t
Process 1: Executing init command: create table t
Process 1: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
-------------------------------------------------------------------------------------------------------------
Time Elapsed Progress QPS DPS CPU Workers Chunks Merging Disk Inserted
-------------------------------------------------------------------------------------------------------------
%{TIME} 5#!/\d\d/!#ms - - - N/A - 0 0B 0
- %{TIME} 1.#!/\d/!#s #!/\d/!#0% #!/\d/!# #!/\d/!# N/A 1 0 465B 2
+ 14:13:13 1.5s 20% 2 2 N/A 1 0 489B 2
- %{TIME} 2.#!/\d/!#s #!/\d/!#0% #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)\s+/!#1 0 465B 3
+ 14:13:14 2.5s 20% 2 2 0% 1 0 489B 3
- %{TIME} 3.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 4
+ 14:13:15 3.5s 40% 1 1 1% 1 0 489B 4
- %{TIME} 4.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 5
+ 14:13:16 4.5s 50% 1 1 1% 1 0 489B 5
- %{TIME} 5.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 6
+ 14:13:17 5.5s 50% 1 1 0% 1 0 489B 6
- %{TIME} 6.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 7
+ 14:13:18 6.5s 70% 1 1 1% 1 0 489B 7
- %{TIME} 7.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 8
+ 14:13:19 7.5s 80% 1 1 0% 1 0 489B 8
- %{TIME} 8.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 9
+ 14:13:20 8.5s 90% 1 1 1% 1 0 489B 9
--------------------------------------------------------------------------------------
Process 1 final statistics:
Init command: create table t
Load commands:
- insert into t values(<increment>)
Total time: 9#!/\.\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 1
QPS avg: 1
QPS 1p: 1
QPS 5p: 1
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 100#!/\d\.\d/!# ms
Latency 50p: 1050.0 ms
Latency 95p: 1050.0 ms
Latency 99p: 1050.0 ms
--------------------------------------------------------------------------------------
|
clt-insert👎 CLT tests in test/clt-tests/basic/worker_lifecycle_order.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
./manticore-load --drop --quiet --json --threads=1 --total=1 --init="CREATE TABLE worker_lifecycle_order(value int)" --worker-init="BEGIN; INSERT INTO worker_lifecycle_order(id,value) VALUES(1,10)" --worker-finalize="INSERT INTO worker_lifecycle_order(id,value) VALUES(3,30); COMMIT" --load="INSERT INTO worker_lifecycle_order(id,value) VALUES(2,20)"
––– output –––
{
"threads": 1,
"batch_size": 1,
"time": "00:00",
"total_operations": 1,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -e "SELECT id,value FROM worker_lifecycle_order WHERE id=1; SELECT id,value FROM worker_lifecycle_order WHERE id=2; SELECT id,value FROM worker_lifecycle_order WHERE id=3"
––– output –––
- id value
+ +------+-------+
- 1 10
+ | id | value |
- id value
+ +------+-------+
- 2 20
+ | 1 | 10 |
- id value
+ +------+-------+
- 3 30
+ +------+-------+
+ | id | value |
+ +------+-------+
+ | 2 | 20 |
+ +------+-------+
+ +------+-------+
+ | id | value |
+ +------+-------+
+ | 3 | 30 |
+ +------+-------+
––– input –––
./manticore-load --drop --quiet --json --threads=4 --total=4 --init="CREATE TABLE worker_lifecycle_parallel(value int)" --worker-init="BEGIN" --worker-finalize="COMMIT" --load="INSERT INTO worker_lifecycle_parallel(id,value) VALUES(<increment>,10)"
––– output –––
{
"threads": 4,
"batch_size": 1,
"time": "00:00",
"total_operations": 4,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -N -e "SELECT COUNT(*),MIN(value),MAX(value) FROM worker_lifecycle_parallel"
––– output –––
- 4\t10\t10
+ +------+------+------+
+ | 4 | 10 | 10 |
+ +------+------+------+
––– input –––
./manticore-load --drop --threads=4 --batch-size=2 --total=8 --init="CREATE TABLE worker_lifecycle_failure(value int)" --worker-init="BEGIN" --worker-finalize="INVALID FINALIZER" --load="INSERT INTO worker_lifecycle_failure(id,value) VALUES(<increment>,10)" > /dev/null; echo $?; mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM worker_lifecycle_failure"
––– output –––
1
- 0
+ +------+
+ | 0 |
+ +------+
––– input –––
)"
––– output –––
- {
+ bash: syntax error near unexpected token `)'
- "threads": 1,
- "batch_size": 2,
- "time": "00:00",
- "total_operations": 4,
- "operations_per_second": #!/\d+/!#,
- "qps": {
- "avg": 0,
- "p99": 0,
- "p95": 0,
- "p5": 0,
- "p1": 0
- },
- "latency": {
- "avg": #!/\d+(\.\d+)?/!#,
- "p50": #!/\d+(\.\d+)?/!#,
- "p95": #!/\d+(\.\d+)?/!#,
- "p99": #!/\d+(\.\d+)?/!#
- }
- }
––– input –––
mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM multiline_insert"
––– output –––
- 4
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'multiline_insert' in search request
––– input –––
cache=$(./manticore-load --verbose --threads=1 --batch-size=2 --total=5 --init="CREATE TABLE empty_cache_regression(value int)" --load="INSERT INTO empty_cache_regression(id,value) VALUES(<increment>,<int/1/100>)" 2>&1 | sed -n 's/.*Generating new data cache \(\/tmp\/manticore_load_[a-f0-9]*\).*/\1/p' | head -1); test -n "$cache"; : > "$cache"; inode_before=$(stat -c %i "$cache"); ./manticore-load --drop --quiet --json --threads=1 --batch-size=2 --total=5 --cache-from-disk --init="CREATE TABLE empty_cache_regression(value int)" --load="INSERT INTO empty_cache_regression(id,value) VALUES(<increment>,<int/1/100>)"; inode_after=$(stat -c %i "$cache"); test "$inode_before" = "$inode_after"; test -s "$cache"; mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM empty_cache_regression"
––– output –––
{
"threads": 1,
"batch_size": 2,
"time": "00:00",
"total_operations": 6,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
- 5
+ +------+
-
+ | 5 |
+ +------+
+
test/clt-tests/basic/worker_lifecycle.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
./manticore-load --drop --quiet --json --threads=1 --batch-size=2 --total=4 --init="CREATE TABLE worker_lifecycle(value int)" --worker-init="BEGIN" --worker-finalize="COMMIT" --load="INSERT INTO worker_lifecycle(id,value) VALUES(<increment>,<int/1/100>)"
––– output –––
{
"threads": 1,
"batch_size": 2,
"time": "00:00",
"total_operations": 4,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) FROM worker_lifecycle; SHOW TABLE worker_lifecycle STATUS LIKE 'ram_chunk_segments_count'"
––– output –––
- COUNT(*)
+ +----------+
- 4
+ | count(*) |
- Variable_name Value
+ +----------+
- ram_chunk_segments_count 2
+ | 4 |
-
+ +----------+
+ +--------------------------+-------+
+ | Variable_name | Value |
+ +--------------------------+-------+
+ | ram_chunk_segments_count | 1 |
+ +--------------------------+-------+
+
test/clt-tests/basic/insert_two.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
PROCESS_1_DELAY=0.5 PROGRESS_DELAY=0.5 ./manticore-load --drop --init="create table t" --total=10 --load="insert into t values(<increment>)" --threads=1 --delay=1 --together --drop --init="create table t2" --total=10 --load="insert into t2 values(<increment>)" --threads=1 --delay=0.5
––– output –––
======================================================================================
Process 2: Running with threads=1, batch-size=1
Process 2: Dropping table: DROP TABLE IF EXISTS t2
Process 2: Executing init command: create table t2
Process 2: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
======================================================================================
Process 1: Running with threads=1, batch-size=1
Process 1: Dropping table: DROP TABLE IF EXISTS t
Process 1: Executing init command: create table t
Process 1: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
-------------------------------------------------------------------------------------------------------------------------------------------
Time Elapsed 1:Progress 2:Progress 1:QPS 2:QPS 1:DPS 2:DPS CPU Workers Chunks Merging Disk Inserted
-------------------------------------------------------------------------------------------------------------------------------------------
%{TIME} #!/\d+/!#ms - - - - - - N/A - 0 0B 0
- %{TIME} 1.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 6
+ 14:13:35 1.5s 20% 30% 2 2 2 1 N/A 1 0 978B 6
- %{TIME} 2.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 9
+ 14:13:36 2.5s 20% 50% 2 2 2 2 0% 1 0 978B 9
- %{TIME} 3.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 12
+ 14:13:37 3.5s 40% 70% 1 2 1 2 0% 1 0 978B 12
--------------------------------------------------------------------------------------
Process 2 final statistics:
Init command: create table t2
Load commands:
- insert into t2 values(<increment>)
Total time: #!/(4|5)\.\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 2
QPS avg: 2
QPS 1p: 2
QPS 5p: 2
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 50#!/\d/!#.#!/\d/!# ms
Latency 50p: 505.#!/\d/!# ms
Latency 95p: 505.#!/\d/!# ms
Latency 99p: 505.#!/\d/!# ms
--------------------------------------------------------------------------------------
- %{TIME} 4.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 5
+ 14:13:38 4.5s 40% - 1 - 1 - 1% 1 0 489B 5
- %{TIME} 5.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 6
+ 14:13:39 5.5s 60% - 1 - 1 - 0% 1 0 489B 6
- %{TIME} 6.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 7
+ 14:13:40 6.5s 60% - 1 - 1 - 1% 1 0 489B 7
- %{TIME} 7.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 8
+ 14:13:41 7.5s 80% - 1 - 1 - 0% 1 0 489B 8
- %{TIME} 8.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 9
+ 14:13:42 8.5s 80% - 1 - 1 - 0% 1 0 489B 9
--------------------------------------------------------------------------------------
Process 1 final statistics:
Init command: create table t
Load commands:
- insert into t values(<increment>)
Total time: 9.#!/\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 1
QPS avg: 1
QPS 1p: 1
QPS 5p: 1
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 100#!/\d/!#.#!/\d/!# ms
Latency 50p: 1050.#!/\d/!# ms
Latency 95p: 1050.#!/\d/!# ms
Latency 99p: 1050.#!/\d/!# ms
--------------------------------------------------------------------------------------
test/clt-tests/basic/insert_basic.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
PROCESS_1_DELAY=0.5 PROGRESS_DELAY=0.5 ./manticore-load --drop --init="create table t" --total=10 --load="insert into t values(<increment>)" --threads=1 --delay=1
––– output –––
======================================================================================
Process 1: Running with threads=1, batch-size=1
Process 1: Dropping table: DROP TABLE IF EXISTS t
Process 1: Executing init command: create table t
Process 1: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
-------------------------------------------------------------------------------------------------------------
Time Elapsed Progress QPS DPS CPU Workers Chunks Merging Disk Inserted
-------------------------------------------------------------------------------------------------------------
%{TIME} 5#!/\d\d/!#ms - - - N/A - 0 0B 0
- %{TIME} 1.#!/\d/!#s #!/\d/!#0% #!/\d/!# #!/\d/!# N/A 1 0 465B 2
+ 14:13:21 1.5s 20% 2 2 N/A 1 0 489B 2
- %{TIME} 2.#!/\d/!#s #!/\d/!#0% #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)\s+/!#1 0 465B 3
+ 14:13:22 2.5s 20% 2 2 1% 1 0 489B 3
- %{TIME} 3.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 4
+ 14:13:23 3.5s 40% 1 1 0% 1 0 489B 4
- %{TIME} 4.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 5
+ 14:13:24 4.5s 50% 1 1 0% 1 0 489B 5
- %{TIME} 5.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 6
+ 14:13:25 5.5s 50% 1 1 0% 1 0 489B 6
- %{TIME} 6.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 7
+ 14:13:26 6.5s 70% 1 1 1% 1 0 489B 7
- %{TIME} 7.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 8
+ 14:13:27 7.5s 70% 1 1 1% 1 0 489B 8
- %{TIME} 8.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 9
+ 14:13:28 8.5s 90% 1 1 0% 1 0 489B 9
--------------------------------------------------------------------------------------
Process 1 final statistics:
Init command: create table t
Load commands:
- insert into t values(<increment>)
Total time: 9#!/\.\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 1
QPS avg: 1
QPS 1p: 1
QPS 5p: 1
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 100#!/\d\.\d/!# ms
Latency 50p: 1050.0 ms
Latency 95p: 1050.0 ms
Latency 99p: 1050.0 ms
--------------------------------------------------------------------------------------
|
clt-insert👎 CLT tests in test/clt-tests/basic/worker_lifecycle_order.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
./manticore-load --drop --quiet --json --threads=1 --total=1 --init="CREATE TABLE worker_lifecycle_order(value int)" --worker-init="BEGIN; INSERT INTO worker_lifecycle_order(id,value) VALUES(1,10)" --worker-finalize="INSERT INTO worker_lifecycle_order(id,value) VALUES(3,30); COMMIT" --load="INSERT INTO worker_lifecycle_order(id,value) VALUES(2,20)"
––– output –––
{
"threads": 1,
"batch_size": 1,
"time": "00:00",
"total_operations": 1,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -e "SELECT id,value FROM worker_lifecycle_order WHERE id=1; SELECT id,value FROM worker_lifecycle_order WHERE id=2; SELECT id,value FROM worker_lifecycle_order WHERE id=3"
––– output –––
- id value
+ +------+-------+
- 1 10
+ | id | value |
- id value
+ +------+-------+
- 2 20
+ | 1 | 10 |
- id value
+ +------+-------+
- 3 30
+ +------+-------+
+ | id | value |
+ +------+-------+
+ | 2 | 20 |
+ +------+-------+
+ +------+-------+
+ | id | value |
+ +------+-------+
+ | 3 | 30 |
+ +------+-------+
––– input –––
./manticore-load --drop --quiet --json --threads=4 --total=4 --init="CREATE TABLE worker_lifecycle_parallel(value int)" --worker-init="BEGIN" --worker-finalize="COMMIT" --load="INSERT INTO worker_lifecycle_parallel(id,value) VALUES(<increment>,10)"
––– output –––
{
"threads": 4,
"batch_size": 1,
"time": "00:00",
"total_operations": 4,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -N -e "SELECT COUNT(*),MIN(value),MAX(value) FROM worker_lifecycle_parallel"
––– output –––
- 4\t10\t10
+ +------+------+------+
+ | 4 | 10 | 10 |
+ +------+------+------+
––– input –––
./manticore-load --drop --threads=4 --batch-size=2 --total=8 --init="CREATE TABLE worker_lifecycle_failure(value int)" --worker-init="BEGIN" --worker-finalize="INVALID FINALIZER" --load="INSERT INTO worker_lifecycle_failure(id,value) VALUES(<increment>,10)" > /dev/null; echo $?; mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM worker_lifecycle_failure"
––– output –––
1
- 0
+ +------+
+ | 0 |
+ +------+
––– input –––
)"
––– output –––
- {
+ bash: syntax error near unexpected token `)'
- "threads": 1,
- "batch_size": 2,
- "time": "00:00",
- "total_operations": 4,
- "operations_per_second": #!/\d+/!#,
- "qps": {
- "avg": 0,
- "p99": 0,
- "p95": 0,
- "p5": 0,
- "p1": 0
- },
- "latency": {
- "avg": #!/\d+(\.\d+)?/!#,
- "p50": #!/\d+(\.\d+)?/!#,
- "p95": #!/\d+(\.\d+)?/!#,
- "p99": #!/\d+(\.\d+)?/!#
- }
- }
––– input –––
mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM multiline_insert"
––– output –––
- 4
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'multiline_insert' in search request
––– input –––
cache=$(./manticore-load --verbose --threads=1 --batch-size=2 --total=5 --init="CREATE TABLE empty_cache_regression(value int)" --load="INSERT INTO empty_cache_regression(id,value) VALUES(<increment>,<int/1/100>)" 2>&1 | sed -n 's/.*Generating new data cache \(\/tmp\/manticore_load_[a-f0-9]*\).*/\1/p' | head -1); test -n "$cache"; : > "$cache"; inode_before=$(stat -c %i "$cache"); ./manticore-load --drop --quiet --json --threads=1 --batch-size=2 --total=5 --cache-from-disk --init="CREATE TABLE empty_cache_regression(value int)" --load="INSERT INTO empty_cache_regression(id,value) VALUES(<increment>,<int/1/100>)"; inode_after=$(stat -c %i "$cache"); test "$inode_before" = "$inode_after"; test -s "$cache"; mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM empty_cache_regression"
––– output –––
{
"threads": 1,
"batch_size": 2,
"time": "00:00",
"total_operations": 6,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
- 5
+ +------+
-
+ | 5 |
+ +------+
+
test/clt-tests/basic/worker_lifecycle.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
./manticore-load --drop --quiet --json --threads=1 --batch-size=2 --total=4 --init="CREATE TABLE worker_lifecycle(value int)" --worker-init="BEGIN" --worker-finalize="COMMIT" --load="INSERT INTO worker_lifecycle(id,value) VALUES(<increment>,<int/1/100>)"
––– output –––
{
"threads": 1,
"batch_size": 2,
"time": "00:00",
"total_operations": 4,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) FROM worker_lifecycle; SHOW TABLE worker_lifecycle STATUS LIKE 'ram_chunk_segments_count'"
––– output –––
- COUNT(*)
+ +----------+
- 4
+ | count(*) |
- Variable_name Value
+ +----------+
- ram_chunk_segments_count 2
+ | 4 |
-
+ +----------+
+ +--------------------------+-------+
+ | Variable_name | Value |
+ +--------------------------+-------+
+ | ram_chunk_segments_count | 1 |
+ +--------------------------+-------+
+
test/clt-tests/basic/insert_two.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
PROCESS_1_DELAY=0.5 PROGRESS_DELAY=0.5 ./manticore-load --drop --init="create table t" --total=10 --load="insert into t values(<increment>)" --threads=1 --delay=1 --together --drop --init="create table t2" --total=10 --load="insert into t2 values(<increment>)" --threads=1 --delay=0.5
––– output –––
======================================================================================
Process 2: Running with threads=1, batch-size=1
Process 2: Dropping table: DROP TABLE IF EXISTS t2
Process 2: Executing init command: create table t2
Process 2: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
======================================================================================
Process 1: Running with threads=1, batch-size=1
Process 1: Dropping table: DROP TABLE IF EXISTS t
Process 1: Executing init command: create table t
Process 1: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
-------------------------------------------------------------------------------------------------------------------------------------------
Time Elapsed 1:Progress 2:Progress 1:QPS 2:QPS 1:DPS 2:DPS CPU Workers Chunks Merging Disk Inserted
-------------------------------------------------------------------------------------------------------------------------------------------
%{TIME} #!/\d+/!#ms - - - - - - N/A - 0 0B 0
- %{TIME} 1.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 6
+ 15:54:17 1.5s 20% 30% 2 2 2 1 N/A 1 0 978B 6
- %{TIME} 2.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 9
+ 15:54:18 2.5s 20% 50% 2 2 2 2 0% 1 0 978B 9
- %{TIME} 3.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 12
+ 15:54:19 3.5s 40% 70% 1 2 1 2 1% 1 0 978B 12
--------------------------------------------------------------------------------------
Process 2 final statistics:
Init command: create table t2
Load commands:
- insert into t2 values(<increment>)
Total time: #!/(4|5)\.\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 2
QPS avg: 2
QPS 1p: 2
QPS 5p: 2
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 50#!/\d/!#.#!/\d/!# ms
Latency 50p: 505.#!/\d/!# ms
Latency 95p: 505.#!/\d/!# ms
Latency 99p: 505.#!/\d/!# ms
--------------------------------------------------------------------------------------
- %{TIME} 4.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 5
+ 15:54:20 4.5s 50% - 1 - 1 - 0% 1 0 489B 5
- %{TIME} 5.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 6
+ 15:54:21 5.5s 60% - 1 - 1 - 1% 1 0 489B 6
- %{TIME} 6.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 7
+ 15:54:22 6.5s 60% - 1 - 1 - 1% 1 0 489B 7
- %{TIME} 7.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 8
+ 15:54:23 7.5s 80% - 1 - 1 - 0% 1 0 489B 8
- %{TIME} 8.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 9
+ 15:54:24 8.5s 80% - 1 - 1 - 0% 1 0 489B 9
--------------------------------------------------------------------------------------
Process 1 final statistics:
Init command: create table t
Load commands:
- insert into t values(<increment>)
Total time: 9.#!/\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 1
QPS avg: 1
QPS 1p: 1
QPS 5p: 1
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 100#!/\d/!#.#!/\d/!# ms
Latency 50p: 1050.#!/\d/!# ms
Latency 95p: 1050.#!/\d/!# ms
Latency 99p: 1050.#!/\d/!# ms
--------------------------------------------------------------------------------------
test/clt-tests/basic/insert_basic.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
PROCESS_1_DELAY=0.5 PROGRESS_DELAY=0.5 ./manticore-load --drop --init="create table t" --total=10 --load="insert into t values(<increment>)" --threads=1 --delay=1
––– output –––
======================================================================================
Process 1: Running with threads=1, batch-size=1
Process 1: Dropping table: DROP TABLE IF EXISTS t
Process 1: Executing init command: create table t
Process 1: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
-------------------------------------------------------------------------------------------------------------
Time Elapsed Progress QPS DPS CPU Workers Chunks Merging Disk Inserted
-------------------------------------------------------------------------------------------------------------
%{TIME} 5#!/\d\d/!#ms - - - N/A - 0 0B 0
- %{TIME} 1.#!/\d/!#s #!/\d/!#0% #!/\d/!# #!/\d/!# N/A 1 0 465B 2
+ 15:54:03 1.5s 20% 2 2 N/A 1 0 489B 2
- %{TIME} 2.#!/\d/!#s #!/\d/!#0% #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)\s+/!#1 0 465B 3
+ 15:54:04 2.5s 20% 2 2 0% 1 0 489B 3
- %{TIME} 3.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 4
+ 15:54:05 3.5s 40% 1 1 0% 1 0 489B 4
- %{TIME} 4.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 5
+ 15:54:06 4.5s 40% 1 1 0% 1 0 489B 5
- %{TIME} 5.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 6
+ 15:54:07 5.5s 60% 1 1 0% 1 0 489B 6
- %{TIME} 6.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 7
+ 15:54:08 6.5s 60% 1 1 1% 1 0 489B 7
- %{TIME} 7.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 8
+ 15:54:09 7.5s 80% 1 1 1% 1 0 489B 8
- %{TIME} 8.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 9
+ 15:54:10 8.5s 80% 1 1 0% 1 0 489B 9
--------------------------------------------------------------------------------------
Process 1 final statistics:
Init command: create table t
Load commands:
- insert into t values(<increment>)
Total time: 9#!/\.\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 1
QPS avg: 1
QPS 1p: 1
QPS 5p: 1
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 100#!/\d\.\d/!# ms
Latency 50p: 1050.0 ms
Latency 95p: 1050.0 ms
Latency 99p: 1050.0 ms
--------------------------------------------------------------------------------------
|
clt-insert👎 CLT tests in test/clt-tests/basic/worker_lifecycle_order.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
./manticore-load --drop --quiet --json --threads=1 --total=1 --init="CREATE TABLE worker_lifecycle_order(value int)" --worker-init="BEGIN; INSERT INTO worker_lifecycle_order(id,value) VALUES(1,10)" --worker-finalize="INSERT INTO worker_lifecycle_order(id,value) VALUES(3,30); COMMIT" --load="INSERT INTO worker_lifecycle_order(id,value) VALUES(2,20)"
––– output –––
{
"threads": 1,
"batch_size": 1,
"time": "00:00",
"total_operations": 1,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -e "SELECT id,value FROM worker_lifecycle_order WHERE id=1; SELECT id,value FROM worker_lifecycle_order WHERE id=2; SELECT id,value FROM worker_lifecycle_order WHERE id=3"
––– output –––
- id value
+ +------+-------+
- 1 10
+ | id | value |
- id value
+ +------+-------+
- 2 20
+ | 1 | 10 |
- id value
+ +------+-------+
- 3 30
+ +------+-------+
+ | id | value |
+ +------+-------+
+ | 2 | 20 |
+ +------+-------+
+ +------+-------+
+ | id | value |
+ +------+-------+
+ | 3 | 30 |
+ +------+-------+
––– input –––
./manticore-load --drop --quiet --json --threads=4 --total=4 --init="CREATE TABLE worker_lifecycle_parallel(value int)" --worker-init="BEGIN" --worker-finalize="COMMIT" --load="INSERT INTO worker_lifecycle_parallel(id,value) VALUES(<increment>,10)"
––– output –––
{
"threads": 4,
"batch_size": 1,
"time": "00:00",
"total_operations": 4,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -N -e "SELECT COUNT(*),MIN(value),MAX(value) FROM worker_lifecycle_parallel"
––– output –––
- 4\t10\t10
+ +------+------+------+
+ | 4 | 10 | 10 |
+ +------+------+------+
––– input –––
./manticore-load --drop --threads=4 --batch-size=2 --total=8 --init="CREATE TABLE worker_lifecycle_failure(value int)" --worker-init="BEGIN" --worker-finalize="INVALID FINALIZER" --load="INSERT INTO worker_lifecycle_failure(id,value) VALUES(<increment>,10)" > /dev/null; echo $?; mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM worker_lifecycle_failure"
––– output –––
1
- 0
+ +------+
+ | 0 |
+ +------+
––– input –––
)"
––– output –––
- {
+ bash: syntax error near unexpected token `)'
- "threads": 1,
- "batch_size": 2,
- "time": "00:00",
- "total_operations": 4,
- "operations_per_second": #!/\d+/!#,
- "qps": {
- "avg": 0,
- "p99": 0,
- "p95": 0,
- "p5": 0,
- "p1": 0
- },
- "latency": {
- "avg": #!/\d+(\.\d+)?/!#,
- "p50": #!/\d+(\.\d+)?/!#,
- "p95": #!/\d+(\.\d+)?/!#,
- "p99": #!/\d+(\.\d+)?/!#
- }
- }
––– input –––
mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM multiline_insert"
––– output –––
- 4
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'multiline_insert' in search request
––– input –––
cache=$(./manticore-load --verbose --threads=1 --batch-size=2 --total=5 --init="CREATE TABLE empty_cache_regression(value int)" --load="INSERT INTO empty_cache_regression(id,value) VALUES(<increment>,<int/1/100>)" 2>&1 | sed -n 's/.*Generating new data cache \(\/tmp\/manticore_load_[a-f0-9]*\).*/\1/p' | head -1); test -n "$cache"; : > "$cache"; inode_before=$(stat -c %i "$cache"); ./manticore-load --drop --quiet --json --threads=1 --batch-size=2 --total=5 --cache-from-disk --init="CREATE TABLE empty_cache_regression(value int)" --load="INSERT INTO empty_cache_regression(id,value) VALUES(<increment>,<int/1/100>)"; inode_after=$(stat -c %i "$cache"); test "$inode_before" = "$inode_after"; test -s "$cache"; mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM empty_cache_regression"
––– output –––
{
"threads": 1,
"batch_size": 2,
"time": "00:00",
"total_operations": 6,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
- 5
+ +------+
-
+ | 5 |
+ +------+
+
test/clt-tests/basic/worker_lifecycle.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
./manticore-load --drop --quiet --json --threads=1 --batch-size=2 --total=4 --init="CREATE TABLE worker_lifecycle(value int)" --worker-init="BEGIN" --worker-finalize="COMMIT" --load="INSERT INTO worker_lifecycle(id,value) VALUES(<increment>,<int/1/100>)"
––– output –––
{
"threads": 1,
"batch_size": 2,
"time": "00:00",
"total_operations": 4,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) FROM worker_lifecycle; SHOW TABLE worker_lifecycle STATUS LIKE 'ram_chunk_segments_count'"
––– output –––
- COUNT(*)
+ +----------+
- 4
+ | count(*) |
- Variable_name Value
+ +----------+
- ram_chunk_segments_count 2
+ | 4 |
-
+ +----------+
+ +--------------------------+-------+
+ | Variable_name | Value |
+ +--------------------------+-------+
+ | ram_chunk_segments_count | 1 |
+ +--------------------------+-------+
+
test/clt-tests/basic/insert_two.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
PROCESS_1_DELAY=0.5 PROGRESS_DELAY=0.5 ./manticore-load --drop --init="create table t" --total=10 --load="insert into t values(<increment>)" --threads=1 --delay=1 --together --drop --init="create table t2" --total=10 --load="insert into t2 values(<increment>)" --threads=1 --delay=0.5
––– output –––
======================================================================================
Process 2: Running with threads=1, batch-size=1
Process 2: Dropping table: DROP TABLE IF EXISTS t2
Process 2: Executing init command: create table t2
Process 2: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
======================================================================================
Process 1: Running with threads=1, batch-size=1
Process 1: Dropping table: DROP TABLE IF EXISTS t
Process 1: Executing init command: create table t
Process 1: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
-------------------------------------------------------------------------------------------------------------------------------------------
Time Elapsed 1:Progress 2:Progress 1:QPS 2:QPS 1:DPS 2:DPS CPU Workers Chunks Merging Disk Inserted
-------------------------------------------------------------------------------------------------------------------------------------------
%{TIME} #!/\d+/!#ms - - - - - - N/A - 0 0B 0
- %{TIME} 1.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 6
+ 04:18:20 1.5s 20% 30% 2 2 2 1 N/A 1 0 978B 6
- %{TIME} 2.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 9
+ 04:18:21 2.5s 30% 50% 1 2 1 2 8% 1 0 978B 9
- %{TIME} 3.#!/\d/!#s #!/\d/!#0% #!/\d/!#0% #!/\d/!# #!/\d/!# #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)/!##!/\s+/!##!/\d/!# 0 930B 12
+ 04:18:22 3.5s 40% 70% 1 2 1 2 5% 1 0 978B 12
--------------------------------------------------------------------------------------
Process 2 final statistics:
Init command: create table t2
Load commands:
- insert into t2 values(<increment>)
Total time: #!/(4|5)\.\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 2
QPS avg: 2
QPS 1p: 2
QPS 5p: 2
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 50#!/\d/!#.#!/\d/!# ms
Latency 50p: 505.#!/\d/!# ms
Latency 95p: 505.#!/\d/!# ms
Latency 99p: 505.#!/\d/!# ms
--------------------------------------------------------------------------------------
- %{TIME} 4.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 5
+ 04:18:23 4.5s 50% - 1 - 1 - 4% 1 0 489B 5
- %{TIME} 5.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 6
+ 04:18:24 5.5s 60% - 1 - 1 - 2% 1 0 489B 6
- %{TIME} 6.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 7
+ 04:18:25 6.5s 70% - 1 - 1 - 3% 1 0 489B 7
- %{TIME} 7.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 8
+ 04:18:26 7.5s 80% - 1 - 1 - 4% 1 0 489B 8
- %{TIME} 8.#!/\d/!#s #!/\d/!#0% - #!/\d/!# - #!/\d/!# - #!/(\d+%|N\/A)/!##!/\s+/!#1 0 465B 9
+ 04:18:27 8.5s 90% - 1 - 1 - 6% 1 0 489B 9
--------------------------------------------------------------------------------------
Process 1 final statistics:
Init command: create table t
Load commands:
- insert into t values(<increment>)
Total time: 9.#!/\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 1
QPS avg: 1
QPS 1p: 1
QPS 5p: 1
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 100#!/\d/!#.#!/\d/!# ms
Latency 50p: 1050.#!/\d/!# ms
Latency 95p: 1050.#!/\d/!# ms
Latency 99p: 1050.#!/\d/!# ms
--------------------------------------------------------------------------------------
test/clt-tests/basic/insert_basic.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
PROCESS_1_DELAY=0.5 PROGRESS_DELAY=0.5 ./manticore-load --drop --init="create table t" --total=10 --load="insert into t values(<increment>)" --threads=1 --delay=1
––– output –––
======================================================================================
Process 1: Running with threads=1, batch-size=1
Process 1: Dropping table: DROP TABLE IF EXISTS t
Process 1: Executing init command: create table t
Process 1: Generating new data cache /tmp/manticore_load_#!/[a-z0-9]+/!# ... 100%
-------------------------------------------------------------------------------------------------------------
Time Elapsed Progress QPS DPS CPU Workers Chunks Merging Disk Inserted
-------------------------------------------------------------------------------------------------------------
%{TIME} 5#!/\d\d/!#ms - - - N/A - 0 0B 0
- %{TIME} 1.#!/\d/!#s #!/\d/!#0% #!/\d/!# #!/\d/!# N/A 1 0 465B 2
+ 04:18:06 1.5s 20% 2 2 N/A 1 0 489B 2
- %{TIME} 2.#!/\d/!#s #!/\d/!#0% #!/\d/!# #!/\d/!# #!/(\d+%|N\/A)\s+/!#1 0 465B 3
+ 04:18:07 2.5s 20% 2 2 10% 1 0 489B 3
- %{TIME} 3.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 4
+ 04:18:08 3.5s 40% 1 1 2% 1 0 489B 4
- %{TIME} 4.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 5
+ 04:18:09 4.5s 40% 1 1 3% 1 0 489B 5
- %{TIME} 5.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 6
+ 04:18:10 5.5s 60% 1 1 3% 1 0 489B 6
- %{TIME} 6.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 7
+ 04:18:11 6.5s 70% 1 1 2% 1 0 489B 7
- %{TIME} 7.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 8
+ 04:18:12 7.5s 80% 1 1 3% 1 0 489B 8
- %{TIME} 8.#!/\d/!#s #!/\d/!#0% 1 1 #!/(\d+%|N\/A)\s+/!#1 0 465B 9
+ 04:18:13 8.5s 90% 1 1 3% 1 0 489B 9
--------------------------------------------------------------------------------------
Process 1 final statistics:
Init command: create table t
Load commands:
- insert into t values(<increment>)
Total time: 9#!/\.\d/!#s
Total queries: 10
Threads: 1
Batch size: 1
Total docs: 10
Docs per sec avg: 1
QPS avg: 1
QPS 1p: 1
QPS 5p: 1
QPS 95p: #!/\d/!#
QPS 99p: #!/\d/!#
Latency avg: 100#!/\d\.\d/!# ms
Latency 50p: 1050.0 ms
Latency 95p: 1050.0 ms
Latency 99p: 1050.0 ms
--------------------------------------------------------------------------------------
|
clt-insert👎 CLT tests in test/clt-tests/basic/worker_lifecycle_order.rec––– input –––
searchd > /dev/null
––– output –––
––– input –––
sleep 3
––– output –––
––– input –––
cd /manticore-load
––– output –––
––– input –––
./manticore-load --drop --quiet --json --threads=1 --total=1 --init="CREATE TABLE worker_lifecycle_order(value int)" --worker-init="BEGIN; INSERT INTO worker_lifecycle_order(id,value) VALUES(1,10)" --worker-finalize="INSERT INTO worker_lifecycle_order(id,value) VALUES(3,30); COMMIT" --load="INSERT INTO worker_lifecycle_order(id,value) VALUES(2,20)"
––– output –––
{
"threads": 1,
"batch_size": 1,
"time": "00:00",
"total_operations": 1,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -e "SELECT id,value FROM worker_lifecycle_order WHERE id=1; SELECT id,value FROM worker_lifecycle_order WHERE id=2; SELECT id,value FROM worker_lifecycle_order WHERE id=3"
––– output –––
+------+-------+
| id | value |
+------+-------+
| 1 | 10 |
+------+-------+
+------+-------+
| id | value |
+------+-------+
| 2 | 20 |
+------+-------+
+------+-------+
| id | value |
+------+-------+
| 3 | 30 |
+------+-------+
––– input –––
./manticore-load --drop --quiet --json --threads=4 --total=4 --init="CREATE TABLE worker_lifecycle_parallel(value int)" --worker-init="BEGIN" --worker-finalize="COMMIT" --load="INSERT INTO worker_lifecycle_parallel(id,value) VALUES(<increment>,10)"
––– output –––
{
"threads": 4,
"batch_size": 1,
"time": "00:00",
"total_operations": 4,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -N -e "SELECT COUNT(*),MIN(value),MAX(value) FROM worker_lifecycle_parallel"
––– output –––
+------+------+------+
| 4 | 10 | 10 |
+------+------+------+
––– input –––
./manticore-load --drop --threads=4 --batch-size=2 --total=8 --init="CREATE TABLE worker_lifecycle_failure(value int)" --worker-init="BEGIN" --worker-finalize="INVALID FINALIZER" --load="INSERT INTO worker_lifecycle_failure(id,value) VALUES(<increment>,10)" > /dev/null; echo $?; mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM worker_lifecycle_failure"
––– output –––
1
+------+
- | 0 |
+ | 0 |
+------+
––– input –––
./manticore-load --drop --quiet --json --threads=1 --batch-size=2 --total=4 --init="CREATE TABLE multiline_insert(value int)" --load="$(printf 'INSERT INTO multiline_insert(\nid,\nvalue\n) VALUES(\n<increment>,\n<int/1/100>\n)')"
––– output –––
{
"threads": 1,
"batch_size": 2,
"time": "00:00",
"total_operations": 4,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
––– input –––
mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM multiline_insert"
––– output –––
+------+
| 4 |
+------+
––– input –––
cache=$(./manticore-load --verbose --threads=1 --batch-size=2 --total=5 --init="CREATE TABLE empty_cache_regression(value int)" --load="INSERT INTO empty_cache_regression(id,value) VALUES(<increment>,<int/1/100>)" 2>&1 | sed -n 's/.*Generating new data cache \(\/tmp\/manticore_load_[a-f0-9]*\).*/\1/p' | head -1); test -n "$cache"; : > "$cache"; inode_before=$(stat -c %i "$cache"); ./manticore-load --drop --quiet --json --threads=1 --batch-size=2 --total=5 --cache-from-disk --init="CREATE TABLE empty_cache_regression(value int)" --load="INSERT INTO empty_cache_regression(id,value) VALUES(<increment>,<int/1/100>)"; inode_after=$(stat -c %i "$cache"); test "$inode_before" = "$inode_after"; test -s "$cache"; mysql -h0 -P9306 -N -e "SELECT COUNT(*) FROM empty_cache_regression"
––– output –––
{
"threads": 1,
"batch_size": 2,
"time": "00:00",
"total_operations": 6,
"operations_per_second": #!/\d+/!#,
"qps": {
"avg": 0,
"p99": 0,
"p95": 0,
"p5": 0,
"p1": 0
},
"latency": {
"avg": #!/\d+(\.\d+)?/!#,
"p50": #!/\d+(\.\d+)?/!#,
"p95": #!/\d+(\.\d+)?/!#,
"p99": #!/\d+(\.\d+)?/!#
}
}
+------+
| 5 |
+------+
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run connection-scoped setup and ordered concurrent finalization SQL around benchmark workloads. Preserve child exit statuses, support multiline cached INSERT templates, and regenerate zero-byte caches in place.