Setup
1 Node cluster on my local laptop: 8core, Xms=8G, Xmx=8G
專注于為中小企業(yè)提供做網(wǎng)站、成都網(wǎng)站建設(shè)服務(wù),電腦端+手機(jī)端+微信端的三站合一,更高效的管理,為中小企業(yè)臨河免費(fèi)做網(wǎng)站提供優(yōu)質(zhì)的服務(wù)。我們立足成都,凝聚了一批互聯(lián)網(wǎng)行業(yè)人才,有力地推動(dòng)了上1000家企業(yè)的穩(wěn)健成長(zhǎng),幫助中小企業(yè)通過網(wǎng)站建設(shè)實(shí)現(xiàn)規(guī)模擴(kuò)充和轉(zhuǎn)變。
Indexing performance (Single index):
10 million payments, each one about 5KB, with batch size = 10000. Each batch takes roughly 2.5 s → 4 s, total time to index 10 million payment is around 50 min
Indexing performance (Multiple indices):
20 separate indices store totally 10 million payments. Indexing execution is slightly faster than single index case. Each batch takes roughly 1.7 s → 3.8 s, total time to index 10 million payment is around 38 min
Parameters required for bulk load operation
Elasticsearch config: http.max_content_length: 500mb
Client time out adjustment:
RestClient.builder(HttpHost("localhost", 9200))
.setRequestConfigCallback {
it.apply {
this.setConnectTimeout(5000)
this.setSocketTimeout(60000)
}
}.setMaxRetryTimeoutMillis(60000))
Initially batch size is set to 100000, elastic search server becomes unstable with high GC frequency, occupying a large percent of CPU time. So larger batch size does not always imply higher performance
Query aggregation performance:
Test query: real aggregation query used by rule engine
{
"aggregations": {
"date_range": {
"range": {
"field": "createdAt",
"ranges": [
{
"key": "LAST_7_DAYS",
"from": 1544400968485,
"to": 1545005768486
}
],
"keyed": false
},
"aggregations": {
"filter_aggregator": {
"filters": {
"filters": {
"602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
"terms": {
"beneficiaryId.keyword": [
"602c7d66-e990-4dfb-b6e2-72b62ff159d5"
],
"boost": 1
}
},
"67cab0c8-2510-443d-8f00-bce19c04815e": {
"terms": {
"bankAccountUserId.keyword": [
"67cab0c8-2510-443d-8f00-bce19c04815e"
],
"boost": 1
}
},
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7": {
"terms": {
"payerId.keyword": [
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
],
"boost": 1
}
},
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
"bool": {
"filter": [
{
"terms": {
"payerId.keyword": [
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
],
"boost": 1
}
},
{
"terms": {
"beneficiaryId.keyword": [
"602c7d66-e990-4dfb-b6e2-72b62ff159d5"
],
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"9a1b4bad-ccf5-4c67-8718-02696cb351e4": {
"terms": {
"clientId.keyword": [
"9a1b4bad-ccf5-4c67-8718-02696cb351e4"
],
"boost": 1
}
},
"9a1b4bad-ccf5-4c67-8718-02696cb351e4_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
"bool": {
"filter": [
{
"terms": {
"clientId.keyword": [
"9a1b4bad-ccf5-4c67-8718-02696cb351e4"
],
"boost": 1
}
},
{
"terms": {
"beneficiaryId.keyword": [
"602c7d66-e990-4dfb-b6e2-72b62ff159d5"
],
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"9a1b4bad-ccf5-4c67-8718-02696cb351e4_8da52e51-eabf-4f6c-b9f0-e222933c1cb7": {
"bool": {
"filter": [
{
"terms": {
"clientId.keyword": [
"9a1b4bad-ccf5-4c67-8718-02696cb351e4"
],
"boost": 1
}
},
{
"terms": {
"payerId.keyword": [
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
],
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"9a1b4bad-ccf5-4c67-8718-02696cb351e4_8da52e51-eabf-4f6c-b9f0-e222933c1cb7_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
"bool": {
"filter": [
{
"terms": {
"clientId.keyword": [
"9a1b4bad-ccf5-4c67-8718-02696cb351e4"
],
"boost": 1
}
},
{
"terms": {
"payerId.keyword": [
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
],
"boost": 1
}
},
{
"terms": {
"beneficiaryId.keyword": [
"602c7d66-e990-4dfb-b6e2-72b62ff159d5"
],
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
}
},
"other_bucket": false,
"other_bucket_key": "_other_"
},
"aggregations": {
"beneficiary_amount": {
"stats": {
"field": "beneficiaryAmountUsd"
}
},
"payer_amount": {
"stats": {
"field": "payerAmountUsd"
}
},
"distinct_count_beneficiary": {
"cardinality": {
"field": "beneficiaryId.keyword"
}
},
"distinct_count_payer": {
"cardinality": {
"field": "payerId.keyword"
}
},
"distinct_count_client": {
"cardinality": {
"field": "clientId.keyword"
}
},
"distinct_count_bank_acc": {
"cardinality": {
"field": "bankAccountUserId.keyword"
}
},
"distinct_count_bene_country": {
"cardinality": {
"field": "beneficiaryCountry.keyword"
}
},
"distinct_count_payer_country": {
"cardinality": {
"field": "payerCountry.keyword"
}
},
"distinct_count_bene_currency": {
"cardinality": {
"field": "beneficiaryCurrency.keyword"
}
},
"distinct_count_payer_currency": {
"cardinality": {
"field": "payerCurrency.keyword"
}
},
"structured_payment_amount_personal": {
"range": {
"field": "payerAmountUsd",
"ranges": [
{
"from": 9000,
"to": 9999.999
}
],
"keyed": false
}
},
"structured_payment_amount_company": {
"range": {
"field": "payerAmountUsd",
"ranges": [
{
"from": 112500000,
"to": 124999999.999
}
],
"keyed": false
}
}
}
}
}
}
}
}
Test result : (Single Index)
Scenario | Number of run | Execution times | Min | Max | Average |
---|---|---|---|---|---|
Single thread Search result hit Result size unset | 10 | 6s, 5.9s, 5.9s, 6s, 5.9s, 6.2s, 6.2s, 6.2s, 6.6s, 8.2s | 5.9s | 8.2s | 6.93s |
Single thread Search result hit Result size = 0 | 10 | 7200ms, 12ms, 19ms, 20ms, 23ms, 22ms, 28ms, 21ms, 19ms, 29ms | 12ms | 7200ms | 19.3ms (First execution takes 7.2s, deviates too much from the rest, so exclude it) |
Single thread Search result none hit Result size unset | 10 | 510ms, 529ms, 549ms, 512ms, 489ms, 520ms, 506ms, 500ms, 493ms, 499ms | 489ms | 549ms | 510.7ms |
Single thread Search result none hit Result size = 0 | 10 | 389ms, 1ms, 2ms, 3ms, 3ms, 1ms, 1ms, 1ms, 2ms, 3ms | 389ms | 1ms | 1.89ms (First execution takes 389ms, deviates too much from the rest, so exclude it) |
20 threads Search result hit Result size unset | 10 | Exception, listener timeout after waiting for [60000] ms | |||
20 threads Search result hit Result size = 0 | 10 | 18ms, 29ms, 61ms, 53ms, 93ms, 20ms, 39ms, 82ms, 17ms, 24ms | 17ms | 82ms | 43.6ms |
20 threads Search result none hit Result size unset | 10 | 4.4s, 5.8s, 4.9s, 5.6s 4.6s, 5.4s, 5.3s 4.7s, 5.8s, 4.7s | 5.8s | 4.4s | 5.1s |
20 threads Search result none hit Result size = 0 | 10 | 1.4s, 25ms, 6ms, 22ms, 276ms, 250ms, 19ms, 30ms, 11ms, 18ms | 1.4s | 11ms | 73ms (First execution takes 1.4s, deviates too much from the rest, so exclude it) |
Test result: (Multiple Index)
Scenario | Number of run | Execution times | Min | Max | Average |
---|---|---|---|---|---|
Single thread Search result hit Result size unset | 10 | 12.6s, 12.7s 12.6s 12.7s, 13.1s, 13.1s, 13.2s, 13,1s, 13.1s, 13.1s | 12.6s | 13.2s | 12.9s |
Single thread Search result hit Result size = 0 | 10 | 212ms, 147ms, 170ms, 272ms, 223ms, 222ms, 207ms, 255ms, 219ms, 219ms | 147ms | 272ms | 214.6ms |
Single thread Search result none hit Result size unset | 10 | 1.2s, 1.1s, 1.1s, 1.1s, 1.1s, 1.2s, 1.1s, 1.1s, 1.2s, 1.1s | 1.1s | 1.2s | 1.13s |
Single thread Search result none hit Result size = 0 | 10 | 909ms, 23ms, 23ms, 35ms, 14ms, 26ms, 19ms, 12ms, 15ms, 22m | 12ms | 909ms | 18.9ms (First execution takes 909ms, deviates too much from the rest, so exclude it) |
20 threads Search result hit Result size unset | 10 | Exception, listener timeout after waiting for [60000] ms | |||
20 threads Search result hit Result size = 0 | 10 | 18ms, 29ms, 61ms, 53ms, 93ms, 20ms, 39ms, 82ms, 17ms, 24ms | 17ms | 93ms | 43.6ms |
20 threads Search result none hit Result size unset | 10 | 7.8s, 7.6s, 7.1s, 6.9s, 7.4s, 8s, 7.3, 7.2s, 7.3s, 7.4s | 6.9s | 8s | 7.4s |
20 threads Search result none hit Result size = 0 | 10 | 202ms, 91ms, 100ms, 146ms, 182ms, 131ms, 158ms, 48ms, 171ms, 152ms | 48ms | 202ms | 138.1ms |
Conclusion:
Aggregation performance hinges on the number of documents that matches the aggregation?
Result size parameter has significant impact on aggregation performance. not only because it skipped returning hit documents, but also because it enables caching for aggregation result, otherwise, you have to force result caching by explicitly setting?request_cache=true
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/shard-request-cache.html
Executing query concurrently can also have negative impact on performance
Increasing number of indices have positive impact on index speed but have large negative impact on aggregation if the aggregation is performed across indices
網(wǎng)站欄目:Elasticsearchperformancetestingfinding
當(dāng)前URL:http://bm7419.com/article18/jdeogp.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián),為您提供虛擬主機(jī)、網(wǎng)頁(yè)設(shè)計(jì)公司、網(wǎng)站策劃、微信公眾號(hào)、網(wǎng)站建設(shè)、網(wǎng)站制作
聲明:本網(wǎng)站發(fā)布的內(nèi)容(圖片、視頻和文字)以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主,如果涉及侵權(quán)請(qǐng)盡快告知,我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如需處理請(qǐng)聯(lián)系客服。電話:028-86922220;郵箱:631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載,或轉(zhuǎn)載時(shí)需注明來源: 創(chuàng)新互聯(lián)