ai_indexWebs_parallel - note: indexes in different order - this makes no sense, should make multiple indexes, then merge [1011919]

< > BotCompany Repo | #1011919 // ai_indexWebs_parallel - note: indexes in different order - this makes no sense, should make multiple indexes, then merge

JavaX fragment (include)

static MultiMap<S, WebNode> ai_indexWebs_parallel(Collection<Web> webs) ctex {
  print("Making index in parallel.");
  final MultiMap<S, WebNode> index = caseInsensitiveMultiMap();
  
  int queueSize = numberOfCores();
  NotifyingBlockingThreadPoolExecutor e = new(numberOfCores(), queueSize, 15, TimeUnit.SECONDS);
    
  try {
    for (final L<Web> l : chunksOf1000(webs)) {
      e.execute(r { pcall {
        for (Web web : l)
          ai_addWebToIndex(index, web);
      }});
    }
    e.shutdown();
    e.awaitTermination(1, TimeUnit.DAYS);
  } finally {
    e.shutdown();
  }
  
  ret index;
}

Author comment

Began life as a copy of #1011347

Travelled to 13 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt

Snippet ID:

#1011919

Snippet name:

ai_indexWebs_parallel - note: indexes in different order - this makes no sense, should make multiple indexes, then merge

Eternal ID of this version:

#1011919/9

Text MD5:

235531fb20d539813096168dc4d983fb

Author:

stefan

Category:

javax / a.i.

Type:

JavaX fragment (include)

Public (visible to everyone):

Yes

Archived (hidden from active list):

Created/modified:

2017-11-10 03:29:30

Source code size:

622 bytes / 22 lines

Pitched / IR pitched:

No / No

Views / Downloads:

410 / 432

Version history:

8 change(s)

Referenced in:

[show references]

1	static MultiMap<S, WebNode> ai_indexWebs_parallel(Collection<Web> webs) ctex {
2	print("Making index in parallel.");
3	final MultiMap<S, WebNode> index = caseInsensitiveMultiMap();
4
5	int queueSize = numberOfCores();
6	NotifyingBlockingThreadPoolExecutor e = new(numberOfCores(), queueSize, 15, TimeUnit.SECONDS);
7
8	try {
9	for (final L<Web> l : chunksOf1000(webs)) {
10	e.execute(r { pcall {
11	for (Web web : l)
12	ai_addWebToIndex(index, web);
13	}});
14	}
15	e.shutdown();
16	e.awaitTermination(1, TimeUnit.DAYS);
17	} finally {
18	e.shutdown();
19	}
20
21	ret index;
22	}