Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

42
LINES

< > BotCompany Repo | #1011783 // webs_readTripleFile_parallel

JavaX fragment (include)

1  
static L<Web> webs_readTripleFile_parallel(File f) ctex {
2  
  if (!f.exists()) ret ll();
3  
  print("Reading " + f2s(f) + " in " + nCore(numberOfCores()));
4  
  time "webs_readTripleFile_parallel" {
5  
    time "Reading names" {
6  
      Iterator<S> it = linesFromFile(f);
7  
    
8  
      final new L<S> names;
9  
      while (it.hasNext()) {
10  
        S s = trim(it.next());
11  
        if (empty(s)) break;
12  
        names.add(unquote(s));
13  
      }
14  
    }
15  
    
16  
    final new L<Pair<Web, Int>> webs;
17  
    
18  
    int poolSize = numberOfCores(), queueSize = 1000;
19  
    NotifyingBlockingThreadPoolExecutor e = new(poolSize, queueSize, 15, TimeUnit.SECONDS);
20  
    
21  
    try {
22  
      int i = 0;
23  
      for (fS s : iterable(it)) {
24  
        ++i;
25  
        final int _i = i;
26  
        e.execute(r { pcall {
27  
          Web web = webs_readTripleFile_line(s, names);
28  
          if (web != null) synchronized(webs) {
29  
            webs.add(pair(web, _i));
30  
          }
31  
        }});
32  
      }
33  
      e.shutdown();
34  
      e.awaitTermination(1, TimeUnit.DAYS);
35  
    } finally {
36  
      e.shutdown();
37  
    }
38  
    
39  
    L<Web> result = firstOfPairs(sortBySecondOfPairs_inPlace(webs));
40  
  }
41  
  ret result;
42  
}

Author comment

Began life as a copy of #1011762

download  show line numbers  debug dex  old transpilations   

Travelled to 13 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID: #1011783
Snippet name: webs_readTripleFile_parallel
Eternal ID of this version: #1011783/11
Text MD5: e80f9c965bf7503fb11285fa7ce781a2
Author: stefan
Category: javax / a.i.
Type: JavaX fragment (include)
Public (visible to everyone): Yes
Archived (hidden from active list): No
Created/modified: 2017-11-05 04:38:45
Source code size: 1169 bytes / 42 lines
Pitched / IR pitched: No / No
Views / Downloads: 362 / 392
Version history: 10 change(s)
Referenced in: [show references]