Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

66
LINES

< > BotCompany Repo | #1012454 // webs_readTripleFileAsTriples_parallel2

JavaX fragment (include)

please include function webs_readTripleFileAsTriples.

static L<TripleWeb> webs_readTripleFileAsTriples_parallel2(File f) ctex {
  if (!f.exists()) ret ll();
  time "Getting names" {
    Iterator<S> it = linesFromFile(f);
    final new L<Symbol> names;
    while (it.hasNext()) {
      S s = trim(it.next());
      if (empty(s)) break;
      names.add(symbol(unquote(s)));
    }
  }
  
  if (webs_readTripleFileAsTriples_debug)
    print("Have " + l(names) + " names");
  
  final L<Pair<L<TripleWeb>, Int>> triples = new L;
  
  int poolSize = numberOfCores(), queueSize = 10000;
  int blockSize = 1000;
  NotifyingBlockingThreadPoolExecutor e = new(poolSize, queueSize, 15, TimeUnit.SECONDS);
  print("Using " + poolSize + " cores, block size: " + blockSize + ".");
  
  try {
    int i = 0;
    L<S> block = emptyListWithCapacity(blockSize);
    while licensed {
      ++i;
      final Int _i = i;
      bool hasNext = it.hasNext();
      if ((!hasNext || l(block) >= blockSize) && nempty(block)) {
        final L<S> _block = cloneList(block);
        e.execute(r { pcall {
          L<TripleWeb> l = emptyListWithCapacity(_block);
          for (S s : _block) {
            TripleWeb w = webs_readTripleFileAsTriples_line(s, names);
            if (w != null)
              l.add(w);
          }
          synchronized(triples) {
            triples.add(pair(l, _i));
          }
        }});
        block.clear();
      }
      if (hasNext)
        block.add(it.next());
      else
        break;
    }
    e.shutdown();
    e.awaitTermination(1, TimeUnit.DAYS);
  } finally {
    e.shutdown();
  }
  
  sortBySecondOfPairs_inPlace(triples);
  int n = 0;
  for (Pair<L<TripleWeb>, Int> p : triples)
    n += l(p.a);
  L<TripleWeb> l = emptyListWithCapacity(n);
  for (Pair<L<TripleWeb>, Int> p : triples)
    l.addAll(p.a);
  ret l;
}

Author comment

Began life as a copy of #1012453

download  show line numbers  debug dex  old transpilations   

Travelled to 13 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID: #1012454
Snippet name: webs_readTripleFileAsTriples_parallel2
Eternal ID of this version: #1012454/10
Text MD5: b6d35ec8cdf426977c05a0a62697eae9
Author: stefan
Category: javax / a.i.
Type: JavaX fragment (include)
Public (visible to everyone): Yes
Archived (hidden from active list): No
Created/modified: 2017-12-31 21:00:22
Source code size: 1906 bytes / 66 lines
Pitched / IR pitched: No / No
Views / Downloads: 405 / 424
Version history: 9 change(s)
Referenced in: [show references]