Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

33
LINES

< > BotCompany Repo | #1025695 // PositionalTokenIndex2 - PositionalTokenIndex grouped by token count [OK]

JavaX fragment (include) [tags: use-pretranspiled]

Libraryless. Click here for Pure Java version (2640L/17K).

// clever index of tokenized strings
// maintaining one CI-sorted list per token position
sclass PositionalTokenIndex2 {
  new L<PositionalTokenIndex> indices; // index = token count

  *() {}
  *(Iterable<LS> toks) { addAll(toks); }
  
  void addAll(Iterable<LS> toks) {
    fOr (LS tok : toks)
      add(tok);
  }
  
  void add(LS tok) {
    listGetOrCreate_full PositionalTokenIndex(indices, l(tok)/2).add(tok);
  }
  
  // return all tokenizations with token t in position i
  // having tokenCount code tokens
  // may return null
  LLS byToken(int i, S t, int tokenCount) {
    PositionalTokenIndex idx = get(indices, tokenCount);
    ret idx == null ? null : idx.byToken(i, t);
  }
  
  PositionalTokenIndex indexForTokenCount(int tokenCount) {
    ret get(indices, tokenCount);
  }
  
  void clean() {
    for (PositionalTokenIndex idx : indices) idx.clean();
  }
}

Author comment

Began life as a copy of #1025688

download  show line numbers  debug dex  old transpilations   

Travelled to 6 computer(s): bhatertpkbcr, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID: #1025695
Snippet name: PositionalTokenIndex2 - PositionalTokenIndex grouped by token count [OK]
Eternal ID of this version: #1025695/5
Text MD5: 7e6789bc590fbe1f2345c9c982b43993
Transpilation MD5: c18c7ce3c15822be879095443303bc2a
Author: stefan
Category: javax / agi.blue
Type: JavaX fragment (include)
Public (visible to everyone): Yes
Archived (hidden from active list): No
Created/modified: 2019-10-13 23:45:55
Source code size: 904 bytes / 33 lines
Pitched / IR pitched: No / No
Views / Downloads: 172 / 530
Version history: 4 change(s)
Referenced in: [show references]