Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

72
LINES

< > BotCompany Repo | #1016606 // LinkedTokenList Spike [OK]

JavaX source code (desktop) [tags: use-pretranspiled] - run with: x30.jar

Download Jar. Uses 3874K of libraries. Click here for Pure Java version (7817L/58K).

!7

sclass LinkedToken {
  S t;
  LinkedToken prev, next;
  LinkedToken prevIdentical, nextIdentical;
  
  *() {}
  *(S *t) {}
}

sclass Tokenization {
  LinkedToken first, last;
  new Map<S, LinkedToken> firstByContent;
  new Map<S, LinkedToken> lastByContent;
  
  bool contains(S t) {
    ret firstByContent.get(t) != null;
  }
  
  int countInstances(S t) {
    int n = 0;
    LinkedToken lt = firstByContent.get(t);
    while (lt != null) {
      ++n;
      lt = lt.nextIdentical;
    }
    ret n;
  }
}

static Tokenization makeTokenization(L<S> tokens) {
  new Tokenization tok;
  for (S t : tokens) {
    LinkedToken lt = new(t);
    lt.t = t;
    if (tok.first == null)
      tok.first = lt;
    else {
      lt.prev = tok.last;
      tok.last.next = lt;
    }
    tok.last = lt;
    lt.prevIdentical = tok.lastByContent.get(t);
    if (lt.prevIdentical == null)
      tok.firstByContent.put(t, lt);
    else {
      lt.prevIdentical.nextIdentical = lt;
    }
    tok.lastByContent.put(t, lt);
  }
  ret tok;
}

p-experiment {
  S text = "hello hello";
  L<S> tokens = javaTok(text);
  
  Tokenization tok = makeTokenization(tokens);
  
  printStruct(tok);
  
  assertVerbose(true);
  assertEquals(true, tok.contains("hello"));
  assertEquals(true, tok.contains(" "));
  assertEquals(false, tok.contains("x"));
  assertEquals(2, tok.countInstances("hello"));
  assertEquals(1, tok.countInstances(" "));
  assertEquals(0, tok.countInstances("x"));
  
  print("OK!");
}

download  show line numbers  debug dex  old transpilations   

Travelled to 14 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, gwrvuhgaqvyk, irmadwmeruwu, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID: #1016606
Snippet name: LinkedTokenList Spike [OK]
Eternal ID of this version: #1016606/10
Text MD5: f8e5c9664d43f394b5001f2dfe05f70e
Transpilation MD5: f077080717a15cfba02374be0b0ef28f
Author: stefan
Category: javax / parsing
Type: JavaX source code (desktop)
Public (visible to everyone): Yes
Archived (hidden from active list): No
Created/modified: 2018-06-25 14:39:19
Source code size: 1547 bytes / 72 lines
Pitched / IR pitched: No / No
Views / Downloads: 296 / 725
Version history: 9 change(s)
Referenced in: [show references]