Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

53
LINES

< > BotCompany Repo | #1004101 // Try scraping medium.com - using streamItems, stopped working

JavaX source code [tags: use-pretranspiled] - run with: x30.jar

Libraryless. Click here for Pure Java version (2435L/17K/54K).

!752

p {
  S username = "stefanreich";
  Map<S, O> data = loadMediumJSON(username);
  printStructureLines(data);
  
  pcall {
    Map user = (Map) data.get("user");
    print("Name: " + user.get("name"));
    print("Bio: " + user.get("bio"));
    print("Image ID: " + user.get("imageid"));
    // Could do more here, like Twitter name
  }
  
  pcall {
    L streamItems = (L) data.get("streamItems");
    for i over streamItems: {
      print("Stream item #" + i);
      Map item = (Map) streamItems.get(i);
      Map preview = (Map) item.get("postPreview");
      psl(preview);
      if (preview != null) {
        S postID = getString(preview, "postId");
        Map post = (Map) preview.get("post");
        //psl(post);
        print("Post ID: " + postID);
        S uniqueSlug = getString(post, "uniqueSlug");
        S link = "https://medium.com/@" + username + "/" + uniqueSlug;
        S title = getString(post, "title");
        print("Title: " + title);
        print("Probable link: " + link);
        psl(post);
      }
      print();
    }
  }
}

sbool loadMediumJSON_verbose;

static Map<S, O> loadMediumJSON(S username) {
  S html = loadPageWithUserAgent("https://medium.com/@" + username, "Mac Safari");
  if (loadMediumJSON_verbose) print(html);
  
  S prefix = [[window["obvInit"](]];
  int i = indexOf(html, prefix);
  if (i < 0) null;
  int j = indexOf(html, "\n", i);
  if (j < 0) null;
  S json = dropSuffix(")", trim(substring(html, i+l(prefix), j)));
  ret jsonDecodeMap(json);
}

download  show line numbers  debug dex  old transpilations   

Travelled to 14 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, ddnzoavkxhuk, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID: #1004101
Snippet name: Try scraping medium.com - using streamItems, stopped working
Eternal ID of this version: #1004101/1
Text MD5: 831f4f1a066139c88a8ef70483af2a44
Transpilation MD5: 3de7994e59212667ede5c3c61ca517e3
Author: stefan
Category: javax / networking
Type: JavaX source code
Public (visible to everyone): Yes
Archived (hidden from active list): No
Created/modified: 2016-10-04 21:48:40
Source code size: 1556 bytes / 53 lines
Pitched / IR pitched: No / No
Views / Downloads: 563 / 632
Referenced in: [show references]