Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

53
LINES

< > BotCompany Repo | #1004101 // Try scraping medium.com - using streamItems, stopped working

JavaX source code [tags: use-pretranspiled] - run with: x30.jar

Libraryless. Click here for Pure Java version (2435L/17K/54K).

1  
!752
2  
3  
p {
4  
  S username = "stefanreich";
5  
  Map<S, O> data = loadMediumJSON(username);
6  
  printStructureLines(data);
7  
  
8  
  pcall {
9  
    Map user = (Map) data.get("user");
10  
    print("Name: " + user.get("name"));
11  
    print("Bio: " + user.get("bio"));
12  
    print("Image ID: " + user.get("imageid"));
13  
    // Could do more here, like Twitter name
14  
  }
15  
  
16  
  pcall {
17  
    L streamItems = (L) data.get("streamItems");
18  
    for i over streamItems: {
19  
      print("Stream item #" + i);
20  
      Map item = (Map) streamItems.get(i);
21  
      Map preview = (Map) item.get("postPreview");
22  
      psl(preview);
23  
      if (preview != null) {
24  
        S postID = getString(preview, "postId");
25  
        Map post = (Map) preview.get("post");
26  
        //psl(post);
27  
        print("Post ID: " + postID);
28  
        S uniqueSlug = getString(post, "uniqueSlug");
29  
        S link = "https://medium.com/@" + username + "/" + uniqueSlug;
30  
        S title = getString(post, "title");
31  
        print("Title: " + title);
32  
        print("Probable link: " + link);
33  
        psl(post);
34  
      }
35  
      print();
36  
    }
37  
  }
38  
}
39  
40  
sbool loadMediumJSON_verbose;
41  
42  
static Map<S, O> loadMediumJSON(S username) {
43  
  S html = loadPageWithUserAgent("https://medium.com/@" + username, "Mac Safari");
44  
  if (loadMediumJSON_verbose) print(html);
45  
  
46  
  S prefix = [[window["obvInit"](]];
47  
  int i = indexOf(html, prefix);
48  
  if (i < 0) null;
49  
  int j = indexOf(html, "\n", i);
50  
  if (j < 0) null;
51  
  S json = dropSuffix(")", trim(substring(html, i+l(prefix), j)));
52  
  ret jsonDecodeMap(json);
53  
}

download  show line numbers  debug dex  old transpilations   

Travelled to 14 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, ddnzoavkxhuk, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID: #1004101
Snippet name: Try scraping medium.com - using streamItems, stopped working
Eternal ID of this version: #1004101/1
Text MD5: 831f4f1a066139c88a8ef70483af2a44
Transpilation MD5: 3de7994e59212667ede5c3c61ca517e3
Author: stefan
Category: javax / networking
Type: JavaX source code
Public (visible to everyone): Yes
Archived (hidden from active list): No
Created/modified: 2016-10-04 21:48:40
Source code size: 1556 bytes / 53 lines
Pitched / IR pitched: No / No
Views / Downloads: 566 / 637
Referenced in: [show references]