From 5329a9868138f261bc60c60e3c95680f69991334 Mon Sep 17 00:00:00 2001 From: Paul Jones Vijayakumar Date: Thu, 19 May 2016 09:07:49 -0500 Subject: [PATCH 1/8] making small changes to documentation --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 37da1a2..334d0ca 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ The Edison Web interface is a Node-based web application that allows one to sear This application requires the following to generate its components and to run: - node - - npm (node package manager) + - npm (node package manager - version > 3.8.3) - bash - java (version > 1.6) - mvn @@ -13,7 +13,7 @@ This application requires the following to generate its components and to run: Steps to run the application: 1) Extract the javadoc from the online javadoc repository. - - Replace the base_javadoc_url variable with the appropriate base_url (The url that produces the API overview page) + - Replace the base_javadoc_url variable in the 'extract_javadoc.sh' script in the javadoc_scraper folder with the appropriate base_url (The url that produces the API overview page) - The current default is: "http://cogcomp.cs.illinois.edu/software/doc/illinois-edison/apidocs/" - Run './extract_javadoc.sh' from the 'javadoc_scraper' directory From ee5cc3cd6cd6388e9856ba949e275e740aca29b0 Mon Sep 17 00:00:00 2001 From: Paul Jones Vijayakumar Date: Thu, 19 May 2016 09:31:19 -0500 Subject: [PATCH 2/8] adding troubleshooting for npmlog error --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 334d0ca..9293a72 100644 --- a/README.md +++ b/README.md @@ -32,4 +32,8 @@ Steps to run the application: - jade - bootstrap - [jquery-searchable](https://github.com/stidges/jquery-searchable) - \ No newline at end of file + +##Troubleshooting + - 'Error: Cannot find module 'npmlog'. + This error usually occurs in outdated versions of npm (typically < 2.5.8) so be sure to run 'sudo npm install npm -g' to fix this issue. + This error could also occur in some Mac installations and the best means to resolve the issue in that case is to a well-supported package installer such as 'homebrew' to handle your npm installation on Mac \ No newline at end of file From 8dfc56dc913c81b9f944c0fd4ad3b60cd56e10e4 Mon Sep 17 00:00:00 2001 From: Paul Jones Vijayakumar Date: Thu, 19 May 2016 09:32:27 -0500 Subject: [PATCH 3/8] adding small doc change --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9293a72..bde5f4b 100644 --- a/README.md +++ b/README.md @@ -35,5 +35,5 @@ Steps to run the application: ##Troubleshooting - 'Error: Cannot find module 'npmlog'. - This error usually occurs in outdated versions of npm (typically < 2.5.8) so be sure to run 'sudo npm install npm -g' to fix this issue. - This error could also occur in some Mac installations and the best means to resolve the issue in that case is to a well-supported package installer such as 'homebrew' to handle your npm installation on Mac \ No newline at end of file + - This error usually occurs in outdated versions of npm (typically < 2.5.8) so be sure to run 'sudo npm install npm -g' to fix this issue. + - This error could also occur in some Mac installations and the best means to resolve the issue in that case is to a well-supported package installer such as 'homebrew' to handle your npm installation on Mac \ No newline at end of file From ae6da4759887733f7af32b1422a2c6726555ff69 Mon Sep 17 00:00:00 2001 From: Paul Jones Vijayakumar Date: Thu, 19 May 2016 09:33:28 -0500 Subject: [PATCH 4/8] small doc change --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index bde5f4b..085a650 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ The Edison Web interface is a Node-based web application that allows one to sear This application requires the following to generate its components and to run: - node - - npm (node package manager - version > 3.8.3) + - npm (node package manager - version >= 3.8.3) - bash - java (version > 1.6) - mvn From c59150556bcf18eaa2d022c3c00ecfb260f016d1 Mon Sep 17 00:00:00 2001 From: Paul Jones Vijayakumar Date: Fri, 20 May 2016 08:35:20 -0500 Subject: [PATCH 5/8] made changes to doc and javadoc_scraper to specify the extraction URL for the raw github files --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 085a650..0559ed2 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,9 @@ Steps to run the application: 2) Use the jdparser maven package to parser and generate the 'data.json' file that will be ingested and presented by the web interface. - Ensure that the base_url variable in the 'src/test/java/cz/zcu/validator/jdparser/JDParserTest.java' file contains the base url with the root package path of the LREC feature classes. - The current default is: "http://cogcomp.cs.illinois.edu/software/doc/apidocs/edu/illinois/cs/cogcomp/edison/features/" + - Ensure that the github_url_base_raw variable in the 'src/test/java/cz/zcu/validator/jdparser/JDParserTest.java' file contains the base raw github url from which we would append a "Test[FeatureExtractorName]" to download the Test file to be presented. + - The current default is: "https://raw.githubusercontent.com/IllinoisCogComp/illinois-cogcomp-nlp/maste\ +r/edison/src/test/java/edu/illinois/cs/cogcomp/edison/features/lrec/" - Run 'mvn test' from the 'javadoc_scraper' directory, and this will generate the 'data.json' file in the base directory of our web application 3) Start the Web Application From 7c8c06d6ed44074b9e79d71c7aab44a7e5961e2d Mon Sep 17 00:00:00 2001 From: Paul Jones Vijayakumar Date: Fri, 20 May 2016 08:35:37 -0500 Subject: [PATCH 6/8] made changes to doc and javadoc_scraper to specify the extraction URL for the raw github files --- javadoc_scraper/robots.txt | 1 + .../jdparser/api/JsoupJavaDocParser.java | 9 +- .../jdparser/api/JsoupJavaDocParser.java~ | 652 ++++++++++++++++++ .../cz/zcu/validator/jdparser/api/Main.java | 2 +- .../cz/zcu/validator/jdparser/api/Main.java~ | 72 ++ .../zcu/validator/jdparser/JDParserTest.java | 7 +- .../zcu/validator/jdparser/JDParserTest.java~ | 38 + 7 files changed, 774 insertions(+), 7 deletions(-) create mode 100644 javadoc_scraper/robots.txt create mode 100755 javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java~ create mode 100755 javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java~ create mode 100755 javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java~ diff --git a/javadoc_scraper/robots.txt b/javadoc_scraper/robots.txt new file mode 100644 index 0000000..7d329b1 --- /dev/null +++ b/javadoc_scraper/robots.txt @@ -0,0 +1 @@ +User-agent: * diff --git a/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java b/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java index 743c0f4..b9d461c 100755 --- a/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java +++ b/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java @@ -109,17 +109,18 @@ public void setJavadoc_url(String javadoc_url) { */ private JavaDocsFactory javaDocFactory = null; - private String base_url, repo_path_string, path_to_write; + private String base_url, repo_path_string, base_github_url_raw, path_to_write; //Old approach to extracting classes that we require. //String[] fexes = {"LabelOneAfter", "LabelOneBefore", "LabelTwoAfter", "POSWindow"}; - public JsoupJavaDocParser(String base_url, String repo_path_string, String path_to_write) { + public JsoupJavaDocParser(String base_url, String repo_path_string, String base_github_url_raw, String path_to_write) { this.skeletonFactory = new SkeletonFactory(); this.javaDocFactory = new JavaDocsFactory(); this.base_url = base_url; this.repo_path_string = repo_path_string; + this.base_github_url_raw = base_github_url_raw; this.path_to_write = path_to_write; } @@ -202,13 +203,13 @@ public Collection parseDataFromJavaDoc( ClassRepresentation a = (ClassRepresentation) reps[i]; String java_code = ""; - String base_raw_url = "https://raw.githubusercontent.com/pauljv92/testedison/master/edison/src/test/java/edu/illinois/cs/cogcomp/edison/features/lrec/"; + //String base_raw_url = "https://raw.githubusercontent.com/pauljv92/testedison/master/edison/src/test/java/edu/illinois/cs/cogcomp/edison/features/lrec/"; JavaDocClass jdoc = a.getJavaDocClass(); try { - URL url = new URL(base_raw_url+"Test"+jdoc.getName()+".java"); + URL url = new URL(this.base_github_url_raw+"Test"+jdoc.getName()+".java"); Scanner s = new Scanner(url.openStream()); while(s.hasNext()) { // operate diff --git a/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java~ b/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java~ new file mode 100755 index 0000000..743c0f4 --- /dev/null +++ b/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java~ @@ -0,0 +1,652 @@ +package cz.zcu.validator.jdparser.api; + +import edu.illinois.cs.cogcomp.core.utilities.StringUtils; + +import java.io.BufferedReader; +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.net.URL; +import java.nio.charset.Charset; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Scanner; +import java.io.Writer; +import java.io.FileWriter; + +import org.jsoup.Jsoup; +import org.jsoup.nodes.Document; +import org.jsoup.nodes.Element; +import org.jsoup.nodes.Node; +import org.jsoup.select.Elements; + +import com.google.gson.Gson; + +import cz.zcu.validator.jdparser.api.javadocs.JavaDocAttribute; +import cz.zcu.validator.jdparser.api.javadocs.JavaDocClass; +import cz.zcu.validator.jdparser.api.javadocs.JavaDocMethod; +import cz.zcu.validator.jdparser.api.javadocs.factory.JavaDocsFactory; +import cz.zcu.validator.jdparser.api.skeletons.SkeletonAttribute; +import cz.zcu.validator.jdparser.api.skeletons.SkeletonClass; +import cz.zcu.validator.jdparser.api.skeletons.SkeletonConstructor; +import cz.zcu.validator.jdparser.api.skeletons.SkeletonMethod; +import cz.zcu.validator.jdparser.api.skeletons.factory.SkeletonFactory; +import cz.zcu.validator.jdparser.data.JavaDocFilesFinder; +import cz.zcu.validator.jdparser.data.JavaDocParsing; +import cz.zcu.validator.jdparser.data.JavaDocParsingException; + +/** + * Parses data from JavaDoc folder into {@code ClassRepresentation} list. An + * implementation of JavaDocParsing. + * + * @author Paul Vijayakumar + * + * adapted from code by Stepan Cais, scais@kiv.zcu.cz, 3.12.2012 + */ +public class JsoupJavaDocParser implements JavaDocParsing { + + private class FexClass{ + + public FexClass(String name, String description, String testcode, String javadoc_url, String annotations){ + this.name = name; + this.description = description; + this.testcode = testcode; + this.javadoc_url = javadoc_url; + this.annotations = annotations; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getTestcode() { + return testcode; + } + + public void setTestcode(String testcode) { + this.testcode = testcode; + } + + public String getJavadoc_url() { + return javadoc_url; + } + + public void setJavadoc_url(String javadoc_url) { + this.javadoc_url = javadoc_url; + } + + String name, description, testcode = null, javadoc_url = "", annotations = ""; + + + } + + /** + * Factory for creating skeleton elements. + */ + private SkeletonFactory skeletonFactory = null; + + /** + * Factory for creating javaDoc elements. + */ + private JavaDocsFactory javaDocFactory = null; + + private String base_url, repo_path_string, path_to_write; + + //Old approach to extracting classes that we require. + //String[] fexes = {"LabelOneAfter", "LabelOneBefore", "LabelTwoAfter", "POSWindow"}; + + public JsoupJavaDocParser(String base_url, String repo_path_string, String path_to_write) { + + this.skeletonFactory = new SkeletonFactory(); + this.javaDocFactory = new JavaDocsFactory(); + this.base_url = base_url; + this.repo_path_string = repo_path_string; + this.path_to_write = path_to_write; + } + + @Override + public Collection parseDataFromJavaDoc( + File javaDocFolder) throws JavaDocParsingException { + + + if (!javaDocFolder.isDirectory()) { + + throw new JavaDocParsingException( + "Given JavaDoc folder is not a folder: " + + javaDocFolder.getName() + + " Cannot continue with parsing."); + } + + Collection javaDocClassFiles = JavaDocFilesFinder + .getJavaDocClassFiles(javaDocFolder); + + Collection classes = new ArrayList(); + + ArrayList listforjson = new ArrayList(); + + for (File javaDocClassFile : javaDocClassFiles) { + + boolean set = false; + + //Extracting on the classes in the LREC package + if(javaDocClassFile.getAbsolutePath().contains("/lrec/")){ + System.out.println("We have found LREC feature extractor."); + set = true; + } + + if(!set){ + continue; + } + + Document jsoupDocument = null; + + try { + // TODO: Maybe usage of + // http://code.google.com/p/juniversalchardet/ + jsoupDocument = Jsoup.parse(javaDocClassFile, "UTF-8"); + + } catch (IOException e) { + + throw new JavaDocParsingException(e); + } + + try { + + SkeletonClass skeletonClass = extractSkeletonClass(jsoupDocument); + + JavaDocClass javaDocClass = extractJavaDocClass(jsoupDocument); + if(set){ + System.out.println(javaDocClass.toString()); + } + + Map classConstructors = extractConstructors(jsoupDocument); + + Map classAttributes = extractAttributes(jsoupDocument); + + Map skeletonMethods = extractMethods(jsoupDocument); + + classes.add(new ClassRepresentation(classConstructors, + classAttributes, skeletonMethods, javaDocClass, + skeletonClass)); + + } catch (Exception e) { + + throw new JavaDocParsingException( + "Error during checking javadoc in file: " + + javaDocClassFile + "\n" + e); + } + } + + Object [] reps = (Object[]) classes.toArray(); + System.out.println(classes.size()); + for(int i = 0; i < reps.length; i++){ + + ClassRepresentation a = (ClassRepresentation) reps[i]; + String java_code = ""; + String base_raw_url = "https://raw.githubusercontent.com/pauljv92/testedison/master/edison/src/test/java/edu/illinois/cs/cogcomp/edison/features/lrec/"; + + JavaDocClass jdoc = a.getJavaDocClass(); + + try { + + URL url = new URL(base_raw_url+"Test"+jdoc.getName()+".java"); + Scanner s = new Scanner(url.openStream()); + while(s.hasNext()) { + // operate + java_code += s.nextLine()+"\n"; + } + // read from your scanner + } + catch(IOException ex) { + // there was some connection problem, or the file did not exist on the server, + // or your URL was not in the right format. + // think about what to do now, and put it here. + ex.printStackTrace(); // for now, simply output it. + } + + //.substring(java_code.indexOf("test(")) + // + //".replaceAll(";", ";\n")"; + listforjson.add(new FexClass(jdoc.getName(),jdoc.getCommentText()+"\nKeywords: "+jdoc.getKeywords().toString(),java_code.substring(java_code.indexOf("test(")),this.base_url+jdoc.getName()+".html",jdoc.getKeywords())); + } + + ClassLoader classLoader = edu.illinois.cs.cogcomp.edison.features.Feature.class.getClassLoader(); + + //InputStream in = edu.illinois.cs.cogcomp.edison.features.Feature.class.getResourceAsStream(package_structure); + //System.out.println(in.toString()); + //BufferedReader rdr = new BufferedReader(new InputStreamReader(in)); + try (Writer writer = new FileWriter(this.path_to_write)) { + new Gson().toJson(listforjson, writer); + } + catch(Exception e){ + e.printStackTrace(); + } + + return classes; + } + + /** + * Extract JavaDoc class from given document representation in jsoup format. + * + * @param jsoupDocument + * Document containing loaded JavaDoc. + * @return + * @return New instance of JavaDocClass. + */ + private JavaDocClass extractJavaDocClass (Document jsoupDocument) { + + Element elementsSiblingsHR = jsoupDocument.getElementsByTag("hr") + .first().parent(); + + Elements blockElements = elementsSiblingsHR.getElementsByClass("block"); + + String name = elementsSiblingsHR.getElementsByTag("span").first() + .text().trim(); + + String deprecated = ""; + String text = ""; + + for (Element el : blockElements) { + + if (el.text().contains("Deprecated")) { + + deprecated = el.text().trim(); + } else { + + text = el.text().trim(); + } + } + + Elements dlElements = elementsSiblingsHR.getElementsByTag("dl"); + + String since = ""; + String version = ""; + String author = ""; + String see = ""; + String keywords = ""; + + if (dlElements.size() != 0) { + + Elements s = dlElements.last().getElementsByClass("simpleTagLabel"); + for (Element el : s) { + System.out.println(el.text()); + if (el.text().contains("Since")) { + + since = el.parent().nextElementSibling().text().trim(); + } else if (el.text().contains("Version")) { + + version = el.parent().nextElementSibling().text().trim(); + } else if (el.text().contains("Author")) { + + author = el.parent().nextElementSibling().text().trim(); + } else if (el.text().contains("See")) { + + see = el.parent().nextElementSibling().text().trim(); + } else if (el.text().contains("Keywords")){ + keywords = el.parent().nextElementSibling().text().trim(); + } + } + }else{ + return null; + } + + return javaDocFactory.makeJavaDocClass(name, text, deprecated, since, + see, author, version, keywords); + } + + /** + * Extract JavaDoc class from given document representation in jsoup format. + * + * @param jsoupDocument + * Document containing loaded JavaDoc. + * @return New instance of {@code SkeletonClass}. + */ + private SkeletonClass extractSkeletonClass(Document jsoupDocument) { + + Elements descriptionElements = jsoupDocument + .getElementsByClass("description"); + + Elements preElements = descriptionElements.get(0).getElementsByTag( + "pre"); + + Node acessModifiers = preElements.first().childNodes().get(0); + + Node abc = preElements.first().childNodes().get(1); + Node className = null; + String classNameString = null; + if(abc.childNodes().size() == 0){ + //System.out.println("abc is NULL"); + + } + else{ + //System.out.println("abc is good"); + className = abc.childNode(0); + + } + //System.out.println("SkeletonClass - getting here 5."); + + if(className == null){ + + classNameString = "null"; + + }else{ + + classNameString = className.toString(); + + } + + return skeletonFactory.makeSkeletonClass(classNameString, + acessModifiers.toString()); + } + + /** + * Extract constructor from given document representation in jsoup format. + * Constructor is a pair of {@code SkeletonConstrucotr} and + * {@code JavadocMethod}. + * + * @param jsoupDocument + * Document containing loaded JavaDoc. + * @return All constructors saved as pairs in hashmap. + */ + private Map extractConstructors( + Document jsoupDocument) { + + Elements counstructorDetailElements = jsoupDocument + .getElementsByAttributeValue("name", "constructor_detail"); + + Map constructors = new HashMap(); + + if (!counstructorDetailElements.isEmpty()) { + + Elements preElements = counstructorDetailElements.parents().first() + .getElementsByTag("pre"); + + for (Element preElement : preElements) { + + JavaDocMethod javaDocConstructor = extractJavaDocConstructor(preElement + .parent()); + SkeletonConstructor skeletonConstructor = skeletonFactory + .makeSkeletonConstructor(preElement.text()); + + constructors.put(skeletonConstructor, javaDocConstructor); + } + } + + return constructors; + } + + /** + * Extract constructor from given element. + * + * @param blistelement + * Element {@code blocklist} which contains information about + * constructor. + * @return New instance of {@code JavaDocMethod} (construcotr). + */ + private JavaDocMethod extractJavaDocConstructor(Element blistelement) { + + String[] absJavaDoc = extractAbstractJavaDoc(blistelement); + + List parameters = extractMethodParam(blistelement); + + return javaDocFactory.makeJavaDocMethod(absJavaDoc[0], absJavaDoc[1], + absJavaDoc[2], absJavaDoc[3], absJavaDoc[4], parameters, + new String()); + } + + /** + * Extract method parameters from given String. + * + * @param blistelement + * Element containing information about parameters. + * @return List of parameters, every parameter is a String containing name - + * type. + */ + private List extractMethodParam(Element blistelement) { + + Elements dlElements = blistelement.parent().getElementsByTag("dl"); + + List parameters = new ArrayList(); + + if (!dlElements.isEmpty()) { + + Elements strongElements = dlElements.first().getElementsByClass( + "strong"); + + for (Element el : strongElements) { + + if (el.text().contains("Parameters")) { + + Elements dlChildren = el.parent().parent().children(); + + boolean foundParameters = false; + for (Element dlChild : dlChildren) { + + if (dlChild.text().contains("Parameters:")) { + + foundParameters = true; + continue; + } + + if (!foundParameters) { + + continue; + } + + if (foundParameters && dlChild.tagName().equals("dt")) { + + break; + } + + parameters.add(dlChild.text().trim()); + } + } + } + } + + return parameters; + } + + /** + * Extract standard {@code AbstractJavaDoc} parameters. + * + * @param blistelement + * Block list element. + * @return An array of Strings. + */ + private String[] extractAbstractJavaDoc(Element blistelement) { + + String name = ""; + String comment = ""; + String deprecated = ""; + String since = ""; + String see = ""; + + name = blistelement.getElementsByTag("h4").text().trim(); + + Elements blockElements = blistelement.getElementsByClass("block"); + + for (Element blockElement : blockElements) { + + if (blockElement.text().contains("Deprecated.")) { + + deprecated = blockElement.text().trim(); + + } else { + + comment = blockElement.text().trim(); + } + } + + Elements dlElements = blistelement.getElementsByTag("dl"); + + if (!dlElements.isEmpty()) { + + Elements strongElements = dlElements.first().getElementsByClass( + "strong"); + + for (Element el : strongElements) { + + if (el.text().contains("Since")) { + + since = el.parent().nextElementSibling().text().trim(); + } else if (el.text().contains("See")) { + + see = el.parent().nextElementSibling().text().trim(); + } + } + } + + // this order has to be met up. + return new String[] { name, comment, deprecated, since, see }; + } + + /** + * Extracts all methods from given jsoup document. + * + * @param jsoupDocument + * Reference to an jsoup document. + * @return HashMap with pairs method - it's javadoc. + */ + private Map extractMethods( + Document jsoupDocument) { + + Map methods = new HashMap(); + + Elements methodDetailElements = jsoupDocument + .getElementsByAttributeValue("name", "method_detail"); + + if (!methodDetailElements.isEmpty()) { + + Elements h4Elements = methodDetailElements.parents().first() + .getElementsByTag("h4"); + + for (Element h4element : h4Elements) { + + JavaDocMethod javaDocMethod = extractJavaDocMethod(h4element + .parent()); + SkeletonMethod skeletonMethod = skeletonFactory + .makeSkeletonMethod(h4element.nextElementSibling() + .text()); + + methods.put(skeletonMethod, javaDocMethod); + } + + } + + return methods; + } + + /** + * Extract one {@code JavaDocMethod} from an element. + * + * @param blistelement + * Element from which is JavaDocMethod being parsed. + * @return New instance of {@code JavaDocMethod}. + */ + private JavaDocMethod extractJavaDocMethod(Element blistelement) { + + String[] absJavaDoc = extractAbstractJavaDoc(blistelement); + + List parameters = extractMethodParam(blistelement); + + Elements dlElements = blistelement.getElementsByTag("dl"); + + String returns = ""; + + if (!dlElements.isEmpty()) { + + Elements strongElements = dlElements.first().getElementsByClass( + "strong"); + + for (Element el : strongElements) { + + if (el.text().contains("Returns")) { + + returns = el.parent().nextElementSibling().text().trim(); + } + } + } + + return javaDocFactory.makeJavaDocMethod(absJavaDoc[0], absJavaDoc[1], + absJavaDoc[2], absJavaDoc[3], absJavaDoc[4], parameters, + returns); + } + + /** + * Extracts all attributes from given jsoup document. + * + * @param jsoupDocument + * Reference to jsoup documents. + * @return Hashmap with pairs attribute - it's javadoc. + */ + private Map extractAttributes( + Document jsoupDocument) { + + Map attributes = new HashMap(); + + Elements fieldDetailElements = jsoupDocument + .getElementsByAttributeValue("name", "field_detail"); + + if (!fieldDetailElements.isEmpty()) { + + Elements preElements = fieldDetailElements.parents().first() + .getElementsByTag("pre"); + + for (Element preElement : preElements) { + + SkeletonAttribute skeletonAttribute = skeletonFactory + .makeSkeletonAttribute(preElement.text()); + JavaDocAttribute javaDocAttribute = extractJavaDocAttribut(preElement + .parent()); + + attributes.put(skeletonAttribute, javaDocAttribute); + } + } + + return attributes; + } + + /** + * Extract one {@code JavaDocAttribute} from an element. + * + * @param blistelement + * Reference to element, from which is {@code JavaDocAttribute} + * parsed. + * @return New instance of {@code JavaDocAttribute}. + */ + private JavaDocAttribute extractJavaDocAttribut(Element blistelement) { + + String[] absJavaDoc = extractAbstractJavaDoc(blistelement); + + return javaDocFactory.makeJavaDocAttribute(absJavaDoc[0], + absJavaDoc[1], absJavaDoc[2], absJavaDoc[3], absJavaDoc[4]); + } + + static String readFile(String path, Charset encoding) + throws IOException + { + byte[] encoded = Files.readAllBytes(Paths.get(path)); + return new String(encoded, encoding); + } + +} diff --git a/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java b/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java index 76f7fa0..a3da04c 100755 --- a/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java +++ b/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java @@ -17,7 +17,7 @@ public class Main { */ public static void main(String[] args) { - JsoupJavaDocParser parser = new JsoupJavaDocParser("None", "/test/","../data.json"); + JsoupJavaDocParser parser = new JsoupJavaDocParser("None", "/test/","../data.json", "http://github.com/test"); try { diff --git a/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java~ b/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java~ new file mode 100755 index 0000000..76f7fa0 --- /dev/null +++ b/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java~ @@ -0,0 +1,72 @@ +package cz.zcu.validator.jdparser.api; + +import java.io.File; + +import cz.zcu.validator.jdparser.data.JavaDocParsingException; + +/** + * ONLY FOR LOCAL TESTING PURPOUSE. + * + * @author Stepan Cais, scais@kiv.zcu.cz, 14.12.2012 + */ +public class Main { + + /** + * @param args + * Arguments from command line - nothing here. + */ + public static void main(String[] args) { + + JsoupJavaDocParser parser = new JsoupJavaDocParser("None", "/test/","../data.json"); + + try { + + // parses javadoc of vogar project http://code.google.com/p/vogar/ + // parser.parseDataFromJavaDoc(new + // File("src/test/javadoc-example/vogar/doc")); + + // parses javadoc of pdfjbim project + // http://code.google.com/p/pdfrecompressor/ + // parser.parseDataFromJavaDoc(new + // File("src/test/javadoc-example/pdfjbim/doc")); + + // parses javadoc of gtkjfilechooser project + // http://code.google.com/p/gtkjfilechooser/ + // parser.parseDataFromJavaDoc(new + // File("src/test/javadoc-example/gtkjfilechooser/doc")); + + // parses javadoc of jflowmap project + // http://code.google.com/p/jflowmap/ + // parser.parseDataFromJavaDoc(new + // File("src/test/javadoc-example/jflowmap/doc")); + + // parses javadoc of openfire project + // http://code.google.com/p/openfire/ + // parser.parseDataFromJavaDoc(new + // File("src/test/javadoc-example/openfire/src/doc")); + + // parses javadoc of zmanim project + // http://code.google.com/p/kosherjava/ + // parser.parseDataFromJavaDoc(new + // File("src/test/javadoc-example/zmanim/doc")); + + // parses javadoc of jdparser project + parser.parseDataFromJavaDoc(new File("src/test/javadoc-example/jdparser/doc")); + +// Collection classRepre = parser +// .parseDataFromJavaDoc(new File( +// "src/test/javadoc-example/jdparser/doc.zip")); + + //parser.parseDataFromJavaDoc(new File("src/test/javadoc-example/oop-cv3/doc")); + + //Collection classRepre = parser.parseDataFromJavaDoc(new File("src/test/javadoc-example/oop-cv3-manualjavadoc/doc")); + //System.out.println(classRepre); + + } catch (JavaDocParsingException e) { + + System.out.println(e); + } + + } + +} diff --git a/javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java b/javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java index 9de8808..84cc7d0 100755 --- a/javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java +++ b/javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java @@ -20,10 +20,13 @@ public void standardJavaDocDocumentsTest() { String base_url = "http://cogcomp.cs.illinois.edu/software/doc/apidocs/edu/illinois/cs/cogcomp/edison/features/"; //This URL will be expected to the base URL of the public repository. We will extract the testcode to be presented on the web interface from here. String repository_path = "/lrec/"; //We will be using string to check for Classes whose path contains this substring. - + + //Raw Github URL base for Tests + String github_url_base_raw = "https://raw.githubusercontent.com/IllinoisCogComp/illinois-cogcomp-nlp/master/edison/src/test/java/edu/illinois/cs/cogcomp/edison/features/lrec/"; + String path_to_write_json = "../data.json"; - JsoupJavaDocParser parser = new JsoupJavaDocParser(base_url, repository_path, path_to_write_json); + JsoupJavaDocParser parser = new JsoupJavaDocParser(base_url, repository_path, github_url_base_raw, path_to_write_json); try { // parses javadoc directory of the Edison project diff --git a/javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java~ b/javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java~ new file mode 100755 index 0000000..9de8808 --- /dev/null +++ b/javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java~ @@ -0,0 +1,38 @@ +package cz.zcu.validator.jdparser; + +import static org.junit.Assert.fail; + +import java.io.File; + +import org.junit.Test; + +import cz.zcu.validator.jdparser.api.JsoupJavaDocParser; +import cz.zcu.validator.jdparser.data.JavaDocParsingException; + +/** + * Unit test for simple App. + */ +public class JDParserTest { + + @Test + public void standardJavaDocDocumentsTest() { + + String base_url = "http://cogcomp.cs.illinois.edu/software/doc/apidocs/edu/illinois/cs/cogcomp/edison/features/"; //This URL will be expected to the base URL of the public repository. We will extract the testcode to be presented on the web interface from here. + + String repository_path = "/lrec/"; //We will be using string to check for Classes whose path contains this substring. + + String path_to_write_json = "../data.json"; + + JsoupJavaDocParser parser = new JsoupJavaDocParser(base_url, repository_path, path_to_write_json); + + try { + // parses javadoc directory of the Edison project + parser.parseDataFromJavaDoc(new File("src/test/apidoc/software/doc/illinois-edison/apidocs")); + + } catch (JavaDocParsingException e) { + + fail(e.getMessage()); + } + + } +} From cbe397fb922d50939479c997af311083fd898bf8 Mon Sep 17 00:00:00 2001 From: Paul Jones Vijayakumar Date: Fri, 20 May 2016 08:36:44 -0500 Subject: [PATCH 7/8] made changes to doc and javadoc_scraper to specify the extraction URL for the raw github files --- .../jdparser/api/JsoupJavaDocParser.java~ | 652 ------------------ .../cz/zcu/validator/jdparser/api/Main.java~ | 72 -- .../zcu/validator/jdparser/JDParserTest.java~ | 38 - 3 files changed, 762 deletions(-) delete mode 100755 javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java~ delete mode 100755 javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java~ delete mode 100755 javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java~ diff --git a/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java~ b/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java~ deleted file mode 100755 index 743c0f4..0000000 --- a/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/JsoupJavaDocParser.java~ +++ /dev/null @@ -1,652 +0,0 @@ -package cz.zcu.validator.jdparser.api; - -import edu.illinois.cs.cogcomp.core.utilities.StringUtils; - -import java.io.BufferedReader; -import java.io.File; -import java.io.IOException; -import java.io.InputStream; -import java.io.InputStreamReader; -import java.net.URL; -import java.nio.charset.Charset; -import java.nio.file.Files; -import java.nio.file.Paths; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collection; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Scanner; -import java.io.Writer; -import java.io.FileWriter; - -import org.jsoup.Jsoup; -import org.jsoup.nodes.Document; -import org.jsoup.nodes.Element; -import org.jsoup.nodes.Node; -import org.jsoup.select.Elements; - -import com.google.gson.Gson; - -import cz.zcu.validator.jdparser.api.javadocs.JavaDocAttribute; -import cz.zcu.validator.jdparser.api.javadocs.JavaDocClass; -import cz.zcu.validator.jdparser.api.javadocs.JavaDocMethod; -import cz.zcu.validator.jdparser.api.javadocs.factory.JavaDocsFactory; -import cz.zcu.validator.jdparser.api.skeletons.SkeletonAttribute; -import cz.zcu.validator.jdparser.api.skeletons.SkeletonClass; -import cz.zcu.validator.jdparser.api.skeletons.SkeletonConstructor; -import cz.zcu.validator.jdparser.api.skeletons.SkeletonMethod; -import cz.zcu.validator.jdparser.api.skeletons.factory.SkeletonFactory; -import cz.zcu.validator.jdparser.data.JavaDocFilesFinder; -import cz.zcu.validator.jdparser.data.JavaDocParsing; -import cz.zcu.validator.jdparser.data.JavaDocParsingException; - -/** - * Parses data from JavaDoc folder into {@code ClassRepresentation} list. An - * implementation of JavaDocParsing. - * - * @author Paul Vijayakumar - * - * adapted from code by Stepan Cais, scais@kiv.zcu.cz, 3.12.2012 - */ -public class JsoupJavaDocParser implements JavaDocParsing { - - private class FexClass{ - - public FexClass(String name, String description, String testcode, String javadoc_url, String annotations){ - this.name = name; - this.description = description; - this.testcode = testcode; - this.javadoc_url = javadoc_url; - this.annotations = annotations; - } - - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - - public String getDescription() { - return description; - } - - public void setDescription(String description) { - this.description = description; - } - - public String getTestcode() { - return testcode; - } - - public void setTestcode(String testcode) { - this.testcode = testcode; - } - - public String getJavadoc_url() { - return javadoc_url; - } - - public void setJavadoc_url(String javadoc_url) { - this.javadoc_url = javadoc_url; - } - - String name, description, testcode = null, javadoc_url = "", annotations = ""; - - - } - - /** - * Factory for creating skeleton elements. - */ - private SkeletonFactory skeletonFactory = null; - - /** - * Factory for creating javaDoc elements. - */ - private JavaDocsFactory javaDocFactory = null; - - private String base_url, repo_path_string, path_to_write; - - //Old approach to extracting classes that we require. - //String[] fexes = {"LabelOneAfter", "LabelOneBefore", "LabelTwoAfter", "POSWindow"}; - - public JsoupJavaDocParser(String base_url, String repo_path_string, String path_to_write) { - - this.skeletonFactory = new SkeletonFactory(); - this.javaDocFactory = new JavaDocsFactory(); - this.base_url = base_url; - this.repo_path_string = repo_path_string; - this.path_to_write = path_to_write; - } - - @Override - public Collection parseDataFromJavaDoc( - File javaDocFolder) throws JavaDocParsingException { - - - if (!javaDocFolder.isDirectory()) { - - throw new JavaDocParsingException( - "Given JavaDoc folder is not a folder: " - + javaDocFolder.getName() - + " Cannot continue with parsing."); - } - - Collection javaDocClassFiles = JavaDocFilesFinder - .getJavaDocClassFiles(javaDocFolder); - - Collection classes = new ArrayList(); - - ArrayList listforjson = new ArrayList(); - - for (File javaDocClassFile : javaDocClassFiles) { - - boolean set = false; - - //Extracting on the classes in the LREC package - if(javaDocClassFile.getAbsolutePath().contains("/lrec/")){ - System.out.println("We have found LREC feature extractor."); - set = true; - } - - if(!set){ - continue; - } - - Document jsoupDocument = null; - - try { - // TODO: Maybe usage of - // http://code.google.com/p/juniversalchardet/ - jsoupDocument = Jsoup.parse(javaDocClassFile, "UTF-8"); - - } catch (IOException e) { - - throw new JavaDocParsingException(e); - } - - try { - - SkeletonClass skeletonClass = extractSkeletonClass(jsoupDocument); - - JavaDocClass javaDocClass = extractJavaDocClass(jsoupDocument); - if(set){ - System.out.println(javaDocClass.toString()); - } - - Map classConstructors = extractConstructors(jsoupDocument); - - Map classAttributes = extractAttributes(jsoupDocument); - - Map skeletonMethods = extractMethods(jsoupDocument); - - classes.add(new ClassRepresentation(classConstructors, - classAttributes, skeletonMethods, javaDocClass, - skeletonClass)); - - } catch (Exception e) { - - throw new JavaDocParsingException( - "Error during checking javadoc in file: " - + javaDocClassFile + "\n" + e); - } - } - - Object [] reps = (Object[]) classes.toArray(); - System.out.println(classes.size()); - for(int i = 0; i < reps.length; i++){ - - ClassRepresentation a = (ClassRepresentation) reps[i]; - String java_code = ""; - String base_raw_url = "https://raw.githubusercontent.com/pauljv92/testedison/master/edison/src/test/java/edu/illinois/cs/cogcomp/edison/features/lrec/"; - - JavaDocClass jdoc = a.getJavaDocClass(); - - try { - - URL url = new URL(base_raw_url+"Test"+jdoc.getName()+".java"); - Scanner s = new Scanner(url.openStream()); - while(s.hasNext()) { - // operate - java_code += s.nextLine()+"\n"; - } - // read from your scanner - } - catch(IOException ex) { - // there was some connection problem, or the file did not exist on the server, - // or your URL was not in the right format. - // think about what to do now, and put it here. - ex.printStackTrace(); // for now, simply output it. - } - - //.substring(java_code.indexOf("test(")) - // - //".replaceAll(";", ";\n")"; - listforjson.add(new FexClass(jdoc.getName(),jdoc.getCommentText()+"\nKeywords: "+jdoc.getKeywords().toString(),java_code.substring(java_code.indexOf("test(")),this.base_url+jdoc.getName()+".html",jdoc.getKeywords())); - } - - ClassLoader classLoader = edu.illinois.cs.cogcomp.edison.features.Feature.class.getClassLoader(); - - //InputStream in = edu.illinois.cs.cogcomp.edison.features.Feature.class.getResourceAsStream(package_structure); - //System.out.println(in.toString()); - //BufferedReader rdr = new BufferedReader(new InputStreamReader(in)); - try (Writer writer = new FileWriter(this.path_to_write)) { - new Gson().toJson(listforjson, writer); - } - catch(Exception e){ - e.printStackTrace(); - } - - return classes; - } - - /** - * Extract JavaDoc class from given document representation in jsoup format. - * - * @param jsoupDocument - * Document containing loaded JavaDoc. - * @return - * @return New instance of JavaDocClass. - */ - private JavaDocClass extractJavaDocClass (Document jsoupDocument) { - - Element elementsSiblingsHR = jsoupDocument.getElementsByTag("hr") - .first().parent(); - - Elements blockElements = elementsSiblingsHR.getElementsByClass("block"); - - String name = elementsSiblingsHR.getElementsByTag("span").first() - .text().trim(); - - String deprecated = ""; - String text = ""; - - for (Element el : blockElements) { - - if (el.text().contains("Deprecated")) { - - deprecated = el.text().trim(); - } else { - - text = el.text().trim(); - } - } - - Elements dlElements = elementsSiblingsHR.getElementsByTag("dl"); - - String since = ""; - String version = ""; - String author = ""; - String see = ""; - String keywords = ""; - - if (dlElements.size() != 0) { - - Elements s = dlElements.last().getElementsByClass("simpleTagLabel"); - for (Element el : s) { - System.out.println(el.text()); - if (el.text().contains("Since")) { - - since = el.parent().nextElementSibling().text().trim(); - } else if (el.text().contains("Version")) { - - version = el.parent().nextElementSibling().text().trim(); - } else if (el.text().contains("Author")) { - - author = el.parent().nextElementSibling().text().trim(); - } else if (el.text().contains("See")) { - - see = el.parent().nextElementSibling().text().trim(); - } else if (el.text().contains("Keywords")){ - keywords = el.parent().nextElementSibling().text().trim(); - } - } - }else{ - return null; - } - - return javaDocFactory.makeJavaDocClass(name, text, deprecated, since, - see, author, version, keywords); - } - - /** - * Extract JavaDoc class from given document representation in jsoup format. - * - * @param jsoupDocument - * Document containing loaded JavaDoc. - * @return New instance of {@code SkeletonClass}. - */ - private SkeletonClass extractSkeletonClass(Document jsoupDocument) { - - Elements descriptionElements = jsoupDocument - .getElementsByClass("description"); - - Elements preElements = descriptionElements.get(0).getElementsByTag( - "pre"); - - Node acessModifiers = preElements.first().childNodes().get(0); - - Node abc = preElements.first().childNodes().get(1); - Node className = null; - String classNameString = null; - if(abc.childNodes().size() == 0){ - //System.out.println("abc is NULL"); - - } - else{ - //System.out.println("abc is good"); - className = abc.childNode(0); - - } - //System.out.println("SkeletonClass - getting here 5."); - - if(className == null){ - - classNameString = "null"; - - }else{ - - classNameString = className.toString(); - - } - - return skeletonFactory.makeSkeletonClass(classNameString, - acessModifiers.toString()); - } - - /** - * Extract constructor from given document representation in jsoup format. - * Constructor is a pair of {@code SkeletonConstrucotr} and - * {@code JavadocMethod}. - * - * @param jsoupDocument - * Document containing loaded JavaDoc. - * @return All constructors saved as pairs in hashmap. - */ - private Map extractConstructors( - Document jsoupDocument) { - - Elements counstructorDetailElements = jsoupDocument - .getElementsByAttributeValue("name", "constructor_detail"); - - Map constructors = new HashMap(); - - if (!counstructorDetailElements.isEmpty()) { - - Elements preElements = counstructorDetailElements.parents().first() - .getElementsByTag("pre"); - - for (Element preElement : preElements) { - - JavaDocMethod javaDocConstructor = extractJavaDocConstructor(preElement - .parent()); - SkeletonConstructor skeletonConstructor = skeletonFactory - .makeSkeletonConstructor(preElement.text()); - - constructors.put(skeletonConstructor, javaDocConstructor); - } - } - - return constructors; - } - - /** - * Extract constructor from given element. - * - * @param blistelement - * Element {@code blocklist} which contains information about - * constructor. - * @return New instance of {@code JavaDocMethod} (construcotr). - */ - private JavaDocMethod extractJavaDocConstructor(Element blistelement) { - - String[] absJavaDoc = extractAbstractJavaDoc(blistelement); - - List parameters = extractMethodParam(blistelement); - - return javaDocFactory.makeJavaDocMethod(absJavaDoc[0], absJavaDoc[1], - absJavaDoc[2], absJavaDoc[3], absJavaDoc[4], parameters, - new String()); - } - - /** - * Extract method parameters from given String. - * - * @param blistelement - * Element containing information about parameters. - * @return List of parameters, every parameter is a String containing name - - * type. - */ - private List extractMethodParam(Element blistelement) { - - Elements dlElements = blistelement.parent().getElementsByTag("dl"); - - List parameters = new ArrayList(); - - if (!dlElements.isEmpty()) { - - Elements strongElements = dlElements.first().getElementsByClass( - "strong"); - - for (Element el : strongElements) { - - if (el.text().contains("Parameters")) { - - Elements dlChildren = el.parent().parent().children(); - - boolean foundParameters = false; - for (Element dlChild : dlChildren) { - - if (dlChild.text().contains("Parameters:")) { - - foundParameters = true; - continue; - } - - if (!foundParameters) { - - continue; - } - - if (foundParameters && dlChild.tagName().equals("dt")) { - - break; - } - - parameters.add(dlChild.text().trim()); - } - } - } - } - - return parameters; - } - - /** - * Extract standard {@code AbstractJavaDoc} parameters. - * - * @param blistelement - * Block list element. - * @return An array of Strings. - */ - private String[] extractAbstractJavaDoc(Element blistelement) { - - String name = ""; - String comment = ""; - String deprecated = ""; - String since = ""; - String see = ""; - - name = blistelement.getElementsByTag("h4").text().trim(); - - Elements blockElements = blistelement.getElementsByClass("block"); - - for (Element blockElement : blockElements) { - - if (blockElement.text().contains("Deprecated.")) { - - deprecated = blockElement.text().trim(); - - } else { - - comment = blockElement.text().trim(); - } - } - - Elements dlElements = blistelement.getElementsByTag("dl"); - - if (!dlElements.isEmpty()) { - - Elements strongElements = dlElements.first().getElementsByClass( - "strong"); - - for (Element el : strongElements) { - - if (el.text().contains("Since")) { - - since = el.parent().nextElementSibling().text().trim(); - } else if (el.text().contains("See")) { - - see = el.parent().nextElementSibling().text().trim(); - } - } - } - - // this order has to be met up. - return new String[] { name, comment, deprecated, since, see }; - } - - /** - * Extracts all methods from given jsoup document. - * - * @param jsoupDocument - * Reference to an jsoup document. - * @return HashMap with pairs method - it's javadoc. - */ - private Map extractMethods( - Document jsoupDocument) { - - Map methods = new HashMap(); - - Elements methodDetailElements = jsoupDocument - .getElementsByAttributeValue("name", "method_detail"); - - if (!methodDetailElements.isEmpty()) { - - Elements h4Elements = methodDetailElements.parents().first() - .getElementsByTag("h4"); - - for (Element h4element : h4Elements) { - - JavaDocMethod javaDocMethod = extractJavaDocMethod(h4element - .parent()); - SkeletonMethod skeletonMethod = skeletonFactory - .makeSkeletonMethod(h4element.nextElementSibling() - .text()); - - methods.put(skeletonMethod, javaDocMethod); - } - - } - - return methods; - } - - /** - * Extract one {@code JavaDocMethod} from an element. - * - * @param blistelement - * Element from which is JavaDocMethod being parsed. - * @return New instance of {@code JavaDocMethod}. - */ - private JavaDocMethod extractJavaDocMethod(Element blistelement) { - - String[] absJavaDoc = extractAbstractJavaDoc(blistelement); - - List parameters = extractMethodParam(blistelement); - - Elements dlElements = blistelement.getElementsByTag("dl"); - - String returns = ""; - - if (!dlElements.isEmpty()) { - - Elements strongElements = dlElements.first().getElementsByClass( - "strong"); - - for (Element el : strongElements) { - - if (el.text().contains("Returns")) { - - returns = el.parent().nextElementSibling().text().trim(); - } - } - } - - return javaDocFactory.makeJavaDocMethod(absJavaDoc[0], absJavaDoc[1], - absJavaDoc[2], absJavaDoc[3], absJavaDoc[4], parameters, - returns); - } - - /** - * Extracts all attributes from given jsoup document. - * - * @param jsoupDocument - * Reference to jsoup documents. - * @return Hashmap with pairs attribute - it's javadoc. - */ - private Map extractAttributes( - Document jsoupDocument) { - - Map attributes = new HashMap(); - - Elements fieldDetailElements = jsoupDocument - .getElementsByAttributeValue("name", "field_detail"); - - if (!fieldDetailElements.isEmpty()) { - - Elements preElements = fieldDetailElements.parents().first() - .getElementsByTag("pre"); - - for (Element preElement : preElements) { - - SkeletonAttribute skeletonAttribute = skeletonFactory - .makeSkeletonAttribute(preElement.text()); - JavaDocAttribute javaDocAttribute = extractJavaDocAttribut(preElement - .parent()); - - attributes.put(skeletonAttribute, javaDocAttribute); - } - } - - return attributes; - } - - /** - * Extract one {@code JavaDocAttribute} from an element. - * - * @param blistelement - * Reference to element, from which is {@code JavaDocAttribute} - * parsed. - * @return New instance of {@code JavaDocAttribute}. - */ - private JavaDocAttribute extractJavaDocAttribut(Element blistelement) { - - String[] absJavaDoc = extractAbstractJavaDoc(blistelement); - - return javaDocFactory.makeJavaDocAttribute(absJavaDoc[0], - absJavaDoc[1], absJavaDoc[2], absJavaDoc[3], absJavaDoc[4]); - } - - static String readFile(String path, Charset encoding) - throws IOException - { - byte[] encoded = Files.readAllBytes(Paths.get(path)); - return new String(encoded, encoding); - } - -} diff --git a/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java~ b/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java~ deleted file mode 100755 index 76f7fa0..0000000 --- a/javadoc_scraper/src/main/java/cz/zcu/validator/jdparser/api/Main.java~ +++ /dev/null @@ -1,72 +0,0 @@ -package cz.zcu.validator.jdparser.api; - -import java.io.File; - -import cz.zcu.validator.jdparser.data.JavaDocParsingException; - -/** - * ONLY FOR LOCAL TESTING PURPOUSE. - * - * @author Stepan Cais, scais@kiv.zcu.cz, 14.12.2012 - */ -public class Main { - - /** - * @param args - * Arguments from command line - nothing here. - */ - public static void main(String[] args) { - - JsoupJavaDocParser parser = new JsoupJavaDocParser("None", "/test/","../data.json"); - - try { - - // parses javadoc of vogar project http://code.google.com/p/vogar/ - // parser.parseDataFromJavaDoc(new - // File("src/test/javadoc-example/vogar/doc")); - - // parses javadoc of pdfjbim project - // http://code.google.com/p/pdfrecompressor/ - // parser.parseDataFromJavaDoc(new - // File("src/test/javadoc-example/pdfjbim/doc")); - - // parses javadoc of gtkjfilechooser project - // http://code.google.com/p/gtkjfilechooser/ - // parser.parseDataFromJavaDoc(new - // File("src/test/javadoc-example/gtkjfilechooser/doc")); - - // parses javadoc of jflowmap project - // http://code.google.com/p/jflowmap/ - // parser.parseDataFromJavaDoc(new - // File("src/test/javadoc-example/jflowmap/doc")); - - // parses javadoc of openfire project - // http://code.google.com/p/openfire/ - // parser.parseDataFromJavaDoc(new - // File("src/test/javadoc-example/openfire/src/doc")); - - // parses javadoc of zmanim project - // http://code.google.com/p/kosherjava/ - // parser.parseDataFromJavaDoc(new - // File("src/test/javadoc-example/zmanim/doc")); - - // parses javadoc of jdparser project - parser.parseDataFromJavaDoc(new File("src/test/javadoc-example/jdparser/doc")); - -// Collection classRepre = parser -// .parseDataFromJavaDoc(new File( -// "src/test/javadoc-example/jdparser/doc.zip")); - - //parser.parseDataFromJavaDoc(new File("src/test/javadoc-example/oop-cv3/doc")); - - //Collection classRepre = parser.parseDataFromJavaDoc(new File("src/test/javadoc-example/oop-cv3-manualjavadoc/doc")); - //System.out.println(classRepre); - - } catch (JavaDocParsingException e) { - - System.out.println(e); - } - - } - -} diff --git a/javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java~ b/javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java~ deleted file mode 100755 index 9de8808..0000000 --- a/javadoc_scraper/src/test/java/cz/zcu/validator/jdparser/JDParserTest.java~ +++ /dev/null @@ -1,38 +0,0 @@ -package cz.zcu.validator.jdparser; - -import static org.junit.Assert.fail; - -import java.io.File; - -import org.junit.Test; - -import cz.zcu.validator.jdparser.api.JsoupJavaDocParser; -import cz.zcu.validator.jdparser.data.JavaDocParsingException; - -/** - * Unit test for simple App. - */ -public class JDParserTest { - - @Test - public void standardJavaDocDocumentsTest() { - - String base_url = "http://cogcomp.cs.illinois.edu/software/doc/apidocs/edu/illinois/cs/cogcomp/edison/features/"; //This URL will be expected to the base URL of the public repository. We will extract the testcode to be presented on the web interface from here. - - String repository_path = "/lrec/"; //We will be using string to check for Classes whose path contains this substring. - - String path_to_write_json = "../data.json"; - - JsoupJavaDocParser parser = new JsoupJavaDocParser(base_url, repository_path, path_to_write_json); - - try { - // parses javadoc directory of the Edison project - parser.parseDataFromJavaDoc(new File("src/test/apidoc/software/doc/illinois-edison/apidocs")); - - } catch (JavaDocParsingException e) { - - fail(e.getMessage()); - } - - } -} From 653e3d9c42765ed886546e95f0f70d0653062e95 Mon Sep 17 00:00:00 2001 From: Paul Jones Vijayakumar Date: Fri, 20 May 2016 08:38:48 -0500 Subject: [PATCH 8/8] made fix to README.md --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 0559ed2..114786c 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,7 @@ Steps to run the application: - Ensure that the base_url variable in the 'src/test/java/cz/zcu/validator/jdparser/JDParserTest.java' file contains the base url with the root package path of the LREC feature classes. - The current default is: "http://cogcomp.cs.illinois.edu/software/doc/apidocs/edu/illinois/cs/cogcomp/edison/features/" - Ensure that the github_url_base_raw variable in the 'src/test/java/cz/zcu/validator/jdparser/JDParserTest.java' file contains the base raw github url from which we would append a "Test[FeatureExtractorName]" to download the Test file to be presented. - - The current default is: "https://raw.githubusercontent.com/IllinoisCogComp/illinois-cogcomp-nlp/maste\ -r/edison/src/test/java/edu/illinois/cs/cogcomp/edison/features/lrec/" + - The current default is: "https://raw.githubusercontent.com/IllinoisCogComp/illinois-cogcomp-nlp/master/edison/src/test/java/edu/illinois/cs/cogcomp/edison/features/lrec/" - Run 'mvn test' from the 'javadoc_scraper' directory, and this will generate the 'data.json' file in the base directory of our web application 3) Start the Web Application