Skip to content

[WIP] HTML SAX parser implementation based on Neko HTML parser #191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
aecio wants to merge 7 commits into
base: master
Choose a base branch
from
Draft

[WIP] HTML SAX parser implementation based on Neko HTML parser #191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
04a7f14
Initial skeleton on HTML SAX parser based on nekohtml (issue #80)
aecio Nov 16, 2017
c0f5942
Added unit tests and implemented URL normalization and validation
aecio Nov 17, 2017
5cf4981
Merge branch 'master' into issue-80-htmlparser
aecio May 9, 2018
60b5582
Merge with master changes
aecio May 9, 2018
1ee7ce4
Merge branch 'master' into issue-80-htmlparser
aecio Jul 16, 2021
27e12d1
Implemented extraction of the text around links
aecio Jul 17, 2021
de8fe4c
Add HTML entity test and clean code
aecio Jul 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion ache/src/main/java/achecrawler/util/Urls.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ public static String removeFragmentsIfAny(String url) {
return url;
}

public static String resolveHttpLink(HttpUrl base, String link) {
public static HttpUrl resolveHttpLink(HttpUrl base, String link) {
HttpUrl resolvedUrl;
try {
if (base == null) {
Expand All @@ -133,6 +133,11 @@ public static String resolveHttpLink(HttpUrl base, String link) {
// The link is invalid or malformed
resolvedUrl = null;
}
return resolvedUrl;
}

public static String resolveHttpLinkAsString(HttpUrl base, String link) {
HttpUrl resolvedUrl = resolveHttpLink(base, link);
if (resolvedUrl == null) {
return null;
} else {
Expand Down
Loading