m-dex
Library for parsing MangaDex data. Written on Crystal.
Why write it in Crystal?
- Efficient memory usage (1mb when idle, 7mb when active)
- Fast processing of data even when only operating in single-core mode
- Compiled language yet with the ease of Ruby
- I haven't made a proper project written in Crystal so why not xD
There is an existing API already, why create one?
Pardon with my shallow reasons but here's what I think why I created one.
- Some fields needs additional request for retrieving the full information.
- Some parts are are not 1:1 the same as the page
- Undocumented.
Web API Demo
You can try the REST API via https://mangadex-api.nedpals.xyz that uses this library for getting the data. This is an alternative to the official API Mangadex is offering and is much more cleaner and readable compared to the former. ~~As of now, expect to have some pages displaying error 500 for fetching reasons.~~
This REST API is running on a Heroku Hobby dyno with Redis for caching data. The dyno that costs $7/per month was made free for one year as part of Github Student perks. If you wish to sponsor a server, contact me through my e-mail which can be found at my Github profile. Not accepting donations for now.
Installation
-
Add the dependency to your
shard.yml
:dependencies: m-dex: github: nedpals/m-dex
-
Run
shards install
Usage
require "m-dex"
mangadex = Mdex::Client.new
# Get genre
mangadex.genre(8)
# Get manga info
mangadex.manga(12345)
# Get groups
mangadex.group(18)
# Get user profile
mangadex.user(22407)
See the endpoint methods found at the src/api.cr file for complete details.
Specification
An API specification has been created to test the library correctly and meet the goals of this project. For contributors, this is a very important guide if you are going to work with creating new endpoints that involvese extracting and displaying the data. Revisions and idea proposals are being discussed and clarified before getting approved so please file an issue first before changing the specification.
Development
Adding a new Endpoint
Endpoints is where the source data is fetched, scraped, and returns a new data very easily. It consists of methods that simplifies the work without repeating blocks of code for fetching and scraping. Right now it only supports the GET
method and eventually will have support for POST
method endpoints.
To create a new endpoint, just create a new class and point it to the Mdex::Endpoint
superclass to inherit the methods.
require "m-dex/endpoint"
class Dummy < Mdex::Endpoint
# Initialize first the class variables (The compiler won't allow instance variables.)
@@id = 0
@@username = ""
# Invoke the superclass method inside the 'self.get' method through
# the `super` method to fetch the data.
def self.get(@id : Int32, @username : String)
# You must pass a path as to what part of the site you want to fetch.
super("/user/#{@@id}/#{@@username}")
end
# From there, you can now access the raw data and parser.
# You need to specify when the data will return an error with the 'self.error_criteria' method
def self.error_criteria
# An error result will appear if the id is less than or equal to 0
# or if the username is empty.
@@id <= 0 || @@username = ""
end
# Next is to insert the variables that we initialized especially the @@id variable
# into the data through the 'self.insert_ids(data)' function.
def self.insert_ids(data)
data["id"] = @@id
data["username"] = @@username
end
# Now, this is where we start scrape and insert the data we have parsed.
# The 'self.parse_and_insert_data' method has access to the data object
# and the HTML parser.
def self.parse_and_insert_data(data, html)
# Scrape the raw data here.
# For a list of methods on how to access the DOM's contents,
# You may refer here: https://github.com/kostya/myhtml
end
end
From there, the data Hash
object will be automatically transformed into a JSON string. To access the endpoint you have created, you can just call the get
method of the Dummy
endpoint in this example.
puts Dummy.get(id: 123, username: "jimmy")
# { "id": 123, "username": "jimmy", "groups": [....] }
Disabling the parser
Some instances such as the Chapter
endpoint use the data from the Mangadex JSON API directly and doesn't use the HTML parser. To disable the parser, you can set the use_parser
variable to false
before you call the super
method inside the self.get
method.
class Dummy > Mdex::Endpoint
# ...
def self.get(...)
use_parser = false
# super(....)
end
# ...
end
Error handling
The endpoint superclass has an error detection already set-up that looks like this:
def self.check_data
html = @@html
error_banner = html.css(".alert.alert-danger.text-center").to_a
if (error_criteria || (error_banner.size == 1))
display_error(404, error_banner.map(&.inner_text).to_a.join("").to_s)
else
display_data(html)
end
end
You can change the error detection by overriding the self.check_data
method. The superclass inherits the data
and html
variables so you can access them inside of it.
Adding the endpoint to the mdex
instance
If you are a contributor who wants to add a new endpoint and want to appear in the list of endpoint methods of the instance, then you can add them to the src/api.cr
file. This is the only way in the meantime but in the future it will be replaced with macro functions.
# src/api.cr
module Mdex
module API
def dummy(id : Int32, username: String) : String
Mdex::Endpoints::Chapter.get(id, username)
end
end
end
# test.cr
require "mdex"
mdex = Mdex::Client.new
mdex.dummy(1234, "jimmy")
# { "id": 123, "username": "jimmy", "groups": [....] }
Roadmap
- [x] Manga Info page
- [x] Updates*1
- [x] Genre*1
- [x] Group Info*2
- [x] User Info*2
- [ ] Search*3&4
- [ ] Authentication
- [ ] MDList
- [ ] Forums
- [ ] Tests
- [ ] Spec implementation*6
(*) - See Issues section.
Issues
- ~~Pagination is yet to be implemented.~~
- Loading of group and user-curated chapters is still in progress.
- Search won't work unless a user logs in. Authentication will be implemented as soon as possible.
- Search is limited right to displaying fields.
- Messy code. Lots of
IndexError
's need to be detected and catched. - A new specification has been created. See SPEC.md for details.
Contributing
- Fork it (https://github.com/nedpals/m-dex/fork)
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request
Contributors
- nedpals - creator and maintainer