w33ble/strain-tools
1
0
Fork 0
Code Issues 2 Pull Requests Releases Wiki Activity

fix: scrape strains alphabetically

Browse Source 
ensures that we get everything
This commit is contained in:
joe fleming
2018-09-13 15:16:42 -07:00
parent a2bea32fb6
commit fa2db8d430
2 changed files with 2 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
packages/scraper/README.md
View File

@@ -8,5 +8,3 @@ Clone repo and run the command. Resulting data can be found in `db.json`.
yarn install yarn install
yarn start yarn start
``` ```
**NOTE**: You may need to run it multiple times (4 or 5 should do it), since some strains will get skipped the first few times. I don't know if it's leafly's endpoint or some weird race condition in the scraping code though.

4
packages/scraper/src/leafly.mjs
View File

@@ -8,14 +8,14 @@ const adapter = new FileAsync('db.json');
const xhr = axios.create({ const xhr = axios.create({
headers: { headers: {
Accept: 'application/json, text/plain, */*', Accept: 'application/json, text/plain, */*',
Referer: 'https://www.leafly.com/explore', Referer: 'https://www.leafly.com/explore/sort-alpha',
}, },
}); });
const pSeries = tasks => tasks.reduce((c, task) => c.then(task), Promise.resolve()); const pSeries = tasks => tasks.reduce((c, task) => c.then(task), Promise.resolve());
const getPage = async num => { const getPage = async num => {
const url = `https://www.leafly.com/explore/page-${num}`; const url = `https://www.leafly.com/explore/page-${num}/sort-alpha`;
const response = await xhr.get(url, { const response = await xhr.get(url, {
responseType: 'json', responseType: 'json',
}); });