c6a69c6ada
Adds a second scraper for bookys-ebooks.com that harvests direct 1fichier download URLs into a .txt file. The existing EBoek scraper core is unchanged; only the GUI gained a Mode menu entry to switch between the two. Bookys specifics: - Visible (never headless) browser, since the site is behind Cloudflare. The challenge is auto-detected: the scraper polls for real listing items and continues on its own once it clears, no button needed. - Persistent Chrome profile in ~/.eboek_scraper/bookys_chrome_profile so the Cloudflare clearance cookie survives between runs. Removed the hardcoded user-agent override, which claimed Chrome/120 against a real Chrome/150 and invalidated that cookie on every launch. - Pop-up ads are avoided by never clicking host links: hrefs are read and navigated to directly, with window.open neutered via CDP as a backstop. - Host links are read with textContent rather than Selenium's .text, which returns empty for these elements because they sit in a collapsed container. - Only 1fichier hosts are followed; other hosts on a page are ignored. Build configs list the new modules as hidden imports in all four places. gui_main imports the Bookys window lazily, so PyInstaller's static analysis would otherwise miss it and the .exe would crash on switching modes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVJc2XuTvqBHSuqotetseS
EBoek.info Scraper
Een moderne PyQt5 GUI applicatie voor het scrapen van EBoek.info met dual scraping modes, real-time voortgangsmonitoring en veilige opslag van inloggegevens.
✨ Functies
- Twee scraping modi: All Comics en Latest Comics
- Gebruiksvriendelijke GUI met real-time voortgang
- Veilige credential opslag in JSON config
- Cross-platform ondersteuning (Windows/macOS)
- Background threading - GUI blijft responsief
- Graceful cancellation tijdens operaties
📋 Vereisten
- Python 3.8+
- Google Chrome browser
- EBoek.info account
🚀 Installatie
Optie 1: Standalone Executable (Aanbevolen)
Geen Python installatie nodig!
Windows:
# Bouw Windows .exe (run in Command Prompt)
scripts\build_exe.bat
# Of cross-platform builder:
python scripts\build_executable.py
macOS/Linux:
# Bouw executable (eenmalig)
python3 scripts/build_executable.py
# Of platform-specific:
./scripts/build_exe.sh
Result: dist/EBoek_Scraper.exe (Windows) of dist/EBoek_Scraper (Unix)
Optie 2: Python Installatie
Windows:
scripts/install_and_run.bat
macOS / Linux:
chmod +x scripts/install_and_run.sh
./scripts/install_and_run.sh
Handmatig:
pip install selenium urllib3 PyQt5
python3 gui_main.py
🎯 Gebruik
- Start de applicatie:
python3 gui_main.py - Voer credentials in: Klik "Change Credentials"
- Kies scraping mode: All Comics of Latest Comics
- Stel pagina bereik in: Start/eind pagina
- Start scraping: Klik "Start Scraping"
📊 Scraping Modi
Mode 0: All Comics
- URL patroon:
stripverhalen-alle/page/X/ - Structuur: Traditionele blog layout
- Selecteer:
h2.post-title a
Mode 1: Latest Comics
- URL patroon:
laatste?_page=X&ref=dw - Structuur: Grid layout met containers
- Selecteer:
.pt-cv-wrapper .pt-cv-ifield h5.pt-cv-title a
🗂️ Project Structuur
├── README.md # Project documentatie
├── requirements.txt # Dependencies
├── gui_main.py # GUI applicatie entry point
├── scripts/ # Build & install scripts
│ ├── build_executable.py # Executable builder (aanbevolen)
│ ├── build_exe.bat # Windows build script
│ ├── build_exe.sh # macOS/Linux build script
│ ├── eboek_scraper.spec # PyInstaller configuration
│ ├── install_and_run.bat # Windows installer
│ └── install_and_run.sh # macOS/Linux installer
├── docs/ # Documentatie
│ └── BUILD_GUIDE.md # Executable build guide
├── core/ # Scraping logic
│ ├── scraper.py # Dual-mode scraper
│ ├── scraper_thread.py # Threading wrapper
│ └── credentials.py # Config management
├── gui/ # GUI components
│ ├── main_window.py # Main interface
│ ├── login_dialog.py # Credential input
│ └── progress_dialog.py # Progress monitoring
├── tests/ # Test scripts
├── utils/ # Helper functions
└── dist/ # Built executables (na build)
└── EBoek_Scraper # Standalone executable
🔧 Troubleshooting
GUI start niet: Controleer PyQt5 installatie
Login problemen: Test credentials via GUI
Download issues: Controleer ~/Downloads folder
💡 Tips
- Distributie: Gebruik executable voor makkelijke delen (geen Python setup nodig)
- Testing: Begin met 1-2 pagina's om de functionaliteit te testen
- Performance: Gebruik headless mode voor optimale snelheid
- Monitoring: Volg de voortgang in de progress dialog met realtime statistieken
Veel succes met scrapen! 🚀