#backend-architecture

Design a web crawler

At the start we seed a few webpage which recursively add more pages from the links found on the seed pages. Uses Cases Used for search engine indexing, web archiving, web minining (scraping), web moni