Skip to content

Commit

Permalink
Make task description more clear on request from Ruslan Namazov
Browse files Browse the repository at this point in the history
  • Loading branch information
loong committed Mar 16, 2017
1 parent b8100c0 commit d6fcf6a
Show file tree
Hide file tree
Showing 2 changed files with 17 additions and 7 deletions.
15 changes: 11 additions & 4 deletions 0-limit-crawler/README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,18 @@
# Limit your crawler

Given is a crawler (modified from the Go tour), which would pull pages excessively. The task is to modify the `main.go` file, so that
Given is a crawler (modified from the Go tour) that requests pages
excessively. However, we don't want to burden the webserver too
much. Your task is to change the code to limit the crawler to at most
one page per second, while maintaining concurrency (in other words,
Crawl() must be called concurrently)

- the crawler cannot crawl more than 2 pages per second
- without losing it's concurrency
## Hint

# Test your solution
This exercise can be solved in 3 lines only. If you can't do
it, have a look at this:
https://github.com/golang/go/wiki/RateLimiting

## Test your solution

Use `go test` to verify if your solution is correct.

Expand Down
9 changes: 6 additions & 3 deletions 0-limit-crawler/main.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
//////////////////////////////////////////////////////////////////////
//
// Your task is to change the code to limit the crawler so that it is
// not possible fo the crawler to crawl more than one page per second.
// Your task is to change the code to limit the crawler to at most one
// page per second, while maintaining concurrency (in other words,
// Crawl() must be called concurrently)
//
// @hint: you can archive that by adding 3 lines
// @hint: you can archive this by adding 3 lines
//

package main
Expand Down Expand Up @@ -32,6 +33,8 @@ func Crawl(url string, depth int, wg *sync.WaitGroup) {

wg.Add(len(urls))
for _, u := range urls {
// Do not remove the `go` keyword, as Crawl() must be
// called concurrently
go Crawl(u, depth-1, wg)
}
return
Expand Down

0 comments on commit d6fcf6a

Please sign in to comment.