-
Notifications
You must be signed in to change notification settings - Fork 1
Reorganize data-fetching code / add tests #4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| run: | ||
| go run main.go | ||
|
|
||
| test: | ||
| go test ./... | ||
|
|
||
| fmt: | ||
| find . -type f -name '*.go' | xargs gofmt -w | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,78 +3,34 @@ package data | |
| import ( | ||
| "encoding/json" | ||
| "fmt" | ||
| "io/ioutil" | ||
| "log" | ||
| "net/http" | ||
| "sort" | ||
| "sync" | ||
| "time" | ||
| ) | ||
|
|
||
| const apiTemplate = "https://api.meetup.com/%s/events?status=upcoming" | ||
|
|
||
| var ( | ||
| meetupNames = []string{ | ||
| "Boulder-Gophers", | ||
| "Denver-Go-Language-User-Group", | ||
| "Denver-Go-Programming-Language-Meetup", | ||
| } | ||
| ) | ||
|
|
||
| // Store contains data for the site. | ||
| type Store struct { | ||
| pollingInterval time.Duration | ||
|
|
||
| mu sync.Mutex | ||
| meetupSchedule *MeetupSchedule | ||
| type Client interface { | ||
| Get(string) ([]byte, error) | ||
| } | ||
|
|
||
| // NewStore creates a new store initialized with a polling interval. | ||
| func NewStore(i time.Duration) *Store { | ||
| return &Store{ | ||
| pollingInterval: i, | ||
| } | ||
| } | ||
| type MeetupClient struct{} | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll skip this altogether based on your other comments. |
||
|
|
||
| // Poll runs forever, polling the meetup API for event data and updating the | ||
| // internal cache. | ||
| func (s *Store) Poll() { | ||
| for { | ||
| events := s.poll() | ||
| s.updateCache(events) | ||
| time.Sleep(s.pollingInterval) | ||
| func (c MeetupClient) Get(url string) (data []byte, err error) { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This type only has one method and the method does not use the receiver for anything. This causes me to think if this could just be a function. Passing a function vs an interface into It might be worth thinking about this: https://www.youtube.com/watch?v=o9pEzgHorH0&t=2m10s Also, the implementation here has nothing to do with the meetup API. It just makes a GET request and returns the body as a
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think I'm bringing too much Ruby over to Go, so creating an // data.go
type DataFetcher func(string) ([]byte, error)
func FetchData(url string) (data []byte, err error) {
resp, err := http.Get(url)
if resp != nil {
defer resp.Body.Close()
}
if err != nil {
return data, err
}
data, err = ioutil.ReadAll(resp.Body)
return data, err
}
func (s *Schedule) FetchEvents(fetcher DataFetcher) (err error) {
url := fmt.Sprintf("https://api.meetup.com/%s/events?status=upcoming", s.key)
data, err := fetcher(url)
if err != nil {
return err
}
err = json.Unmarshal(data, &s.Events)
if err != nil {
return err
}
sort.SliceStable(s.Events, func(i, j int) bool {
return s.Events[i].Time < s.Events[j].Time
})
return err
}Which simplifies the testing side as you mentioned: // data_test.go
func TestFetchEventsStoresEvents(t *testing.T) {
s := Schedule{key: "Boulder-Gophers"}
s.FetchEvents(func(url string) (data []byte, error) {
return []byte(`[{"id":"id","name":"Event","time":400}]`), nil
})
expected := []Event{Event{ID: "id", Name: "Event", Time: 400}}
if !reflect.DeepEqual(s.Events, expected) {
t.Fail()
}
}I like it. |
||
| resp, err := http.Get(url) | ||
|
|
||
| if err != nil { | ||
| return data, err | ||
| } | ||
| } | ||
|
|
||
| func (s *Store) updateCache(schedule *MeetupSchedule) { | ||
| s.mu.Lock() | ||
| defer s.mu.Unlock() | ||
| s.meetupSchedule = schedule | ||
| } | ||
| defer resp.Body.Close() | ||
|
|
||
| func (s *Store) poll() *MeetupSchedule { | ||
| schedule := NewMeetupSchedule() | ||
| for _, meetup := range meetupNames { | ||
| eds, err := events(meetup) | ||
| if err != nil { | ||
| log.Printf("error fetching events for %s: %s", meetup, err) | ||
| continue | ||
| } | ||
| sort.Slice(eds, func(i, j int) bool { | ||
| return eds[i].Time < eds[j].Time | ||
| }) | ||
| schedule.Add(meetup, eds) | ||
| } | ||
| return schedule | ||
| } | ||
| data, err = ioutil.ReadAll(resp.Body) | ||
|
|
||
| // AllEvents returns the current meetup events in CO. | ||
| func (s *Store) AllEvents() *MeetupSchedule { | ||
| s.mu.Lock() | ||
| defer s.mu.Unlock() | ||
| return s.meetupSchedule | ||
| return data, err | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Line 29 and 31 can be merged. return ioutil.ReadAll(resp.Body)vs data, err = ioutil.ReadAll(resp.Body)
return data, err
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. derp -- good call. |
||
| } | ||
|
|
||
| // Event contains information about a meetup event. | ||
| type Event struct { | ||
| ID string `json:"id"` | ||
| Name string `json:"name"` | ||
|
|
@@ -86,52 +42,86 @@ func (e Event) HumanTime() string { | |
| return time.Unix(e.Time/1000, 0).Format(time.RFC1123) | ||
| } | ||
|
|
||
| func NewMeetupSchedule() *MeetupSchedule { | ||
| return &MeetupSchedule{ | ||
| events: make(map[string][]Event), | ||
| } | ||
| } | ||
| type Schedule struct { | ||
| key string | ||
|
|
||
| type MeetupSchedule struct { | ||
| events map[string][]Event | ||
| Label string | ||
| Events []Event | ||
| } | ||
|
|
||
| func (m *MeetupSchedule) Add(name string, events []Event) { | ||
| m.events[name] = events | ||
| } | ||
| type Schedules []*Schedule | ||
|
|
||
| func (s *Schedule) FetchEvents(client Client) (err error) { | ||
| url := fmt.Sprintf("https://api.meetup.com/%s/events?status=upcoming", s.key) | ||
|
|
||
| data, err := client.Get(url) | ||
|
|
||
| func (m *MeetupSchedule) BoulderEvents() []Event { | ||
| return nextThree(m.events["Boulder-Gophers"]) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| err = json.Unmarshal(data, &s.Events) | ||
|
|
||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| sort.SliceStable(s.Events, func(i, j int) bool { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm curious as to why you chose to change to a stable sort. It is not needed here and has a performance cost. I ran some benchmarks to demonstrate this: https://gist.github.com/jasonkeene/3c4d20275ca5c131e63c5f34f0a03815#file-results-txt
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I looked up
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Stable sort sounds like a good idea then. From what I recall the meetup API does not return very precise timestamps. They are rounded significantly. This increases the chance of timestamps being the same. If that is the case preserving order makes sense. Good catch! |
||
| return s.Events[i].Time < s.Events[j].Time | ||
| }) | ||
|
|
||
| return err | ||
| } | ||
|
|
||
| func (m *MeetupSchedule) DenverEvents() []Event { | ||
| return nextThree(m.events["Denver-Go-Language-User-Group"]) | ||
| func (s *Schedule) Next(count int) []Event { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So I think a solution here would be to not modify This may be something best suited for reviewing during a pairing session. |
||
| if len(s.Events) > count { | ||
| return s.Events[0:count] | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This can be
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 😎 |
||
| } | ||
|
|
||
| return s.Events | ||
| } | ||
|
|
||
| func (m *MeetupSchedule) DTCEvents() []Event { | ||
| return nextThree(m.events["Denver-Go-Programming-Language-Meetup"]) | ||
| // Store contains data for the site. | ||
| type Store struct { | ||
| pollingInterval time.Duration | ||
| mu sync.Mutex | ||
|
|
||
| Schedules Schedules | ||
| } | ||
|
|
||
| func nextThree(events []Event) []Event { | ||
| if len(events) < 3 { | ||
| return events | ||
| // NewStore creates a new store initialized with a polling interval. | ||
| func NewStore(i time.Duration) *Store { | ||
| return &Store{ | ||
| pollingInterval: i, | ||
| Schedules: Schedules{ | ||
| &Schedule{key: "Boulder-Gophers", Label: "Boulder"}, | ||
| &Schedule{key: "Denver-Go-Language-User-Group", Label: "Denver"}, | ||
| &Schedule{key: "Denver-Go-Programming-Language-Meetup", Label: "Denver Tech Center"}, | ||
| }, | ||
| } | ||
|
|
||
| return events[0:3] | ||
| } | ||
|
|
||
| func events(name string) ([]Event, error) { | ||
| resp, err := http.Get(fmt.Sprintf(apiTemplate, name)) | ||
| if err != nil { | ||
| log.Fatal(err) | ||
| // Poll runs forever, polling the meetup API for event data and updating the | ||
| // internal cache. | ||
| func (s *Store) Poll() { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| for { | ||
| s.mu.Lock() | ||
| defer s.mu.Unlock() | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This has a few problems:
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The link you provided describes exactly why I implemented it this way -- I think this is another instance of me making changes w/o understanding the underlying design decisions.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's a very common mistake. |
||
| s.refresh() | ||
|
|
||
| time.Sleep(s.pollingInterval) | ||
| } | ||
| defer resp.Body.Close() | ||
| } | ||
|
|
||
| decoder := json.NewDecoder(resp.Body) | ||
| var data []Event | ||
| err = decoder.Decode(&data) | ||
| if err != nil { | ||
| return nil, err | ||
| func (s *Store) refresh() { | ||
| client := MeetupClient{} | ||
|
|
||
| for _, s := range s.Schedules { | ||
| err := s.FetchEvents(client) | ||
|
|
||
| if err != nil { | ||
| log.Printf("error fetching events for %s: %s", s.key, err) | ||
| continue | ||
| } | ||
| } | ||
| return data, nil | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| package data | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Using
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That makes sense -- I have done this when using Ginkgo, but I wasn't sure if it was a general practice.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I prefer it. Using |
||
|
|
||
| import ( | ||
| "errors" | ||
| "reflect" | ||
| "testing" | ||
| ) | ||
|
|
||
| type TestClient struct { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixture types and functions should be lowercase. They won't be exported if they are uppercase but keeping them lowercase reduces confusion.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Coupled with |
||
| *MeetupClient | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Embedding
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Must have been a vestige of a previous approach, this is moot now anyway. |
||
|
|
||
| data []byte | ||
| err error | ||
| } | ||
|
|
||
| func NewClient(data string, err error) TestClient { | ||
| return TestClient{ | ||
| data: []byte(data), | ||
| err: err, | ||
| } | ||
| } | ||
|
|
||
| func (c TestClient) Get(key string) ([]byte, error) { | ||
| if c.err != nil { | ||
| return []byte{}, c.err | ||
| } | ||
|
|
||
| return c.data, c.err | ||
| } | ||
|
|
||
| func TestScheduleHasNoEvents(t *testing.T) { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This test isn't useful, it covers no code in
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can delete. |
||
| s := Schedule{} | ||
| if len(s.Events) != 0 { | ||
| t.Fail() | ||
| } | ||
| } | ||
|
|
||
| func TestFetchEventsStoresEvents(t *testing.T) { | ||
| s := Schedule{key: "Boulder-Gophers"} | ||
|
|
||
| c := NewClient(`[{"id":"id","name":"Event","time":400}]`, nil) | ||
|
|
||
| s.FetchEvents(c) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You should assert that the error returned her is nil.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||
|
|
||
| expected := []Event{Event{ID: "id", Name: "Event", Time: 400}} | ||
|
|
||
| if !reflect.DeepEqual(s.Events, expected) { | ||
| t.Fail() | ||
| } | ||
| } | ||
|
|
||
| func TestFetchEventsReturnsErrorWhenRequestFails(t *testing.T) { | ||
| s := Schedule{key: "Boulder-Gophers"} | ||
| c := NewClient(``, errors.New("Failed to connect")) | ||
|
|
||
| err := s.FetchEvents(c) | ||
|
|
||
| if err == nil { | ||
| t.Fail() | ||
| } | ||
| } | ||
|
|
||
| func TestFetchEventsReturnsErrorWhenUnmarshalFails(t *testing.T) { | ||
| s := Schedule{key: "Boulder-Gophers"} | ||
| c := NewClient(`<>`, nil) | ||
|
|
||
| err := s.FetchEvents(c) | ||
|
|
||
| if err == nil { | ||
| t.Fail() | ||
| } | ||
| } | ||
|
|
||
| func TestFetchEventsReturnsEventsInOrderOfTime(t *testing.T) { | ||
| s := Schedule{key: "Boulder-Gophers"} | ||
| c := NewClient(` | ||
| [ | ||
| {"id":"two","name":"Two","time":2}, | ||
| {"id":"one","name":"One","time":1} | ||
| ] | ||
| `, nil) | ||
|
|
||
| s.FetchEvents(c) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You should assert that the error returned her is nil.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||
|
|
||
| expected := []Event{ | ||
| Event{ID: "one", Name: "One", Time: 1}, | ||
| Event{ID: "two", Name: "Two", Time: 2}, | ||
| } | ||
|
|
||
| if !reflect.DeepEqual(s.Events, expected) { | ||
| t.Fail() | ||
| } | ||
| } | ||
|
|
||
| func TestNextReturnsSubsetOfEvents(t *testing.T) { | ||
| s := Schedule{Events: []Event{ | ||
| Event{ID: "one", Name: "One", Time: 1}, | ||
| Event{ID: "two", Name: "Two", Time: 2}, | ||
| Event{ID: "three", Name: "Three", Time: 3}, | ||
| }} | ||
|
|
||
| expected := []Event{ | ||
| Event{ID: "one", Name: "One", Time: 1}, | ||
| Event{ID: "two", Name: "Two", Time: 2}, | ||
| } | ||
|
|
||
| if !reflect.DeepEqual(s.Next(2), expected) { | ||
| t.Fail() | ||
| } | ||
| } | ||
|
|
||
| func TestNextReturnsAllWhenLimitGreaterThanLen(t *testing.T) { | ||
| s := Schedule{Events: []Event{ | ||
| Event{ID: "one", Name: "One", Time: 1}, | ||
| }} | ||
|
|
||
| expected := []Event{ | ||
| Event{ID: "one", Name: "One", Time: 1}, | ||
| } | ||
|
|
||
| if !reflect.DeepEqual(s.Next(2), expected) { | ||
| t.Fail() | ||
| } | ||
| } | ||
|
|
||
| func TestHumanTimeReturnsTimeInProperFormat(t *testing.T) { | ||
| e := Event{Time: 1533121600000} | ||
|
|
||
| if e.HumanTime() != "Wed, 01 Aug 2018 05:06:40 MDT" { | ||
| t.Fail() | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,11 +8,11 @@ import ( | |
| "github.com/milehighgophers/website/ui" | ||
| ) | ||
|
|
||
| type Store interface { | ||
| AllEvents() *data.MeetupSchedule | ||
| } | ||
| // type Store interface { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I wouldn't commit code that is commented out. If it isn't needed just delete it. It is in version control so if it is needed in the future it can be brought back.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think I just missed this when I committed -- I can rebase this out. |
||
| // AllEvents() *data.MeetupSchedule | ||
| // } | ||
|
|
||
| func Start(addr string, s Store) error { | ||
| func Start(addr string, s *data.Store) error { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm wondering why
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I didn't see the advantage of an interface in this instance -- what's the best practice here?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Typically, I receive small interfaces. This decouples your code, allows for callers to provide alternative implementations which is good for testing. That said, passing concrete types is more performant. I only cross that bridge when it is necessary tho. |
||
| log.Printf("listening on %s", addr) | ||
|
|
||
| mux := http.NewServeMux() | ||
|
|
@@ -22,17 +22,18 @@ func Start(addr string, s Store) error { | |
| } | ||
|
|
||
| type IndexHandler struct { | ||
| store Store | ||
| store data.Store | ||
| } | ||
|
|
||
| func NewIndexHandler(s Store) *IndexHandler { | ||
| func NewIndexHandler(s *data.Store) *IndexHandler { | ||
| return &IndexHandler{ | ||
| store: s, | ||
| store: *s, | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Dereferencing here makes a copy of the
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not at all, will need to fix this. |
||
| } | ||
| } | ||
|
|
||
| func (h *IndexHandler) ServeHTTP(rw http.ResponseWriter, r *http.Request) { | ||
| html := ui.Render(h.store.AllEvents()) | ||
| html := ui.Render(h.store.Schedules) | ||
|
|
||
| _, err := rw.Write(html) | ||
| if err != nil { | ||
| log.Print("error occured with /:", err) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can use
goimports -w .to do the same thing here.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🆒 TIL