Endpoint and relation to track downsample datapoints. by Gooner91 · Pull Request #1 · Gooner91/labforward-skill-test

Gooner91 · 2021-08-22T14:22:49Z

Summary

Changes in this pull request, introduce a new endpoint to request persisted downsampled datapoints.

Description

Motivation
This was my first time working with downsampling or analysis of time series data. After doing some R&D on how to achieve it, I came across the rollups gem (you may also see me using this gem in a couple of commits to get an idea). This gem helped, me with my design of the solution.

Solution
For the solution, I followed the pattern I observed in the rollups gem and came up with a model to store the downsampled data. The reason to come up with a separate model was to persist the downsampled data (in background or via cron job) and avoid large computations while processing an http request. Also, having a model gives us an opportunity to analyze historical down sampled data.

Sample data set
For this solution I have created a seed file to generate sample datapoints , 10000 per channel which can be executed using:

rails db:seed VERSION=001_sample_data_points.rb
This data set is created with the assumption that the timeseries data has been in coming at a frequency of 1 minute.

Implementation of downsampling_datapoints model

the new model I came up with to store the downsampled data is associated with channel. The reason being, we may want to see downsampled data against each channel.

Implementation of downsampling logic

For the implementation I have written a service which will downsample all the datapoints to a frequency of 5 minutes
Once, the data is downsampled it is inserted in bulk inside the db (to avoid multiple insertion queires)
This service is added in the crontab via rake task, the idea is that downsampling will occur periodically on newly added datapoints

Implementation of api endpoint

a new endpoint is added which will return paginated downsampled data in json api format
have also added a couple of specs against the end point

Improvements

Due to the time constraint I was able to come up with this PR, but there are some improvements that I can think of, would have implemented if there was more time

I can see some room for improvement in the downsampling logic, I came up with. There is room for some code clean up a well.
Better and extensive test coverage, especially of the service that generates the downsampling
General code clean up in the overall approach
improving quality of sample data set

…tion based on the motivation taken from the gem.

github-actions · 2021-08-23T21:08:33Z

app/controllers/api/data_points_controller.rb

      respond_with data_points
    end
+
+    def downsampled


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Style/EmptyMethod: Put empty method definitions on a single line.

github-actions · 2021-08-23T21:08:33Z

app/controllers/api/data_points_controller.rb

    end
+
+    def downsampled
+


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Layout/EmptyLinesAroundMethodBody: Extra empty line detected at method body beginning.

github-actions · 2021-08-23T21:08:33Z

app/models/data_point.rb

-  after_commit :broadcast
+  attr_accessor :skip_broadcast
+
+  after_commit :broadcast, unless: Proc.new{ |data_point| data_point.skip_broadcast }


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Style/Proc: Use proc instead of Proc.new.

github-actions · 2021-08-23T21:08:33Z

app/models/data_point.rb

-  after_commit :broadcast
+  attr_accessor :skip_broadcast
+
+  after_commit :broadcast, unless: Proc.new{ |data_point| data_point.skip_broadcast }


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Layout/SpaceBeforeBlockBraces: Space missing to the left of {.

github-actions · 2021-08-23T21:08:33Z

app/models/downsampled_datapoint.rb

@@ -0,0 +1,5 @@
+class DownsampledDatapoint < ApplicationRecord


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Style/FrozenStringLiteralComment: Missing frozen string literal comment.

github-actions · 2021-08-23T21:08:34Z

app/services/data_points/rollup.rb

+    end
+
+  end
+end


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Layout/TrailingEmptyLines: Final newline missing.

github-actions · 2021-08-23T21:08:34Z

spec/factories/downsampled_datapoints.rb

@@ -0,0 +1,5 @@
+FactoryBot.define do


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Style/FrozenStringLiteralComment: Missing frozen string literal comment.

github-actions · 2021-08-23T21:08:34Z

spec/factories/downsampled_datapoints.rb

@@ -0,0 +1,5 @@
+FactoryBot.define do
+  factory :downsampled_datapoint do


⚠️ [rubocop] _{reported by reviewdog 🐶}
Lint/EmptyBlock: Empty block detected.

github-actions · 2021-08-23T21:08:34Z

spec/factories/downsampled_datapoints.rb

@@ -0,0 +1,5 @@
+FactoryBot.define do
+  factory :downsampled_datapoint do
+


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Layout/TrailingWhitespace: Trailing whitespace detected.

github-actions · 2021-08-23T21:08:34Z

spec/models/downsampled_datapoint_spec.rb

@@ -0,0 +1,5 @@
+require 'rails_helper'


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Style/FrozenStringLiteralComment: Missing frozen string literal comment.

…he interval.

github-actions · 2021-08-24T18:41:56Z

app/controllers/api/data_points_controller.rb

+    def down_sampled
+      down_sampled_data = DownsampledDatapoint.paginate(page: params[:page],
+                                                        per_page: 20)
+


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Layout/TrailingWhitespace: Trailing whitespace detected.

github-actions · 2021-08-24T18:41:56Z

app/controllers/api/data_points_controller.rb

+                                                        per_page: 20)
+
+      render json: DataPoints::DownSampledDataSerializer
+                     .new(down_sampled_data)


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Layout/MultilineMethodCallIndentation: Use 2 (not 15) spaces for indenting an expression spanning multiple lines.

github-actions · 2021-08-24T18:41:56Z

app/controllers/api/data_points_controller.rb

+
+      render json: DataPoints::DownSampledDataSerializer
+                     .new(down_sampled_data)
+                     .serializable_hash


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Layout/MultilineMethodCallIndentation: Use 2 (not 15) spaces for indenting an expression spanning multiple lines.

github-actions · 2021-08-24T18:41:56Z

app/controllers/api/data_points_controller.rb

+                     .new(down_sampled_data)
+                     .serializable_hash
+    end
+


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Layout/EmptyLinesAroundClassBody: Extra empty line detected at class body end.

github-actions · 2021-08-24T18:41:56Z

app/models/downsampled_datapoint.rb

+class DownsampledDatapoint < ApplicationRecord
+  belongs_to :channel
+
+  enum interval: %w[5m 10m 15m]


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Rails/EnumHash: Enum defined as an array found in interval enum declaration. Use hash syntax instead.

github-actions · 2021-08-24T18:41:57Z

spec/controllers/api/data_points_controller_spec.rb

+  let!(:down_sampled_data){ create_list(:downsampled_datapoint,
+                                       100,
+                                       channel: channel,
+                                        value: rand(1.0..999.9)) }


[rubocop] _{reported by reviewdog 🐶}
[Correctable] RSpec/EmptyLineAfterFinalLet: Add an empty line after the last let!.

github-actions · 2021-08-24T18:41:57Z

spec/controllers/api/data_points_controller_spec.rb

+  let!(:down_sampled_data){ create_list(:downsampled_datapoint,
+                                       100,
+                                       channel: channel,
+                                        value: rand(1.0..999.9)) }


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Layout/BlockEndNewline: Expression at 11, 66 should be on its own line.

github-actions · 2021-08-24T18:41:57Z

spec/controllers/api/data_points_controller_spec.rb

+    it 'returns appropriate attributes' do
+      get :down_sampled, format: :json, params: { page: 1 }
+      parsed_response = JSON.parse(response.body)['data']
+      expect(parsed_response.first["attributes"].keys)


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Style/StringLiterals: Prefer single-quoted strings when you don't need string interpolation or special symbols.

github-actions · 2021-08-24T18:41:58Z

spec/factories/downsampled_datapoints.rb

@@ -0,0 +1,7 @@
+FactoryBot.define do
+  factory :downsampled_datapoint do
+    interval { '5m' }


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Style/BlockDelimiters: Prefer do...end over {...} for procedural blocks.

github-actions · 2021-08-24T18:41:58Z

spec/factories/downsampled_datapoints.rb

+FactoryBot.define do
+  factory :downsampled_datapoint do
+    interval { '5m' }
+    time_interval { Time.now + 5.minutes }


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Rails/TimeZone: Do not use Time.now without zone. Use one of Time.zone.now, Time.current, Time.now.in_time_zone, Time.now.utc, Time.now.getlocal, Time.now.xmlschema, Time.now.iso8601, Time.now.jisx0301, Time.now.rfc3339, Time.now.httpdate, Time.now.to_i, Time.now.to_f instead.

github-actions · 2021-08-24T18:53:52Z

app/services/data_points/rollup.rb

+      @time_lower_bound = set_time_lower_bound
+    end
+
+    def call


[rubocop] _{reported by reviewdog 🐶}
Metrics/AbcSize: Assignment Branch Condition size for call is too high. [<7, 16, 5> 18.17/17]

github-actions · 2021-08-24T18:53:52Z

app/services/data_points/rollup.rb

+                                value: average.to_f,
+                                time_interval: time_upper_bound,
+                                created_at: Time.now.utc,
+                                updated_at: Time.now.utc, }


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Style/TrailingCommaInHashLiteral: Avoid comma after the last item of a hash.

github-actions · 2021-08-24T18:53:52Z

spec/controllers/api/data_points_controller_spec.rb

 describe Api::DataPointsController, type: :controller do
+  let(:device) { create(:device) }
+  let(:channel) { create(:channel, device: device) }
+  let!(:down_sampled_data) do


[rubocop] _{reported by reviewdog 🐶}
RSpec/LetSetup: Do not use let! to setup objects not referenced in tests.

github-actions · 2021-08-24T20:36:34Z

lib/tasks/data_points.rake

  end

+  task down_sample: :environment do
+    DataPoints::Rollup.new().call


[rubocop] _{reported by reviewdog 🐶}
[Correctable] Style/MethodCallWithoutArgsParentheses: Do not use parentheses for method calls with no arguments.

Gooner91 added 2 commits August 22, 2021 19:22

Added seed file to add dummy data_points for each device.

58cd75b

Added a gem to rollup datapoints as well as added a custom implementa…

7d54322

…tion based on the motivation taken from the gem.

github-actions bot reviewed Aug 23, 2021

View reviewed changes

Added serializer to respond to the request, added default value for t…

3280076

…he interval.

github-actions bot reviewed Aug 24, 2021

View reviewed changes

Added minor refactoring to resolve robocops comments

5b01c2d

github-actions bot reviewed Aug 24, 2021

View reviewed changes

Gooner91 added 2 commits August 25, 2021 00:10

Fixed robocop offenses

a358e13

Removed Rollup gem and its relation.

3b9d817

Gooner91 changed the title ~~Added seed file to add dummy data_points for each device.~~ Endpoint and relation to track downsample datapoints. Aug 24, 2021

Gooner91 self-assigned this Aug 24, 2021

Incorporated the downsampling with cron task.

04a893d

github-actions bot reviewed Aug 24, 2021

View reviewed changes

		@@ -0,0 +1,5 @@
		class DownsampledDatapoint < ApplicationRecord

+                  end
+                end
+              end

                
                    No newline at end of file

		@@ -0,0 +1,5 @@
		FactoryBot.define do
		factory :downsampled_datapoint do

Conversation

Gooner91 commented Aug 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Description

Improvements

Uh oh!

github-actions bot Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 24, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Gooner91 commented Aug 22, 2021 •

edited

Loading