Metric Injection Data Items

[joeledwards]

Once James' Issue (#2) is complete, the required inputs should be known, and the values which need to be passed in to the inject function of asl.seedscan.database.MetricDatabase can be added to the method signature (arguments and return).

[mikehagerty]

I pulled your latest changes to catbox2 but I am unable to connect to the
database server:
[java] java.sql.SQLException: No suitable driver found for
jdbc:postgresql://136.177.121.210:5432/dqaOct 25, 2012 11:19:36 AM
asl.seedscan.database.MetricDatabase
[java] SEVERE: Could not open station database.

Is this something James is working on today ?
I can wait until tomorrow to try this ...

Thanks!

-Mike

On Thu, Oct 25, 2012 at 10:33 AM, Joel Edwards notifications@github.comwrote:

Once James' Issue (#2#2)
is complete, the required inputs should be known, and the values which need
to be passed in to the inject function of
asl.seedscan.database.MetricDatabase can be added to the method signature
(arguments and return).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3.

[joeledwards]

This is related to our decision to de-couple the database driver from the committed source. You can download the latest JCDB driver for PostgreSQL from the web. We are using version 9.1 of Postgres.

Downloads page: http://jdbc.postgresql.org/download.html

It is up to you whether you pull this completely out of the build process, and simply link to various jars, rather than placing these in the libs directory where they will be broken into class files, and included in the final jar.

[joeledwards]

The new class MetricInjector is a Runnable which is launched as a separate thread before running through the Scanners, and is now handed to each Scanner as it is created.

Injections are performed via a call to this class, which adds the Metric to a queue for injection. This way MetricDatabase doesn't need to worry about concurrent write requests.

[mikehagerty]

Hey guys,
just a quick update/check-in.

I'm working from home today as we're awaiting the super storm that's
suppose to arrive some time today
and the power at my office often gets knocked out when trees are felled by
the wind.

I'm working on MetricData.hashChanged() right now, computing the
multiDigests for a given channel (or
channel pair). When this is ready I'll clean up all the metric classes
that call it, then pull the latest
from Joel's Github, apply my changes and push up.

I can try to do a voice chat from my home in an hour or two.

Thanks,

-Mike

On Sat, Oct 27, 2012 at 2:45 AM, Joel Edwards notifications@github.comwrote:

The new class MetricInjector is a Runnable which is launched as a separate
thread before running through the Scanners, and is now handed to each
Scanner as it is created.

Injections are performed via a call to this class, which adds the Metric
to a queue for injection. This way MetricDatabase doesn't need to worry
about concurrent write requests.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-9833195.

[joeledwards]

James and I ran an injection test today, and confirmed that SeedScan can now push metrics into the database.

James is still working on the digest comparison SQL function. This can be used by hashChanged() when it is ready.

[jamesfholland]

I finished the comparison function. Its call is nearly identical to the other function, but without a value. It returns a boolean value. If they match it returns a TRUE, if they don't match it returns FALSE

SELECT spcomparehash( '09-17-1985', --Or any other standard date format DATE 'Metric', --Metric name VARCHAR(64) 'IU', --Network ID VARCHAR(64) 64 chars is long for network names, but the network name is stored in the same column as other groups with longer names 'STAT', --station Name VARCHAR(16) 'loc', --Location VARCHAR(16) 'ch1', --channel Name VARCHAR(16) E'\\xe4d909c290d0fb1ca068ffaddf22cbd0' -- hash in hex format );

[jamesfholland]

If there is no hash it returns false as well.

[mikehagerty]

I just finished MetricData.hashChanged()
It computes the digest and puts it into a byte[] to pass back to the
calling metric.

I just pushed up to joel-github and I'm about to pull everything down to
catbox2 to
see if the database parts still work.

-M

On Mon, Oct 29, 2012 at 3:25 PM, Joel Edwards notifications@github.comwrote:

James and I ran an injection test today, and confirmed that SeedScan can
now push metrics into the database.

James is still working on the digest comparison SQL function. This can be
used by hashChanged() when it is ready.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-9881922.

[jamesfholland]

Finished the get hash function
SELECT spGethash( '09-17-1985', --Or any other standard date format DATE 'Metric', --Metric name VARCHAR(64) 'IU', --Network ID VARCHAR(64) 'STAT', --station Name VARCHAR(16) 'loc', --Location VARCHAR(16) 'ch1' --channel Name VARCHAR(16) );

[jamesfholland]

Renamed spGethash to spGetDigest

[jamesfholland]

Renamed spGetDigest to spGetMetricValueDigest

[mikehagerty]

I cleaned up the metric classes so they now all call
MetricData.hashChanged() like:
// Check to see that we have data + metadata & see if the digest
has changed wrt the database:
ByteBuffer digest = metricData.hashChanged(channel);
or, in the case of CoherencePBM:
ByteBuffer digest = metricData.hashChanged(channelX, channelY);

At the end of hashChanged() I added the logic to test the current (just)
calculated digest
against the digest returned from the database (e.g., this is where you will
put MetricQuery() or whatever):
ByteBuffer newDigest = MemberDigest.multiBuffer(digests);
// This is just a place holder until the old digest can be retrieved from
the database:
ByteBuffer oldDigest = ByteBuffer.allocate(16);

    if (newDigest.compareTo(oldDigest) == 0){
        System.out.format("=== ByteBuffers are Equal !!\n");
        return null;
    }
    else {
        return newDigest;
    }

I guess that I'll need to supply you with a String to use to lookup the old
digest inside the hashChanged() method.
For now I'll add a line into each metric, something like:
CoherencePBM: String dbKey = MetricResult.createResultId( new Channel("00",
"LHND"), new Channel("10", "LHND") );
ByteBuffer digest =
MetricData.hashChanged(channelArray, dbKey);
and then inside hashChanged we'll have a line like:
hashChanged(): ByteBuffer oldDigest = MetricQuery(dbKey);

Okay, I see you've got something called getMetricValueDigest, so maybe I
would do something like:

hashChanged(): ByteBuffer oldDigest = getMetricValueDigest(date,
metricName, station, channelId);

What I have to figure out now is how to get the metricName and station from
inside MetricData.hashChanged()
(these are both available inside MetricResult ...)

Let me know if I'm missing something you intended.

Thanks!

-Mike

P.S. The hurricane is upon us with full force!

On Mon, Oct 29, 2012 at 6:12 PM, James Holland notifications@github.comwrote:

Renamed spGetDigest to spGetMetricValueDigest

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-9887612.

[mikehagerty]

Okay, ignore all previous mails :)

I just pushed up to the Github a version that has all the logic in place to
compare the
current digest to the digest stored in the database.

This happens in MetricData.hashChanged()

Right now I just have a placeholder for:
//ByteBuffer oldDigest = getMetricValueDigest(date, metricName,
station, channelId);

I have all the arguments (date, metricName, etc.) but within MetricData I
do NOT have
a handle to the database so I can't invoke this method.

I'm not sure how we want to handle this (?) We used to pass the database
handle in to Scanner
but this has changed to an Injector class.

Joel, can you work on this ? Once this is done we'll be able to test the
newDigest against the oldDigest.
After that I guess we just need to verify that Metrics are being properly
injected into the database
and we've got our end-to-end test ...

Thanks!

-Mike

On Mon, Oct 29, 2012 at 7:50 PM, Mike Hagerty
michael.t.hagerty@gmail.comwrote:

I cleaned up the metric classes so they now all call
MetricData.hashChanged() like:
// Check to see that we have data + metadata & see if the digest
has changed wrt the database:
ByteBuffer digest = metricData.hashChanged(channel);
or, in the case of CoherencePBM:
ByteBuffer digest = metricData.hashChanged(channelX, channelY);

At the end of hashChanged() I added the logic to test the current (just)
calculated digest
against the digest returned from the database (e.g., this is where you
will put MetricQuery() or whatever):
ByteBuffer newDigest = MemberDigest.multiBuffer(digests);
// This is just a place holder until the old digest can be retrieved from
the database:
ByteBuffer oldDigest = ByteBuffer.allocate(16);

    if (newDigest.compareTo(oldDigest) == 0){
        System.out.format("=== ByteBuffers are Equal !!\n");
        return null;
    }
    else {
        return newDigest;
    }

I guess that I'll need to supply you with a String to use to lookup the
old digest inside the hashChanged() method.
For now I'll add a line into each metric, something like:
CoherencePBM: String dbKey = MetricResult.createResultId( new
Channel("00", "LHND"), new Channel("10", "LHND") );
ByteBuffer digest =
MetricData.hashChanged(channelArray, dbKey);
and then inside hashChanged we'll have a line like:
hashChanged(): ByteBuffer oldDigest = MetricQuery(dbKey);

Okay, I see you've got something called getMetricValueDigest, so maybe I
would do something like:

hashChanged(): ByteBuffer oldDigest = getMetricValueDigest(date,
metricName, station, channelId);

What I have to figure out now is how to get the metricName and station
from inside MetricData.hashChanged()
(these are both available inside MetricResult ...)

Let me know if I'm missing something you intended.

Thanks!

-Mike

P.S. The hurricane is upon us with full force!

On Mon, Oct 29, 2012 at 6:12 PM, James Holland notifications@github.comwrote:

Renamed spGetDigest to spGetMetricValueDigest

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-9887612.

[joeledwards]

I pushed in changes that include a separate runnable for handling database read requests. Just realized that I forgot to place this in a thread, so that needs to be done tomorrow before we run tests.

Also important is the addition of a MetricValueIdentifier class which is used extensively for database queries. It can be created for one or two channels by calling Metric's createIdentifier() method.

The hashChanged() methods have been renamed to valueDigestChanged() for consistency. A metricDigestChanged() method may be added in the future to perform a pre-check for the entire metric before doing individual value checks. This could net some serious performance improvements, but is not a major objective.