Allow visibility into errors

[mrroach]

At the moment, all ingest errors are silently swallowed:

                try:
                    self._post(self._batch_data(datapoints_list),
                               '{0}/{1}'.format(
                                   self._endpoint,
                                   self._INGEST_ENDPOINT_DATAPOINT_SUFFIX))
                except:
                    _logger.exception('Posting data to SignalFx failed.')

I would like to be able to at least have a count of the types of exceptions that occur so that I can track them via other mechanisms (statsd, other internal health check code)

An approach I am using with other metric sending systems is something like this:

    ┆   # A mapping of error type to error count                                                                                                                                                                                       
    ┆   self.error_lock = threading.Lock()
    ┆   self.error_counters = collections.defaultdict(lambda: 0)

    def inc_error(self, error_type):
    ┆   """Increment internal counter of errors encountered.
    ┆   :param error_type: str, Exception class or other descriptor of error.
    ┆   """
    ┆   with self.error_lock:
    ┆   ┆   self.error_counters[error_type] += 1

    def reset_error_counters(self):
    ┆   """Reset error counters to 0 and return the previous values.
    ┆   :return: dict, Mapping of error type to count.
    ┆   """
    ┆   with self.error_lock:
    ┆   ┆   previous = self.error_counters
    ┆   ┆   self.error_counters = collections.defaultdict(lambda: 0)
    ┆   return previous

And then to use this in the sending code:

    def send(self):
    ...
    ┆   try:
    ┆   ┆   response = send_metrics(self.session, self.config, content)
    ┆   except RequestException as err:
    ┆   ┆   self.inc_error(err.__class__.__name__)

[mrroach]

I'll add that if this approach is acceptable, I'd be happy to send a PR

[mmusik]

Hi (a disclaimer - I'm an employee of Splunk/SignalFx). What you propose sounds reasonable to me. I think it is a good suggestion and I'll be happy to help with reviews if you prepare a PR. Thanks!

[mrroach]

Great, thank you! I'll put it together shortly