Updating tie client documentation

doc/sdk/cachingreputations.rst

@@ -0,0 +1,5 @@
+Caching Reputations
+===================
+
+Any client who request reputations from TIE should cache results they receive for a period of time to prevent unnecessary repeated reputation calls to the TIE service. Whenever a reputation changes within the TIE service a Reputation Change Event is sent out to all subscribed clients who have previously requested the reputation of the item. These reputation change events include the old and new reputations to allow updating of the local cache without sending an additional reputation request. For further information please refer to Test Scenarios for TIE Server Consumer section caching in :ref:`caching`.
+If  the connection to the DXL fabric is lost, a client sends a Reputation Update Request with the last time in which they received a successful response from the TIE Server. This request retrieves the hashes of all the items that changed their reputations since the last response time. These hashes can then be evicted from the local cache to be queried again the next time they are needed. It is recommended that the serverTime obtained from the last time a reputation request was issued is kept locally by the client so that this timestamp is used in the reputation updates request.

doc/sdk/index.rst

@@ -9,6 +9,7 @@ Introduction
 	:maxdepth: 1
 
 	overview
+	messageflow
 
 Installation
 ------------
@@ -65,3 +66,12 @@ Advanced
         :maxdepth: 1
 
         advancedgetreputationexample
+
+Guidance
+--------
+
+    .. toctree::
+        :maxdepth: 1
+
+        cachingreputations
+        testscenarios

doc/sdk/messageflow.rst

@@ -0,0 +1,16 @@
+General message flow
+====================
+
+In a real world environment it is expected that the TIE service will be consuming reputations from multiple products and providing reputations to multiple products, each with their own set of services, events, and requests. One of the key concerns for the design of a new integrating product is that it properly publishes and requests reputations from the TIE service. TIE will also store and share details about threat prevalence and spread.
+
+The TIE Server uses two types of communication:
+
+Service-based (Request\Response)
+ The TIE service is exposed via DXL allowing for DXL clients to request reputations for files and certificates. The TIE service responds to requests via a one-to-one communication, meaning the communication is solely between an invoking client and the TIE service.
+
+Event-based (Publish\Subscribe)
+    The TIE service provides topics where DXL clients can subscribe to and the TIE service publishes events to those specific topics. When the TIE service publishes events to its topics, all subscribed clients receive the event.
+
+The following diagram shows a generic overview of the typical flow of messages across a TIE server and its clients.
+
+.. image:: ./images/TIE-flow.jpg

doc/sdk/testscenarios.rst

@@ -0,0 +1,41 @@
+Best Practices and Test Scenarios for a TIE Server Consumer
+===========================================================
+
+General practices
+*****************
+* Consumer must not expect any given order of object attributes in the TIE server JSON response (see RFC4627 for more details).
+* Consumer must provide as much metadata as available if requested to do so by the TIE server in the reputation response ("submitMetaData": 1). Metadata is meaningful context used by the Security Administrator to make an informed decision about the reputation of an object.
+* Consumer must ignore any attributes or values provided in TIE server responses which the consumer is not expecting or capable to address. This is to ensure forwards compatibility of the TIE server API.
+* Consumer must properly handle error responses and timeout
+
+.. _caching:
+
+Caching
+*******
+
+* Consumer must cache reputations of files.
+* Consumer must cache reputations of certificates.
+* Consumer must invalidate cache entries after a configurable amount of days if the entry is not used.
+* Consumer must not refresh reputation proactively and fully rely on notifications.
+* Consumer must ask for certificate reputation when a file is signed.
+* Consumer must check for has-override flag and use file reputation although the file being signed.
+* Consumer should provide MD5, SHA1 and SHA256 file hashes when asking for reputations.
+
+Notifications
+*************
+
+* Consumer must subscribe to file reputation change notifications to keep cached items refreshed.
+* Consumer must subscribe to certificate reputation change notifications to keeps cached items refreshed properly.
+* Consumer must refresh cache entries only already present in cache (note that after 5k endpoints the reputation change topic behaves as a broadcast).
+* Consumer must reuse data in the notification instead of querying again for reputations.
+* Consumer should promptly react after reputation change updates (e.g. a running matching binary is killed).
+
+Cache Invalidation Updates
+**************************
+
+* Consumer cache must be indexed for fast access, but it must nor rely on presence of a given hash type.
+* Consumer must ask for reputation updates after becoming online to invalidates cached entries.
+* Consumer must not refresh updated reputations after becoming online until they are needed.
+* Consumer should limit each reputation updates request. Note that it will default to 100 if omitted.
+* Consumer should preserve the serverTime of the response to the last reputation request issued and use it as sinceTime in the first reputation updates request after becoming online.
+* Consumer should ask rolling requests following last time-stamp in response until getting all available updates.