Microservices

DataWarehouse provides a way to let external services KCIDB objects updates.

Design

To communicate between the DataWarehouse and the services running, RabbitMQ is used as the message broker.

DataWarehouse publishes all the events described on this page to the cki.exchange.datawarehouse.kcidb exchange. Services that want to consume these messages must create a queue and subscribe to this exchange. Using the cki-lib MessageQueue implementation is strongly recommended.

              application
              +---------------+
              | DataWarehouse |
              +-------+-------+
                      |
      exchange        v
      +---------------+------------------+
+-----+ cki.exchange.datawarehouse.kcidb |
|     +----------------------------------+
|
|
|  queues                                           services
|  +-------------------------------------------+    +-----------+
+->+ cki.queue.datawarehouse.kcidb.{service_1} +--->+ service_1 |
|  +-------------------------------------------+    +-----------+
|  +-------------------------------------------+    +-----------+
+->+ cki.queue.datawarehouse.kcidb.{service_2} +--->+ service_2 |
|  +-------------------------------------------+    +-----------+
.
.
|  +-------------------------------------------+    +-----------+
+->+ cki.queue.datawarehouse.kcidb.{service_n} +--->+ service_n |
   +-------------------------------------------+    +-----------+

Events

Different events are broadcasted over this method:

New new

This event is sent when a new object is received by the DataWarehouse.

Needs Triage needs_triage

This message indicates that an object needs to be triaged. These are sent when a change is done on the Issue Regexes, and the last 2 weeks objects are requeued to be analized for failures.

Ready to Report ready_to_report

This message indicates that the conditions to report this object were met. Note: The conditions under which this message needs to be sent are still undefined.

Updated updated

This message is sent when a existing object is modified by the DataWarehouse.

Build setups finised build_setups_finished

This message is sent when all the setup for all the builds of a checkout are finished. Note: This event always relates to a checkout.

Tests finished tests_finished

This message is sent when all the tests for a checkout are finished. Note: This event always relates to a checkout.

Messages

The body of the message contains a JSON payload as it follows:

message = {
    'timestamp': 'Timestamp of the message in ISO format',
    'status': '(new|needs_triage|ready_to_report|..)'
    'object_type': '(checkout|build|test)',
    'object': 'Json serialization of the object data *',
    'id': 'Id of the object *',
    'iid': 'IID of the object *'
}

• Check Checkout, Build and Test API docs to get more information.

Acking messages

The action of acking a message should only be done when the service finished processing the pipeline. This allows that if a failure happens while working with the message, the message will be still available on the queue the next time.