User guide

This service provides a REST-compliant API for the BUS Methodology data. The fundamentals of what a REST API is, and what it is designed to achieve are available here.

This API provides a full range of CRUD operations

Overriding HTTP Methods

As a fallback for the odd client not supporting any of these methods, the API will gladly honor X-HTTP-Method-Override requests. For example a client not supporting the PATCH method could send a POST request with a X-HTTP-Method-Override: PATCH header. The API would then perform a PATCH, overriding the original request method.

Throughout this guide we will use the python bulib library to access and manipulate data using the BUS methodology API. This is only one of many way to access the data. Below is a quick walkthrough of how to initialise your python environment and login so that you can make requests to the API.

Obtaining an API key

To make a request you will need to start off by getting an API key. You can do this by:

• Logging into the BUS platform,
• Click the 'Account' link (top-right) and select 'API_KEY'

Making a request using the python client library buslib

Start off by installing the buslib library, preferably in a virtualenv, using pip install buslib

Now in a python terminal (we recommend jupyter notebook)

  >> # API_KEY holds the API key obtained above
  >> # PROXIES contains any proxy settings you need for your network
  >> # For ARUP this is "{'https' : 'https://proxy.ha.arup.com:80'}" - which is the default
  >> import buslib
  >> client = buslib.Client(API_KEY, host='%s/api' % "https://api.busmethodology.org.uk", proxies=PROXIES)
  >> client.partners.get()

You have successfully fetched all the partners you are allowed access to!

GET

As an example, if we make the following request using our python environment:

  >> client.partners.get(parse=False).json()

The response payload will look something like this:

{'_items': [{'_created': '24 Apr 2018 15:07:32 GMT',
  '_deleted': False,
  '_etag': '70f9e831c2384ee8807e212b5ea5764e85bdd152',
  '_id': '5adf483468b91d1d51ddc88e',
  '_latest_version': 1,
  '_links': {'self': {'href': 'partners/5adf483468b91d1d51ddc88e',
    'title': 'Partner object'}},
  '_updated': '24 Apr 2018 15:07:32 GMT',
  '_version': 1,
  'activePartner': True,
  'admin': False,
  'invoiceContact': {'address': {'addressLines': 'Flat 64v\nMarcus cliffs, Port Christine',
    'country': 'UK',
    'postcode': 'S1K 5FN'},
    'email': 'meganshort@gmail.com',
    'name': 'Dr. Scott Brown',
    'phoneNumber': '+441314960710'},
  'name': 'Walker, Hutchinson and Reid',
  'surveys': []}],
  '_links': {'parent': {'href': '/', 'title': 'home'},
  'self': {'href': 'partners?max_results=100', 'title': 'partners'}},
  '_meta': {'max_results': 100, 'page': 1, 'total': 1}}

The _items list contains the requested data. Along with its own fields, each item provides some important, additional fields:

These additional fields are automatically handled by the API (clients don’t need to provide them when adding/editing resources).

The _meta field provides pagination data and will only be there if pagination has been enabled (it is by default) and there is at least one document being returned. The _links list provides HATEOAS directives.

The client library conveniently removes most of the (usually) unneccesary information. The default configuration is to parse this response to provide:

    >> client.partners.get()
  

The response payload will look something like this:

[{'_id': '5adf483468b91d1d51ddc88e',
  'activePartner': True,
  'admin': False,
  'invoiceContact': {'address': {'addressLines': 'Flat 64v\nMarcus cliffs, Port Christine',
    'country': 'UK',
    'postcode': 'S1K 5FN'},
    'email': 'meganshort@gmail.com',
    'name': 'Dr. Scott Brown',
    'phoneNumber': '+441314960710'},
  'name': 'Walker, Hutchinson and Reid',
  'surveys': []}]
  

Resource endpoints allow consumers to retrieve multiple documents. Query strings are supported, allowing for filtering and sorting. Two query syntaxes are supported. The mongo query syntax:

  https://api.busmethodology.org.uk/api/accounts?where={"name": "Doe"}

and the native Python syntax:

  https://api.busmethodology.org.uk/api/accounts?where=name=="Doe"

Both query formats allow for conditional and logical And/Or operators, however nested and combined.

Using the buslib library this can be acheived using:

  >> client.accounts.get(where={"name": "Doe"})

Filters are mostly enabled on all document fields.

Please note

Always use double quotes to wrap field names and values. Using single quotes will result in 400 Bad Request responses.

Also note that when filtering date columns the format is as follows. %d %b %Y %H:%M:%S GMT. e.g. 16 May 2016 18:00:00 GMT. Note that this is a different format to that used when making conditional requests (as described later)

Sorting is supported as well:

    https://api.busmethodology.org.uk/accounts?sort=name,-email

Would return documents sorted by city and then by lastname (descending). As you can see you simply prepend a minus to the field name if you need the sort order to be reversed for a field.

The MongoDB data layer also supports native MongoDB syntax:

    https://api.busmethodology.org.uk/accounts?sort=[("email", -1)]

Would return documents sorted by lastname in descending order.

Using the buslib library this can be acheived using:

  >> client.accounts.get(sort="name,-email")

Please note

Always use double quotes to wrap field names and values. Using single quotes will result in 400 Bad Request responses.

This feature allows you to create dynamic views of collections and documents, or more precisely, to decide what fields should or should not be returned, using a "projection". Put another way, Projections are conditional queries where the client dictates which fields should be returned by the API.

    >> client.accounts.get(projection={"name": 1})

The query above will only return the name out of all the fields available in the 'accounts' resource. You can also exclude fields:

    >> client.accounts.get(projection={"name": 0})

The above will return all fields but name. Please note that key fields such as ID_FIELD, DATE_CREATED, DATE_UPDATED etc. will still be included with the payload. Also keep in mind that the database does not allow for mixing of inclusive and exclusive selections.

If a document field is referencing a document in another resource, clients can request the referenced document to be embedded within the requested document.

Clients have the power to activate document embedding on per-request basis by means of a query parameter. For example, survey items reference building and question items.

A GET like this: /surveys/?embedded={"building":1} would return a fully embedded building document, whereas the same request without the embedded argument would just return the building ObjectId.

The embedded documents may reference a specific version of a document (e.g. question items within a survey definition. See the schema definition),

Resource pagination is enabled by default in order to improve performance and preserve bandwidth. When a consumer requests a resource, the first N items matching the query are served, and links to subsequent/previous pages are provided with the response. There is a maximum page size, and consumers can request specific pages via the query string:

  https://api.busmethodology.org.uk/api/accounts?max_results=20&page=5

Of course you can mix all the available query parameters:

  https://api.busmethodology.org.uk/api/accounts?where={"lastname": "Doe}"&sort=[("firstname", 1)]&page=5

Hypermedia as the Engine of Application State (HATEOAS) is enabled by default. Each GET response includes a _links section. Links provide details on their relation relative to the resource being accessed, and a title. Relations and titles can then be used by clients to dynamically updated their UI, or to navigate the API without knowing its structure beforehand. An example:

      {'_links': {'child': [{'href': 'partners', 'title': 'partners'},
        {'href': 'accounts', 'title': 'accounts'},
        {'href': 'surveys', 'title': 'surveys'},
        {'href': 'survey_statistics', 'title': 'survey_statistics'},
        {'href': 'benchmarks', 'title': 'benchmarks'},
        {'href': 'questions', 'title': 'questions'},
        {'href': 'occupant_responses', 'title': 'occupant_responses'}]}}
  

A GET request to the API home page (the API entry point) will be served with a list of links to accessible resources. From there, any client could navigate the API just by following the links provided with every response.

A client may submit a single document for insertion:

  >> partner = {'name': 'Arup London', 'admin': False, 'activePartner': True}
  >> client.partners.post(building)
  {u'_id': u'5739b26069b17d648d18707c'}

In this case the response payload will just contain the relevant document metadata:

However, in order to reduce the number of loopbacks, a client might also submit multiple documents with a single request. All it needs to do is enclose the documents in a JSON list:

  >> client.partners.post[partner, partner, partner])
  [{u'_id': u'5739b2f269b17d648d18707d'}, {u'_id': u'5739b2f269b17d648d18707e'}, {u'_id': u'5739b2f269b17d648d18707f'}]

Data validation is performed on every resource managed by the API. Data sent to the API to be inserted/updated will be validated against the endpoint shema, and a resource will only be updated if validation passes.

This API provides automatic version control of documents. The API automatically tracks changes to documents and adds the fields _version and _latest_version when retrieving documents.

Behind the scenes, the API stores document versions in shadow collections that parallels the collection of each primary resource. New document versions are automatically added to this collection during normal POST, PUT, and PATCH operations. A query parameter is available when GETing an item that provides access to the document versions. Access a specific version with ?version=VERSION, access all versions with ?version=all, and access diffs of all versions with ?version=diffs. Collection query features like projections, pagination, and sorting work with all and diff except for sorting which does not work on diff.

Deleted documents continue to be stored in the database and are able to be restored, but still act as removed items in response to API requests.

Behavior

DELETE requests to individual items and resources respond just as they do for a traditional hard delete. Behind the scenes, however, Eve does not remove deleted items from the database, but instead patches the document with a _deleted meta field set to true.All requests made when soft delete is enabled filter against or otherwise account for the _deleted field.

The _deleted field is automatically added and initialized to false for all documents created.

Responses to GET requests for soft deleted documents vary slightly from responses to missing or hard deleted documents. GET requests for soft deleted documents will still respond with 404 Not Found status codes, but the response body will contain the soft deleted document with _deleted: true. Documents embedded in the deleted document will not be expanded in the response, regardless of any default settings or the contents of the requests embedded query param. This is to ensure that soft deleted documents included in 404 responses reflect the state of a document when it was deleted, and do not to change if embedded documents are updated.

By default, resource level GET requests will not include soft deleted items in their response. This behavior matches that of requests after a hard delete. If including deleted items in the response is desired, the show_deleted query param can be added to the request. The API will respond with all documents, deleted or not, and it is up to the client to parse returned documents _deleted field. The _deleted field can also be explicitly filtered against in a request, allowing only deleted documents to be returned using a ?where={"_deleted": true} query.

Restoring Soft Deleted Items

PUT or PATCH requests made to a soft deleted document will restore it, automatically setting _deleted to false in the database. Modifying the _deleted field directly is not necessary (or allowed). For example, using PATCH requests, only the fields to be changed in the restored version would be specified, or an empty request would be made to restore the document as is. The request must be made with proper authorization for write permission to the soft deleted document or it will be refused.

Be aware that, should a previously soft deleted document be restored, there is a chance that an eventual unique field might end up being now duplicated in two different documents: the restored one, and another which might have been stored with the same field value while the original (now restored) was in deleted state. This is because soft deleted documents are ignored when validating the unique rule for new or updated documents.

Versioning

Soft deleting a versioned document creates a new version of that document with _deleted set to true. A GET request to the deleted version will receive a 404 Not Found response as described above, while previous versions will continue to respond with 200 OK. Responses to ?version=diff or ?version=all will include the deleted version as if it were any other.

Data Relations

The API data_relation validator will not allow references to documents that have been soft deleted. Attempting to create or update a document with a reference to a soft deleted document will fail just as if that document had been hard deleted. Existing data relations to documents that are soft deleted remain in the database, but requests requiring embedded document serialization of those relations will resolve to a null value. Again, this matches the behavior of relations to hard deleted documents.

Versioned data relations to a deleted document version will also fail to validate, but relations to versions prior to deletion or after restoration of the document are allowed and will continue to resolve successfully.