Monitoring cloud data in verifiable ways - an introduction
Increasing amounts of crucial and sensitive information, from companies’ financial records to biometric data about individuals, are being stored in cloud platforms. One example is Amazon’s Simple Storage Service (S3), which is part of Amazon Web Services (AWS) platform.
It’s important to ensure that this data is secure, and that any intruder who gets into this system and adds, deletes or changes anything is detected. At IF, we believe that technologies should be designed to be publicly accountable, understood and regulated. Sometimes, making that happen is going to require building new tools. That’s why Emily and I spent a few weeks building a monitoring tool which uses S3 as an example to show teams how they can prove statements about how they handle data. Emily has written a post for developers and other specialists about how we did it.
In the longer term, tools like this will strengthen the information security of cloud computing so companies can prove sensitive data remains secure.
We built a tool to try this out: IF’s S3-monitor
Amazon has a security system called CloudTrail which regularly audits logs of all activity on the AWS platform. It works well and doesn’t need replacing. However, we used the example of S3 to come up with a prototype for a different type of monitoring tool. It could have many other potential applications. We’ll be writing about some of these soon.
We’re calling the tool we built “S3-monitor”. It uses open source software called Trillian, which produces datasets that are transparent and verifiable. No one can break into the system and change or add anything without leaving footprints that alert others to their presence.
How Trillian creates proofs
It’s possible to provide proofs about data stored by Trillian because it uses a cryptographic data structure. For example, it’s possible to prove that data has been stored in an ‘append-only’ way. This means the data set can be added to, but previous entries can’t be changed or deleted. These proofs are small, allowing them to be verified quickly without using much space.
Proofs can be used in ways that don’t reveal the underlying data. For instance, an auditing company could access a proof that a company’s financial information hasn’t been hacked or secretly altered without accessing the actual details of the company’s finances.
Trillian can prove three things:
Consistency: it’s possible to continuously monitor the ‘append-only’ nature of the log by requesting Trillian to issue a consistency proof. This is useful to show an audit log is made up of all previous events which have appeared in the log.
Containment: Trillian proofs can also show whether a particular event is contained in the log. This is useful for issuing a receipt when a file is included in a data set.
Negative events: Trillian can also verify (using the Map component) that an event is not contained in the log. For example, if you had a data set of photos and an individual requested photos of them be removed, a proof of non-inclusion could be given to prove photos of them aren't being stored.
The S3-monitor tool that we created is a prototype that is intended to lead to more experimentation and discovery. It’s open-source and we are keen for developers to come up with their own applications for it that we might not have thought of.
We'd love to hear your thoughts and feedback. Write to us at email@example.com
Through doing this work, we’ve identified a few potential use cases in different industries. We’ll be sharing these in the next few weeks.
Thanks to Jess Holland and Ella Fitzsimmons for their contributions to this post.