When building an OLTP system, the first item to consider is consistency. Consistency in this article talks about mutual consistency. Mutual consistency is consistency obtained across servers. Traditional single server applications do not have to worry about mutual consistency. These applications only have to consider consistency in the context of correctness mostly related to transactions. Deciding what consistency models the application wants to use is a tough job in itself. Most vendors offer options for eventual consistency and/or strong consistency.
First let’s talk about what each option means through examples and then I’ll explain how the requirement for consistency impacts concurrency and availability of the application.
Case 1: Both Read and Write are Eventually Consistent
If a write is sent to the system, the system sends a command to replicate to the other servers, i.e., write sent to server 3, replicated to server 2 and server 1. However as soon as an ACK is received from the initial server, the client obtains an acknowledgement that the operation succeeded. However while the operation did not succeed on server 2 and server 3 yet, if a read operation is triggered, it can read the value from server 1 and get an older value.
Case 2: Both Read and Write are Strongly Consistent
The write will return to the client only if all the replicas have the new value. For read, the value read will always be the value, which is committed across replicas. If a write operation is going on while read is triggered, then the read will wait for the write to finish.
Case 3: Read is Eventually Consistent and Write is Strongly Consistent
The write will commit only if all the replicas have the same value. If a read is triggered while write is going on, then the read will deliver the last committed value.
Case 4: Read is Strongly Consistent and Write is Eventually Consistent
The write will return as soon as one server gets the value. If read is triggered while write is updating that value, the read will wait until all the replicas get the value.
The consistency decision in an application heavily impacts concurrency of the system. If the system is eventually consistent, there is no waiting for operations and hence improved concurrency. This leads to improved throughput. If the consistency in the system is strong, then a lot of time is spent waiting for other operations to finish, which reduces the concurrency of the system considerably. This leads to decline in throughput of the system.
The consistency decision also impacts availability of the system. CAP theorem proves that consistency, availability, and partition tolerance cannot all exist at the same time. Partition tolerance is assumed in a distributed system. That means a user has to choose between a completely consistent system and availability of the system. Having strong consistency can impact availability of the system by making the system unavailable to clients. Having eventual consistency configured in the system can allow accessibility to the system from the client even in the case of partition failure and allow for system availability even though the data in one partition is not consistent with data in another partition at a particular point in time.