Kafka and Blockchain. Kafka and KEEP network

Kafka can provide the foundation for a simple framework for KEEP network experimentation.

While scaling transactions in a public setting presents one set of issues, within a private network or consortium, where real-world trust is already established, transaction scaling might be achieved via an implementation which takes advantage of Kafka concepts.

There are two different ways to use kafka in blockchain projects. Either you connect Kafka to one or more blockchains, or you implement a Kafka-native blockchain.

Kafka and blockchain:

Kubernetes and apache Kafka technologies will help it scale faster, maintain top-notch records, give real-time services.

Kafka and blockchain:

Another option is to use or build a Kafka-native blockchain infrastructure. High scalability and high volume data processing in real-time for mission-critical deployments is a crucial differentiator of using Kafka compared to “traditional blockchain deployments” like Bitcoin or Ethereum.

Kafka enables a blockchain for real-time processing and analysis of historical data with one single platform.

AI Blockchain — A Tamper-proof Kafka-native Blockchain Implementation

AI Blockchain has implemented and patented a Kafka-native blockchain.

I decided to share how you can use this library — install and connect to the KEEP project

The project KafkaBlockchain provides a Java library for tamper-evidence using Kafka. Messages are optionally encrypted and hashed sequentially. The library methods are called within a Kafka application’s message producer code to wrap messages and called within the application’s consumer code to unwrap messages.

Because blockchains must be strictly sequentially ordered, Kafka blockchain topics must either have a single partition, or consumers for each partition must cooperate to sequence the records:

Kafka already implements checksums for message streams to detect data loss. However, an attacker can provide false records that have correct checksums. Cryptographic hashes such as the standard SHA-256 algorithm are very difficult to falsify, which makes them ideal for tamper-evidence despite being a bit more computation than checksums:

The provided sample in the Github project stores the first (genesis) message SHA-256 hash for each blockchain topic in ZooKeeper.

In this guide I decided to share how you can use this library — install and configure to the KEEP project

Installation and configuration

1. Compile and test the library

sudo git clone https://github.com/ai-coin/KafkaBlockchain.git  

2. Install Kafka and Zookeeper

sudo yum install curl which -y
sudo yum install java-1*openjdk-devel -y
sudo rpm --import https://packages.confluent.io/rpm/6.0/archive.key

3. Create confluent.repo file in /etc/yum.repos.d/ directory:

[Confluent.dist]
name=Confluent repository (dist)
baseurl=https://packages.confluent.io/rpm/6.0/7
gpgcheck=1
gpgkey=https://packages.confluent.io/rpm/6.0/archive.key
enabled=1
[Confluent]
name=Confluent repository
baseurl=https://packages.confluent.io/rpm/6.0
gpgcheck=1
gpgkey=https://packages.confluent.io/rpm/6.0/archive.key
enabled=1

4. Clean the cache and install kafka

sudo yum clean all && sudo yum install confluent-community-2.13

5. Configure kafka config: server.properties — file:

broker.id=1
listeners=PLAINTEXT://:9092
num.partitions=1
offsets.topic.replication.factor=1
 transaction.state.log.replication.factor=1
 transaction.state.log.min.isr=1
zookeeper.connect=localhost:2181

6. Configure zookeeper config: zookeeper.properties — file:

clientPort=2181
 tickTime=2000
 initLimit=5
 syncLimit=3
 maxClientCnxns=0
 admin.enableServer=false
 server.1=localhost:2888:3888

7. Configure Kafka Consumers config-file consumer.properties

bootstrap.server=localhost:9092
# this parameter defines the name of consumer group to be launched by run-kafka-blockchain-demo.sh script
group.id=demo-blockchain-consumers

8. Launch ZooKeeper in a terminal session

/bin/zookeeper-server-start.sh config/zookeeper.properties

9. Launch Kafka (on broker.id=1)

/bin/kafka-server-start.sh config/server.properties

10. Create kafka-topic “quickstart-events”

bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092

11. Launch Kafka Producer and send some messages in topic (partition)

kafka-console-producer --topic quickstart-events --bootstrap-server localhost:9092

12. Launch Kafka Consumer and read the messages received in the previous step

kafka-console-consumer --topic quickstart-events --from-beginning --bootstrap-server localhost:9092

13. Kafka blockchain demonstration.

The program produces four payloads on the blockchain and then consumes them. A second program verifies the whole blockchain.

Navigate to the KafkaBlockchain directory and launch this script

scripts/run-kafka-blockchain-demo.sh

Before re-running this demonstration, remove the previous blockchain messages by running this script first

scripts/run-kafka-blockchain-demo-reset.sh

14. Kafka blockchain multiple partition demonstration.

The program produces four payloads on the blockchain and then consumes them. A second program verifies the whole blockchain

Navigate to the KafkaBlockchain directory and launch this script

scripts/run-kafka-blockchain-demo.sh

Before re-running this demonstration, remove the previous blockchain messages by running this script first

scripts/run-kafka-blockchain-multiple-partition-demo-reset.sh

15. Kafka encrypted blockchain demonstration.

The program produces four encrypted payloads on the blockchain and then consumes them with decryption. A second program verifies the whole blockchain without needing to decrypt it.

Navigate to the KafkaBlockchain directory and launch this script

scripts/run-kafka-blockchain-encryption-demo.sh

Before re-running this demonstration, remove the previous blockchain messages by running this script first

scripts/run-kafka-blockchain-demo-encryption-reset.sh

16. Shut down the Kafka session and the Zookeeper session with Ctrl-C in both terminal sessions.