null record error on inserting a new message or append a new message into a topic- Apache Kafka

So, I have been working on Apache Kafka using CDH5.4 with parcels.
Scenario: I have four Linux servers of which one is Master and remaining three are slaves.
Task: To configure one of the slaves to act as a Kafka messaging server.
Command: When I execute this command to append a new message to the topic

hadoop jar /opt/camus/camus-example/target/camus-example-0.1.0-SNAPSHOT-shaded.jar com.linkedin.camus.etl.kafka.CamusJob -P /opt/camus/camus.properties

I get the error "java.lang.RunTimeException job failed nullrecord

camus- null record errorMistake: I overlooked the camus.properties file in Kafka and did not properly configure it which caused this error
Solution:
camus.message.timestamp.field=created_at
camus.message.timestamp.format=ISO-8601
etl.hourly=hourly
etl.daily=daily
(etl.hourly and etl.daily were grayed out, I only enabled them)
etl.default.timezone=Singapore (The default timezone was not set, I set it to Singapore)

This solved the null record error
Help: camus_etl@googlegroups.com was instrumental in providing the solution

What you need to ensure is the timezone where you are in and most important is the camus.message.timestamp.field=created_at

The complete conf.properties file is listed at my github page

References

1. http://alvincjin.blogspot.com/2014/12/trouble-shooting-for-kafka-camus-example.html

2. http://saurzcode.in/2015/02/integrate-kafka-hdfs-using-camus-twitter-stream-example/

Advertisements